Local-first π§ memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly π΄ dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
v0.10.0: Brain search β SQLite + FTS5 + optional semantic over the vault
Sol Aitken2ad0af506253
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.jsonindex e8660784..247dba69 100644--- a/.claude-plugin/plugin.json+++ b/.claude-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "0.9.1",+ "version": "0.10.0","description": "Plugin-first second brain package for AI agents and humans.","author": {"name": "Open Second Brain contributors"diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.jsonindex 319bbf03..1f5f3ccc 100644--- a/.codex-plugin/plugin.json+++ b/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "0.9.1",+ "version": "0.10.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","skills": "./skills","hooks": "./hooks/hooks.json",diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 0d8caadc..52b0ffc6 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,6 +5,76 @@ All notable changes to this project will be documented in this file.The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).+## [0.10.0] - 2026-05-16++Full-text search over the vault as a deterministic, filesystem-first+layer. Index lives at `<vault>/.open-second-brain/brain.sqlite`+(SQLite + FTS5, schema versioned). Optional semantic layer via+`sqlite-vec` plus any OpenAI-compatible `/v1/embeddings` provider+(OpenRouter, OpenAI, Together, Google's OpenAI-compat endpoint,+local Ollama, Hermes proxy). Closes design plan+`docs/plans/2026-05-16-brain-search-design.md`.++### Added++- **Core module `src/core/search/`** with isolated walker, chunker,+ store (the only SQL boundary), FTS query, links, ranker, indexer,+ search, and embedding providers (`null-provider`, `openai-compat`).+ Public surface: `resolveSearchConfig`, `indexVault`, `reindexVault`,+ `indexStatus`, `indexCheck`, `search`, plus typed `SearchError`+ codes.+- **CLI** namespace `o2b search` with verbs `query` (default),+ `index`, `reindex`, `status`, `check`. Human and `--json` output;+ `--auto-refresh` for read-time incremental indexing.+- **MCP tool** `brain_search` (read-only, agent-facing). Diagnostic+ score components (`keywordScore`, `semanticScore`, `linkBoost`,+ `recencyBoost`) are intentionally absent from the MCP shape; they+ live in CLI `--verbose` only. Content per chunk is truncated to+ 600 characters. Index-management verbs are NOT exposed over MCP+ (operator business, never agent business β design Β§3 principle 5).+- **`second_brain_status`** gains a `search.*` block: index path,+ schema version, document/chunk/embedding counts, embedding model+ and dimension, sqlite-vec status, key presence (redacted),+ `last_indexed_at`, `last_full_index_at`. Reports+ `{ exists: false, hint }` when the index has not been built yet.+- **Ranking** combines min-max-normalised BM25, cosine similarity+ on unit-normalised vectors, in-result wikilink boost (capped at+ 0.03), shared-tag boost (capped at 0.02), and recency steps+ (β€7d β 0.05, β€30d β 0.025, β€90d β 0.01). Tie-break on equal+ final score: keyword desc, mtime desc, chunk id asc.+- **Atomic reindex** via `brain.sqlite.new` + same-directory rename+ swap with `brain.sqlite.bak` retention. Auto-restore from `.bak`+ on open if the main file is missing.+- **Embedding-model fingerprint** stored in `index_state`. Changing+ `embedding_model` or `embedding_dimension` drops `embeddings`,+ `chunk_vec`, and `chunk_vec_map` on next open, logs one line, and+ preserves `chunks` + `chunk_fts`. The next+ `o2b search index --embeddings` repopulates vectors.+- **Concurrency guard** via `proper-lockfile` on the index path:+ three attempts, 1s backoff, then `INDEX_LOCKED`. Readers do not+ take the lock; WAL handles concurrent reads safely.+- **Semantic-unavailable policy** (design Β§7): implicit semantic+ (config default) warns and falls back to keyword-only; explicit+ `--semantic` / `semantic: true` throws a typed `SearchError` so+ the failure cannot hide. Data-state cases (no embeddings yet)+ warn and skip even when explicit β running+ `o2b search index --embeddings` is the right answer there, not a+ panic.++### Changed++- **`sqlite-vec`** added to `optionalDependencies`. The runtime+ detects whether the loadable extension is present on disk and+ records availability in `index_state.vec_extension_available`;+ no failure if the platform package is missing.++### Notes++- v0.10.0 ships schema version 1 only. Future migrations follow the+ same `MIGRATIONS[]` pattern in `src/core/search/schema.ts`.+- `o2b index` (the Markdown index generator at `AI Wiki/index.md`)+ is unchanged. The new system is `o2b search index`.+## [0.9.1] - 2026-05-15Active-preferences digest, MCP Resources, and a visibility expansion@@ -1081,6 +1151,11 @@ Hermes / Claude Code / Codex / OpenClaw configurations do not change.- Sandbox vault and plugin manifest fixtures for tests.- GitHub release workflow for tag-based and manually dispatched releases.+[0.10.0]: https://github.com/itechmeat/open-second-brain/compare/v0.9.1...v0.10.0+[0.9.1]: https://github.com/itechmeat/open-second-brain/compare/v0.9.0...v0.9.1+[0.9.0]: https://github.com/itechmeat/open-second-brain/compare/v0.8.1...v0.9.0+[0.8.1]: https://github.com/itechmeat/open-second-brain/compare/v0.8.0...v0.8.1+[0.8.0]: https://github.com/itechmeat/open-second-brain/compare/v0.7.0...v0.8.0[0.7.0]: https://github.com/itechmeat/open-second-brain/compare/v0.6.2...v0.7.0[0.6.2]: https://github.com/itechmeat/open-second-brain/compare/v0.6.1...v0.6.2[0.6.1]: https://github.com/itechmeat/open-second-brain/compare/v0.6.0...v0.6.1diff --git a/README.md b/README.mdindex f07e00e1..232f3016 100644--- a/README.md+++ b/README.md@@ -13,6 +13,36 @@ and **not** an LLM-driven knowledge store β the `dream` consolidationpass is pure deterministic counters. The plugin never writes hiddenstate outside the configured vault and config directory.+## How it works++Three loops cooperate over plain Markdown:++- **Capture.** Agents call `brain_feedback` to drop taste signals into+ `Brain/inbox/`.+- **Accretion.** A deterministic `dream` pass clusters repeat signals+ into rules β counters and atomic file moves, no LLM.+- **Application.** Agents log whether each rule was `applied` /+ `violated` / `outdated`. `Brain/active.md` is auto-regenerated and+ injected at session start; `brain_search` exposes full-text search+ across the whole vault.++```mermaid+flowchart LR+ Agent[Agent / human]+ Agent -- brain_feedback --> Inbox[(Brain/inbox/)]+ Inbox -. dream .-> Pref[(Brain/preferences/)]+ Agent -- brain_apply_evidence --> Log[(Brain/log/)]+ Log -. dream .-> Pref+ Pref -. regenerate .-> Active[Brain/active.md]+ Active -- SessionStart hook --> Agent+ Vault[Vault Markdown] -- o2b search index --> FTS[(SQLite + FTS5<br/>brain_search)]+ FTS --> Agent+```++Mechanics, dream pipeline, state diagram, snapshot model, hygiene+lints, MCP resources, the v0.10.0 search layer, and the full set of+safety invariants are in [`docs/how-it-works.md`](docs/how-it-works.md).+## Supported runtimes| Runtime | Integration | Notes |diff --git a/bun.lock b/bun.lockindex 0a7d3740..4f7c5812 100644--- a/bun.lock+++ b/bun.lock@@ -13,6 +13,9 @@"@types/proper-lockfile": "^4.1.4","typescript": "^5.5.0",},+ "optionalDependencies": {+ "sqlite-vec": "^0.1.9",+ },},},"packages": {@@ -34,6 +37,18 @@"signal-exit": ["[email protected]", "", {}, "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ=="],+ "sqlite-vec": ["[email protected]", "", { "optionalDependencies": { "sqlite-vec-darwin-arm64": "0.1.9", "sqlite-vec-darwin-x64": "0.1.9", "sqlite-vec-linux-arm64": "0.1.9", "sqlite-vec-linux-x64": "0.1.9", "sqlite-vec-windows-x64": "0.1.9" } }, "sha512-L7XJWRIBNvR9O5+vh1FQ+IGkh/3D2AzVksW5gdtk28m78Hy8skFD0pqReKH1Yp0/BUKRGcffgKvyO/EON5JXpA=="],++ "sqlite-vec-darwin-arm64": ["[email protected]", "", { "os": "darwin", "cpu": "arm64" }, "sha512-jSsZpE42OfBkGL/ItyJTVCUwl6o6Ka3U5rc4j+UBDIQzC1ulSSKMEhQLthsOnF/MdAf1MuAkYhkdKmmcjaIZQg=="],++ "sqlite-vec-darwin-x64": ["[email protected]", "", { "os": "darwin", "cpu": "x64" }, "sha512-KDlVyqQT7pnOhU1ymB9gs7dMbSoVmKHitT+k1/xkjarcX8bBqPxWrGlK/R+C5WmWkfvWwyq5FfXfiBYCBs6PlA=="],++ "sqlite-vec-linux-arm64": ["[email protected]", "", { "os": "linux", "cpu": "arm64" }, "sha512-5wXVJ9c9kR4CHm/wVqXb/R+XUHTdpZ4nWbPHlS+gc9qQFVHs92Km4bPnCKX4rtcPMzvNis+SIzMJR1SCEwpuUw=="],++ "sqlite-vec-linux-x64": ["[email protected]", "", { "os": "linux", "cpu": "x64" }, "sha512-w3tCH8xK2finW8fQJ/m8uqKodXUZ9KAuAar2UIhz4BHILfpE0WM/MTGCRfa7RjYbrYim5Luk3guvMOGI7T7JQA=="],++ "sqlite-vec-windows-x64": ["[email protected]", "", { "os": "win32", "cpu": "x64" }, "sha512-y3gEIyy/17bq2QFPQOWLE68TYWcRZkBQVA2XLrTPHNTOp55xJi/BBBmOm40tVMDMjtP+Elpk6UBUXdaq+46b0Q=="],+"typescript": ["[email protected]", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],"undici-types": ["[email protected]", "", {}, "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ=="],diff --git a/docs/how-it-works.md b/docs/how-it-works.mdindex 2b923b36..e6092411 100644--- a/docs/how-it-works.md+++ b/docs/how-it-works.md@@ -23,25 +23,29 @@ no surprise, no hallucinated memory.## Vault layoutThe vault holds three top-level agent-facing directories. Brain owns-its own.+its own, plus a sibling derived-index directory.```text<vault>/βββ Brain/ # observing memory (agent-writable)β βββ _brain.yaml # schema, thresholds, retentionβ βββ _BRAIN.md # operating manual for agents+β βββ active.md # derived: confirmed + quarantine + recently retiredβ βββ inbox/ # raw taste signalsβ β βββ sig-<date>-<slug>.md-β β βββ processed/ # signals already folded into rules+β β βββ processed/ # signals already folded into rulesβ βββ preferences/ # active rules-β β βββ pref-<slug>.md # status: unconfirmed | confirmed+β β βββ pref-<slug>.md # status: unconfirmed | confirmed | quarantineβ βββ retired/ # archived rules-β β βββ ret-<slug>.md # retired_reason: stale-no-evidence | expired-unconfirmed | rebutted | user-rejected+β β βββ ret-<slug>.md # retired_reason: stale-no-evidence | expired-unconfirmed | rebutted | user-rejected | quarantine-violated | superseded-by-contextβ βββ log/ # daily event logβ β βββ YYYY-MM-DD.md # append-only, typed eventsβ βββ .snapshots/ # pre-dream snapshotsβ βββ dream-<run-id>.tar.zstβ+βββ .open-second-brain/ # derived search index (rebuildable)+β βββ brain.sqlite # SQLite + FTS5 + optional sqlite-vec+ββββ AI Wiki/ # curated knowledge surfaceβ βββ identity/ # user.md, agents.mdβ βββ index.md / hot.md@@ -52,6 +56,11 @@ its own.βββ YYYY.MM.DD.md```+`Brain/active.md` and `.open-second-brain/brain.sqlite` are **derived** β+both can be deleted and rebuilt at any time (`o2b brain dream` and+`o2b search reindex` respectively). They are excluded from the agent's+write contract.+```mermaidflowchart LRAgent[Agent / MCP client]@@ -75,7 +84,7 @@ there through the `payment_*` tools).## A preference's lifecycle-A preference moves between four states from first signal to retirement:+A preference moves between five states from first signal to retirement:```mermaidstateDiagram-v2@@ -83,28 +92,49 @@ stateDiagram-v2Inbox --> Unconfirmed: dream (β₯candidate_threshold<br/>same-sign signals on topic)Unconfirmed --> Confirmed: brain_apply_evidence applied<br/>(first such event)Unconfirmed --> Retired_Expired: dream (now > unconfirmed_until,<br/>no applied evidence)+ Confirmed --> Quarantine: dream (violated_count β₯ applied_count<br/>AND applied_count > low_max_applied)+ Quarantine --> Confirmed: dream (applied_count > violated_count)+ Quarantine --> Retired_Quarantine: dream (one further violated)Confirmed --> Retired_Stale: dream (last_evidence_at<br/>older than stale_evidence_days)Confirmed --> Retired_Rebutted: dream (β₯candidate_threshold<br/>opposite-sign signals collected)+ Confirmed --> Retired_Superseded: dream (one apply-evidence with<br/>result: outdated)+ Quarantine --> Retired_Superseded: dream (one apply-evidence with<br/>result: outdated)Confirmed --> Retired_UserRejected: o2b brain reject+ Quarantine --> Retired_UserRejected: o2b brain rejectUnconfirmed --> Retired_UserRejected: o2b brain rejectRetired_Expired --> [*]Retired_Stale --> [*]Retired_Rebutted --> [*]+ Retired_Quarantine --> [*]+ Retired_Superseded --> [*]Retired_UserRejected --> [*]note right of ConfirmedPinned preferences (pinned: true)- skip all three automatic retire- transitions. Only o2b brain reject- can retire them.+ skip the three "natural decay"+ retire transitions (stale,+ expired, rebutted) and the+ quarantine downgrade β dream+ emits retain-pinned instead.+ A context-driven outdated still+ retires a pinned pref; only+ o2b brain reject and outdated+ evidence can move it.end note```-`Inbox` is not really a state of the preference β it's the staging+`Inbox` is not really a state of the preference β it is the stagingarea for the signals that will eventually create one. The first realstate is `Unconfirmed`: the rule exists but has not yet been appliedin real work.+`Quarantine` (added in v0.9.1) is a probation state for confirmed rules+whose recent evidence has turned dominantly negative without yet+crossing the rebuttal threshold. A quarantined rule is still active β+the agent reads it β but one further `violated` retires it with+`quarantine-violated`, and recovery happens automatically when `applied`+events overtake the violated count.+## End-to-end signal β rule flowA typical sequence from a user remark to a confirmed preference:@@ -168,9 +198,9 @@ A single dream invocation is a deterministic pipeline:flowchart TDStart([o2b brain dream]) --> Load[Load _brain.yaml]Load --> Scan[Scan inbox/, preferences/, retired/, log/]- Scan --> Plan[Plan transitions:<br/>new unconfirmed, promotions,<br/>retires, signal moves]+ Scan --> Plan[Plan transitions:<br/>new unconfirmed, promotions,<br/>quarantine in/out, retires, signal moves]Plan --> Changed{Any changes planned?}- Changed -- no --> NoOp([return changed: false<br/>no snapshot, no log])+ Changed -- no --> NoOp([regenerate active.md<br/>return changed: false])Changed -- yes --> Snap["createSnapshot to<br/>.snapshots/dream-RUNID.tar.zst"]Snap --> SnapErr{Snapshot OK?}SnapErr -- fail --> Abort([throw; no state changed])@@ -184,23 +214,34 @@ flowchart TDA3 -- yes --> A4[Promote to confirmed,<br/>stamp confirmed_at]A3 -- no --> A5A4 --> A5[Recompute confidence]- A5 --> A6{Pinned?}- A6 -- yes --> A7[Skip auto-retire,<br/>log retain-pinned]- A6 -- no --> A8{Retire conditions?}- A8 -- stale --> A9[Move to retired/<br/>reason: stale-no-evidence]- A8 -- expired --> A10[Move to retired/<br/>reason: expired-unconfirmed]- A8 -- rebutted --> A11[Move to retired/<br/>reason: rebutted]- A8 -- none --> A12[Leave in preferences/]- A9 --> A13- A10 --> A13- A11 --> A13- A12 --> A13- A7 --> A13- A13[Move consumed signals to processed/]+ A5 --> A6{outdated evidence?}+ A6 -- yes --> A6r[Move to retired/<br/>reason: superseded-by-context]+ A6 -- no --> A7{Pinned?}+ A7 -- yes --> A7l[Skip auto-retire + quarantine,<br/>log retain-pinned]+ A7 -- no --> A8{Quarantine conditions?}+ A8 -- enter --> A8q[status: quarantine]+ A8 -- exit --> A8c[status: confirmed]+ A8 -- already-quarantine + violated --> A8r[Move to retired/<br/>reason: quarantine-violated]+ A8 -- none --> A9{Retire conditions?}+ A9 -- stale --> A10[Move to retired/<br/>reason: stale-no-evidence]+ A9 -- expired --> A11[Move to retired/<br/>reason: expired-unconfirmed]+ A9 -- rebutted --> A12[Move to retired/<br/>reason: rebutted]+ A9 -- none --> A13[Leave in preferences/]+ A6r --> A14+ A7l --> A14+ A8q --> A14+ A8c --> A14+ A8r --> A14+ A10 --> A14+ A11 --> A14+ A12 --> A14+ A13 --> A14+ A14[Move consumed signals to processed/]endApply --> LogEvent["Append dream event to log/today.md"]- LogEvent --> Prune[pruneSnapshots to retention_count]+ LogEvent --> ActMd[regenerate Brain/active.md]+ ActMd --> Prune[pruneSnapshots to retention_count]Prune --> Done([return changed: true with summary])```@@ -244,6 +285,42 @@ Defaults from `_brain.yaml`:All four are tunable per vault in `Brain/_brain.yaml`.+## Active preferences injection++A confirmed rule is useless if the agent does not see it during work.+v0.9.1 closes that gap with three cooperating surfaces, all derived+from the same source of truth (`Brain/preferences/`):++```mermaid+flowchart LR+ Dream[dream pass] -- regenerate --> Active[Brain/active.md]+ Pin[o2b brain pin / unpin] -- regenerate --> Active+ Active -- SessionStart hook --> ClaudeStart["Claude Code / Codex<br/>session start"]+ Active -- PostCompact hook --> ClaudeCompact["Claude Code / Codex<br/>after /compact"]+ Active -- "osb://preferences/active" --> MCPHost["MCP host (Hermes, others)"]+```++- **`Brain/active.md`** is a derived Markdown digest: confirmed+ preferences (id, scope, confidence, principle), quarantined+ preferences with their applied / violated counters, and the three+ most recently retired entries. The writer is idempotent β if the+ rendered body matches the file on disk, no I/O happens.+- **SessionStart hook** (`startup | resume | clear`) and+ **PostCompact hook** (`manual | auto`) inject the body as+ `additionalContext` so the agent sees current rules at the start of+ every session and again after `/compact` (otherwise the+ hook-injected context is dropped from long sessions). Both fail+ closed β any error path exits 0 with no output so the runtime+ proceeds unaffected.+- **MCP Resources** expose the same content for hosts that prefer+ pull access (`osb://preferences/active` and friends in the table+ above). The MCP `initialize` reply advertises the `resources`+ capability so clients know to enumerate them.++`active.md` is **not** edited by hand β `o2b brain dream` and pin /+unpin re-derive it from `preferences/` and `retired/`. Deleting the+file is harmless: the next dream regenerates it.+## CLI / MCP surfaceThe same operations are reachable through two channels. Read columns@@ -252,20 +329,41 @@ are mirrored in MCP; destructive operations are CLI-only by design.| Operation | CLI verb | MCP tool | Side effect ||--------------------------|----------------------------|-----------------------|------------------------|| Bootstrap layer | `o2b brain init` | β | creates `Brain/` skeleton |-| Record taste signal | `o2b brain feedback` | `brain_feedback` | writes signal + log event |-| Consolidation pass | `o2b brain dream` | `brain_dream` | mutates preferences/retired, atomic snapshot |-| Record application | `o2b brain apply-evidence` | `brain_apply_evidence`| appends log event |-| Render summary | `o2b brain digest` | `brain_digest` | read-only |+| Record taste signal | `o2b brain feedback` | `brain_feedback` | writes signal + log event (sanitised) |+| Consolidation pass | `o2b brain dream` | `brain_dream` | mutates preferences/retired, atomic snapshot, regenerates `Brain/active.md` |+| Record application | `o2b brain apply-evidence` | `brain_apply_evidence`| appends log event; `result β {applied, violated, outdated}`; artifact supports `[[file:120-145]]` ranges |+| Render summary | `o2b brain digest` | `brain_digest` | read-only; includes `top_applied` and `top_referenced` sections || Inspect state | `o2b brain query` | `brain_query` | read-only |-| Validate invariants | `o2b brain doctor` | `brain_doctor` | read-only |+| Computed backlinks | `o2b brain backlinks <id>` | `brain_backlinks` | read-only; inverted reference map across `preferences/`, `retired/`, `log/` |+| Validate invariants | `o2b brain doctor` | `brain_doctor` | read-only; six lint rules |+| Full-text search | `o2b search "<query>"` | `brain_search` | read-only; FTS5 + optional semantic |+| Manage search index | `o2b search index \| reindex \| status \| check` | β (CLI-only) | builds / inspects `<vault>/.open-second-brain/brain.sqlite` |+| Operational snapshot | `o2b status` | `second_brain_status` | read-only; `brain.*` + `search.*` blocks || Retire manually | `o2b brain reject` | β (CLI-only) | moves pref β retired/ |-| Toggle pin | `o2b brain pin / unpin` | β (CLI-only) | flips `pinned` field |+| Toggle pin | `o2b brain pin / unpin` | β (CLI-only) | flips `pinned` field; regenerates `Brain/active.md` || Restore snapshot | `o2b brain rollback` | β (CLI-only) | overwrites Brain/ from snapshot |Operations that change the **protected set** (`pin`, `unpin`,-`reject`, `rollback`) are kept off the MCP surface so an autonomous-agent cannot quietly alter what is shielded from automatic retire or-roll back state.+`reject`, `rollback`) and the **index lifecycle** (`search index`,+`reindex`, `check`) are kept off the MCP surface β protected-set+moves so an autonomous agent cannot quietly alter what is shielded+from automatic retire, and index management because it is operator+business, never agent business.++In addition to the tools above, MCP hosts can pull structured Brain+content as **resources** (added in v0.9.1):++| Resource URI | Body |+|-----------------------------------|---------------------------------------------------|+| `osb://preferences/active` | `Brain/active.md` (auto-regenerated on first read if missing) |+| `osb://digest/latest` | most recent `brain_digest` rendering |+| `osb://status` | Markdown form of the `second_brain_status.brain` block |+| `osb://preference/{id}` | one `pref-` / `ret-` file |+| `osb://topic/{slug}` | every signal + active/retired pref for a topic |+| `osb://log/{date}` | one day's `Brain/log/YYYY-MM-DD.md` |+| `osb://backlinks/{id}` | inverted reference map for `<id>` |++Resources are pure read; mutating verbs stay on `tools/call`.## Snapshots and rollback@@ -289,6 +387,45 @@ A snapshot captures every file under `Brain/` **except** `.snapshots/`itself β otherwise rollback would erase any snapshots taken afterthis one. Retention defaults to ten newest archives.+## Hygiene: sanitisation and lints++Two cheap, deterministic layers keep the data clean.++**Input sanitisation** (v0.9.1). Every Brain writer routes text fields+through `src/core/redactor.ts` (promoted from Pay Memory) plus a+`normaliseTextField` / `sanitiseTextField` pair:++- C0 control characters are stripped (except `\t` and `\n`); `U+2028`+ and `U+2029` collapse to `\n`; text is NFC-normalised.+- Length caps per field: `principle` β€ 512 (single-line), `scope` β€+ 128, `raw` β€ 4096, `source[]` items β€ 512, `artifact` β€ 512+ (single-line), `note` β€ 4096.+- Secret-shaped substrings (`api_key=β¦`, `token: β¦`, `bearer β¦`,+ `authorization: β¦`, `password=β¦`, `private_key=β¦`, etc.) are masked+ with `***REDACTED***`. The `raw` field is the only verbatim-quote+ surface and is intentionally **not** redacted β it carries the+ original user phrasing.+- Inputs that sanitise down to empty fall into the existing "missing+ field" error branch, so the writer never persists a placeholder+ signal.++**Doctor lints** (`o2b brain doctor`, `brain_doctor` MCP). All six are+pure functions over the on-disk Brain state β no LLM, no network:++| Code | Triggers when |+|---|---|+| `broken-backlinks` | a `[[pref-β¦]]` / `[[ret-β¦]]` / `[[sig-β¦]]` reference exists in `preferences/`, `retired/`, `inbox/`, or `log/`, but the target file does not |+| `duplicate-preferences` | pairwise Jaccard β₯ 0.7 on principle tokens within the same `(topic, scope)` bucket |+| `low-evidence-confirmed` | `status: confirmed` with `applied_count β€ low_max_applied` AND `confirmed_at` older than `unconfirmed_window_days` |+| `pinned-without-recent-evidence` | `pinned: true` with no evidence or evidence older than `stale_evidence_days` |+| `malformed-evidence-range` | `apply-evidence` `artifact` uses `[[file:β¦]]` range syntax but fails validation (`:abc-def`, `:120-100`, bare `:`) |+| `orphan-evidence` | `apply-evidence` `artifact` wikilink does not resolve to any file in the vault |++With `--strict`, warnings demote `ok` to `false` so CI can gate on+hygiene. The lints are read-only β `brain_doctor --fix` is explicitly+out of scope because auto-modifying state runs against the+"explicit-driven" invariant.+```mermaidflowchart LRBrainRoot[Brain/]@@ -335,6 +472,89 @@ graph LR- **OpenClaw** runs tools natively in the plugin's Node.js process(no subprocess, by security-scanner requirement).+## Full-text search++v0.10.0 adds a deterministic search layer over the entire vault β not+just `Brain/`. The index lives at `<vault>/.open-second-brain/brain.sqlite`+(SQLite + FTS5) and is fully rebuildable from the Markdown files.+Semantic search is optional and pluggable: `sqlite-vec` + any+OpenAI-compatible `/v1/embeddings` endpoint when configured.++```mermaid+flowchart LR+ subgraph Build [o2b search index]+ direction LR+ Walker[walker<br/>vault-relative paths] --> Chunker[chunker<br/>structural split + token packer]+ Chunker --> Store[(SQLite store<br/>documents / chunks / links / embeddings)]+ Chunker --> Links[extract wikilinks /<br/>markdown links / tags]+ Links --> Store+ Store --> FTS[(chunk_fts<br/>FTS5 BM25)]+ Store -.optional.-> Vec[(chunk_vec<br/>sqlite-vec)]+ Texts[chunk content] -. on --embeddings .-> Provider[OpenAI-compat<br/>embeddings provider]+ Provider --> Vec+ end++ subgraph Query [search]+ direction LR+ Q["o2b search '<q>'<br/>brain_search"] --> KW[FTS5 keyword<br/>limit Γ 3 candidates]+ Q -. semantic on .-> SemQ[provider.embed query]+ SemQ --> SemS[vec_search<br/>limit Γ 5 candidates]+ KW --> Ranker[ranker<br/>min-max BM25 + cosine<br/>+ link boost + recency boost]+ SemS --> Ranker+ Ranker --> Results[ranked results]+ end+```++Key behaviours, all driven from `Brain/_brain.yaml`-free `search_*` /+`embedding_*` config keys:++- **Chunker.** Two passes: a structural split (headings, fenced code,+ lists, tables, frontmatter), then a token-budget packer with+ overlap. YAML frontmatter becomes synthetic `chunk_index: 0` so+ `tags:` and other front-matter fields are searchable through FTS5+ alongside the body.+- **Store boundary.** `src/core/search/store.ts` is the single SQL+ home; every other module talks to it through a typed surface. WAL+ mode for concurrent reads, `proper-lockfile` on the index path for+ writer exclusivity (three attempts, 1 s backoff, then+ `INDEX_LOCKED`).+- **Ranking.** `final_score = clamp01(keyword_weightΒ·norm_BM25 ++ semantic_weightΒ·cosine + link_boost + recency_boost)`. BM25 is+ min-max-normalised within the candidate set; cosine is+ `1 - L2Β² / 2` on unit-normalised vectors; link boost rewards+ candidates that other candidates reference via `[[wikilink]]` /+ markdown link (capped at 0.03) or share a tag with (capped at+ 0.02); recency is a step function on `mtime`.+- **Semantic policy.** Implicit semantic (config default) warns and+ falls back to keyword-only when sqlite-vec is unavailable, the key+ is missing, the provider is down, or no embeddings exist yet.+ Explicit `--semantic` / `semantic: true` on infrastructure failure+ throws a typed `SearchError` so a misconfigured run cannot hide.+ The data-state case (zero embeddings) always warns and skips β+ running `o2b search index --embeddings` is the right answer there.+- **Atomic reindex.** `o2b search reindex` writes to+ `brain.sqlite.new`, renames to `brain.sqlite`, and keeps the+ previous file as `brain.sqlite.bak`. If the main file is missing on+ open and a `.bak` exists, it is auto-restored with a stderr notice.+- **Embedding model fingerprint.** Model + dimension are recorded in+ the index. Changing either drops the `embeddings` and `chunk_vec`+ tables on next open, logs one line, and preserves `chunks` and+ `chunk_fts`; the next `o2b search index --embeddings` repopulates+ vectors.++The MCP `brain_search` tool returns at most 50 results with each+chunk's `content` truncated to 600 characters; diagnostic score+components (`keywordScore`, `semanticScore`, `linkBoost`,+`recencyBoost`) are intentionally absent from the MCP shape β they+live in the CLI's `--verbose` output only, to keep the agent context+small. Index-management verbs (`index`, `reindex`, `check`) are+CLI-only.++Full design and migration notes:+[`docs/plans/2026-05-16-brain-search-design.md`](plans/2026-05-16-brain-search-design.md)+and the matching implementation plan+[`docs/plans/2026-05-16-brain-search-impl.md`](plans/2026-05-16-brain-search-impl.md).+## Safety propertiesThese are invariants of the system, not configuration to enable.@@ -433,9 +653,15 @@ re-renders it from the current template.- **`docs/architecture.md`** β layered system architecture beyondBrain (vault model, runtime adapters, configuration model).-- **`docs/plans/2026-05-15-brain-observing-memory.md`** β the design- document, including the full file-format specs and test strategy.+- **`docs/plans/2026-05-15-brain-observing-memory.md`** β the Brain+ design document, including the full file-format specs and test+ strategy.- **`docs/plans/2026-05-15-brain-roadmap.md`** β trigger-based roadmapof future capabilities (`BRAIN-FUT-NNN` entries).+- **`docs/plans/2026-05-16-brain-search-design.md`** and+ **`docs/plans/2026-05-16-brain-search-impl.md`** β design and+ implementation plan for the v0.10.0 full-text search layer.+- **`docs/mcp.md`** β protocol, schemas, lifecycle, and resource+ surface of the MCP server.- **`Brain/_BRAIN.md`** (in any initialised vault) β the operatingmanual for agents working with that specific vault.diff --git a/docs/plans/2026-05-16-brain-search-design.md b/docs/plans/2026-05-16-brain-search-design.mdnew file mode 100644index 00000000..f8564cbe--- /dev/null+++ b/docs/plans/2026-05-16-brain-search-design.md@@ -0,0 +1,973 @@+# Brain Search β Full-text Index over the Vault++Status: design+Target: Open Second Brain v0.10.0+Authors: Sergey Eroshenkov (product), Claude (drafting)++## 1. Overview++Open Second Brain stores every durable artifact as Markdown in an+Obsidian-compatible vault. The Brain layer adds preferences, signals, and a+daily log; Pay Memory adds an auditable trail of paid actions; AI Wiki and+Daily areas accumulate notes that may be either agent- or user-authored. Until+now, the only way to find anything inside the vault has been to read the file+the caller already names: `brain_query` resolves a known slug, `vault_health`+walks paths, the Read tool requires an absolute path. There is no lookup by+content.++This document specifies **Brain Search** β a deterministic, filesystem-first+search layer over the entire vault. Keyword retrieval is the contract; a+semantic layer is optional and pluggable. The implementation must hold to the+same invariants as the rest of the project: no daemon, no external service is+required to run search, Markdown files remain the source of truth, the index+is a fully rebuildable derived artifact.++This plan covers Tier-S item Β§2 of `Projects/OpenSecondBrain/Features/_summary`+and supersedes the more sketch-like `Projects/OpenSecondBrain/Features/search`+note. Items Β§9 (inline `@osb` markers), Β§12 (merge suggestions), Β§14 (HTML+explorer) are explicitly **not** included.++## 2. Scope of v0.10.0++In scope:++- New module `src/core/search/` with isolated responsibilities (walker,+ chunker, store, FTS, links, ranker, embedding providers).+- New CLI namespace `o2b search` with verbs `query` (default) / `index` /+ `reindex` / `status` / `check`.+- New MCP tool `brain_search` (read-only).+- Extension of the existing `second_brain_status` MCP tool with a+ `search.*` block.+- SQLite index at `<vault>/.open-second-brain/brain.sqlite` (overridable),+ WAL mode, schema versioned through `index_state.schema_version`.+- Optional semantic layer through `sqlite-vec` and any OpenAI-compatible+ `/v1/embeddings` endpoint (OpenRouter, OpenAI, Google's OpenAI-compat+ endpoint, Together, local Ollama, Hermes proxy when authenticated).+- Vault-wide indexing with a configurable ignore list (defaults: `.git`,+ `node_modules`, `.open-second-brain`, `.obsidian/cache`, `.trash`,+ `.stversions`).+- Incremental reindexing on file modification and removal.++Out of scope:++- Inline `@osb` marker parsing.+- Merge-suggestion detection in `brain_dream` or `brain_digest`.+- HTML/graph explorer.+- Long-running file watcher / daemon.+- LLM-based answer synthesis.+- Multi-vault aggregation.+- A non-OpenAI-shape embedding provider (e.g. Google's native+ `:embedContent` endpoint). Users route Google through OpenRouter or+ Google's OpenAI-compatible endpoint.++## 3. Architectural Principles++1. **Filesystem-first.** Markdown is the source of truth. The SQLite index+ is recoverable: drop the file, run `o2b search reindex`, recover.+2. **Single SQL boundary.** Only `src/core/search/store.ts` imports+ `bun:sqlite`. Every other module talks to the store through a typed+ surface. This keeps backend substitution feasible.+3. **No daemon, no watcher.** Index is updated by explicit commands. An+ opt-in `--auto-refresh` flag on `o2b search query` runs incremental+ indexing once before the read.+4. **Optional semantic, never silent.** Semantic search is off unless the+ user enables it. When enabled but unavailable (extension missing,+ provider unreachable, no compatible embeddings), the system emits an+ explicit `warnings` entry rather than degrading silently.+5. **Read/write asymmetry on MCP.** The MCP surface only exposes+ `brain_search` and a status enrichment. Index management is operator+ business, never agent business.+6. **No prompt-shaped fallbacks.** A misconfigured provider or a missing+ key produces a typed `SearchError`, not a heuristic re-route.++## 4. Module Layout++```+src/core/search/+ types.ts // BrainSearchResult, IndexStats, SearchError, IndexCheckReport+ paths.ts // resolveIndexPath(vault, override?)+ store.ts // bun:sqlite wrapper; only file holding SQL+ schema.ts // DDL + MIGRATIONS[]+ walker.ts // vault traversal with ignore-paths+ chunker.ts // Markdown β chunks { content, start_line, end_line }+ indexer.ts // indexVault({...}) coordinating walker/chunker/store/embeddings+ fts.ts // FTS5 query construction and BM25 retrieval+ links.ts // wikilink / markdown-link / tag extraction+ ranker.ts // pure: merge keyword + semantic + boosts+ embeddings/+ provider.ts // EmbeddingProvider interface, makeProvider(config)+ openai-compat.ts // OpenAICompatProvider+ null-provider.ts // NullProvider (raises EmbeddingDisabledError)+ index.ts // public re-exports for CLI and MCP++src/cli/search.ts // verb dispatcher: query | index | reindex | status | check+src/mcp/search-tools.ts // brain_search tool, second_brain_status enrichment+```++Dependency direction is strictly downward:++```+cli/mcp β indexer/search β fts/links/ranker/embeddings β store β schema+```++There are no cycles. `embeddings/*` does not import `store` β embeddings are+computed outside and passed in to `store.vecUpsert(...)`.++### Invariants++- `store.ts` is the only module importing `bun:sqlite`.+- `NullProvider.embed()` throws `SearchError("EMBEDDING_DISABLED", ...)` β+ not a no-op β so a caller that reaches it has a configuration bug.+- Search reads chunk content from SQLite, not from disk. The disk is only+ read by the indexer when refreshing.+- The existing `o2b index` (Markdown index generator at+ `AI Wiki/index.md`) is not modified.++## 5. SQLite Schema++Location: `<vault>/.open-second-brain/brain.sqlite`. Overridable through+`search_db_path` in config and `--db` CLI flag. Pragmas applied on open:+`journal_mode=WAL`, `foreign_keys=ON`, `synchronous=NORMAL`.++```sql+CREATE TABLE documents (+ id INTEGER PRIMARY KEY,+ path TEXT NOT NULL UNIQUE,+ title TEXT,+ content_hash TEXT NOT NULL,+ mtime INTEGER NOT NULL,+ size INTEGER NOT NULL,+ created_at TEXT NOT NULL,+ updated_at TEXT NOT NULL,+ indexed_at TEXT NOT NULL+);+CREATE INDEX idx_documents_path ON documents(path);+CREATE INDEX idx_documents_mtime ON documents(mtime);++CREATE TABLE chunks (+ id INTEGER PRIMARY KEY,+ document_id INTEGER NOT NULL REFERENCES documents(id) ON DELETE CASCADE,+ chunk_index INTEGER NOT NULL,+ content TEXT NOT NULL,+ content_hash TEXT NOT NULL,+ start_line INTEGER NOT NULL,+ end_line INTEGER NOT NULL,+ token_count INTEGER NOT NULL,+ created_at TEXT NOT NULL,+ updated_at TEXT NOT NULL,+ UNIQUE(document_id, chunk_index)+);+CREATE INDEX idx_chunks_document ON chunks(document_id);++CREATE VIRTUAL TABLE chunk_fts USING fts5(+ content,+ content='chunks',+ content_rowid='id',+ tokenize='unicode61 remove_diacritics 2'+);+-- Triggers chunks_ai / chunks_ad / chunks_au keep chunk_fts in sync.+-- FTS holds only `content` (the single source column on `chunks`). Title+-- and path are not separate FTS columns; they reach the result set via+-- JOIN to `documents` at query time, and title text remains searchable+-- because Β§6's synthetic frontmatter chunk embeds it as part of the+-- chunk body.++CREATE TABLE embeddings (+ chunk_id INTEGER PRIMARY KEY REFERENCES chunks(id) ON DELETE CASCADE,+ model TEXT NOT NULL,+ dimension INTEGER NOT NULL,+ embedding_hash TEXT NOT NULL,+ created_at TEXT NOT NULL,+ updated_at TEXT NOT NULL+);+CREATE INDEX idx_embeddings_model ON embeddings(model);++-- Only when sqlite-vec is loaded:+CREATE VIRTUAL TABLE chunk_vec USING vec0(embedding float[<DIM>]);+CREATE TABLE chunk_vec_map (+ chunk_id INTEGER PRIMARY KEY REFERENCES chunks(id) ON DELETE CASCADE,+ vec_rowid INTEGER NOT NULL UNIQUE+);++CREATE TABLE links (+ id INTEGER PRIMARY KEY,+ source_document_id INTEGER NOT NULL REFERENCES documents(id) ON DELETE CASCADE,+ source_chunk_id INTEGER REFERENCES chunks(id) ON DELETE CASCADE,+ target_path TEXT,+ target_document_id INTEGER REFERENCES documents(id) ON DELETE SET NULL,+ link_text TEXT,+ link_type TEXT NOT NULL CHECK(link_type IN ('wikilink','markdown_link','tag')),+ created_at TEXT NOT NULL+);+CREATE INDEX idx_links_source ON links(source_document_id);+CREATE INDEX idx_links_target_doc ON links(target_document_id);+CREATE INDEX idx_links_target_path ON links(target_path);++CREATE TABLE index_state (+ key TEXT PRIMARY KEY,+ value TEXT NOT NULL,+ updated_at TEXT NOT NULL+);+-- Known keys: schema_version, embedding_model, embedding_dimension,+-- last_full_index_at, vec_extension_available.+```++### Decisions++- `documents.path` is vault-relative with POSIX separators. The walker+ normalises `\` to `/` on upsert. This keeps the schema portable across+ Syncthing peers.+- `content_hash` exists on two levels (document and chunk). File hash is+ the fast-path skip for incremental reindex; chunk hash is the skip key+ for recomputing embeddings.+- FTS5 uses `content='chunks'` (external content) so the chunk text is+ not duplicated. Standard `_ai/_ad/_au` triggers keep the FTS table in+ sync.+- Tokeniser is `unicode61 remove_diacritics 2`. This covers Latin and+ Cyrillic; stemming is not enabled because it would introduce a+ dependency and break determinism across machines.+- `sqlite-vec` is loaded via `Database.loadExtension(...)`. If it+ succeeds, the virtual table `chunk_vec` and its map are created.+ Dimension is fixed at first index and recorded in `index_state`.+ Changing the embedding model or dimension drops `embeddings`,+ `chunk_vec`, and `chunk_vec_map`; `chunks` and `chunk_fts` are+ preserved.+- **`chunk_vec` is a virtual `vec0` table; SQLite foreign-key cascade+ does not reach it.** Only `chunk_vec_map` cascades from `chunks`.+ Every `store` mutation that deletes chunks therefore takes the+ explicit two-step shape: first+ `DELETE FROM chunk_vec WHERE rowid IN (SELECT vec_rowid FROM chunk_vec_map WHERE chunk_id IN (...))`,+ then delete from `chunks` (which cascades to `chunk_vec_map` and via+ triggers removes the FTS rows). This contract is enforced by+ `store.deleteChunks(ids)` / `store.replaceChunks(documentId, ...)` /+ `store.deleteDocument(path)`; no caller talks to `chunk_vec`+ directly. `tests/core/search/store.vec.test.ts` verifies that+ deleting a document leaves zero rows in `chunk_vec` and+ `chunk_vec_map`.+- WAL mode lets readers (`o2b search query`) run while a writer+ (`o2b search index`) holds an exclusive lock. The writer lock is+ enforced through `proper-lockfile` on `brain.sqlite.lock`.++## 6. Chunking++Two passes.++**Pass 1 β structural split.** A single linewise traversal of the file+produces blocks. The block boundaries are:++- ATX heading (`^#{1,6}\s`) β starts a new block.+- Blank line separating non-empty content β paragraph boundary.+- Fenced code (`` ``` `` or `~~~`) β atomic block. An unclosed fence runs+ to EOF.+- YAML frontmatter (`---` β¦ `---` at start of file) β atomic block.+- Bulleted or numbered list of one nesting level β atomic until first+ non-list line.+- Markdown table (line with `|` followed by a `|---` separator) β atomic+ until blank line.++**Pass 2 β pack blocks into chunks.** The packer accumulates blocks while+`tokens(chunk) + tokens(next_block) β€ max_tokens`. Atomic blocks larger+than `max_tokens` become single chunks unmodified. A heading-only block+starts a new chunk only if the current chunk has a non-heading and+`tokens(chunk) β₯ min_tokens`; otherwise the heading attaches to the+current chunk. Each chunk begins with an overlap of the last+`overlap_tokens` lines from the previous chunk; `start_line`/`end_line`+are recorded from the first non-overlap line so line ranges remain+truthful.++Parameters for v0.10.0:++| Parameter | Value | Rationale |+| ----------------- | ----: | ------------------------------------ |+| `max_tokens` | 800 | mid of 500β1000 recommendation |+| `min_tokens` | 100 | smaller chunks return uninformative hits |+| `overlap_tokens` | 100 | ~12.5% β preserves context cheaply |++Tokens are approximated by whitespace-separated word count+(`text.split(/\s+/).filter(Boolean).length`). Deterministic, dependency-free.++### Title resolution++First non-empty wins: `title:` in YAML frontmatter β first H1 in body β+filename with `-`/`_` replaced by spaces.++### Frontmatter as synthetic chunk++If YAML frontmatter exists and is non-empty, it becomes `chunk_index=0`+with the raw frontmatter text as content. This makes `tags:` and other+frontmatter fields searchable through FTS5. Files without frontmatter+start at `chunk_index=0` from their first structural chunk. Malformed+frontmatter (unterminated block, broken syntax) does not abort the+file: it is omitted from chunking, the file is otherwise indexed+normally, and a warning is recorded in `IndexStats.errors` for the+operator to act on.++### Edge cases++- Empty file / whitespace-only: `documents` row is written for tracking,+ zero `chunks`.+- Non-UTF8 bytes: read fails fatally, file is logged as an error in+ `IndexStats.errors`, no `documents` row is written.+- Symlinks: followed only when the target is inside the vault (verified+ via `realpath`). External symlinks are skipped with a warning.++## 7. Ranking++Final score:++```+final_score = clamp01(+ keyword_weight * keyword_score+ + semantic_weight * semantic_score+ + link_boost+ + recency_boost+)+```++Defaults: `keyword_weight = 0.6`, `semantic_weight = 0.4`. When semantic+is disabled the weights collapse to `1.0 / 0.0`. CLI/MCP/config may+override; the ranker validates that the sum is `β€ 1.0`.++**keyword_score** is min-max normalised BM25 within the query's result+set: `keyword_score = (bm25_raw - min) / (max - min)` over the top-K+keyword candidates; if `max == min`, every candidate gets `1.0`. This+removes the dependency on absolute BM25 magnitudes (which depend on+corpus size).++**semantic_score** is cosine similarity, computed via L2 on+unit-normalised vectors. Vectors are normalised to unit length at write+time, so `cosine_similarity = 1 - (l2_distanceΒ² / 2)`, mapped to+`[0, 1]` with `max(0, similarity)`.++**link_boost** β `[0, 0.05]`. Per chunk: `+0.02` if another result in+the same query references it via `[[wikilink]]` (cap `0.03`); `+0.01` if+it shares a tag with another result (cap `0.02`).++**recency_boost** β `[0, 0.05]`. Step function on `mtime`:+β€ 7d β `0.05`, β€ 30d β `0.025`, β€ 90d β `0.01`, older β `0`.++### Candidate sets++Keyword candidates: top `K_kw = limit * 3`. Semantic candidates: top+`K_sem = max(limit * 5, 50)`. The two sets are unioned by `chunk_id`;+absent scores are zero on the other side. Tie-break order on equal+`final_score`: `keyword_score desc`, `mtime desc`, `chunk_id asc`.++### `searchType`++Returned as diagnostic only:++- `"hybrid"` if the chunk came from both sets.+- `"keyword"` if only FTS5.+- `"semantic"` if only sqlite-vec.++### Semantic-unavailable behaviour++A semantic request is **explicit** when the caller actively asks for+semantic search: CLI `--semantic`, MCP `semantic: true`. Anything else+(`--keyword-only`, MCP `semantic: false`, the implicit default that+honours `search_semantic_enabled` from config) is **implicit**.++The rule:++- **Implicit + anything missing β warn + keyword-only fallback, exit 0.**+ The caller did not pin the search to semantic; degrading is safe.+- **Explicit + infrastructure missing β fail hard.** sqlite-vec did not+ load, or the embedding provider returned an error / timed out: the+ ranker cannot honour the request, so it surfaces a typed+ `SearchError`. CLI exit 1, MCP `INTERNAL_ERROR`.+- **Explicit + data state recoverable β warn + keyword-only fallback,+ exit 0.** The index simply has no embeddings yet for the current+ model. No infrastructure is broken; the user needs+ `o2b search index --embeddings`, not a panic.++Concretely:++| Trigger | Implicit | Explicit |+|---|---|---|+| `search_semantic_enabled=false` | keyword-only, no warning | n/a (config gate doesn't apply when explicit) |+| sqlite-vec failed to load | warn `"sqlite-vec unavailable, semantic disabled this session"`, exit 0 | `VEC_EXTENSION_UNAVAILABLE`, exit 1 |+| Embedding provider unreachable / timeout | warn `"embedding provider unavailable: <reason>"`, exit 0 | `EMBEDDING_PROVIDER_HTTP` / `_TIMEOUT`, exit 1 |+| Embedding key missing | warn `"embedding key not configured; semantic disabled"`, exit 0 | `EMBEDDING_KEY_MISSING`, exit 1 |+| Index has no compatible embeddings | warn `"no compatible embeddings; run: o2b search index --embeddings"`, exit 0 | same warning, exit 0 (data state, not infra) |++## 8. CLI++All verbs sit under `o2b search`. The default verb is `query`+(positional argument is the query string).++### `o2b search <query>`++Flags: `--vault`, `--db`, `--limit` (1..100, default 10), `--semantic`,+`--keyword-only`, `--path` (prefix filter), `--keyword-weight`,+`--semantic-weight`, `--auto-refresh`, `--json`, `--verbose`.++Human format:++```+[1] Brain/preferences/pref-russian-replies.md β’ 0.81+ line 12-38 β’ hybrid β’ mtime 2026-05-14+ ΠΡΠ°Π²ΠΈΠ»Π° ΠΎΠ±ΡΠ΅Π½ΠΈΡ: Π²ΡΠ΅Π³Π΄Π° ΠΎΡΠ²Π΅ΡΠ°ΡΡ ΠΏΠΎ-ΡΡΡΡΠΊΠΈ, ΠΊΡΠΎΠΌΠ΅ ΠΊΠΎΠ³Π΄Π° ΡΠ΅ΡΡ ΠΎβ¦+```++JSON format: object+`{ results: BrainSearchResult[]; warnings: string[]; total: number }`.+`results` is the array of ranked hits, `warnings` is empty when nothing+to report, `total` is `min(matched, limit)`.++Exit codes: `0` always for success including zero results, `1` for runtime+errors, `2` for bad flags.++### `o2b search index`++Incremental indexing. Flags: `--vault`, `--db`, `--embeddings`,+`--force` (recompute everything, do not delete file), `--concurrency`+(1..16, default 4), `--verbose`, `--json`.++The writer holds an exclusive lock through `proper-lockfile`. A+concurrent run fails fast with `INDEX_LOCKED`.++Output (human):++```+indexing vault: /root/vault+ added: 23 files, 187 chunks+ updated: 4 files, 31 chunks+ unchanged: 412 files+ deleted: 2 files (chunks purged)+ embeddings: 218/218 OK (4 retries)+done in 12.4s+```++JSON: `{stats: {...}, errors: [...], duration_ms}`.++### `o2b search reindex`++Full rebuild. Writes to `brain.sqlite.new`, then `rename(.new, .)`+atomically, then the old file is preserved as `.bak` until the next+successful reindex. On startup, an orphaned `.bak` is restored if the+main file is missing or unreadable.++### `o2b search status`++Read-only summary. Does not take the writer lock. Human and `--json`+forms. Reports index path, schema version, document/chunk/embedding+counts, embedding model and dimension, sqlite-vec status, embedding key+presence (not value), `last_indexed_at`, `last_full_index_at`,+`stale_embeddings` count, and warnings. With no index: prints+`index: not initialised. Run: o2b search index` and exits 0.++### `o2b search check`++Pre-flight diagnostic. Verifies vault readable, index directory writable,+`bun:sqlite` open and FTS5 supported, sqlite-vec loadable (if semantic+enabled), embedding key resolvable (config or env), and β if key is+present β performs a single `embed(["check"])` probe with a 5-second+timeout. Exits 0 unless a hard precondition fails (1/2/3); semantic+issues are reported as warnings, exit 0.++### Parity table++| Verb | CLI | MCP |+| -------- | ------------------------------------ | ------------------------- |+| query | `o2b search "..."` | `brain_search` |+| index | `o2b search index [--embeddings]` | β |+| reindex | `o2b search reindex` | β |+| status | `o2b search status` | `second_brain_status.search` |+| check | `o2b search check` | β |++Mutating and HTTP-spending verbs are operator-only.++## 9. MCP++### `brain_search`++Input schema:++```json+{+ "type": "object",+ "properties": {+ "query": { "type": "string", "minLength": 1, "maxLength": 2000 },+ "limit": { "type": "integer", "minimum": 1, "maximum": 50, "default": 10 },+ "semantic": { "type": "boolean", "default": null },+ "keyword_only": { "type": "boolean", "default": false },+ "path_prefix": { "type": "string", "maxLength": 256 }+ },+ "required": ["query"],+ "additionalProperties": false+}+```++`semantic: null` uses the config default; `true` or `false` overrides for+the call. `path_prefix` is passed through `ensureInsideVault` to reject+escape attempts.++Output:++```json+{+ "results": [+ {+ "path": "Brain/preferences/pref-russian-replies.md",+ "title": "Russian replies",+ "content": "...",+ "score": 0.81,+ "startLine": 12,+ "endLine": 38,+ "searchType": "hybrid"+ }+ ],+ "warnings": [],+ "total": 1+}+```++`content` is truncated to 600 characters per chunk with `β¦` suffix on+overflow. The diagnostic score components (`keywordScore`,+`semanticScore`, `linkBoost`, `recencyBoost`) are intentionally absent β+they are noise for the agent and live in CLI `--verbose` only.++`limit` is capped at 50 (not 100 as for CLI) to keep the agent context+small.++### Error mapping++| Situation | MCP code | Message |+| ------------------------------------------ | ----------------- | -------------------------------------------------------------------- |+| missing/empty `query` | `INVALID_PARAMS` | `missing required argument: query` |+| `path_prefix` escapes vault | `INVALID_PARAMS` | `path_prefix escapes vault` |+| index file missing | `INTERNAL_ERROR` | `search index not initialised. Run: o2b search index` |+| index file unreadable | `INTERNAL_ERROR` | `search index unreadable: <reason>` |+| explicit `semantic=true` without vec | `INTERNAL_ERROR` | `semantic search unavailable: sqlite-vec extension not loaded` |+| explicit `semantic=true`, key missing | `INTERNAL_ERROR` | `embedding key not configured` |+| explicit `semantic=true`, provider down | `INTERNAL_ERROR` | `embedding provider unavailable: <reason>` |+| operation > 10s | `INTERNAL_ERROR` | `search timeout after 10000ms` |++Implicit `semantic: null` never produces an error from any of the+infrastructure rows above β those become entries in the `warnings`+array per Β§7's table.++### `second_brain_status` enrichment++The existing tool gains a `search` block when `search_enabled !== false`:++```json+{+ "search": {+ "index_path": "/root/vault/.open-second-brain/brain.sqlite",+ "exists": true,+ "schema_version": 1,+ "documents": 441,+ "chunks": 1827,+ "embeddings": 1827,+ "stale_embeddings": 0,+ "embedding_model": "google/gemini-embedding-2-preview",+ "embedding_dimension": 768,+ "vec_extension": "loaded",+ "semantic_enabled": true,+ "embedding_key_present": true,+ "last_indexed_at": "2026-05-16T14:22:17Z",+ "last_full_index_at": "2026-05-10T09:01:03Z"+ }+}+```++`embedding_key_present` is a boolean; the key itself is redacted by the+existing `redactMapping` helper. When the index does not exist, the+block is `{ "exists": false, "hint": "run: o2b search index" }`.++## 10. Configuration++The existing config parser handles flat `key: value` YAML only. All new+keys use prefixes `search_*` and `embedding_*`:++```yaml+# Existing+vault: "/root/vault"+timezone: "Europe/Belgrade"+agent_name: "@claude-vps-agent"++# New (all optional)+search_enabled: "true"+search_db_path: "" # blank β default <vault>/.open-second-brain/brain.sqlite+search_chunk_size: "800"+search_chunk_overlap: "100"+search_keyword_weight: "0.6"+search_semantic_weight: "0.4"+search_ignore_paths: ".git,node_modules,.open-second-brain,.obsidian/cache,.trash,.stversions"++search_semantic_enabled: "false"+embedding_provider: "openai-compat"+embedding_base_url: "https://openrouter.ai/api/v1"+embedding_model: "google/gemini-embedding-2-preview"+embedding_api_key: "" # blank β check env; if env empty, semantic disabled+embedding_dimension: "" # blank β autodetect from first response+embedding_timeout_ms: "10000"+embedding_concurrency: "4"+embedding_batch_size: "32"+```++ENV resolution order is `env β config β default` per key:++| Config key | ENV | Default |+| ----------------------- | -------------------------------------------- | ------------------------------------------------ |+| `search_db_path` | `OPEN_SECOND_BRAIN_SEARCH_DB` | `<vault>/.open-second-brain/brain.sqlite` |+| `search_semantic_enabled` | `OPEN_SECOND_BRAIN_SEARCH_SEMANTIC` | `false` |+| `embedding_provider` | `OPEN_SECOND_BRAIN_EMBEDDING_PROVIDER` | `openai-compat` |+| `embedding_base_url` | `OPEN_SECOND_BRAIN_EMBEDDING_BASE_URL` | β (required if semantic on) |+| `embedding_model` | `OPEN_SECOND_BRAIN_EMBEDDING_MODEL` | β (required if semantic on) |+| `embedding_api_key` | `OPEN_SECOND_BRAIN_EMBEDDING_KEY` | β (required if semantic on) |+| `embedding_dimension` | `OPEN_SECOND_BRAIN_EMBEDDING_DIM` | autodetect |++`search_ignore_paths` is comma-separated, exact directory-name match+(not glob). Numeric and boolean fields are parsed with explicit+validation; invalid values cause `INVALID_INPUT` (CLI exit 2) rather+than a silent fallback.++`embedding_api_key` falls under the existing `SECRET_KEY_PARTS = ["key",+"token", ...]` heuristic and is redacted to `[REDACTED]` by+`redactMapping`. The exact name is also added to the redactor's list as+belt-and-braces.++### `o2b init` instruction block++When `search_semantic_enabled` is `true` (in config or env) but no key+is resolvable, and when the user runs `o2b init`, the bootstrap output+appends:++```+Search is enabled (keyword-only).++To enable semantic search, set the following (config file path is+printed below; alternatively use the environment variables):++ search_semantic_enabled: "true"+ embedding_base_url: "https://openrouter.ai/api/v1"+ embedding_model: "google/gemini-embedding-2-preview"+ embedding_api_key: "<your key>"++Config file: ~/.config/open-second-brain/config.yaml+Or via env: OPEN_SECOND_BRAIN_SEARCH_SEMANTIC=true+ OPEN_SECOND_BRAIN_EMBEDDING_BASE_URL=...+ OPEN_SECOND_BRAIN_EMBEDDING_MODEL=...+ OPEN_SECOND_BRAIN_EMBEDDING_KEY=...++Then:+ o2b search check+ o2b search index --embeddings+```++The block prints once, only during `o2b init`. No nagging on every CLI+invocation. `o2b search check` is the dedicated diagnostic command.++## 11. Embedding Provider++```ts+export interface EmbeddingProvider {+ readonly name: string;+ readonly model: string;+ readonly dimension: number | null;+ embed(texts: readonly string[]): Promise<number[][]>;+ ping(): Promise<{ ok: true; dimension: number } | { ok: false; reason: string }>;+}++export function makeProvider(config: ResolvedEmbeddingConfig): EmbeddingProvider;+```++Two concrete implementations:++- `OpenAICompatProvider` for `embedding_provider: openai-compat`. Posts+ `{ model, input: texts, encoding_format: "float" }` to+ `${base_url}/embeddings` with `Authorization: Bearer <api_key>`.+ Batches `texts` into groups of `embedding_batch_size`. Concurrency is+ limited by a semaphore of width `embedding_concurrency`. Vectors are+ unit-normalised before being returned (so cosine works through L2 on+ `chunk_vec`). Retries on `429/500/502/503/504` and network errors+ with exponential backoff `1s/2s/4s + jitter`, three attempts total;+ other 4xx fail fast.+- `NullProvider` for `embedding_provider: disabled` and the implicit+ case when semantic is off. Its `embed()` throws+ `SearchError("EMBEDDING_DISABLED", ...)`; this exists to surface+ configuration bugs rather than silently no-op.++`makeProvider` throws `SearchError("INVALID_INPUT", ...)` synchronously+for an unknown `embedding_provider` value β eagerly at construction+time, not lazily on the first `embed()` call.++The single OpenAI-compatible class is intentional. OpenRouter, OpenAI,+Together, Mistral, Google's OpenAI-compat endpoint, local Ollama+(`http://localhost:11434/v1`), and a future authenticated Hermes proxy+all conform to this shape. Adding a Google-native `:embedContent`+adapter is deferred until a concrete user need surfaces; users can+already reach Google models through OpenRouter.++## 12. Public API++All public functions take a fully-resolved `ResolvedSearchConfig`.+Resolution happens once at the CLI/MCP boundary through+`resolveSearchConfig({...})`; the search module itself never reads+env or `config.yaml`. This keeps CLI / MCP / tests on a single+resolution path and isolates the search module from process-wide+state.++```ts+// types.ts+export interface ResolvedSearchConfig {+ readonly vault: string;+ readonly dbPath: string;+ readonly ignorePaths: ReadonlyArray<string>;+ readonly chunkSize: number;+ readonly chunkOverlap: number;+ readonly keywordWeight: number;+ readonly semanticWeight: number;+ readonly semantic: {+ readonly enabled: boolean;+ readonly provider: "openai-compat" | "disabled";+ readonly baseUrl: string | null;+ readonly model: string | null;+ readonly apiKey: string | null; // never logged; redactMapping handles display+ readonly dimension: number | null; // null = autodetect at first call+ readonly timeoutMs: number;+ readonly concurrency: number;+ readonly batchSize: number;+ };+}++// index.ts+export function resolveSearchConfig(opts: {+ vault?: string;+ configPath?: string;+ overrides?: Partial<ResolvedSearchConfig>; // CLI flag overrides go here+}): ResolvedSearchConfig;++export async function indexVault(+ config: ResolvedSearchConfig,+ opts?: {+ embeddings?: boolean;+ force?: boolean;+ onFile?: (event: {+ path: string;+ kind: "added" | "updated" | "unchanged" | "deleted" | "error";+ message?: string;+ }) => void;+ },+): Promise<IndexStats>;++export async function reindexVault(+ config: ResolvedSearchConfig,+ opts?: { embeddings?: boolean },+): Promise<IndexStats>;++export async function search(+ config: ResolvedSearchConfig,+ opts: SearchOptions,+): Promise<SearchOutcome>;++export async function indexStatus(+ config: ResolvedSearchConfig,+): Promise<IndexStatusSnapshot>;++export async function indexCheck(+ config: ResolvedSearchConfig,+): Promise<IndexCheckReport>;+```++`resolveSearchConfig` implements the env β config β default chain+documented in Β§10 and validates numeric/boolean shapes; an invalid+field raises `SearchError("INVALID_INPUT", ...)`. `overrides` is+how `o2b search query --keyword-weight 0.8` reaches the ranker+without redundant flag-parsing inside the search module.++Result types β `BrainSearchResult`, `IndexStats`, `IndexStatusSnapshot`,+`IndexCheckReport`, `SearchOutcome` β are exported from `types.ts` and+returned `Object.freeze`-d. The `onFile` callback is the sole progress+channel; CLI consumes it for `--verbose` output, tests intercept it for+assertions.++## 13. Migrations and Recovery++Schema version is tracked in `index_state.schema_version`. Migrations+are an array of `{ version, up(db) }` functions applied sequentially in+a transaction. v0.10.0 ships `version 1` only (the initial schema).+Future minor versions add migrations; major schema changes that cannot+be migrated raise `SCHEMA_MISMATCH` with a `o2b search reindex` hint.++Embedding-model or dimension changes do not go through the migration+list. On open, if `index_state.embedding_model` or `embedding_dimension`+mismatches the resolved config, the store drops `embeddings`,+`chunk_vec`, `chunk_vec_map`, logs one line `embedding model changed+from X to Y, embeddings cleared`, and updates `index_state`. `chunks`+and `chunk_fts` are preserved. The next `o2b search index --embeddings`+repopulates vectors.++`o2b search reindex` writes to `brain.sqlite.new`, atomically renames to+`brain.sqlite`, and keeps the previous file as `brain.sqlite.bak` until+the next successful reindex. If `brain.sqlite` is missing or unreadable+at open time and a `.bak` exists, the `.bak` is restored automatically+with a stderr notice.++## 14. Errors++```ts+export class SearchError extends Error {+ readonly code: SearchErrorCode;+}++export type SearchErrorCode =+ | "INDEX_MISSING"+ | "INDEX_UNREADABLE"+ | "SCHEMA_MISMATCH"+ | "VEC_EXTENSION_UNAVAILABLE"+ | "EMBEDDING_DISABLED"+ | "EMBEDDING_KEY_MISSING"+ | "EMBEDDING_PROVIDER_HTTP"+ | "EMBEDDING_PROVIDER_TIMEOUT"+ | "EMBEDDING_DIMENSION_MISMATCH"+ | "INDEX_LOCKED"+ | "INVALID_INPUT";+```++CLI mapping: `INVALID_INPUT β exit 2`, everything else β exit 1.++MCP mapping: `INVALID_INPUT β INVALID_PARAMS`, everything else β+`INTERNAL_ERROR`.++Non-`SearchError` exceptions (sqlite-vec DDL crashes, malformed+extensions) bubble up wrapped as `INDEX_UNREADABLE` with the original+message preserved.++## 15. Concurrency and Locking++- Writers (`index`, `reindex`) hold an exclusive `proper-lockfile` on+ `brain.sqlite.lock`. Retry policy: three attempts, 1s backoff. After+ exhaustion: `INDEX_LOCKED`.+- Readers do not take the lock; WAL is sufficient.+- A single search invocation uses one connection inside a `BEGIN`/+ `COMMIT` pair so keyword and semantic queries see a consistent+ snapshot.++## 16. Idempotency++- `o2b search index` repeated with no changes: `unchanged: N`,+ `added/updated/deleted = 0`, `embeddings_computed = 0`.+- `o2b search reindex` repeated: both runs succeed; the second `.bak`+ replaces the first.+- `documents.path` is unique; double-indexing a file replaces its row+ rather than duplicating.++## 17. Testing++Test framework: `bun test` (already the project standard). Helpers:++```+tests/helpers/+ search-fixtures.ts // build temporary vaults with fixed content+ mock-embedding.ts // MockEmbeddingProvider β deterministic sha256-derived vectors+ fake-http.ts // local OpenAI-compatible server for provider tests+```++Test files:++| File | Coverage |+| --------------------------------------------- | ------------------------------------------------------------------------------------- |+| `tests/core/search/walker.test.ts` | ignore list, symlinks, hidden dirs, non-md files |+| `tests/core/search/chunker.test.ts` | empty file, single line, code fence > max, frontmatter + H1, heading without body, lists, tables, UTF-8 Cyrillic |+| `tests/core/search/store.test.ts` | open/migrate, upsert/replace/delete, FTS triggers, schema-mismatch error, INDEX_LOCKED |+| `tests/core/search/store.vec.test.ts` | when sqlite-vec loads: round-trip; when not: graceful skip |+| `tests/core/search/fts.test.ts` | BM25 ordering, path-prefix filter, Cyrillic, escape of query special chars |+| `tests/core/search/links.test.ts` | wikilink, markdown link, tag; resolve `target_document_id` when target is indexed |+| `tests/core/search/ranker.test.ts` | union, weights, clamp, link/recency boost, tie-break, keyword-only fallback |+| `tests/core/search/indexer.test.ts` | E2E: index β incremental skip β modify β delete β reindex |+| `tests/core/search/indexer.embeddings.test.ts`| `onFile` events, embeddings computed for new/updated, not for unchanged, model-change clears embeddings |+| `tests/core/search/embeddings.test.ts` | `OpenAICompatProvider` against fake-http: batching, sort by `index`, retry, timeout, dim-mismatch, 4xx fail-fast |+| `tests/core/search/search.test.ts` | keyword / semantic / hybrid / keyword-only / path-filter; warning when vec unavailable |+| `tests/cli/search.test.ts` | every verb; flag parsing; exit codes 0/1/2; JSON output |+| `tests/mcp/search.test.ts` | `brain_search` happy path; `INVALID_PARAMS` cases; `INTERNAL_ERROR` cases; warning case |++### Performance bench (informative)++`tests/core/search/bench.test.ts` is opt-in (`bun test --filter="bench"`).+Baselines, captured in this document on first run:++- Indexing 1000 synthetic Markdown files: under 30s without embeddings;+ under 5 minutes with embeddings (network-bound).+- Keyword search across five representative queries: average under+ 100 ms.+- Hybrid search with a mocked vector store: average under 200 ms.++CI does not run the bench.++## 18. Acceptance Criteria++A reviewer marks the PR ready to merge when all 20 items below are+verified, either by an automated test or by a documented manual check:++1. `o2b search index` creates `<vault>/.open-second-brain/brain.sqlite`+ with all tables and `schema_version=1`.+2. A second `o2b search index` with no file changes reports+ `unchanged: N`, `added/updated/deleted = 0`,+ `embeddings_computed = 0`.+3. Modifying a file's content and `mtime` produces an `updated` entry,+ deletes the old chunks, and indexes the new chunks.+4. Removing a file from disk produces a `deleted` entry on the next+ `index`; queries no longer return its content.+5. `o2b search "<Cyrillic>"` returns hits against a Cyrillic-content+ vault.+6. `o2b search reindex` rebuilds atomically; if killed mid-run, the next+ open restores from `.bak`.+7. `o2b search status` without an index prints `not initialised`; with+ an index prints accurate counts.+8. `o2b search check` without `embedding_api_key` prints MISSING and the+ key-setup instructions (exit 0); with an unreachable provider it+ prints the provider error (exit 0).+9. `o2b search index --embeddings` without a resolvable key fails with+ `EMBEDDING_KEY_MISSING` (exit 1).+10. With a working provider (`fake-http` in the test), `o2b search+ index --embeddings` writes vectors into `chunk_vec` and populates+ `embeddings.embedding_hash` correctly.+11. `o2b search "..." --semantic` with sqlite-vec loaded but empty+ `embeddings` returns a keyword-only result plus the+ `no compatible embeddings` warning, exit 0 (data-state case,+ treated identically for implicit and explicit per Β§7).+12. Implicit semantic (no `--semantic`, semantic enabled in config)+ with sqlite-vec unloaded β keyword-only result + warning, exit 0.+ Explicit `o2b search "..." --semantic` with sqlite-vec unloaded+ β exit 1, error code `VEC_EXTENSION_UNAVAILABLE`. Explicit+ `--semantic` with provider unreachable β exit 1, error code+ `EMBEDDING_PROVIDER_HTTP` or `_TIMEOUT`.+13. Changing `embedding_model` in config drops `embeddings`,+ `chunk_vec`, `chunk_vec_map`; logs one line; preserves FTS5 and+ `chunks`; next `index --embeddings` repopulates.+14. MCP `brain_search` returns results with `content β€ 600` chars,+ without diagnostic score components, with `limit β€ 50`.+15. Two concurrent `o2b search index` processes: the first succeeds,+ the second exits with `INDEX_LOCKED`.+16. The existing `o2b index` (Markdown index generator) continues to+ work unchanged; its tests remain green.+17. MCP `second_brain_status` exposes a `search.*` block with accurate+ counters.+18. Performance benchmark (informative, not a merge gate): keyword+ search across 1000 chunks below 100 ms; hybrid below 500 ms+ excluding the query-embedding round trip.+19. All new TypeScript files pass `tsc --noEmit` with the project's+ strict configuration.+20. `bun test` is green end to end; existing Brain, Pay Memory, and CLI+ suites do not regress.++## 19. Versioning and CHANGELOG++Release: `0.10.0` (minor bump β new feature, backwards compatible). One+CHANGELOG entry under `## 0.10.0`:++```+### Added+- Brain search: SQLite + FTS5 index over the entire vault with optional+ semantic layer via sqlite-vec and any OpenAI-compatible embeddings+ provider. CLI verbs `o2b search query|index|reindex|status|check`,+ MCP tool `brain_search`. See docs/plans/2026-05-16-brain-search-design.md.+```++No `[Unreleased]` section (project convention). The version is updated+in `package.json` and `pyproject.toml` as part of the merge through+`bun run sync-version`.diff --git a/docs/plans/2026-05-16-brain-search-impl.md b/docs/plans/2026-05-16-brain-search-impl.mdnew file mode 100644index 00000000..2ae64325--- /dev/null+++ b/docs/plans/2026-05-16-brain-search-impl.md@@ -0,0 +1,656 @@+# Brain Search Implementation Plan++> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.++**Goal:** Ship a deterministic full-text search over the vault with optional semantic layer, exposed via CLI (`o2b search`) and MCP (`brain_search`).++**Architecture:** New isolated module `src/core/search/` owns walker, chunker, SQLite store (single SQL boundary), FTS query, links, ranker, embedding providers. CLI and MCP consume a resolved config plus the public API. Keyword search is always available; semantic search loads `sqlite-vec` and any OpenAI-compatible `/v1/embeddings` provider when configured.++**Tech Stack:** TypeScript on Bun, `bun:sqlite` (FTS5 built-in), `sqlite-vec` extension (optional), `proper-lockfile`, OpenAI-compatible HTTP embeddings.++**Source of truth for behaviour:** [`docs/plans/2026-05-16-brain-search-design.md`](./2026-05-16-brain-search-design.md). Every task here implements a slice of that spec β if a conflict appears, the spec wins and the plan must be updated.++---++## Plan-wide conventions++These apply to every task; do not re-state them per step.++- **Imports.** Production code uses `node:`-prefixed Node builtins (`node:fs`, `node:path`) and `bun:sqlite`. Tests use `import { test, expect, describe, beforeEach, afterEach } from "bun:test"`. Path: always `.ts` extensions in imports (project convention, `allowImportingTsExtensions: true`).+- **Result shape.** Every public-API return value is `Object.freeze`-d at the call site that produces it. This mirrors `src/core/brain/query.ts`.+- **Errors.** All user-facing errors from this module are instances of `SearchError` (defined in Task 1). Non-`SearchError` exceptions bubble up as-is β do not wrap them.+- **No git from this plan.** The user does git work themselves. Each task ends with **Pause for review (no commit)** instead of `git add`/`git commit`. The reviewer either runs `bun test` themselves to confirm green, or relies on the in-task `bun test ...` output.+- **No bait fallbacks.** "Silent semantic downgrade" is allowed only on the implicit code path (config-default semantic). On the explicit code path (CLI `--semantic`, MCP `semantic: true`) the implementation MUST throw `SearchError` and let CLI/MCP translate to the right exit code.+- **Atomic writes** for any file touched outside SQLite go through `src/core/fs-atomic.ts:atomicWriteFileSync`. SQLite files are mutated through prepared statements inside `BEGIN`/`COMMIT`; the temp-file rename pattern is reserved for `o2b search reindex` (Task 24).+- **Verification.** Every task ends with `bun test tests/path/to/file.test.ts` and an expected pass count. If the suite has > 10 tests, expect `(pass N)` exactly. CI green is `bun test` across the repo β confirmed at the end of each Phase.++## File map++Create:++```+src/core/search/types.ts β SearchError, BrainSearchResult, IndexStats, IndexStatusSnapshot, IndexCheckReport, SearchOutcome, ResolvedSearchConfig, SearchOptions+src/core/search/paths.ts β resolveIndexPath(config) β string+src/core/search/schema.ts β DDL, MIGRATIONS[], applyV1+src/core/search/store.ts β Store class wrapping bun:sqlite (only SQL home)+src/core/search/walker.ts β walkVault(config) async-iterates *.md paths+src/core/search/chunker.ts β chunkMarkdown(text) β BlockChunk[]+src/core/search/links.ts β extractLinks(content) β ExtractedLink[]+src/core/search/fts.ts β runFtsQuery(store, query, opts) β FtsHit[]+src/core/search/ranker.ts β mergeResults(kw, sem, opts) β BrainSearchResult[]+src/core/search/indexer.ts β indexVault(config, opts), reindexVault, etc.+src/core/search/embeddings/provider.ts β EmbeddingProvider interface + makeProvider()+src/core/search/embeddings/null-provider.ts β NullProvider+src/core/search/embeddings/openai-compat.ts β OpenAICompatProvider+src/core/search/index.ts β Public re-export surface for CLI + MCP+src/cli/search.ts β `o2b search` verb dispatcher+src/mcp/search-tools.ts β brain_search MCP tool + search status enrichment+tests/helpers/search-fixtures.ts β createTempVault(name, files), writeMd helper+tests/helpers/mock-embedding.ts β MockEmbeddingProvider (deterministic sha256 vectors)+tests/helpers/fake-http.ts β startFakeOpenAi() returns {url, close, setResponse}+tests/core/search/types.test.ts+tests/core/search/paths.test.ts+tests/core/search/config.test.ts+tests/core/search/schema.test.ts+tests/core/search/store.test.ts+tests/core/search/store.vec.test.ts+tests/core/search/walker.test.ts+tests/core/search/chunker.test.ts+tests/core/search/links.test.ts+tests/core/search/fts.test.ts+tests/core/search/ranker.test.ts+tests/core/search/indexer.test.ts+tests/core/search/indexer.embeddings.test.ts+tests/core/search/embeddings.test.ts+tests/core/search/search.test.ts+tests/cli/search.test.ts+tests/mcp/search.test.ts+```++Modify:++```+src/core/config.ts β extend SECRET_KEY_PARTS list to include 'embedding_api_key' explicitly (belt-and-braces); add config keys to discovery types if needed.+src/core/init.ts β print semantic-instructions block after vault bootstrap (Task 52).+src/cli/main.ts β register 'search' verb in dispatcher (Task 30).+src/mcp/tools.ts β register brain_search + extend second_brain_status payload (Tasks 32, 33).+package.json β add `sqlite-vec` to optionalDependencies; bump version 0.9.1 β 0.10.0 (Task 53).+pyproject.toml β bump version 0.9.1 β 0.10.0 via `bun run sync-version` (Task 53).+CHANGELOG.md β `## 0.10.0` entry (Task 53).+```++---++## Phase 1 β Foundation++Three small modules that everything else depends on. No SQLite yet.++### Task 1: Types and SearchError++**Files:**+- Create: `src/core/search/types.ts`+- Test: `tests/core/search/types.test.ts`++- [ ] **Step 1: Write the failing test for SearchError shape**++`tests/core/search/types.test.ts`:++```ts+import { test, expect } from "bun:test";+import { SearchError } from "../../../src/core/search/types.ts";++test("SearchError carries a typed code", () => {+ const err = new SearchError("INDEX_MISSING", "no index at /tmp/x");+ expect(err).toBeInstanceOf(Error);+ expect(err.name).toBe("SearchError");+ expect(err.code).toBe("INDEX_MISSING");+ expect(err.message).toBe("no index at /tmp/x");+});++test("SearchError preserves stack trace", () => {+ const err = new SearchError("INVALID_INPUT", "bad");+ expect(err.stack).toContain("SearchError");+});+```++- [ ] **Step 2: Run test to verify it fails**++Run: `bun test tests/core/search/types.test.ts`+Expected: FAIL with "Cannot find module" or similar.++- [ ] **Step 3: Implement types.ts**++`src/core/search/types.ts`:++```ts+/**+ * Public types for `src/core/search/*`. Plain data β no behaviour, no I/O.+ *+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§12, Β§14.+ */++export const SEARCH_ERROR_CODES = [+ "INDEX_MISSING",+ "INDEX_UNREADABLE",+ "SCHEMA_MISMATCH",+ "VEC_EXTENSION_UNAVAILABLE",+ "EMBEDDING_DISABLED",+ "EMBEDDING_KEY_MISSING",+ "EMBEDDING_PROVIDER_HTTP",+ "EMBEDDING_PROVIDER_TIMEOUT",+ "EMBEDDING_DIMENSION_MISMATCH",+ "INDEX_LOCKED",+ "INVALID_INPUT",+] as const;+export type SearchErrorCode = (typeof SEARCH_ERROR_CODES)[number];++export class SearchError extends Error {+ readonly code: SearchErrorCode;+ constructor(code: SearchErrorCode, message: string) {+ super(message);+ this.name = "SearchError";+ this.code = code;+ }+}++export interface BrainSearchResult {+ readonly documentId: number;+ readonly chunkId: number;+ readonly path: string;+ readonly title: string | null;+ readonly content: string;+ readonly startLine: number;+ readonly endLine: number;+ readonly score: number;+ readonly keywordScore: number;+ readonly semanticScore: number;+ readonly linkBoost: number;+ readonly recencyBoost: number;+ readonly searchType: "keyword" | "semantic" | "hybrid";+}++export interface IndexStats {+ readonly added: number;+ readonly updated: number;+ readonly unchanged: number;+ readonly deleted: number;+ readonly chunksTotal: number;+ readonly embeddingsComputed: number;+ readonly embeddingsRetries: number;+ readonly errors: ReadonlyArray<{ readonly path: string; readonly message: string }>;+ readonly durationMs: number;+}++export interface IndexStatusSnapshot {+ readonly indexPath: string;+ readonly exists: boolean;+ readonly schemaVersion: number | null;+ readonly documents: number;+ readonly chunks: number;+ readonly embeddings: number;+ readonly staleEmbeddings: number;+ readonly embeddingModel: string | null;+ readonly embeddingDimension: number | null;+ readonly vecExtension: "loaded" | "unavailable" | "unknown";+ readonly semanticEnabled: boolean;+ readonly embeddingKeyPresent: boolean;+ readonly lastIndexedAt: string | null;+ readonly lastFullIndexAt: string | null;+ readonly warnings: ReadonlyArray<string>;+}++export interface IndexCheckReport {+ readonly vaultReadable: boolean;+ readonly indexDirWritable: boolean;+ readonly sqliteOk: boolean;+ readonly fts5Ok: boolean;+ readonly vecExtension: "loaded" | "unavailable" | "not-attempted";+ readonly embeddingKeyResolved: boolean;+ readonly providerReachable: boolean | null; // null = not probed+ readonly providerReason: string | null;+ readonly warnings: ReadonlyArray<string>;+ readonly fatal: ReadonlyArray<string>;+}++export interface SearchOptions {+ readonly query: string;+ readonly limit?: number;+ readonly semantic?: boolean | null;+ readonly keywordOnly?: boolean;+ readonly pathPrefix?: string;+ readonly keywordWeight?: number;+ readonly semanticWeight?: number;+}++export interface SearchOutcome {+ readonly results: ReadonlyArray<BrainSearchResult>;+ readonly warnings: ReadonlyArray<string>;+ readonly total: number;+}++export interface ResolvedEmbeddingConfig {+ readonly enabled: boolean;+ readonly provider: "openai-compat" | "disabled";+ readonly baseUrl: string | null;+ readonly model: string | null;+ readonly apiKey: string | null;+ readonly dimension: number | null;+ readonly timeoutMs: number;+ readonly concurrency: number;+ readonly batchSize: number;+}++export interface ResolvedSearchConfig {+ readonly vault: string;+ readonly dbPath: string;+ readonly ignorePaths: ReadonlyArray<string>;+ readonly chunkSize: number;+ readonly chunkOverlap: number;+ readonly keywordWeight: number;+ readonly semanticWeight: number;+ readonly semantic: ResolvedEmbeddingConfig;+}+```++- [ ] **Step 4: Run test to verify pass**++Run: `bun test tests/core/search/types.test.ts`+Expected: `2 pass, 0 fail`.++- [ ] **Step 5: Typecheck**++Run: `bun run typecheck`+Expected: no errors. (Other modules may still error if they were partially implemented in earlier work β at this point only types.ts is new.)++- [ ] **Step 6: Pause for review (no commit)**++### Task 2: paths.ts β resolveIndexPath++**Files:**+- Create: `src/core/search/paths.ts`+- Test: `tests/core/search/paths.test.ts`++- [ ] **Step 1: Write the failing test**++`tests/core/search/paths.test.ts`:++```ts+import { test, expect } from "bun:test";+import { resolveIndexPath } from "../../../src/core/search/paths.ts";++test("default points under <vault>/.open-second-brain", () => {+ expect(resolveIndexPath("/v", null)).toBe("/v/.open-second-brain/brain.sqlite");+});++test("explicit override wins", () => {+ expect(resolveIndexPath("/v", "/tmp/custom.sqlite")).toBe("/tmp/custom.sqlite");+});++test("blank override falls back to default", () => {+ expect(resolveIndexPath("/v", "")).toBe("/v/.open-second-brain/brain.sqlite");+});+```++- [ ] **Step 2: Run test to verify it fails**++Run: `bun test tests/core/search/paths.test.ts`+Expected: FAIL with "Cannot find module".++- [ ] **Step 3: Implement paths.ts**++`src/core/search/paths.ts`:++```ts+/**+ * Resolve the on-disk location of the search index file.+ *+ * Default: `<vault>/.open-second-brain/brain.sqlite`. Overridable+ * through CLI `--db` or config `search_db_path`.+ */++import { join } from "node:path";++export function resolveIndexPath(vault: string, override: string | null): string {+ if (override && override.trim() !== "") return override;+ return join(vault, ".open-second-brain", "brain.sqlite");+}+```++- [ ] **Step 4: Run test to verify pass**++Run: `bun test tests/core/search/paths.test.ts`+Expected: `3 pass, 0 fail`.++- [ ] **Step 5: Pause for review (no commit)**++### Task 3: resolveSearchConfig++**Files:**+- Create: `src/core/search/index.ts` (just the `resolveSearchConfig` export; will grow later)+- Test: `tests/core/search/config.test.ts`++- [ ] **Step 1: Write the failing test**++`tests/core/search/config.test.ts`:++```ts+import { test, expect, beforeEach, afterEach } from "bun:test";+import { mkdtempSync, writeFileSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";+import { resolveSearchConfig } from "../../../src/core/search/index.ts";+import { SearchError } from "../../../src/core/search/types.ts";++let tmp: string;+let configPath: string;+let origEnv: Record<string, string | undefined>;++const ENV_KEYS = [+ "OPEN_SECOND_BRAIN_SEARCH_DB",+ "OPEN_SECOND_BRAIN_SEARCH_SEMANTIC",+ "OPEN_SECOND_BRAIN_EMBEDDING_PROVIDER",+ "OPEN_SECOND_BRAIN_EMBEDDING_BASE_URL",+ "OPEN_SECOND_BRAIN_EMBEDDING_MODEL",+ "OPEN_SECOND_BRAIN_EMBEDDING_KEY",+ "OPEN_SECOND_BRAIN_EMBEDDING_DIM",+];++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "osb-cfg-"));+ configPath = join(tmp, "config.yaml");+ origEnv = {};+ for (const k of ENV_KEYS) {+ origEnv[k] = process.env[k];+ delete process.env[k];+ }+});++afterEach(() => {+ for (const k of ENV_KEYS) {+ if (origEnv[k] === undefined) delete process.env[k];+ else process.env[k] = origEnv[k];+ }+ rmSync(tmp, { recursive: true, force: true });+});++test("defaults are returned when config and env are empty", () => {+ writeFileSync(configPath, `vault: "${tmp}"\n`);+ const cfg = resolveSearchConfig({ vault: tmp, configPath });+ expect(cfg.vault).toBe(tmp);+ expect(cfg.dbPath).toBe(join(tmp, ".open-second-brain", "brain.sqlite"));+ expect(cfg.chunkSize).toBe(800);+ expect(cfg.chunkOverlap).toBe(100);+ expect(cfg.keywordWeight).toBe(0.6);+ expect(cfg.semanticWeight).toBe(0.4);+ expect(cfg.semantic.enabled).toBe(false);+ expect(cfg.semantic.provider).toBe("openai-compat");+ expect(cfg.semantic.apiKey).toBeNull();+ expect(cfg.ignorePaths).toContain(".git");+ expect(cfg.ignorePaths).toContain(".open-second-brain");+});++test("env overrides config which overrides defaults", () => {+ writeFileSync(+ configPath,+ [+ `vault: "${tmp}"`,+ `search_chunk_size: "500"`,+ `search_semantic_enabled: "true"`,+ `embedding_base_url: "https://config.example/v1"`,+ `embedding_model: "config-model"`,+ `embedding_api_key: "cfg-key"`,+ ].join("\n") + "\n",+ );+ process.env["OPEN_SECOND_BRAIN_EMBEDDING_MODEL"] = "env-model";+ const cfg = resolveSearchConfig({ vault: tmp, configPath });+ expect(cfg.chunkSize).toBe(500);+ expect(cfg.semantic.enabled).toBe(true);+ expect(cfg.semantic.baseUrl).toBe("https://config.example/v1");+ expect(cfg.semantic.model).toBe("env-model"); // env wins+ expect(cfg.semantic.apiKey).toBe("cfg-key");+});++test("overrides win over both env and config", () => {+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_chunk_size: "500"\n`);+ process.env["OPEN_SECOND_BRAIN_SEARCH_SEMANTIC"] = "false";+ const cfg = resolveSearchConfig({+ vault: tmp,+ configPath,+ overrides: { keywordWeight: 0.9, semanticWeight: 0.1 },+ });+ expect(cfg.keywordWeight).toBe(0.9);+ expect(cfg.semanticWeight).toBe(0.1);+});++test("invalid numeric value throws INVALID_INPUT", () => {+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_chunk_size: "not-a-number"\n`);+ let err: SearchError | null = null;+ try {+ resolveSearchConfig({ vault: tmp, configPath });+ } catch (e) {+ err = e as SearchError;+ }+ expect(err).not.toBeNull();+ expect(err?.code).toBe("INVALID_INPUT");+ expect(err?.message).toContain("search_chunk_size");+});++test("weights summing above 1 is rejected", () => {+ writeFileSync(+ configPath,+ `vault: "${tmp}"\nsearch_keyword_weight: "0.7"\nsearch_semantic_weight: "0.5"\n`,+ );+ expect(() => resolveSearchConfig({ vault: tmp, configPath })).toThrow(/sum.*1/);+});+```++- [ ] **Step 2: Run test to verify it fails**++Run: `bun test tests/core/search/config.test.ts`+Expected: FAIL with "Cannot find module" (resolveSearchConfig not exported yet).++- [ ] **Step 3: Implement resolveSearchConfig**++`src/core/search/index.ts`:++```ts+/**+ * Public surface for `src/core/search/*`. Currently exports the config+ * resolver only; indexer/search/status/check are added in later phases.+ */++import { discoverConfig } from "../config.ts";+import { resolveIndexPath } from "./paths.ts";+import { SearchError } from "./types.ts";+import type { ResolvedSearchConfig, ResolvedEmbeddingConfig } from "./types.ts";++const DEFAULT_IGNORE_PATHS = [+ ".git",+ "node_modules",+ ".open-second-brain",+ ".obsidian/cache",+ ".trash",+ ".stversions",+];++const DEFAULTS = {+ chunkSize: 800,+ chunkOverlap: 100,+ keywordWeight: 0.6,+ semanticWeight: 0.4,+ provider: "openai-compat" as const,+ timeoutMs: 10_000,+ concurrency: 4,+ batchSize: 32,+};++function envOrConfig(+ env: NodeJS.ProcessEnv,+ config: Record<string, string>,+ envKey: string,+ configKey: string,+): string | null {+ const e = env[envKey];+ if (e !== undefined && e !== "") return e;+ const c = config[configKey];+ if (c !== undefined && c !== "") return c;+ return null;+}++function parseInteger(raw: string | null, fallback: number, fieldName: string): number {+ if (raw === null) return fallback;+ const n = Number(raw);+ if (!Number.isFinite(n) || !Number.isInteger(n)) {+ throw new SearchError("INVALID_INPUT", `${fieldName} must be an integer, got '${raw}'`);+ }+ return n;+}++function parseFloat01(raw: string | null, fallback: number, fieldName: string): number {+ if (raw === null) return fallback;+ const n = Number(raw);+ if (!Number.isFinite(n) || n < 0 || n > 1) {+ throw new SearchError("INVALID_INPUT", `${fieldName} must be a number in [0, 1], got '${raw}'`);+ }+ return n;+}++function parseBool(raw: string | null, fallback: boolean, fieldName: string): boolean {+ if (raw === null) return fallback;+ if (raw === "true" || raw === "1") return true;+ if (raw === "false" || raw === "0") return false;+ throw new SearchError("INVALID_INPUT", `${fieldName} must be 'true' or 'false', got '${raw}'`);+}++function parseProvider(raw: string | null): ResolvedEmbeddingConfig["provider"] {+ if (raw === null) return DEFAULTS.provider;+ if (raw === "openai-compat" || raw === "disabled") return raw;+ throw new SearchError("INVALID_INPUT", `embedding_provider must be 'openai-compat' or 'disabled', got '${raw}'`);+}++function parseIgnorePaths(raw: string | null): string[] {+ if (raw === null) return DEFAULT_IGNORE_PATHS;+ return raw.split(",").map((s) => s.trim()).filter((s) => s.length > 0);+}++export function resolveSearchConfig(opts: {+ vault: string;+ configPath?: string;+ overrides?: Partial<ResolvedSearchConfig>;+}): ResolvedSearchConfig {+ const env = process.env;+ const config = opts.configPath+ ? discoverConfig(opts.configPath).data+ : {} as Record<string, string>;++ const dbPath = resolveIndexPath(+ opts.vault,+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_DB", "search_db_path"),+ );++ const chunkSize = parseInteger(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_CHUNK_SIZE", "search_chunk_size"),+ DEFAULTS.chunkSize,+ "search_chunk_size",+ );+ const chunkOverlap = parseInteger(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_CHUNK_OVERLAP", "search_chunk_overlap"),+ DEFAULTS.chunkOverlap,+ "search_chunk_overlap",+ );+ const keywordWeight = parseFloat01(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_KW_WEIGHT", "search_keyword_weight"),+ DEFAULTS.keywordWeight,+ "search_keyword_weight",+ );+ const semanticWeight = parseFloat01(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_SEM_WEIGHT", "search_semantic_weight"),+ DEFAULTS.semanticWeight,+ "search_semantic_weight",+ );+ if (keywordWeight + semanticWeight > 1.0 + 1e-9) {+ throw new SearchError(+ "INVALID_INPUT",+ `keyword_weight + semantic_weight must sum to <= 1, got ${keywordWeight} + ${semanticWeight}`,+ );+ }++ const ignorePaths = parseIgnorePaths(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_IGNORE", "search_ignore_paths"),+ );++ const semanticEnabled = parseBool(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_SEMANTIC", "search_semantic_enabled"),+ false,+ "search_semantic_enabled",+ );+ const provider = parseProvider(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_PROVIDER", "embedding_provider"),+ );+ const baseUrl = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_BASE_URL", "embedding_base_url");+ const model = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_MODEL", "embedding_model");+ const apiKey = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_KEY", "embedding_api_key");+ const dimRaw = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_DIM", "embedding_dimension");+ const dimension = dimRaw === null ? null : parseInteger(dimRaw, 0, "embedding_dimension");+ const timeoutMs = parseInteger(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_TIMEOUT", "embedding_timeout_ms"),+ DEFAULTS.timeoutMs,+ "embedding_timeout_ms",+ );+ const concurrency = parseInteger(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_CONCURRENCY", "embedding_concurrency"),+ DEFAULTS.concurrency,+ "embedding_concurrency",+ );+ const batchSize = parseInteger(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_BATCH", "embedding_batch_size"),+ DEFAULTS.batchSize,+ "embedding_batch_size",+ );++ const semantic: ResolvedEmbeddingConfig = Object.freeze({+ enabled: semanticEnabled,+ provider,+ baseUrl,+ model,+ apiKey,+ dimension,+ timeoutMs,+ concurrency,+ batchSize,+ });++ const base: ResolvedSearchConfig = Object.freeze({+ vault: opts.vault,+ dbPath,+ ignorePaths: Object.freeze([...ignorePaths]),+ chunkSize,+ chunkOverlap,+ keywordWeight,+ semanticWeight,+ semantic,+ });++ if (!opts.overrides) return base;+ return Object.freeze({+ ...base,+ ...opts.overrides,+ semantic: Object.freeze({ ...base.semantic, ...(opts.overrides.semantic ?? {}) }),+ ignorePaths: opts.overrides.ignorePaths+ ? Object.freeze([...opts.overrides.ignorePaths])+ : base.ignorePaths,+ });+}+```++- [ ] **Step 4: Run test to verify pass**++Run: `bun test tests/core/search/config.test.ts`+Expected: `5 pass, 0 fail`.++- [ ] **Step 5: Run wider typecheck**++Run: `bun run typecheck`+Expected: no errors.++- [ ] **Step 6: Pause for review (no commit)**diff --git a/openclaw.plugin.json b/openclaw.plugin.jsonindex 36b1793b..0b2cddc5 100644--- a/openclaw.plugin.json+++ b/openclaw.plugin.json@@ -2,7 +2,7 @@"id": "open-second-brain","name": "Open Second Brain","description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",- "version": "0.9.1",+ "version": "0.10.0","activation": { "onStartup": true },"skills": ["./skills"],"contracts": {diff --git a/package.json b/package.jsonindex 352c01dc..763ad572 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "0.9.1",+ "version": "0.10.0","type": "module","private": false,"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.",@@ -47,5 +47,8 @@},"openclaw": {"extensions": ["./openclaw/index.js"]+ },+ "optionalDependencies": {+ "sqlite-vec": "^0.1.9"}}diff --git a/plugin.yaml b/plugin.yamlindex 3d11dd35..a76abe97 100644--- a/plugin.yaml+++ b/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "0.9.1"+version: "0.10.0"description: "Hermes plugin entrypoint for OpenSecondBrain vault health checks and the optional MCP tool server."author: "OpenSecondBrain contributors"kind: standalonediff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.jsonindex ca4bc433..20153cf1 100644--- a/plugins/codex/.codex-plugin/plugin.json+++ b/plugins/codex/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "0.9.1",+ "version": "0.10.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yamlindex 62c134b1..b6e76575 100644--- a/plugins/hermes/plugin.yaml+++ b/plugins/hermes/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "0.9.1"+version: "0.10.0"description: "Hermes adapter for OpenSecondBrain vault health checks and optional MCP tool server."author: "OpenSecondBrain contributors"kind: standalonediff --git a/pyproject.toml b/pyproject.tomlindex 32469c6e..3d183f0b 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"# CLI entry points (those moved to `package.json` `bin`).[project]name = "open-second-brain"-version = "0.9.1"+version = "0.10.0"description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."readme = "README.md"requires-python = ">=3.11"diff --git a/src/cli/main.ts b/src/cli/main.tsindex 386ce326..6c38e19c 100644--- a/src/cli/main.ts+++ b/src/cli/main.ts@@ -44,6 +44,7 @@ import type { ReceiptPolicyStatus } from "../core/pay-memory/types.ts";import { listVaultPages, writeFrontmatter } from "../core/vault.ts";import { CliError, parseFlags } from "./argparse.ts";import { handleBrainSubcommand } from "./brain.ts";+import { handleSearchSubcommand } from "./search.ts";import {installCli,renderInstallResult,@@ -223,9 +224,67 @@ async function cmdInit(argv: string[]): Promise<number> {process.stdout.write(`timezone registered: ${timezone}\n`);process.stdout.write(`timezone persisted to: ${configPath}\n`);}+ writeSearchInitBlock(configPath);return 0;}+/**+ * Print the post-init search-onboarding block (design Β§10).+ *+ * Always advertises `o2b search index`. When the user has already+ * flipped `search_semantic_enabled` to true but no embedding key is+ * resolvable, the detailed configuration template is appended. The+ * block prints once, only during `o2b init` β no nagging on other+ * CLI invocations (the dedicated diagnostic is `o2b search check`).+ */+function writeSearchInitBlock(configPath: string): void {+ process.stdout.write("\nSearch:\n");+ process.stdout.write(" next: o2b search index # build the vault search index\n");++ const data = discoverConfig(configPath).data;+ // Accept "true" / "True" / "TRUE" / "1" from both the config file+ // and the env override β matches the parseBool helper inside+ // resolveSearchConfig so the init banner does not lie about state.+ const truthy = (v: unknown): boolean => {+ if (typeof v !== "string") return false;+ const s = v.trim().toLowerCase();+ return s === "true" || s === "1";+ };+ const enabled =+ truthy(data["search_semantic_enabled"]) ||+ truthy(process.env["OPEN_SECOND_BRAIN_SEARCH_SEMANTIC"]);+ const keyPresent = !!(+ (data["embedding_api_key"] && data["embedding_api_key"] !== "") ||+ process.env["OPEN_SECOND_BRAIN_EMBEDDING_KEY"]+ );+ if (!enabled || keyPresent) return;++ process.stdout.write(+ [+ "",+ "Semantic search is enabled but no embedding key is configured.",+ "",+ "Either set in the config file (printed above), or via env vars:",+ "",+ ` search_semantic_enabled: "true"`,+ ` embedding_base_url: "https://openrouter.ai/api/v1"`,+ ` embedding_model: "google/gemini-embedding-2-preview"`,+ ` embedding_api_key: "<your key>"`,+ "",+ "Env equivalents:",+ " OPEN_SECOND_BRAIN_SEARCH_SEMANTIC=true",+ " OPEN_SECOND_BRAIN_EMBEDDING_BASE_URL=...",+ " OPEN_SECOND_BRAIN_EMBEDDING_MODEL=...",+ " OPEN_SECOND_BRAIN_EMBEDDING_KEY=...",+ "",+ "Then:",+ " o2b search check",+ " o2b search index --embeddings",+ "",+ ].join("\n"),+ );+}+async function cmdDoctor(argv: string[]): Promise<number> {const { flags } = parseFlags(argv, {vault: { type: "string" },@@ -1193,6 +1252,13 @@ Brain (observing memory):brain unpin Clear the pinned flagbrain rollback Restore Brain/ from a snapshot (--list / <run_id>)brain doctor Validate Brain invariants (--strict promotes warnings)++Search:+ search "<query>" Search the vault index (default verb is 'query')+ search index Incrementally update the index from the vault+ search reindex Rebuild the index atomically (.new -> rename -> .bak)+ search status Print index summary (counts, model, vec extension)+ search check Pre-flight diagnostics (SQLite, FTS5, vec, provider)`;function sortedReplacer(_key: string, value: unknown): unknown {@@ -1274,6 +1340,8 @@ export async function main(argv: ReadonlyArray<string>): Promise<number> {return await cmdPaymentDigest(rest);case "brain":return await handleBrainSubcommand(rest);+ case "search":+ return await handleSearchSubcommand(rest);default:process.stderr.write(`error: unknown command: ${command}\n`);process.stderr.write(HELP);diff --git a/src/cli/search.ts b/src/cli/search.tsnew file mode 100644index 00000000..0f5b3ffa--- /dev/null+++ b/src/cli/search.ts@@ -0,0 +1,422 @@+/**+ * `o2b search` subcommand dispatcher.+ *+ * Routes the five Brain Search verbs (design doc Β§8) to thin wrappers+ * over `src/core/search/*`. The core modules own all I/O; this file+ * only parses flags, resolves the vault, shapes exit codes, and renders+ * either human-readable or JSON output.+ *+ * o2b search "<query>" β cmdSearchQuery (default verb)+ * o2b search index β cmdSearchIndex+ * o2b search reindex β cmdSearchReindex+ * o2b search status β cmdSearchStatus+ * o2b search check β cmdSearchCheck+ */++import { defaultConfigPath, resolveVault } from "../core/config.ts";+import {+ indexCheck,+ indexStatus,+ indexVault,+ reindexVault,+ resolveSearchConfig,+ search,+ SearchError,+} from "../core/search/index.ts";+import type {+ IndexCheckReport,+ IndexProgressEvent,+ IndexStats,+ IndexStatusSnapshot,+ ResolvedSearchConfig,+ SearchOutcome,+} from "../core/search/index.ts";+import { CliError, parseFlags } from "./argparse.ts";++const KNOWN_VERBS = new Set(["query", "index", "reindex", "status", "check"]);++export async function handleSearchSubcommand(argv: ReadonlyArray<string>): Promise<number> {+ // First positional is verb iff it matches a known verb. Otherwise the+ // default verb is `query` and the positional is the query string.+ let verb = "query";+ let rest = argv;+ if (argv.length > 0 && KNOWN_VERBS.has(argv[0]!)) {+ verb = argv[0]!;+ rest = argv.slice(1);+ }++ try {+ switch (verb) {+ case "query":+ return await cmdSearchQuery(rest);+ case "index":+ return await cmdSearchIndex(rest);+ case "reindex":+ return await cmdSearchReindex(rest);+ case "status":+ return await cmdSearchStatus(rest);+ case "check":+ return await cmdSearchCheck(rest);+ default:+ process.stderr.write(`error: unknown search verb: ${verb}\n`);+ return 2;+ }+ } catch (e) {+ if (e instanceof CliError) {+ process.stderr.write(`error: ${e.message}\n`);+ return 2;+ }+ if (e instanceof SearchError) {+ process.stderr.write(`error: ${e.message} [${e.code}]\n`);+ return e.code === "INVALID_INPUT" ? 2 : 1;+ }+ process.stderr.write(`error: ${e instanceof Error ? e.message : String(e)}\n`);+ return 1;+ }+}++function resolveConfig(+ flags: Record<string, string | boolean | string[] | undefined>,+): ResolvedSearchConfig {+ const flagVault = typeof flags["vault"] === "string" ? (flags["vault"] as string) : undefined;+ const configPath =+ typeof flags["config"] === "string" ? (flags["config"] as string) : defaultConfigPath();+ const vault = flagVault ?? resolveVault(configPath) ?? null;+ if (!vault) {+ throw new CliError(+ "no vault configured. Pass --vault <path> or run `o2b init --vault <path> ...` first.",+ );+ }+ const dbFlag = typeof flags["db"] === "string" ? (flags["db"] as string) : undefined;+ const kwFlag = typeof flags["keyword-weight"] === "string" ? Number(flags["keyword-weight"]) : undefined;+ const semFlag = typeof flags["semantic-weight"] === "string" ? Number(flags["semantic-weight"]) : undefined;+ const concurrencyFlag =+ typeof flags["concurrency"] === "string" ? Number(flags["concurrency"]) : undefined;+ const overrides = {+ ...(dbFlag !== undefined ? { dbPath: dbFlag } : {}),+ ...(kwFlag !== undefined ? { keywordWeight: kwFlag } : {}),+ ...(semFlag !== undefined ? { semanticWeight: semFlag } : {}),+ ...(concurrencyFlag !== undefined ? { semantic: { concurrency: concurrencyFlag } } : {}),+ };+ return resolveSearchConfig({ vault, configPath, overrides });+}++// βββ query ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++async function cmdSearchQuery(argv: ReadonlyArray<string>): Promise<number> {+ const { flags, positional } = parseFlags(argv, {+ vault: { type: "string" },+ config: { type: "string" },+ db: { type: "string" },+ limit: { type: "string", default: "10" },+ semantic: { type: "boolean" },+ "keyword-only": { type: "boolean" },+ path: { type: "string" },+ "keyword-weight": { type: "string" },+ "semantic-weight": { type: "string" },+ "auto-refresh": { type: "boolean" },+ json: { type: "boolean" },+ verbose: { type: "boolean" },+ });++ if (positional.length === 0) {+ throw new CliError("query string is required");+ }+ if (flags["semantic"] === true && flags["keyword-only"] === true) {+ throw new CliError("--semantic and --keyword-only are mutually exclusive");+ }+ const query = positional.join(" ");+ const limitNum = Number(flags["limit"] ?? "10");+ if (!Number.isInteger(limitNum) || limitNum < 1 || limitNum > 100) {+ throw new CliError("--limit must be an integer in 1..100");+ }++ const cfg = resolveConfig(flags);++ if (flags["auto-refresh"]) {+ try {+ await indexVault(cfg);+ } catch (e) {+ const msg = e instanceof Error ? e.message : String(e);+ process.stderr.write(`auto-refresh failed: ${msg}\n`);+ }+ }++ // Pass `undefined` when no explicit flag is set so `search()` falls back+ // to the config default. Passing `null` works by accident today but blurs+ // the implicit/explicit policy boundary in Β§7 of the search design.+ const semanticOverride: boolean | undefined =+ flags["semantic"] === true ? true : flags["keyword-only"] === true ? false : undefined;++ const outcome = await search(cfg, {+ query,+ limit: limitNum,+ semantic: semanticOverride,+ keywordOnly: flags["keyword-only"] === true,+ pathPrefix: typeof flags["path"] === "string" ? (flags["path"] as string) : undefined,+ });++ if (flags["json"]) {+ process.stdout.write(JSON.stringify(jsonForOutcome(outcome)) + "\n");+ return 0;+ }+ process.stdout.write(renderOutcomeHuman(outcome, flags["verbose"] === true));+ return 0;+}++function jsonForOutcome(o: SearchOutcome): unknown {+ return {+ results: o.results.map((r) => ({+ path: r.path,+ title: r.title,+ content: r.content,+ score: r.score,+ keyword_score: r.keywordScore,+ semantic_score: r.semanticScore,+ link_boost: r.linkBoost,+ recency_boost: r.recencyBoost,+ start_line: r.startLine,+ end_line: r.endLine,+ search_type: r.searchType,+ document_id: r.documentId,+ chunk_id: r.chunkId,+ })),+ warnings: o.warnings,+ total: o.total,+ };+}++function renderOutcomeHuman(o: SearchOutcome, verbose: boolean): string {+ const lines: string[] = [];+ if (o.results.length === 0) {+ lines.push("(no results)");+ }+ o.results.forEach((r, i) => {+ const score = r.score.toFixed(2);+ lines.push(`[${i + 1}] ${r.path} β’ ${score}`);+ lines.push(+ ` line ${r.startLine}-${r.endLine} β’ ${r.searchType}` ++ (verbose+ ? ` β’ kw=${r.keywordScore.toFixed(2)} sem=${r.semanticScore.toFixed(2)} link=${r.linkBoost.toFixed(2)} rec=${r.recencyBoost.toFixed(2)}`+ : ""),+ );+ const snippet = r.content.trim().replace(/\s+/g, " ").slice(0, 140);+ lines.push(` ${snippet}${r.content.length > 140 ? "β¦" : ""}`);+ lines.push("");+ });+ for (const w of o.warnings) lines.push(`warning: ${w}`);+ return lines.join("\n") + (lines.length > 0 ? "" : "\n");+}++// βββ index ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++async function cmdSearchIndex(argv: ReadonlyArray<string>): Promise<number> {+ const { flags } = parseFlags(argv, {+ vault: { type: "string" },+ config: { type: "string" },+ db: { type: "string" },+ embeddings: { type: "boolean" },+ force: { type: "boolean" },+ concurrency: { type: "string" },+ verbose: { type: "boolean" },+ json: { type: "boolean" },+ });+ const cfg = resolveConfig(flags);++ const events: IndexProgressEvent[] = [];+ const stats = await indexVault(cfg, {+ embeddings: flags["embeddings"] === true,+ force: flags["force"] === true,+ onFile: (e) => {+ events.push(e);+ if (flags["verbose"]) {+ const msg = e.message ? ` ${e.message}` : "";+ process.stderr.write(`${e.kind}\t${e.path}${msg}\n`);+ }+ },+ });++ if (flags["json"]) {+ process.stdout.write(JSON.stringify(jsonForStats(stats, cfg)) + "\n");+ return 0;+ }+ process.stdout.write(renderStatsHuman(stats, cfg));+ return 0;+}++function jsonForStats(stats: IndexStats, cfg: ResolvedSearchConfig): unknown {+ return {+ stats: {+ added: stats.added,+ updated: stats.updated,+ unchanged: stats.unchanged,+ deleted: stats.deleted,+ chunks_total: stats.chunksTotal,+ embeddings_computed: stats.embeddingsComputed,+ embeddings_retries: stats.embeddingsRetries,+ },+ errors: stats.errors.map((e) => ({ path: e.path, message: e.message })),+ duration_ms: stats.durationMs,+ vault: cfg.vault,+ db_path: cfg.dbPath,+ };+}++function renderStatsHuman(stats: IndexStats, cfg: ResolvedSearchConfig): string {+ const lines: string[] = [];+ lines.push(`indexing vault: ${cfg.vault}`);+ lines.push(` added: ${stats.added} files, ${stats.chunksTotal} chunks`);+ lines.push(` updated: ${stats.updated} files`);+ lines.push(` unchanged: ${stats.unchanged} files`);+ lines.push(` deleted: ${stats.deleted} files`);+ if (stats.embeddingsComputed > 0 || stats.embeddingsRetries > 0) {+ lines.push(+ ` embeddings: ${stats.embeddingsComputed} computed (${stats.embeddingsRetries} retries)`,+ );+ }+ if (stats.errors.length > 0) {+ lines.push(` errors:`);+ for (const e of stats.errors) lines.push(` - ${e.path}: ${e.message}`);+ }+ lines.push(`done in ${(stats.durationMs / 1000).toFixed(1)}s`);+ return lines.join("\n") + "\n";+}++// βββ reindex ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++async function cmdSearchReindex(argv: ReadonlyArray<string>): Promise<number> {+ const { flags } = parseFlags(argv, {+ vault: { type: "string" },+ config: { type: "string" },+ db: { type: "string" },+ embeddings: { type: "boolean" },+ concurrency: { type: "string" },+ json: { type: "boolean" },+ verbose: { type: "boolean" },+ });+ const cfg = resolveConfig(flags);+ const stats = await reindexVault(cfg, {+ embeddings: flags["embeddings"] === true,+ onFile: flags["verbose"]+ ? (e) => process.stderr.write(`${e.kind}\t${e.path}\n`)+ : undefined,+ });+ if (flags["json"]) {+ process.stdout.write(JSON.stringify(jsonForStats(stats, cfg)) + "\n");+ return 0;+ }+ process.stdout.write(renderStatsHuman(stats, cfg));+ return 0;+}++// βββ status βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++async function cmdSearchStatus(argv: ReadonlyArray<string>): Promise<number> {+ const { flags } = parseFlags(argv, {+ vault: { type: "string" },+ config: { type: "string" },+ db: { type: "string" },+ json: { type: "boolean" },+ });+ const cfg = resolveConfig(flags);+ const status = await indexStatus(cfg);+ if (flags["json"]) {+ process.stdout.write(JSON.stringify(jsonForStatus(status)) + "\n");+ return 0;+ }+ process.stdout.write(renderStatusHuman(status));+ return 0;+}++function jsonForStatus(s: IndexStatusSnapshot): unknown {+ return {+ index_path: s.indexPath,+ exists: s.exists,+ schema_version: s.schemaVersion,+ documents: s.documents,+ chunks: s.chunks,+ embeddings: s.embeddings,+ stale_embeddings: s.staleEmbeddings,+ embedding_model: s.embeddingModel,+ embedding_dimension: s.embeddingDimension,+ vec_extension: s.vecExtension,+ semantic_enabled: s.semanticEnabled,+ embedding_key_present: s.embeddingKeyPresent,+ last_indexed_at: s.lastIndexedAt,+ last_full_index_at: s.lastFullIndexAt,+ warnings: s.warnings,+ };+}++function renderStatusHuman(s: IndexStatusSnapshot): string {+ if (!s.exists) {+ return `index: not initialised. Run: o2b search index\n path: ${s.indexPath}\n`;+ }+ const lines: string[] = [];+ lines.push(`index: ${s.indexPath}`);+ lines.push(`schema_version: ${s.schemaVersion}`);+ lines.push(`documents: ${s.documents}`);+ lines.push(`chunks: ${s.chunks}`);+ lines.push(`embeddings: ${s.embeddings} (stale: ${s.staleEmbeddings})`);+ lines.push(`embedding_model: ${s.embeddingModel ?? "(none)"}`);+ lines.push(`embedding_dimension: ${s.embeddingDimension ?? "(none)"}`);+ lines.push(`vec_extension: ${s.vecExtension}`);+ lines.push(`semantic_enabled: ${s.semanticEnabled}`);+ lines.push(`embedding_key: ${s.embeddingKeyPresent ? "present" : "missing"}`);+ lines.push(`last_indexed_at: ${s.lastIndexedAt ?? "(never)"}`);+ lines.push(`last_full_index_at: ${s.lastFullIndexAt ?? "(never)"}`);+ for (const w of s.warnings) lines.push(`warning: ${w}`);+ return lines.join("\n") + "\n";+}++// βββ check ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++async function cmdSearchCheck(argv: ReadonlyArray<string>): Promise<number> {+ const { flags } = parseFlags(argv, {+ vault: { type: "string" },+ config: { type: "string" },+ db: { type: "string" },+ json: { type: "boolean" },+ });+ const cfg = resolveConfig(flags);+ const report = await indexCheck(cfg);+ if (flags["json"]) {+ process.stdout.write(JSON.stringify(jsonForCheck(report)) + "\n");+ } else {+ process.stdout.write(renderCheckHuman(report));+ }+ return report.fatal.length > 0 ? 1 : 0;+}++function jsonForCheck(r: IndexCheckReport): unknown {+ return {+ vault_readable: r.vaultReadable,+ index_dir_writable: r.indexDirWritable,+ sqlite_ok: r.sqliteOk,+ fts5_ok: r.fts5Ok,+ vec_extension: r.vecExtension,+ embedding_key_resolved: r.embeddingKeyResolved,+ provider_reachable: r.providerReachable,+ provider_reason: r.providerReason,+ warnings: r.warnings,+ fatal: r.fatal,+ };+}++function renderCheckHuman(r: IndexCheckReport): string {+ const lines: string[] = [];+ const ok = (b: boolean) => (b ? "OK" : "MISSING");+ lines.push(`vault_readable: ${ok(r.vaultReadable)}`);+ lines.push(`index_dir_writable: ${ok(r.indexDirWritable)}`);+ lines.push(`sqlite_ok: ${ok(r.sqliteOk)}`);+ lines.push(`fts5_ok: ${ok(r.fts5Ok)}`);+ lines.push(`vec_extension: ${r.vecExtension}`);+ lines.push(`embedding_key: ${ok(r.embeddingKeyResolved)}`);+ if (r.providerReachable !== null) {+ lines.push(`provider_reachable: ${r.providerReachable ? "OK" : "FAIL"}`);+ if (r.providerReason) lines.push(`provider_reason: ${r.providerReason}`);+ }+ for (const w of r.warnings) lines.push(`warning: ${w}`);+ for (const f of r.fatal) lines.push(`fatal: ${f}`);+ return lines.join("\n") + "\n";+}diff --git a/src/core/search/chunker.ts b/src/core/search/chunker.tsnew file mode 100644index 00000000..8e20dc1c--- /dev/null+++ b/src/core/search/chunker.ts@@ -0,0 +1,432 @@+/**+ * Markdown chunker β two-pass structural split + token-budget packer.+ *+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§6.+ *+ * Token approximation: whitespace word count. Deterministic across+ * Bun/Node, dependency-free, machine-independent β so the same vault+ * hashes the same chunks on every Syncthing peer.+ */++const DEFAULT_MAX_TOKENS = 800;+const DEFAULT_MIN_TOKENS = 100;+const DEFAULT_OVERLAP_TOKENS = 100;++export interface ChunkOptions {+ readonly maxTokens?: number;+ readonly minTokens?: number;+ readonly overlapTokens?: number;+}++export interface MarkdownChunk {+ readonly chunkIndex: number;+ readonly content: string;+ readonly startLine: number;+ readonly endLine: number;+ readonly tokenCount: number;+}++export interface ChunkResult {+ readonly title: string | null;+ readonly chunks: ReadonlyArray<MarkdownChunk>;+ readonly warnings: ReadonlyArray<string>;+}++interface Line {+ readonly num: number;+ readonly text: string;+}++type BlockKind = "frontmatter" | "code" | "heading" | "list" | "table" | "paragraph";++interface Block {+ readonly kind: BlockKind;+ readonly lines: ReadonlyArray<Line>;+ readonly tokenCount: number;+}++function countTokens(text: string): number {+ if (text.length === 0) return 0;+ let count = 0;+ let inWord = false;+ for (let i = 0; i < text.length; i++) {+ const c = text.charCodeAt(i);+ const isSpace = c === 0x20 || c === 0x09 || c === 0x0a || c === 0x0d;+ if (!isSpace) {+ if (!inWord) {+ count++;+ inWord = true;+ }+ } else {+ inWord = false;+ }+ }+ return count;+}++function tokensOfLines(lines: ReadonlyArray<Line>): number {+ let total = 0;+ for (const l of lines) total += countTokens(l.text);+ return total;+}++function makeBlock(kind: BlockKind, lines: Line[]): Block {+ return { kind, lines, tokenCount: tokensOfLines(lines) };+}++function isBlank(s: string): boolean {+ return s.trim() === "";+}++const FENCE_RE = /^(?:```|~~~)/;+const HEADING_RE = /^#{1,6}\s/;+const LIST_ITEM_RE = /^\s*(?:[-*+]\s|\d+\.\s)/;+const TABLE_SEP_RE = /^\s*\|?\s*:?-{3,}:?\s*(?:\|\s*:?-{3,}:?\s*)+\|?\s*$/;++/** Split the raw text into a 1-indexed line array. */+function splitLines(text: string): Line[] {+ const out: Line[] = [];+ let lineNum = 1;+ let start = 0;+ for (let i = 0; i < text.length; i++) {+ const c = text.charCodeAt(i);+ if (c === 0x0a /* \n */) {+ const seg = text.slice(start, i);+ const cleaned = seg.endsWith("\r") ? seg.slice(0, -1) : seg;+ out.push({ num: lineNum++, text: cleaned });+ start = i + 1;+ }+ }+ if (start <= text.length) {+ const seg = text.slice(start);+ if (seg !== "" || out.length === 0) {+ const cleaned = seg.endsWith("\r") ? seg.slice(0, -1) : seg;+ out.push({ num: lineNum, text: cleaned });+ }+ }+ return out;+}++function extractFrontmatter(+ lines: ReadonlyArray<Line>,+): { block: Block | null; remaining: ReadonlyArray<Line>; warnings: string[] } {+ if (lines.length === 0 || lines[0]!.text.trim() !== "---") {+ return { block: null, remaining: lines, warnings: [] };+ }+ for (let i = 1; i < lines.length; i++) {+ if (lines[i]!.text.trim() === "---") {+ const fmLines = lines.slice(0, i + 1);+ return {+ block: makeBlock("frontmatter", [...fmLines]),+ remaining: lines.slice(i + 1),+ warnings: [],+ };+ }+ }+ // Unterminated frontmatter: omit FM block, keep file otherwise indexable.+ return {+ block: null,+ remaining: lines.slice(1),+ warnings: ["malformed frontmatter (no closing '---')"],+ };+}++function readCodeBlock(lines: ReadonlyArray<Line>, start: number): { end: number } {+ // `lines[start]` is the opening fence.+ const fenceText = lines[start]!.text.trim();+ const fenceMark = fenceText.startsWith("```") ? "```" : "~~~";+ for (let i = start + 1; i < lines.length; i++) {+ if (lines[i]!.text.trim().startsWith(fenceMark)) return { end: i };+ }+ return { end: lines.length - 1 }; // unclosed: run to EOF+}++function readList(lines: ReadonlyArray<Line>, start: number): { end: number } {+ let i = start + 1;+ while (i < lines.length) {+ const t = lines[i]!.text;+ if (isBlank(t)) return { end: i - 1 };+ if (HEADING_RE.test(t)) return { end: i - 1 };+ if (FENCE_RE.test(t)) return { end: i - 1 };+ // Continuation of list: list item or indented continuation.+ if (LIST_ITEM_RE.test(t) || /^\s{2,}/.test(t)) {+ i++;+ continue;+ }+ return { end: i - 1 };+ }+ return { end: lines.length - 1 };+}++function isTableStart(lines: ReadonlyArray<Line>, start: number): boolean {+ const a = lines[start];+ const b = lines[start + 1];+ if (!a || !b) return false;+ if (!a.text.includes("|")) return false;+ return TABLE_SEP_RE.test(b.text);+}++function readTable(lines: ReadonlyArray<Line>, start: number): { end: number } {+ // Header (start), separator (start+1), then rows until blank or non-pipe line.+ let i = start + 2;+ while (i < lines.length) {+ const t = lines[i]!.text;+ if (isBlank(t)) return { end: i - 1 };+ if (!t.includes("|")) return { end: i - 1 };+ i++;+ }+ return { end: lines.length - 1 };+}++function readParagraph(lines: ReadonlyArray<Line>, start: number): { end: number } {+ let i = start + 1;+ while (i < lines.length) {+ const t = lines[i]!.text;+ if (isBlank(t)) return { end: i - 1 };+ if (HEADING_RE.test(t)) return { end: i - 1 };+ if (FENCE_RE.test(t)) return { end: i - 1 };+ if (LIST_ITEM_RE.test(t)) return { end: i - 1 };+ if (isTableStart(lines, i)) return { end: i - 1 };+ i++;+ }+ return { end: lines.length - 1 };+}++function splitIntoBlocks(lines: ReadonlyArray<Line>): { blocks: Block[]; warnings: string[] } {+ const blocks: Block[] = [];+ const warnings: string[] = [];+ const { block: fm, remaining, warnings: fmWarn } = extractFrontmatter(lines);+ warnings.push(...fmWarn);+ if (fm) blocks.push(fm);++ let i = 0;+ while (i < remaining.length) {+ const text = remaining[i]!.text;+ if (isBlank(text)) {+ i++;+ continue;+ }+ if (FENCE_RE.test(text)) {+ const { end } = readCodeBlock(remaining, i);+ blocks.push(makeBlock("code", [...remaining.slice(i, end + 1)]));+ i = end + 1;+ continue;+ }+ if (HEADING_RE.test(text)) {+ blocks.push(makeBlock("heading", [remaining[i]!]));+ i++;+ continue;+ }+ if (LIST_ITEM_RE.test(text)) {+ const { end } = readList(remaining, i);+ blocks.push(makeBlock("list", [...remaining.slice(i, end + 1)]));+ i = end + 1;+ continue;+ }+ if (isTableStart(remaining, i)) {+ const { end } = readTable(remaining, i);+ blocks.push(makeBlock("table", [...remaining.slice(i, end + 1)]));+ i = end + 1;+ continue;+ }+ const { end } = readParagraph(remaining, i);+ blocks.push(makeBlock("paragraph", [...remaining.slice(i, end + 1)]));+ i = end + 1;+ }++ return { blocks, warnings };+}++interface DraftChunk {+ blocks: Block[];+ startLine: number;+ endLine: number;+ tokens: number;+}++function newDraft(): DraftChunk {+ return { blocks: [], startLine: 0, endLine: 0, tokens: 0 };+}++function takeOverlap(prev: DraftChunk, overlapTokens: number): Line[] {+ if (overlapTokens <= 0 || prev.blocks.length === 0) return [];+ // Flatten lines of previous chunk's body in order, then take tail until token budget.+ const allLines: Line[] = [];+ for (const b of prev.blocks) for (const l of b.lines) allLines.push(l);+ let acc = 0;+ const tail: Line[] = [];+ for (let i = allLines.length - 1; i >= 0; i--) {+ const t = countTokens(allLines[i]!.text);+ if (acc + t > overlapTokens && tail.length > 0) break;+ tail.unshift(allLines[i]!);+ acc += t;+ }+ return tail;+}++function emitChunk(+ index: number,+ overlap: ReadonlyArray<Line>,+ draft: DraftChunk,+): MarkdownChunk {+ const bodyLines: Line[] = [];+ for (const b of draft.blocks) for (const l of b.lines) bodyLines.push(l);+ const allText = [+ ...overlap.map((l) => l.text),+ ...bodyLines.map((l) => l.text),+ ].join("\n");+ const tokenCount = countTokens(allText);+ return Object.freeze({+ chunkIndex: index,+ content: allText,+ startLine: bodyLines[0]?.num ?? overlap[0]?.num ?? 1,+ endLine: bodyLines[bodyLines.length - 1]?.num ?? overlap[overlap.length - 1]?.num ?? 1,+ tokenCount,+ });+}++function packBlocks(+ blocks: ReadonlyArray<Block>,+ opts: { maxTokens: number; minTokens: number; overlapTokens: number },+): MarkdownChunk[] {+ const out: MarkdownChunk[] = [];+ let draft = newDraft();+ let prevForOverlap: DraftChunk | null = null;+ let pendingOverlap: Line[] = [];++ const flush = () => {+ if (draft.blocks.length === 0) return;+ const chunk = emitChunk(out.length, pendingOverlap, draft);+ out.push(chunk);+ prevForOverlap = draft;+ pendingOverlap = takeOverlap(prevForOverlap, opts.overlapTokens);+ draft = newDraft();+ };++ for (const block of blocks) {+ if (block.kind === "frontmatter") {+ // Flush any in-progress (shouldn't happen β frontmatter is first), then+ // emit the frontmatter as its own chunk *without* overlap since nothing+ // precedes it.+ flush();+ const fmContent = block.lines.map((l) => l.text).join("\n");+ out.push(+ Object.freeze({+ chunkIndex: out.length,+ content: fmContent,+ startLine: block.lines[0]!.num,+ endLine: block.lines[block.lines.length - 1]!.num,+ tokenCount: countTokens(fmContent),+ }),+ );+ // The next chunk should NOT include frontmatter text as overlap.+ prevForOverlap = null;+ pendingOverlap = [];+ continue;+ }++ // Atomic large block β emit as its own chunk.+ if (block.tokenCount > opts.maxTokens) {+ flush();+ const standalone: DraftChunk = {+ blocks: [block],+ startLine: block.lines[0]!.num,+ endLine: block.lines[block.lines.length - 1]!.num,+ tokens: block.tokenCount,+ };+ const chunk = emitChunk(out.length, pendingOverlap, standalone);+ out.push(chunk);+ prevForOverlap = standalone;+ pendingOverlap = takeOverlap(prevForOverlap, opts.overlapTokens);+ continue;+ }++ if (block.kind === "heading") {+ const hasNonHeading = draft.blocks.some((b) => b.kind !== "heading");+ if (hasNonHeading && draft.tokens >= opts.minTokens) {+ flush();+ }+ draft.blocks.push(block);+ draft.tokens += block.tokenCount;+ continue;+ }++ if (draft.tokens + block.tokenCount > opts.maxTokens && draft.blocks.length > 0) {+ flush();+ }+ draft.blocks.push(block);+ draft.tokens += block.tokenCount;+ }++ flush();+ return out;+}++function stripQuotes(s: string): string {+ const t = s.trim();+ if (t.length >= 2 && ((t.startsWith('"') && t.endsWith('"')) || (t.startsWith("'") && t.endsWith("'")))) {+ return t.slice(1, -1);+ }+ return t;+}++function resolveTitle(+ frontmatterBlock: Block | null,+ blocks: ReadonlyArray<Block>,+ filenameBase: string | null,+): string | null {+ if (frontmatterBlock) {+ for (const l of frontmatterBlock.lines) {+ const m = l.text.match(/^title:\s*(.*)$/);+ if (m) {+ const v = stripQuotes(m[1] ?? "");+ if (v) return v;+ }+ }+ }+ for (const b of blocks) {+ if (b.kind === "heading") {+ const text = b.lines[0]!.text.replace(/^#{1,6}\s+/, "").trim();+ if (text) return text;+ }+ if (b.kind !== "frontmatter") break;+ }+ if (filenameBase) return filenameBase.replace(/[-_]+/g, " ").trim() || filenameBase;+ return null;+}++/**+ * Two-pass markdown chunker. Pure function: same input β same output.+ *+ * Returns `chunks: []` for empty or whitespace-only input; the indexer+ * still records a `documents` row so the file is tracked.+ */+export function chunkMarkdown(+ text: string,+ filenameBase: string | null,+ opts?: ChunkOptions,+): ChunkResult {+ const maxTokens = opts?.maxTokens ?? DEFAULT_MAX_TOKENS;+ const minTokens = opts?.minTokens ?? DEFAULT_MIN_TOKENS;+ const overlapTokens = opts?.overlapTokens ?? DEFAULT_OVERLAP_TOKENS;++ const allLines = splitLines(text);+ if (allLines.every((l) => isBlank(l.text))) {+ return Object.freeze({+ title: filenameBase ? filenameBase.replace(/[-_]+/g, " ") : null,+ chunks: Object.freeze([]),+ warnings: Object.freeze([]),+ });+ }++ const { blocks, warnings } = splitIntoBlocks(allLines);+ const frontmatter = blocks.find((b) => b.kind === "frontmatter") ?? null;+ const title = resolveTitle(frontmatter, blocks, filenameBase);+ const chunks = packBlocks(blocks, { maxTokens, minTokens, overlapTokens });++ return Object.freeze({+ title,+ chunks: Object.freeze(chunks),+ warnings: Object.freeze(warnings),+ });+}diff --git a/src/core/search/embeddings/null-provider.ts b/src/core/search/embeddings/null-provider.tsnew file mode 100644index 00000000..50cdab22--- /dev/null+++ b/src/core/search/embeddings/null-provider.ts@@ -0,0 +1,30 @@+/**+ * Sentinel embedding provider that surfaces configuration bugs.+ *+ * `embed()` throwing is intentional β a caller that reaches this path+ * has set semantic on the call but the runtime has no provider wired+ * up. Better to fail loud than to silently no-op and persist no+ * vectors.+ */++import { SearchError } from "../types.ts";+import type { EmbeddingProvider } from "./provider.ts";++export class NullProvider implements EmbeddingProvider {+ readonly name = "null";+ readonly model = "";+ readonly dimension = null;++ embed(_texts: ReadonlyArray<string>): Promise<number[][]> {+ return Promise.reject(+ new SearchError(+ "EMBEDDING_DISABLED",+ "embeddings disabled: no provider configured. Set embedding_provider, embedding_base_url, embedding_model, embedding_api_key.",+ ),+ );+ }++ async ping(): Promise<{ ok: false; reason: string }> {+ return { ok: false, reason: "embeddings disabled" };+ }+}diff --git a/src/core/search/embeddings/openai-compat.ts b/src/core/search/embeddings/openai-compat.tsnew file mode 100644index 00000000..406bdb2a--- /dev/null+++ b/src/core/search/embeddings/openai-compat.ts@@ -0,0 +1,343 @@+/**+ * OpenAI-compatible `/v1/embeddings` provider.+ *+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§11.+ *+ * Network rules:+ * - One concurrent batch per semaphore slot (`embedding_concurrency`).+ * - Each batch contains up to `embedding_batch_size` texts.+ * - Per-request timeout: `embedding_timeout_ms`.+ * - Retry on `429`, `5xx`, and network/timeout errors with exponential+ * backoff `1s/2s` (+ Β±25% jitter), three attempts total. Other 4xx+ * fail fast.+ * - Vectors are unit-normalised so cosine similarity equals+ * `1 - L2Β² / 2` (see ranker).+ *+ * Provider-shaped, never-prompt-shaped errors. The caller decides+ * whether to surface (`--semantic`) or warn (implicit).+ */++import { SearchError } from "../types.ts";+import type { ResolvedEmbeddingConfig } from "../types.ts";+import type { EmbeddingProvider } from "./provider.ts";++const RETRYABLE_STATUSES = new Set([429, 500, 502, 503, 504]);++interface OpenAiEmbeddingResponse {+ readonly data: ReadonlyArray<{ readonly embedding: ReadonlyArray<number>; readonly index: number }>;+ readonly model?: string;+}++function sleep(ms: number): Promise<void> {+ return new Promise((res) => setTimeout(res, ms));+}++function jittered(base: number): number {+ const jitter = base * 0.25 * (Math.random() * 2 - 1); // Β±25%+ return Math.max(0, base + jitter);+}++class Semaphore {+ private permits: number;+ private readonly waiters: Array<() => void> = [];+ constructor(n: number) {+ this.permits = Math.max(1, n | 0);+ }+ async acquire(): Promise<void> {+ if (this.permits > 0) {+ this.permits--;+ return;+ }+ await new Promise<void>((res) => this.waiters.push(res));+ this.permits--;+ }+ release(): void {+ this.permits++;+ const next = this.waiters.shift();+ if (next) next();+ }+}++function unitNormaliseInPlace(v: number[]): number[] {+ let s = 0;+ for (const x of v) s += x * x;+ const norm = Math.sqrt(s);+ if (norm === 0) return v;+ for (let i = 0; i < v.length; i++) v[i] = (v[i] ?? 0) / norm;+ return v;+}++function chunkArray<T>(arr: ReadonlyArray<T>, size: number): T[][] {+ const step = Math.max(1, size | 0);+ const out: T[][] = [];+ for (let i = 0; i < arr.length; i += step) out.push(arr.slice(i, i + step) as T[]);+ return out;+}++interface ResolvedHttp {+ readonly url: string;+ readonly apiKey: string;+}++function resolveHttp(config: ResolvedEmbeddingConfig): ResolvedHttp {+ if (!config.baseUrl) {+ throw new SearchError("INVALID_INPUT", "embedding_base_url is required when semantic is enabled");+ }+ if (!config.model) {+ throw new SearchError("INVALID_INPUT", "embedding_model is required when semantic is enabled");+ }+ if (!config.apiKey) {+ throw new SearchError(+ "EMBEDDING_KEY_MISSING",+ "embedding_api_key is required when semantic is enabled",+ );+ }+ const base = config.baseUrl.replace(/\/+$/, "");+ return { url: `${base}/embeddings`, apiKey: config.apiKey };+}++export interface OpenAICompatProviderOptions {+ /** Override default `[1000, 2000]` ms backoffs (used by tests). */+ readonly backoffMs?: ReadonlyArray<number>;+}++export class OpenAICompatProvider implements EmbeddingProvider {+ readonly name = "openai-compat";+ readonly model: string;+ private _dimension: number | null;+ private readonly config: ResolvedEmbeddingConfig;+ private readonly http: ResolvedHttp;+ private readonly backoffMs: ReadonlyArray<number>;++ constructor(config: ResolvedEmbeddingConfig, opts?: OpenAICompatProviderOptions) {+ this.config = config;+ this.http = resolveHttp(config);+ this.model = config.model!;+ this._dimension = config.dimension;+ this.backoffMs = opts?.backoffMs ?? [1000, 2000];+ }++ get dimension(): number | null {+ return this._dimension;+ }++ async embed(texts: ReadonlyArray<string>): Promise<number[][]> {+ if (texts.length === 0) return [];+ const batches = chunkArray(+ texts.map((t, i) => ({ text: t, originalIndex: i })),+ this.config.batchSize,+ );+ const sem = new Semaphore(this.config.concurrency);+ const out: number[][] = new Array(texts.length);++ // Shared abort controller cancels in-flight siblings and skips+ // queued ones the moment any batch fails. Without this, a 4xx on+ // batch #1 still bills the remaining N-1 batches that were already+ // scheduled by Promise.all.+ const cancel = new AbortController();++ const tasks = batches.map(async (batch) => {+ await sem.acquire();+ try {+ if (cancel.signal.aborted) return;+ const vectors = await this.embedBatchWithRetry(batch.map((b) => b.text), {+ parentSignal: cancel.signal,+ });+ for (let i = 0; i < vectors.length; i++) {+ out[batch[i]!.originalIndex] = vectors[i]!;+ }+ } finally {+ sem.release();+ }+ });++ try {+ // Promise.all rejects on first rejection. Abort siblings so they+ // don't keep hitting the provider; then await the cancelled tasks+ // via allSettled to avoid unhandled-rejection warnings.+ try {+ await Promise.all(tasks);+ } catch (firstError) {+ cancel.abort();+ await Promise.allSettled(tasks);+ if (firstError instanceof SearchError) throw firstError;+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", String(firstError));+ }+ } finally {+ // No-op if already settled; keeps semaphore + listeners cleaned up.+ cancel.abort();+ }+ return out;+ }++ async ping(): Promise<{ ok: true; dimension: number } | { ok: false; reason: string }> {+ try {+ const vectors = await this.embedBatchWithRetry(["check"], { maxAttempts: 1 });+ const v = vectors[0];+ if (!v) return { ok: false, reason: "empty response" };+ return { ok: true, dimension: v.length };+ } catch (e) {+ const reason = e instanceof Error ? e.message : String(e);+ return { ok: false, reason };+ }+ }++ private async embedBatchWithRetry(+ texts: string[],+ opts?: { maxAttempts?: number; parentSignal?: AbortSignal },+ ): Promise<number[][]> {+ const maxAttempts = opts?.maxAttempts ?? 3;+ let lastError: SearchError | null = null;+ for (let attempt = 1; attempt <= maxAttempts; attempt++) {+ if (opts?.parentSignal?.aborted) {+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", "embed cancelled");+ }+ try {+ return await this.embedBatchOnce(texts, opts?.parentSignal);+ } catch (e) {+ const err = this.classifyError(e);+ if (!err.retriable || attempt >= maxAttempts) throw err.error;+ lastError = err.error;+ this.retriesSeen++;+ const wait = jittered(this.backoffMs[attempt - 1] ?? (this.backoffMs[this.backoffMs.length - 1] ?? 4000));+ await sleep(wait);+ }+ }+ throw lastError ?? new SearchError("EMBEDDING_PROVIDER_HTTP", "retry loop exhausted");+ }++ /** Retries since construction; `consumeRetryCount()` resets the tally. */+ retriesSeen = 0;++ /** Read-and-reset retry counter β indexer uses this to populate IndexStats. */+ consumeRetryCount(): number {+ const n = this.retriesSeen;+ this.retriesSeen = 0;+ return n;+ }++ private async embedBatchOnce(+ texts: string[],+ parentSignal?: AbortSignal,+ ): Promise<number[][]> {+ const controller = new AbortController();+ const timer = setTimeout(() => controller.abort(), this.config.timeoutMs);+ const onParentAbort = () => controller.abort();+ if (parentSignal) {+ if (parentSignal.aborted) controller.abort();+ else parentSignal.addEventListener("abort", onParentAbort, { once: true });+ }+ let response: Response;+ try {+ response = await fetch(this.http.url, {+ method: "POST",+ headers: {+ "content-type": "application/json",+ authorization: `Bearer ${this.http.apiKey}`,+ },+ body: JSON.stringify({+ model: this.model,+ input: texts,+ encoding_format: "float",+ }),+ signal: controller.signal,+ });+ } catch (e) {+ const cause = e instanceof Error ? e : new Error(String(e));+ if (cause.name === "AbortError") {+ if (parentSignal?.aborted) {+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", "embed cancelled");+ }+ throw new SearchError(+ "EMBEDDING_PROVIDER_TIMEOUT",+ `embedding request timed out after ${this.config.timeoutMs}ms`,+ );+ }+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", `network error: ${cause.message}`);+ } finally {+ clearTimeout(timer);+ if (parentSignal) parentSignal.removeEventListener("abort", onParentAbort);+ }++ if (!response.ok) {+ const body = await response.text().catch(() => "");+ const head = body.slice(0, 300);+ throw new SearchError(+ "EMBEDDING_PROVIDER_HTTP",+ `embedding HTTP ${response.status}: ${head || response.statusText}`,+ );+ }++ let json: OpenAiEmbeddingResponse;+ try {+ json = (await response.json()) as OpenAiEmbeddingResponse;+ } catch (e) {+ const msg = e instanceof Error ? e.message : String(e);+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", `embedding response not JSON: ${msg}`);+ }++ if (!Array.isArray(json.data) || json.data.length !== texts.length) {+ throw new SearchError(+ "EMBEDDING_PROVIDER_HTTP",+ `embedding response shape: expected ${texts.length} vectors, got ${json.data?.length ?? "none"}`,+ );+ }++ const ordered: number[][] = new Array(texts.length);+ for (const item of json.data) {+ if (typeof item.index !== "number" || item.index < 0 || item.index >= texts.length) {+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", `embedding response: out-of-range index ${item.index}`);+ }+ if (!Array.isArray(item.embedding)) {+ throw new SearchError(+ "EMBEDDING_PROVIDER_HTTP",+ `embedding response: data[${item.index}].embedding is not an array`,+ );+ }+ const arr = (item.embedding as ReadonlyArray<number>).slice();+ if (this._dimension === null) {+ this._dimension = arr.length;+ } else if (this._dimension !== arr.length) {+ throw new SearchError(+ "EMBEDDING_DIMENSION_MISMATCH",+ `embedding dimension changed mid-batch: expected ${this._dimension}, got ${arr.length}`,+ );+ }+ ordered[item.index] = unitNormaliseInPlace(arr);+ }+ for (let i = 0; i < ordered.length; i++) {+ if (!Array.isArray(ordered[i])) {+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", `embedding response: missing vector for index ${i}`);+ }+ }+ return ordered;+ }++ private classifyError(e: unknown): { retriable: boolean; error: SearchError } {+ if (e instanceof SearchError) {+ // Parent-cancelled batches surface as a synthetic+ // EMBEDDING_PROVIDER_HTTP "embed cancelled". Retrying them would+ // race against the abort and re-spend budget on a batch the+ // outer Promise already gave up on.+ if (e.message.includes("embed cancelled")) {+ return { retriable: false, error: e };+ }+ if (e.code === "EMBEDDING_PROVIDER_TIMEOUT") return { retriable: true, error: e };+ if (e.code === "EMBEDDING_PROVIDER_HTTP") {+ // Parse status from message if present.+ const m = e.message.match(/HTTP (\d+)/);+ if (m) {+ const status = Number(m[1]);+ return { retriable: RETRYABLE_STATUSES.has(status), error: e };+ }+ // Network errors fall here β retriable.+ return { retriable: true, error: e };+ }+ return { retriable: false, error: e };+ }+ return {+ retriable: false,+ error: new SearchError("EMBEDDING_PROVIDER_HTTP", String(e)),+ };+ }+}diff --git a/src/core/search/embeddings/provider.ts b/src/core/search/embeddings/provider.tsnew file mode 100644index 00000000..10f519d9--- /dev/null+++ b/src/core/search/embeddings/provider.ts@@ -0,0 +1,44 @@+/**+ * Embedding-provider abstraction. The store does not call into HTTP;+ * the indexer drives provider.embed() and passes raw vectors to+ * store.vecUpsert().+ *+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§11.+ */++import { SearchError } from "../types.ts";+import type { ResolvedEmbeddingConfig } from "../types.ts";++export interface EmbeddingProvider {+ readonly name: string;+ readonly model: string;+ readonly dimension: number | null;+ embed(texts: ReadonlyArray<string>): Promise<number[][]>;+ ping(): Promise<{ ok: true; dimension: number } | { ok: false; reason: string }>;+ /**+ * Optional read-and-reset of provider-internal retry tally. The+ * indexer consumes this after each `embed()` to populate+ * `IndexStats.embeddingsRetries`. Providers that never retry+ * (NullProvider, MockEmbeddingProvider) leave this undefined.+ */+ consumeRetryCount?(): number;+}++export function makeProvider(config: ResolvedEmbeddingConfig): EmbeddingProvider {+ // Lazy imports to keep the module graph small for users who never+ // enable semantic search.+ if (config.provider === "disabled" || !config.enabled) {+ // eslint-disable-next-line @typescript-eslint/no-require-imports+ const { NullProvider } = require("./null-provider.ts") as typeof import("./null-provider.ts");+ return new NullProvider();+ }+ if (config.provider === "openai-compat") {+ // eslint-disable-next-line @typescript-eslint/no-require-imports+ const { OpenAICompatProvider } = require("./openai-compat.ts") as typeof import("./openai-compat.ts");+ return new OpenAICompatProvider(config);+ }+ throw new SearchError(+ "INVALID_INPUT",+ `unknown embedding_provider '${String((config as { provider?: string }).provider)}'`,+ );+}diff --git a/src/core/search/fts.ts b/src/core/search/fts.tsnew file mode 100644index 00000000..09e47439--- /dev/null+++ b/src/core/search/fts.ts@@ -0,0 +1,37 @@+/**+ * FTS5 query construction and keyword retrieval.+ *+ * The user-typed query is treated as a bag of phrase tokens joined by+ * implicit AND. Each token is wrapped in double quotes so FTS5's+ * operator words (`AND`, `OR`, `NOT`, `NEAR`) and metacharacters+ * (`*`, `(`, `)`, `:`, `^`) lose their special meaning. Internal `"`+ * is escaped as `""` per the FTS5 grammar.+ *+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§7.+ */++import type { KeywordHit, Store } from "./store.ts";++export function buildFtsMatch(rawQuery: string): string {+ const tokens = rawQuery+ .split(/\s+/)+ .map((t) => t.trim())+ .filter((t) => t.length > 0);+ if (tokens.length === 0) return "";+ return tokens.map((t) => `"${t.replace(/"/g, '""')}"`).join(" ");+}++export interface RunFtsOptions {+ readonly limit: number;+ readonly pathPrefix?: string | null;+}++export function runFtsQuery(+ store: Store,+ rawQuery: string,+ opts: RunFtsOptions,+): KeywordHit[] {+ const match = buildFtsMatch(rawQuery);+ if (match === "") return [];+ return store.keywordTopK(match, opts);+}diff --git a/src/core/search/index.ts b/src/core/search/index.tsnew file mode 100644index 00000000..2b33a487--- /dev/null+++ b/src/core/search/index.ts@@ -0,0 +1,293 @@+/**+ * Public surface for `src/core/search/*`: config resolution plus the+ * index/search/status/check entry points used by CLI and MCP layers.+ */++import { discoverConfig } from "../config.ts";+import { resolveIndexPath } from "./paths.ts";+import { SearchError } from "./types.ts";+import type { ResolvedSearchConfig, ResolvedEmbeddingConfig } from "./types.ts";++type SearchConfigOverrides = Partial<Omit<ResolvedSearchConfig, "ignorePaths" | "semantic">> & {+ readonly ignorePaths?: ReadonlyArray<string>;+ readonly semantic?: Partial<ResolvedEmbeddingConfig>;+};++export type {+ BrainSearchResult,+ IndexCheckReport,+ IndexStats,+ IndexStatusSnapshot,+ ResolvedEmbeddingConfig,+ ResolvedSearchConfig,+ SearchErrorCode,+ SearchOptions,+ SearchOutcome,+} from "./types.ts";+export { SearchError, SEARCH_ERROR_CODES } from "./types.ts";++export { resolveIndexPath } from "./paths.ts";+export {+ indexVault,+ reindexVault,+ indexStatus,+ indexCheck,+ type IndexVaultOptions,+ type IndexProgressEvent,+} from "./indexer.ts";+export { search } from "./search.ts";++const DEFAULT_IGNORE_PATHS = [+ ".git",+ "node_modules",+ ".open-second-brain",+ ".obsidian/cache",+ ".trash",+ ".stversions",+];++const DEFAULTS = {+ chunkSize: 800,+ chunkOverlap: 100,+ keywordWeight: 0.6,+ semanticWeight: 0.4,+ provider: "openai-compat" as const,+ timeoutMs: 10_000,+ concurrency: 4,+ batchSize: 32,+};++function envOrConfig(+ env: NodeJS.ProcessEnv,+ config: Readonly<Record<string, string>>,+ envKey: string,+ configKey: string,+): string | null {+ const e = env[envKey];+ if (e !== undefined && e !== "") return e;+ const c = config[configKey];+ if (c !== undefined && c !== "") return c;+ return null;+}++function parseInteger(+ raw: string | null,+ fallback: number,+ fieldName: string,+ opts?: { readonly min?: number; readonly max?: number },+): number {+ if (raw === null) return fallback;+ const n = Number(raw);+ if (!Number.isFinite(n) || !Number.isInteger(n)) {+ throw new SearchError("INVALID_INPUT", `${fieldName} must be an integer, got '${raw}'`);+ }+ validateIntegerRange(n, fieldName, opts);+ return n;+}++function validateIntegerRange(+ n: number,+ fieldName: string,+ opts?: { readonly min?: number; readonly max?: number },+): void {+ if (opts?.min !== undefined && n < opts.min) {+ throw new SearchError("INVALID_INPUT", `${fieldName} must be >= ${opts.min}, got ${n}`);+ }+ if (opts?.max !== undefined && n > opts.max) {+ throw new SearchError("INVALID_INPUT", `${fieldName} must be <= ${opts.max}, got ${n}`);+ }+}++function parseFloat01(raw: string | null, fallback: number, fieldName: string): number {+ if (raw === null) return fallback;+ const n = Number(raw);+ if (!Number.isFinite(n) || n < 0 || n > 1) {+ throw new SearchError(+ "INVALID_INPUT",+ `${fieldName} must be a number in [0, 1], got '${raw}'`,+ );+ }+ return n;+}++function validateWeight(n: number, fieldName: string): void {+ if (!Number.isFinite(n) || n < 0 || n > 1) {+ throw new SearchError("INVALID_INPUT", `${fieldName} must be a number in [0, 1], got '${n}'`);+ }+}++function validateResolvedConfig(config: ResolvedSearchConfig): void {+ validateIntegerRange(config.chunkSize, "search_chunk_size", { min: 1 });+ validateIntegerRange(config.chunkOverlap, "search_chunk_overlap", { min: 0 });+ if (config.chunkOverlap >= config.chunkSize) {+ throw new SearchError(+ "INVALID_INPUT",+ `search_chunk_overlap must be smaller than search_chunk_size, got ${config.chunkOverlap} >= ${config.chunkSize}`,+ );+ }+ validateWeight(config.keywordWeight, "search_keyword_weight");+ validateWeight(config.semanticWeight, "search_semantic_weight");+ if (config.keywordWeight + config.semanticWeight > 1.0 + 1e-9) {+ throw new SearchError(+ "INVALID_INPUT",+ `keyword_weight + semantic_weight must sum to <= 1, got ${config.keywordWeight} + ${config.semanticWeight}`,+ );+ }+ if (config.semantic.dimension !== null) {+ validateIntegerRange(config.semantic.dimension, "embedding_dimension", { min: 1 });+ }+ validateIntegerRange(config.semantic.timeoutMs, "embedding_timeout_ms", { min: 1 });+ validateIntegerRange(config.semantic.concurrency, "embedding_concurrency", { min: 1 });+ validateIntegerRange(config.semantic.batchSize, "embedding_batch_size", { min: 1 });+}++function parseBool(raw: string | null, fallback: boolean, fieldName: string): boolean {+ if (raw === null) return fallback;+ if (raw === "true" || raw === "1") return true;+ if (raw === "false" || raw === "0") return false;+ throw new SearchError(+ "INVALID_INPUT",+ `${fieldName} must be 'true' or 'false', got '${raw}'`,+ );+}++function parseProvider(raw: string | null): ResolvedEmbeddingConfig["provider"] {+ if (raw === null) return DEFAULTS.provider;+ if (raw === "openai-compat" || raw === "disabled") return raw;+ throw new SearchError(+ "INVALID_INPUT",+ `embedding_provider must be 'openai-compat' or 'disabled', got '${raw}'`,+ );+}++function parseIgnorePaths(raw: string | null): string[] {+ if (raw === null) return [...DEFAULT_IGNORE_PATHS];+ return raw+ .split(",")+ .map((s) => s.trim())+ .filter((s) => s.length > 0);+}++export function resolveSearchConfig(opts: {+ vault: string;+ configPath?: string;+ overrides?: SearchConfigOverrides;+}): ResolvedSearchConfig {+ const env = process.env;+ const config: Readonly<Record<string, string>> = opts.configPath+ ? discoverConfig(opts.configPath).data+ : {};++ const dbPath = resolveIndexPath(+ opts.vault,+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_DB", "search_db_path"),+ );++ const chunkSize = parseInteger(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_CHUNK_SIZE", "search_chunk_size"),+ DEFAULTS.chunkSize,+ "search_chunk_size",+ { min: 1 },+ );+ const chunkOverlap = parseInteger(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_CHUNK_OVERLAP", "search_chunk_overlap"),+ DEFAULTS.chunkOverlap,+ "search_chunk_overlap",+ { min: 0 },+ );+ const keywordWeight = parseFloat01(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_KW_WEIGHT", "search_keyword_weight"),+ DEFAULTS.keywordWeight,+ "search_keyword_weight",+ );+ const semanticWeight = parseFloat01(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_SEM_WEIGHT", "search_semantic_weight"),+ DEFAULTS.semanticWeight,+ "search_semantic_weight",+ );+ if (keywordWeight + semanticWeight > 1.0 + 1e-9) {+ throw new SearchError(+ "INVALID_INPUT",+ `keyword_weight + semantic_weight must sum to <= 1, got ${keywordWeight} + ${semanticWeight}`,+ );+ }++ const ignorePaths = parseIgnorePaths(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_IGNORE", "search_ignore_paths"),+ );++ const semanticEnabled = parseBool(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_SEMANTIC", "search_semantic_enabled"),+ false,+ "search_semantic_enabled",+ );+ const provider = parseProvider(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_PROVIDER", "embedding_provider"),+ );+ const baseUrl = envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_EMBEDDING_BASE_URL",+ "embedding_base_url",+ );+ const model = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_MODEL", "embedding_model");+ const apiKey = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_KEY", "embedding_api_key");+ const dimRaw = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_DIM", "embedding_dimension");+ const dimension =+ dimRaw === null ? null : parseInteger(dimRaw, 0, "embedding_dimension", { min: 1 });+ const timeoutMs = parseInteger(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_TIMEOUT", "embedding_timeout_ms"),+ DEFAULTS.timeoutMs,+ "embedding_timeout_ms",+ { min: 1 },+ );+ const concurrency = parseInteger(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_CONCURRENCY", "embedding_concurrency"),+ DEFAULTS.concurrency,+ "embedding_concurrency",+ { min: 1 },+ );+ const batchSize = parseInteger(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_BATCH", "embedding_batch_size"),+ DEFAULTS.batchSize,+ "embedding_batch_size",+ { min: 1 },+ );++ const semantic: ResolvedEmbeddingConfig = Object.freeze({+ enabled: semanticEnabled,+ provider,+ baseUrl,+ model,+ apiKey,+ dimension,+ timeoutMs,+ concurrency,+ batchSize,+ });++ const base: ResolvedSearchConfig = Object.freeze({+ vault: opts.vault,+ dbPath,+ ignorePaths: Object.freeze([...ignorePaths]),+ chunkSize,+ chunkOverlap,+ keywordWeight,+ semanticWeight,+ semantic,+ });++ validateResolvedConfig(base);++ if (!opts.overrides) return base;+ const merged = Object.freeze({+ ...base,+ ...opts.overrides,+ semantic: Object.freeze({ ...base.semantic, ...(opts.overrides.semantic ?? {}) }),+ ignorePaths: opts.overrides.ignorePaths+ ? Object.freeze([...opts.overrides.ignorePaths])+ : base.ignorePaths,+ });+ validateResolvedConfig(merged);+ return merged;+}diff --git a/src/core/search/indexer.ts b/src/core/search/indexer.tsnew file mode 100644index 00000000..59b43354--- /dev/null+++ b/src/core/search/indexer.ts@@ -0,0 +1,525 @@+/**+ * Index lifecycle orchestrator.+ *+ * `indexVault` is the incremental path: walk β diff against stored+ * documents β upsert/replace/delete. `reindexVault` is the atomic+ * rebuild: write to `brain.sqlite.new`, then a same-file rename swap+ * with `.bak` retention.+ *+ * `indexStatus` and `indexCheck` are the read-side diagnostics that+ * power `o2b search status|check` and the MCP status enrichment.+ *+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§6, Β§8,+ * Β§13, Β§15.+ */++import { createHash } from "node:crypto";+import {+ accessSync,+ constants,+ mkdirSync,+ readFileSync,+ renameSync,+ statSync,+ unlinkSync,+} from "node:fs";+import { basename, dirname } from "node:path";++import { chunkMarkdown } from "./chunker.ts";+import { makeProvider } from "./embeddings/provider.ts";+import { extractLinks } from "./links.ts";+import { Store } from "./store.ts";+import { SearchError } from "./types.ts";+import { walkVault } from "./walker.ts";+import { withTimeout } from "./with-timeout.ts";+import type { ChunkInput, LinkInput } from "./store.ts";+import type {+ IndexCheckReport,+ IndexStats,+ IndexStatusSnapshot,+ ResolvedSearchConfig,+} from "./types.ts";++// βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++export interface IndexProgressEvent {+ readonly path: string;+ readonly kind: "added" | "updated" | "unchanged" | "deleted" | "error";+ readonly message?: string;+}++export interface IndexVaultOptions {+ readonly embeddings?: boolean;+ /** When true, every file is reindexed even if hash + mtime match. */+ readonly force?: boolean;+ readonly onFile?: (event: IndexProgressEvent) => void;+}++interface MutableStats {+ added: number;+ updated: number;+ unchanged: number;+ deleted: number;+ chunksTotal: number;+ embeddingsComputed: number;+ embeddingsRetries: number;+ errors: Array<{ readonly path: string; readonly message: string }>;+}++function newStats(): MutableStats {+ return {+ added: 0,+ updated: 0,+ unchanged: 0,+ deleted: 0,+ chunksTotal: 0,+ embeddingsComputed: 0,+ embeddingsRetries: 0,+ errors: [],+ };+}++function freezeStats(s: MutableStats, durationMs: number): IndexStats {+ return Object.freeze({+ added: s.added,+ updated: s.updated,+ unchanged: s.unchanged,+ deleted: s.deleted,+ chunksTotal: s.chunksTotal,+ embeddingsComputed: s.embeddingsComputed,+ embeddingsRetries: s.embeddingsRetries,+ errors: Object.freeze([...s.errors]),+ durationMs,+ });+}++function sha256(text: string): string {+ return createHash("sha256").update(text).digest("hex");+}++const UTF8_FATAL = new TextDecoder("utf-8", { fatal: true });++function readUtf8(absPath: string): string {+ const buf = readFileSync(absPath);+ try {+ return UTF8_FATAL.decode(buf);+ } catch (e) {+ const msg = e instanceof Error ? e.message : String(e);+ throw new SearchError("INVALID_INPUT", `file is not valid UTF-8: ${absPath} (${msg})`);+ }+}++// βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ+// indexVault β incremental+// βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++export async function indexVault(+ config: ResolvedSearchConfig,+ opts?: IndexVaultOptions,+): Promise<IndexStats> {+ return indexInto(config, opts);+}++async function indexInto(+ config: ResolvedSearchConfig,+ opts?: IndexVaultOptions,+ storeOverride?: Store,+): Promise<IndexStats> {+ const t0 = Date.now();+ const ownsStore = !storeOverride;+ const store = storeOverride ?? (await Store.open(config, { mode: "write" }));+ const stats = newStats();++ try {+ const existing = store.listDocuments();+ const seen = new Set<string>();++ for (const file of walkVault(config)) {+ // Mark seen FIRST. If anything downstream throws (read fault,+ // chunker bug, transient FS error), the file must not look+ // "missing" to the deletion sweep below β that would wipe a+ // present file from the index just because a single read failed.+ seen.add(file.relPath);+ try {+ const content = readUtf8(file.absPath);+ const contentHash = sha256(content);+ const mtimeSec = Math.floor(file.stat.mtimeMs / 1000);+ const prev = existing.get(file.relPath);++ if (!opts?.force && prev && prev.contentHash === contentHash && prev.mtime === mtimeSec) {+ stats.unchanged++;+ opts?.onFile?.({ path: file.relPath, kind: "unchanged" });+ continue;+ }++ const filenameBase = basename(file.relPath, ".md");+ const chunkResult = chunkMarkdown(content, filenameBase, {+ maxTokens: config.chunkSize,+ overlapTokens: config.chunkOverlap,+ });+ for (const w of chunkResult.warnings) {+ stats.errors.push({ path: file.relPath, message: w });+ }++ const docId = store.upsertDocument({+ path: file.relPath,+ title: chunkResult.title,+ contentHash,+ mtime: mtimeSec,+ size: file.stat.size,+ });++ const chunkInputs: ChunkInput[] = chunkResult.chunks.map((c) => ({+ chunkIndex: c.chunkIndex,+ content: c.content,+ contentHash: sha256(c.content),+ startLine: c.startLine,+ endLine: c.endLine,+ tokenCount: c.tokenCount,+ }));+ const chunkIds = store.replaceChunks(docId, chunkInputs);++ const links: LinkInput[] = [];+ for (let i = 0; i < chunkResult.chunks.length; i++) {+ const cid = chunkIds[i]!;+ const extracted = extractLinks(chunkResult.chunks[i]!.content);+ for (const l of extracted) {+ links.push({+ sourceChunkId: cid,+ targetPath: l.targetPath,+ linkText: l.linkText,+ linkType: l.linkType,+ });+ }+ }+ store.replaceLinks(docId, links);++ stats.chunksTotal += chunkInputs.length;+ if (!prev) {+ stats.added++;+ opts?.onFile?.({ path: file.relPath, kind: "added" });+ } else {+ stats.updated++;+ opts?.onFile?.({ path: file.relPath, kind: "updated" });+ }+ } catch (e) {+ const msg = e instanceof Error ? e.message : String(e);+ stats.errors.push({ path: file.relPath, message: msg });+ opts?.onFile?.({ path: file.relPath, kind: "error", message: msg });+ }+ }++ for (const [path] of existing) {+ if (!seen.has(path)) {+ store.deleteDocument(path);+ stats.deleted++;+ opts?.onFile?.({ path, kind: "deleted" });+ }+ }++ store.resolveLinkTargets();++ if (opts?.embeddings) {+ await populateEmbeddings(store, config, stats);+ }++ const now = new Date().toISOString();+ store.setState("last_indexed_at", now);+ if (opts?.force) store.setState("last_full_index_at", now);++ return freezeStats(stats, Date.now() - t0);+ } finally {+ if (ownsStore) await store.close();+ }+}++async function populateEmbeddings(+ store: Store,+ config: ResolvedSearchConfig,+ stats: MutableStats,+): Promise<void> {+ if (!config.semantic.enabled) {+ throw new SearchError(+ "EMBEDDING_DISABLED",+ "set search_semantic_enabled=true and embedding_* keys to compute embeddings",+ );+ }+ if (!store.vecLoaded()) {+ throw new SearchError(+ "VEC_EXTENSION_UNAVAILABLE",+ "sqlite-vec did not load; cannot store embeddings",+ );+ }+ if (!config.semantic.apiKey) {+ throw new SearchError(+ "EMBEDDING_KEY_MISSING",+ "embedding_api_key is required when computing embeddings",+ );+ }++ const pending = store.findChunksWithoutEmbeddings();+ if (pending.length === 0) return;++ const provider = makeProvider(config.semantic);+ const model = config.semantic.model ?? provider.model;+ const batchSize = Math.max(1, config.semantic.batchSize);+ // Hand the provider a super-batch sized to fully saturate its+ // internal `embedding_concurrency` semaphore. Without this multiplier+ // the indexer's outer loop would serialise provider.embed() calls and+ // the configured concurrency would never kick in.+ const superBatch = batchSize * Math.max(1, config.semantic.concurrency);++ for (let i = 0; i < pending.length; i += superBatch) {+ const batch = pending.slice(i, i + superBatch);+ const texts = batch.map((p) => p.content);+ const vectors = await provider.embed(texts);+ stats.embeddingsRetries += provider.consumeRetryCount?.() ?? 0;+ if (vectors.length !== batch.length) {+ throw new SearchError(+ "EMBEDDING_PROVIDER_HTTP",+ `provider returned ${vectors.length} vectors for ${batch.length} inputs`,+ );+ }+ // Lock in the auto-detected dimension on the very first batch.+ const dim = provider.dimension ?? vectors[0]?.length ?? 0;+ if (dim <= 0) {+ throw new SearchError("EMBEDDING_DIMENSION_MISMATCH", "provider returned vectors of zero length");+ }+ store.ensureEmbeddingModel(model, dim);+ for (let j = 0; j < batch.length; j++) {+ const chunkId = batch[j]!.chunkId;+ const vec = vectors[j]!;+ const embHash = sha256(vec.map((x) => x.toFixed(8)).join(","));+ store.vecUpsert(chunkId, vec, model, dim, embHash);+ stats.embeddingsComputed++;+ }+ }+}++// βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ+// reindexVault β atomic full rebuild+// βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++export async function reindexVault(+ config: ResolvedSearchConfig,+ opts?: IndexVaultOptions,+): Promise<IndexStats> {+ const newPath = config.dbPath + ".new";+ const bakPath = config.dbPath + ".bak";++ mkdirSync(dirname(config.dbPath), { recursive: true });+ tryUnlink(newPath);++ // Build into the temp file with an override config.+ const tempConfig: ResolvedSearchConfig = Object.freeze({+ ...config,+ dbPath: newPath,+ });+ const stats = await indexVault(tempConfig, { ...opts, force: true });++ // Same-directory rename swap. The two renames are each atomic on+ // POSIX; the gap between them is the only crash window, which the+ // .bak restore on Store.open handles.+ tryUnlink(bakPath);+ tryRename(config.dbPath, bakPath); // no-op (ENOENT) on fresh reindex+ renameSync(newPath, config.dbPath); // must succeed β `newPath` was just built+ return stats;+}++/** `unlinkSync` that tolerates ENOENT (file already absent). */+function tryUnlink(p: string): void {+ try {+ unlinkSync(p);+ } catch (e) {+ if (!isEnoent(e)) throw e;+ }+}++/** `renameSync` that tolerates ENOENT on the source. */+function tryRename(from: string, to: string): void {+ try {+ renameSync(from, to);+ } catch (e) {+ if (!isEnoent(e)) throw e;+ }+}++function isEnoent(e: unknown): boolean {+ return typeof e === "object" && e !== null && (e as { code?: string }).code === "ENOENT";+}++// βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ+// indexStatus+// βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++export async function indexStatus(config: ResolvedSearchConfig): Promise<IndexStatusSnapshot> {+ let store: Store;+ try {+ store = await Store.open(config, { mode: "read" });+ } catch (e) {+ if (e instanceof SearchError && e.code === "INDEX_MISSING") {+ return Object.freeze({+ indexPath: config.dbPath,+ exists: false,+ schemaVersion: null,+ documents: 0,+ chunks: 0,+ embeddings: 0,+ staleEmbeddings: 0,+ embeddingModel: null,+ embeddingDimension: null,+ vecExtension: "unknown" as const,+ semanticEnabled: config.semantic.enabled,+ embeddingKeyPresent: !!config.semantic.apiKey,+ lastIndexedAt: null,+ lastFullIndexAt: null,+ warnings: Object.freeze([]),+ });+ }+ throw e;+ }+ try {+ const counts = store.counts();+ const model = store.getState("embedding_model");+ const dimRaw = store.getState("embedding_dimension");+ const dim = dimRaw ? Number(dimRaw) : null;+ const last = store.getState("last_indexed_at");+ const full = store.getState("last_full_index_at");++ const warnings: string[] = [];+ if (config.semantic.enabled && !store.vecLoaded()) {+ warnings.push("sqlite-vec unavailable; semantic search disabled this session");+ }+ if (config.semantic.enabled && !config.semantic.apiKey) {+ warnings.push("embedding_api_key not configured; semantic search disabled");+ }++ return Object.freeze({+ indexPath: config.dbPath,+ exists: true,+ schemaVersion: store.schemaVersion(),+ documents: counts.documents,+ chunks: counts.chunks,+ embeddings: counts.embeddings,+ staleEmbeddings: counts.staleEmbeddings,+ embeddingModel: model,+ embeddingDimension: dim,+ vecExtension: store.vecLoaded() ? ("loaded" as const) : ("unavailable" as const),+ semanticEnabled: config.semantic.enabled,+ embeddingKeyPresent: !!config.semantic.apiKey,+ lastIndexedAt: last,+ lastFullIndexAt: full,+ warnings: Object.freeze(warnings),+ });+ } finally {+ await store.close();+ }+}++// βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ+// indexCheck+// βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++function isDirectoryWritable(dir: string): boolean {+ try {+ // `recursive: true` is idempotent β no separate existsSync check.+ mkdirSync(dir, { recursive: true });+ accessSync(dir, constants.W_OK);+ return true;+ } catch {+ return false;+ }+}++export async function indexCheck(config: ResolvedSearchConfig): Promise<IndexCheckReport> {+ const warnings: string[] = [];+ const fatal: string[] = [];++ let vaultReadable = false;+ try {+ if (statSync(config.vault).isDirectory()) {+ vaultReadable = true;+ } else {+ fatal.push(`vault path exists but is not a directory: ${config.vault}`);+ }+ } catch {+ fatal.push(`vault not readable: ${config.vault}`);+ }++ const dir = dirname(config.dbPath);+ const indexDirWritable = isDirectoryWritable(dir);+ if (!indexDirWritable) fatal.push(`index directory not writable: ${dir}`);++ let sqliteOk = false;+ let fts5Ok = false;+ let vecExtension: "loaded" | "unavailable" | "not-attempted" = "not-attempted";+ try {+ // Use an in-memory DB so the check never touches the real index.+ const { Database } = await import("bun:sqlite");+ const db = new Database(":memory:");+ sqliteOk = true;+ try {+ db.exec(+ "CREATE VIRTUAL TABLE probe USING fts5(content, tokenize='unicode61 remove_diacritics 2')",+ );+ fts5Ok = true;+ } catch (e) {+ const msg = e instanceof Error ? e.message : String(e);+ fatal.push(`FTS5 not available: ${msg}`);+ }+ if (config.semantic.enabled) {+ try {+ // eslint-disable-next-line @typescript-eslint/no-require-imports+ const vec = require("sqlite-vec") as { getLoadablePath(): string };+ db.loadExtension(vec.getLoadablePath());+ db.query("SELECT vec_version()").get();+ vecExtension = "loaded";+ } catch (e) {+ vecExtension = "unavailable";+ warnings.push(`sqlite-vec unavailable: ${e instanceof Error ? e.message : String(e)}`);+ }+ }+ db.close();+ } catch (e) {+ fatal.push(`bun:sqlite open failed: ${e instanceof Error ? e.message : String(e)}`);+ }++ const embeddingKeyResolved = !!(config.semantic.enabled && config.semantic.apiKey);+ if (config.semantic.enabled && !config.semantic.apiKey) {+ warnings.push("embedding_api_key not configured");+ }++ let providerReachable: boolean | null = null;+ let providerReason: string | null = null;+ if (config.semantic.enabled && embeddingKeyResolved) {+ try {+ const provider = makeProvider(config.semantic);+ const probe = await withTimeout(provider.ping(), 5_000);+ if (probe.ok) {+ providerReachable = true;+ } else {+ providerReachable = false;+ providerReason = probe.reason;+ warnings.push(`embedding provider check failed: ${probe.reason}`);+ }+ } catch (e) {+ providerReachable = false;+ providerReason = e instanceof Error ? e.message : String(e);+ warnings.push(`embedding provider check failed: ${providerReason}`);+ }+ }++ return Object.freeze({+ vaultReadable,+ indexDirWritable,+ sqliteOk,+ fts5Ok,+ vecExtension,+ embeddingKeyResolved,+ providerReachable,+ providerReason,+ warnings: Object.freeze(warnings),+ fatal: Object.freeze(fatal),+ });+}+diff --git a/src/core/search/links.ts b/src/core/search/links.tsnew file mode 100644index 00000000..eaea43a3--- /dev/null+++ b/src/core/search/links.ts@@ -0,0 +1,100 @@+/**+ * Extract wikilinks, markdown links, and tags from a chunk of Markdown.+ *+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§5 (links+ * table). Returns the link rows that `store.replaceLinks` expects, but+ * leaves the source-chunk binding to the indexer (it knows which+ * chunk produced the content).+ *+ * Code fences and inline-code spans are stripped before extraction so+ * a code sample mentioning `[[foo]]` or `#hash` does not become a real+ * link. This is best-effort: nested fenced fences are rare and we err+ * on the side of stripping too much rather than capturing junk links.+ */++export type LinkType = "wikilink" | "markdown_link" | "tag";++export interface ExtractedLink {+ readonly targetPath: string | null;+ readonly linkText: string | null;+ readonly linkType: LinkType;+}++const CODE_FENCE_RE = /(^|\n)(```|~~~)[^\n]*\n[\s\S]*?(?:\n(?:```|~~~)[^\n]*|$)/g;+const INLINE_CODE_RE = /`[^`\n]*`/g;+const WIKILINK_RE = /\[\[([^\]\n|]+?)(?:\|([^\]\n]+))?\]\]/g;+// Negative lookbehind so `` image embeds are NOT captured as+// markdown_link. CodeRabbit caught this regression on PR #15.+const MD_LINK_RE = /(?<!!)\[([^\]\n]*)\]\(([^)\n\s]+)(?:\s+"[^"\n]*")?\)/g;+// Obsidian-style tag: #word where word starts with a letter/_ and may contain+// letters, digits, dashes, underscores, and '/' for hierarchy.+const TAG_RE = /(^|[^\w\/])#([A-Za-z_][\w\-/]*)/g;++function stripCode(text: string): string {+ let out = text.replace(CODE_FENCE_RE, "\n");+ out = out.replace(INLINE_CODE_RE, " ");+ return out;+}++function isUrl(target: string): boolean {+ return /^[a-z][a-z0-9+.\-]*:/i.test(target) || target.startsWith("//");+}++function isMailto(target: string): boolean {+ return target.toLowerCase().startsWith("mailto:");+}++function dedupe(links: ExtractedLink[]): ExtractedLink[] {+ const seen = new Set<string>();+ const out: ExtractedLink[] = [];+ for (const l of links) {+ const key = `${l.linkType}|${l.targetPath ?? ""}|${l.linkText ?? ""}`;+ if (seen.has(key)) continue;+ seen.add(key);+ out.push(l);+ }+ return out;+}++export function extractLinks(content: string): ExtractedLink[] {+ const cleaned = stripCode(content);+ const out: ExtractedLink[] = [];++ for (const m of cleaned.matchAll(WIKILINK_RE)) {+ const target = (m[1] ?? "").trim();+ const alt = m[2] ? m[2].trim() : null;+ if (target === "") continue;+ out.push({+ targetPath: target,+ linkText: alt,+ linkType: "wikilink",+ });+ }++ for (const m of cleaned.matchAll(MD_LINK_RE)) {+ const text = (m[1] ?? "").trim();+ const target = (m[2] ?? "").trim();+ if (target === "" || isUrl(target) || isMailto(target)) continue;+ // Strip the optional `#anchor` fragment for path matching.+ const hashIdx = target.indexOf("#");+ const path = hashIdx >= 0 ? target.slice(0, hashIdx) : target;+ if (path === "") continue;+ out.push({+ targetPath: path,+ linkText: text || null,+ linkType: "markdown_link",+ });+ }++ for (const m of cleaned.matchAll(TAG_RE)) {+ const tag = (m[2] ?? "").trim();+ if (tag === "") continue;+ out.push({+ targetPath: null,+ linkText: tag,+ linkType: "tag",+ });+ }++ return dedupe(out);+}diff --git a/src/core/search/paths.ts b/src/core/search/paths.tsnew file mode 100644index 00000000..eb658ac0--- /dev/null+++ b/src/core/search/paths.ts@@ -0,0 +1,16 @@+/**+ * Resolve the on-disk location of the search index file.+ *+ * Default: `<vault>/.open-second-brain/brain.sqlite`. Overridable+ * through CLI `--db` or config `search_db_path`.+ */++import { join } from "node:path";++export function resolveIndexPath(vault: string, override: string | null): string {+ if (override !== null) {+ const trimmed = override.trim();+ if (trimmed !== "") return trimmed;+ }+ return join(vault, ".open-second-brain", "brain.sqlite");+}diff --git a/src/core/search/ranker.ts b/src/core/search/ranker.tsnew file mode 100644index 00000000..2e1bf6d8--- /dev/null+++ b/src/core/search/ranker.ts@@ -0,0 +1,216 @@+/**+ * Pure ranking function. Combines normalised BM25, cosine semantic+ * similarity, link-graph boost, and recency boost into the final score.+ *+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§7.+ *+ * The ranker imports no I/O modules. Callers (search.ts) gather the+ * inputs from the store and pass them in. This makes it trivially+ * testable and substitutable.+ */++import type { KeywordHit, SemanticHit, HydratedChunk } from "./store.ts";+import type { BrainSearchResult } from "./types.ts";++export interface RankerInputs {+ readonly keyword: ReadonlyArray<KeywordHit>;+ readonly semantic: ReadonlyArray<SemanticHit>;+ readonly hydrated: ReadonlyMap<number, HydratedChunk>;+ /** For each chunkId: set of OTHER document ids linking to its document. */+ readonly inboundLinkSources: ReadonlyMap<number, ReadonlySet<number>>;+ /** For each chunkId: the tag set of its document. */+ readonly tagsByDoc: ReadonlyMap<number, ReadonlySet<string>>;+}++export interface RankerOptions {+ readonly keywordWeight: number;+ readonly semanticWeight: number;+ readonly limit: number;+ /** Unix-ms reference time for recency. Defaults to Date.now(). */+ readonly nowMs?: number;+ /** When false, semantic_score is ignored regardless of inputs. */+ readonly semanticEnabled?: boolean;+}++const DAY_MS = 24 * 60 * 60 * 1000;++function clamp01(x: number): number {+ if (!Number.isFinite(x)) return 0;+ if (x < 0) return 0;+ if (x > 1) return 1;+ return x;+}++/** Min-max normalise BM25 within the candidate set. Lower BM25 is better. */+function normalizeBm25(hits: ReadonlyArray<KeywordHit>): Map<number, number> {+ const out = new Map<number, number>();+ if (hits.length === 0) return out;+ // FTS5 bm25() returns smaller-is-better values (often negative). We invert+ // to "larger is better" by negating, then min-max.+ const scores = hits.map((h) => -h.bm25);+ const min = Math.min(...scores);+ const max = Math.max(...scores);+ if (max === min) {+ for (const h of hits) out.set(h.chunkId, 1);+ return out;+ }+ hits.forEach((h, i) => {+ out.set(h.chunkId, (scores[i]! - min) / (max - min));+ });+ return out;+}++/** Map L2-on-unit-vectors distance β cosine similarity in [0, 1]. */+function semanticFromDistance(distance: number): number {+ const sim = 1 - (distance * distance) / 2;+ return clamp01(sim);+}++function recencyBoost(mtime: number, nowMs: number): number {+ const ageMs = Math.max(0, nowMs - mtime * 1000);+ const ageDays = ageMs / DAY_MS;+ if (ageDays <= 7) return 0.05;+ if (ageDays <= 30) return 0.025;+ if (ageDays <= 90) return 0.01;+ return 0;+}++interface Candidate {+ chunkId: number;+ documentId: number;+ keywordScore: number;+ semanticScore: number;+ searchType: "keyword" | "semantic" | "hybrid";+ mtime: number;+}++export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSearchResult[] {+ const nowMs = opts.nowMs ?? Date.now();+ const semanticEnabled = opts.semanticEnabled !== false;++ const kwNorm = normalizeBm25(inputs.keyword);++ const semNorm = new Map<number, number>();+ if (semanticEnabled) {+ for (const h of inputs.semantic) {+ semNorm.set(h.chunkId, semanticFromDistance(h.distance));+ }+ }++ const candidates = new Map<number, Candidate>();+ for (const h of inputs.keyword) {+ candidates.set(h.chunkId, {+ chunkId: h.chunkId,+ documentId: h.documentId,+ keywordScore: kwNorm.get(h.chunkId) ?? 0,+ semanticScore: 0,+ searchType: "keyword",+ mtime: inputs.hydrated.get(h.chunkId)?.mtime ?? 0,+ });+ }+ if (semanticEnabled) {+ for (const h of inputs.semantic) {+ const existing = candidates.get(h.chunkId);+ if (existing) {+ existing.semanticScore = semNorm.get(h.chunkId) ?? 0;+ existing.searchType = "hybrid";+ } else {+ candidates.set(h.chunkId, {+ chunkId: h.chunkId,+ documentId: h.documentId,+ keywordScore: 0,+ semanticScore: semNorm.get(h.chunkId) ?? 0,+ searchType: "semantic",+ mtime: inputs.hydrated.get(h.chunkId)?.mtime ?? 0,+ });+ }+ }+ }++ // Cross-result tables for boosts.+ const candidateChunks = Array.from(candidates.values());+ const candidateDocIds = new Set(candidateChunks.map((c) => c.documentId));++ // Build a per-document tag map so the tag boost counts distinct docs,+ // not chunks. Without this dedup a doc with K candidate chunks would+ // inflate every other candidate's tag count K-fold.+ const tagsByDocId = new Map<number, ReadonlySet<string>>();+ for (const c of candidateChunks) {+ if (tagsByDocId.has(c.documentId)) continue;+ const t = inputs.tagsByDoc.get(c.chunkId);+ if (t && t.size > 0) tagsByDocId.set(c.documentId, t);+ }++ function linkBoostFor(c: Candidate): number {+ const sources = inputs.inboundLinkSources.get(c.chunkId);+ if (!sources || sources.size === 0) return 0;+ let count = 0;+ for (const s of sources) {+ if (s === c.documentId) continue;+ if (candidateDocIds.has(s)) count++;+ }+ const raw = count * 0.02;+ return Math.min(0.03, raw);+ }++ function tagBoostFor(c: Candidate): number {+ const mine = tagsByDocId.get(c.documentId);+ if (!mine || mine.size === 0) return 0;+ let count = 0;+ for (const [otherDocId, theirs] of tagsByDocId) {+ if (otherDocId === c.documentId) continue;+ for (const tag of mine) {+ if (theirs.has(tag)) {+ count++;+ break;+ }+ }+ }+ const raw = count * 0.01;+ return Math.min(0.02, raw);+ }++ const ranked: BrainSearchResult[] = [];+ for (const c of candidateChunks) {+ const hyd = inputs.hydrated.get(c.chunkId);+ if (!hyd) continue;+ const link = linkBoostFor(c);+ const tag = tagBoostFor(c);+ const linkBoost = Math.min(0.05, link + tag);+ const recency = recencyBoost(c.mtime, nowMs);+ const weighted =+ opts.keywordWeight * c.keywordScore ++ (semanticEnabled ? opts.semanticWeight : 0) * c.semanticScore;+ const score = clamp01(weighted + linkBoost + recency);++ ranked.push(+ Object.freeze({+ documentId: c.documentId,+ chunkId: c.chunkId,+ path: hyd.path,+ title: hyd.title,+ content: hyd.content,+ startLine: hyd.startLine,+ endLine: hyd.endLine,+ score,+ keywordScore: c.keywordScore,+ semanticScore: c.semanticScore,+ linkBoost,+ recencyBoost: recency,+ searchType: c.searchType,+ }),+ );+ }++ // Tie-break per design Β§7: final_score desc, keywordScore desc, mtime desc, chunkId asc.+ ranked.sort((a, b) => {+ if (b.score !== a.score) return b.score - a.score;+ if (b.keywordScore !== a.keywordScore) return b.keywordScore - a.keywordScore;+ const am = inputs.hydrated.get(a.chunkId)?.mtime ?? 0;+ const bm = inputs.hydrated.get(b.chunkId)?.mtime ?? 0;+ if (bm !== am) return bm - am;+ return a.chunkId - b.chunkId;+ });++ return ranked.slice(0, Math.max(1, opts.limit));+}diff --git a/src/core/search/schema.ts b/src/core/search/schema.tsnew file mode 100644index 00000000..2a17e7bc--- /dev/null+++ b/src/core/search/schema.ts@@ -0,0 +1,212 @@+/**+ * DDL and migrations for the search index. The SQLite database itself+ * is managed by `store.ts`; this module is data + pure functions over+ * a connection so future minor versions can append migrations without+ * touching the store class.+ *+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§5.+ */++import type { Database } from "bun:sqlite";++import { SearchError } from "./types.ts";++/**+ * Latest schema version this code understands. A DB with a value above+ * this raises `SCHEMA_MISMATCH` on open β the operator must reindex+ * with a newer binary.+ */+export const LATEST_SCHEMA_VERSION = 1;++const DDL_V1 = `+CREATE TABLE IF NOT EXISTS documents (+ id INTEGER PRIMARY KEY,+ path TEXT NOT NULL UNIQUE,+ title TEXT,+ content_hash TEXT NOT NULL,+ mtime INTEGER NOT NULL,+ size INTEGER NOT NULL,+ created_at TEXT NOT NULL,+ updated_at TEXT NOT NULL,+ indexed_at TEXT NOT NULL+);+CREATE INDEX IF NOT EXISTS idx_documents_path ON documents(path);+CREATE INDEX IF NOT EXISTS idx_documents_mtime ON documents(mtime);++CREATE TABLE IF NOT EXISTS chunks (+ id INTEGER PRIMARY KEY,+ document_id INTEGER NOT NULL REFERENCES documents(id) ON DELETE CASCADE,+ chunk_index INTEGER NOT NULL,+ content TEXT NOT NULL,+ content_hash TEXT NOT NULL,+ start_line INTEGER NOT NULL,+ end_line INTEGER NOT NULL,+ token_count INTEGER NOT NULL,+ created_at TEXT NOT NULL,+ updated_at TEXT NOT NULL,+ UNIQUE(document_id, chunk_index)+);+CREATE INDEX IF NOT EXISTS idx_chunks_document ON chunks(document_id);++CREATE VIRTUAL TABLE IF NOT EXISTS chunk_fts USING fts5(+ content,+ content='chunks',+ content_rowid='id',+ tokenize='unicode61 remove_diacritics 2'+);++CREATE TRIGGER IF NOT EXISTS chunks_ai AFTER INSERT ON chunks BEGIN+ INSERT INTO chunk_fts(rowid, content) VALUES (new.id, new.content);+END;+CREATE TRIGGER IF NOT EXISTS chunks_ad AFTER DELETE ON chunks BEGIN+ INSERT INTO chunk_fts(chunk_fts, rowid, content) VALUES('delete', old.id, old.content);+END;+CREATE TRIGGER IF NOT EXISTS chunks_au AFTER UPDATE ON chunks BEGIN+ INSERT INTO chunk_fts(chunk_fts, rowid, content) VALUES('delete', old.id, old.content);+ INSERT INTO chunk_fts(rowid, content) VALUES (new.id, new.content);+END;++CREATE TABLE IF NOT EXISTS embeddings (+ chunk_id INTEGER PRIMARY KEY REFERENCES chunks(id) ON DELETE CASCADE,+ model TEXT NOT NULL,+ dimension INTEGER NOT NULL,+ embedding_hash TEXT NOT NULL,+ created_at TEXT NOT NULL,+ updated_at TEXT NOT NULL+);+CREATE INDEX IF NOT EXISTS idx_embeddings_model ON embeddings(model);++CREATE TABLE IF NOT EXISTS chunk_vec_map (+ chunk_id INTEGER PRIMARY KEY REFERENCES chunks(id) ON DELETE CASCADE,+ vec_rowid INTEGER NOT NULL UNIQUE+);++CREATE TABLE IF NOT EXISTS links (+ id INTEGER PRIMARY KEY,+ source_document_id INTEGER NOT NULL REFERENCES documents(id) ON DELETE CASCADE,+ source_chunk_id INTEGER REFERENCES chunks(id) ON DELETE CASCADE,+ target_path TEXT,+ target_document_id INTEGER REFERENCES documents(id) ON DELETE SET NULL,+ link_text TEXT,+ link_type TEXT NOT NULL CHECK(link_type IN ('wikilink','markdown_link','tag')),+ created_at TEXT NOT NULL+);+CREATE INDEX IF NOT EXISTS idx_links_source ON links(source_document_id);+CREATE INDEX IF NOT EXISTS idx_links_target_doc ON links(target_document_id);+CREATE INDEX IF NOT EXISTS idx_links_target_path ON links(target_path);++CREATE TABLE IF NOT EXISTS index_state (+ key TEXT PRIMARY KEY,+ value TEXT NOT NULL,+ updated_at TEXT NOT NULL+);+`;++interface Migration {+ readonly version: number;+ readonly up: (db: Database) => void;+}++export const MIGRATIONS: ReadonlyArray<Migration> = Object.freeze([+ {+ version: 1,+ up(db) {+ db.exec(DDL_V1);+ },+ },+]);++/** Read the current schema version from `index_state`. Returns 0 if not yet recorded. */+export function readSchemaVersion(db: Database): number {+ const row = db+ .query<{ value: string }, []>(+ "SELECT value FROM index_state WHERE key = 'schema_version' LIMIT 1",+ )+ .get();+ if (!row) return 0;+ const n = Number(row.value);+ if (!Number.isFinite(n) || !Number.isInteger(n) || n < 0) {+ throw new SearchError(+ "INDEX_UNREADABLE",+ `index_state.schema_version is not an integer: '${row.value}'`,+ );+ }+ return n;+}++function setSchemaVersion(db: Database, version: number): void {+ const now = new Date().toISOString();+ db.run(+ "INSERT INTO index_state(key, value, updated_at) VALUES('schema_version', ?, ?) " ++ "ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at",+ [String(version), now],+ );+}++/**+ * Apply every pending migration in a single transaction. On a fresh+ * database (no `index_state`), v1 creates the entire schema. On an+ * existing one we run only those with `version > current`.+ *+ * Returns the version after migration.+ */+export function applyMigrations(db: Database): number {+ const hasIndexState = db+ .query<{ name: string }, []>(+ "SELECT name FROM sqlite_master WHERE type='table' AND name='index_state'",+ )+ .get();++ const current = hasIndexState ? readSchemaVersion(db) : 0;++ if (current > LATEST_SCHEMA_VERSION) {+ throw new SearchError(+ "SCHEMA_MISMATCH",+ `index schema version ${current} is newer than this binary supports (${LATEST_SCHEMA_VERSION}). ` ++ `Run: o2b search reindex`,+ );+ }++ if (current === LATEST_SCHEMA_VERSION) return current;++ db.exec("BEGIN");+ try {+ for (const migration of MIGRATIONS) {+ if (migration.version <= current) continue;+ migration.up(db);+ setSchemaVersion(db, migration.version);+ }+ db.exec("COMMIT");+ } catch (e) {+ db.exec("ROLLBACK");+ throw e;+ }+ return LATEST_SCHEMA_VERSION;+}++/**+ * Create the `chunk_vec` virtual table at the given dimension. Caller+ * must have already loaded sqlite-vec; this only issues the DDL.+ *+ * No-op if the table already exists.+ */+export function ensureVecTable(db: Database, dimension: number): void {+ if (!Number.isInteger(dimension) || dimension <= 0) {+ throw new SearchError(+ "INVALID_INPUT",+ `vec dimension must be a positive integer, got ${dimension}`,+ );+ }+ const exists = db+ .query<{ name: string }, []>(+ "SELECT name FROM sqlite_master WHERE type='table' AND name='chunk_vec'",+ )+ .get();+ if (exists) return;+ db.exec(`CREATE VIRTUAL TABLE chunk_vec USING vec0(embedding float[${dimension}])`);+}++/** Drop the `chunk_vec` virtual table if it exists. */+export function dropVecTable(db: Database): void {+ db.exec("DROP TABLE IF EXISTS chunk_vec");+}diff --git a/src/core/search/search.ts b/src/core/search/search.tsnew file mode 100644index 00000000..9c797104--- /dev/null+++ b/src/core/search/search.ts@@ -0,0 +1,203 @@+/**+ * Public search query: orchestrates FTS5, semantic vector search, link+ * + recency boosts, and the keyword-only fallback policy.+ *+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§7, Β§9.+ *+ * The function is async because semantic search requires an HTTP call+ * (the provider embeds the query). Keyword-only paths stay sync inside+ * the store but the public surface stays uniform.+ */++import { makeProvider } from "./embeddings/provider.ts";+import { runFtsQuery } from "./fts.ts";+import { rankResults } from "./ranker.ts";+import { Store } from "./store.ts";+import { SearchError } from "./types.ts";+import type {+ BrainSearchResult,+ ResolvedSearchConfig,+ SearchOptions,+ SearchOutcome,+} from "./types.ts";++interface SemanticPolicy {+ /** caller asked for semantic on or off (true), or accepted the default (false). */+ readonly explicit: boolean;+ /** does the caller want semantic at all? */+ readonly wantSemantic: boolean;+}++function resolveSemanticPolicy(+ config: ResolvedSearchConfig,+ opts: SearchOptions,+): SemanticPolicy {+ if (opts.keywordOnly === true) {+ return { explicit: true, wantSemantic: false };+ }+ if (opts.semantic === true) return { explicit: true, wantSemantic: true };+ if (opts.semantic === false) return { explicit: true, wantSemantic: false };+ return { explicit: false, wantSemantic: config.semantic.enabled };+}++function assertSafePathPrefix(prefix: string | undefined): string | undefined {+ if (!prefix) return undefined;+ if (prefix.includes("..") || prefix.startsWith("/")) {+ throw new SearchError("INVALID_INPUT", "path_prefix escapes vault");+ }+ return prefix;+}++export async function search(+ config: ResolvedSearchConfig,+ opts: SearchOptions,+): Promise<SearchOutcome> {+ const query = (opts.query ?? "").trim();+ if (!query) {+ throw new SearchError("INVALID_INPUT", "missing required argument: query");+ }+ const limit = Math.max(1, Math.min(100, opts.limit ?? 10));+ const pathPrefix = assertSafePathPrefix(opts.pathPrefix);+ const policy = resolveSemanticPolicy(config, opts);+ const warnings: string[] = [];++ const store = await Store.open(config, { mode: "read" });+ try {+ // Keyword candidates.+ const kwHits = runFtsQuery(store, query, {+ limit: limit * 3,+ pathPrefix,+ });++ // Semantic candidates (may be skipped).+ let semHits: ReturnType<Store["semanticTopK"]> = [];+ let semanticAttempted = false;+ if (policy.wantSemantic) {+ const semOutcome = await runSemanticPhase(store, config, query, {+ limit: Math.max(limit * 5, 50),+ pathPrefix,+ explicit: policy.explicit,+ });+ semanticAttempted = semOutcome.attempted;+ semHits = semOutcome.hits;+ for (const w of semOutcome.warnings) warnings.push(w);+ }++ // Hydrate.+ const allChunkIds = new Set<number>();+ for (const h of kwHits) allChunkIds.add(h.chunkId);+ for (const h of semHits) allChunkIds.add(h.chunkId);+ const idsList = Array.from(allChunkIds);+ if (idsList.length === 0) {+ return Object.freeze({+ results: Object.freeze([] as ReadonlyArray<BrainSearchResult>),+ warnings: Object.freeze(warnings),+ total: 0,+ });+ }++ const hydrated = store.hydrateChunks(idsList);+ const inboundLinkSources = store.inboundLinkSources(idsList);+ const tagsByDoc = store.tagsByChunkDocument(idsList);++ const ranked = rankResults(+ {+ keyword: kwHits,+ semantic: semHits,+ hydrated,+ inboundLinkSources,+ tagsByDoc,+ },+ {+ keywordWeight: opts.keywordWeight ?? config.keywordWeight,+ semanticWeight: opts.semanticWeight ?? config.semanticWeight,+ limit,+ semanticEnabled: policy.wantSemantic && semanticAttempted,+ },+ );++ return Object.freeze({+ results: Object.freeze(ranked),+ warnings: Object.freeze(warnings),+ total: ranked.length,+ });+ } finally {+ await store.close();+ }+}++interface SemanticPhaseOutcome {+ readonly attempted: boolean;+ readonly hits: ReturnType<Store["semanticTopK"]>;+ readonly warnings: string[];+}++async function runSemanticPhase(+ store: Store,+ config: ResolvedSearchConfig,+ query: string,+ opts: { limit: number; pathPrefix: string | undefined; explicit: boolean },+): Promise<SemanticPhaseOutcome> {+ const warnings: string[] = [];++ if (!store.vecLoaded()) {+ if (opts.explicit) {+ throw new SearchError(+ "VEC_EXTENSION_UNAVAILABLE",+ "semantic search unavailable: sqlite-vec extension not loaded",+ );+ }+ warnings.push("sqlite-vec unavailable, semantic disabled this session");+ return { attempted: false, hits: [], warnings };+ }+ if (!config.semantic.enabled) {+ // Defensive: should be handled at policy layer, but in case caller+ // forced wantSemantic without enabling, treat as implicit warning.+ warnings.push("semantic not enabled in config; using keyword-only");+ return { attempted: false, hits: [], warnings };+ }+ if (!config.semantic.apiKey) {+ if (opts.explicit) {+ throw new SearchError("EMBEDDING_KEY_MISSING", "embedding key not configured");+ }+ warnings.push("embedding key not configured; semantic disabled");+ return { attempted: false, hits: [], warnings };+ }++ // Data-state check: are there embeddings to search against at all?+ const counts = store.counts();+ if (counts.embeddings === 0) {+ warnings.push("no compatible embeddings; run: o2b search index --embeddings");+ return { attempted: false, hits: [], warnings };+ }++ let queryVec: number[];+ try {+ const provider = makeProvider(config.semantic);+ const vectors = await provider.embed([query]);+ queryVec = vectors[0] ?? [];+ } catch (e) {+ if (opts.explicit) {+ // Defensive: provider methods are expected to throw SearchError,+ // but wrap anything else (e.g. an unexpected runtime failure)+ // so callers always see a typed code rather than a bare Error.+ if (e instanceof SearchError) throw e;+ const msg = e instanceof Error ? e.message : String(e);+ throw new SearchError(+ "EMBEDDING_PROVIDER_HTTP",+ `embedding provider failure: ${msg}`,+ );+ }+ const msg = e instanceof Error ? e.message : String(e);+ warnings.push(`embedding provider unavailable: ${msg}`);+ return { attempted: false, hits: [], warnings };+ }++ if (queryVec.length === 0) {+ warnings.push("embedding provider returned an empty vector; semantic skipped");+ return { attempted: false, hits: [], warnings };+ }++ const hits = store.semanticTopK(queryVec, { limit: opts.limit, pathPrefix: opts.pathPrefix });+ return { attempted: true, hits, warnings };+}diff --git a/src/core/search/store.ts b/src/core/search/store.tsnew file mode 100644index 00000000..f7847df6--- /dev/null+++ b/src/core/search/store.ts@@ -0,0 +1,960 @@+/**+ * Single SQL boundary for `src/core/search/*`. Every read or write of+ * the index passes through this module. Other modules use the typed+ * surface defined here so that:+ *+ * - the SQLite backend can be swapped without touching callers;+ * - the explicit two-step deletion of `chunk_vec` rows (which the+ * SQLite FK cascade does NOT reach, see design Β§5) is centralised+ * in one place;+ * - the embedding-model fingerprint check runs in one place at open+ * time.+ *+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§5, Β§15.+ */++import { Database } from "bun:sqlite";+import { existsSync, mkdirSync, renameSync } from "node:fs";+import { dirname } from "node:path";+import lockfile from "proper-lockfile";++import { SearchError } from "./types.ts";+import type { ResolvedSearchConfig } from "./types.ts";+import {+ applyMigrations,+ dropVecTable,+ ensureVecTable,+ LATEST_SCHEMA_VERSION,+ readSchemaVersion,+} from "./schema.ts";++// βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ+// Types+// βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++export interface StoreOpenOptions {+ /** "read" never locks; "write" acquires an exclusive proper-lockfile. */+ readonly mode: "read" | "write";+ /** When false, the vec extension is not auto-loaded (used by tests). */+ readonly loadVec?: boolean;+}++export interface DocumentInput {+ readonly path: string; // vault-relative POSIX+ readonly title: string | null;+ readonly contentHash: string;+ readonly mtime: number; // unix seconds+ readonly size: number;+}++export interface DocumentSummary {+ readonly id: number;+ readonly contentHash: string;+ readonly mtime: number;+}++export interface ChunkInput {+ readonly chunkIndex: number;+ readonly content: string;+ readonly contentHash: string;+ readonly startLine: number;+ readonly endLine: number;+ readonly tokenCount: number;+}++export interface ChunkRow {+ readonly id: number;+ readonly documentId: number;+ readonly chunkIndex: number;+ readonly content: string;+ readonly contentHash: string;+ readonly startLine: number;+ readonly endLine: number;+ readonly tokenCount: number;+}++export interface LinkInput {+ readonly sourceChunkId: number | null;+ readonly targetPath: string | null;+ readonly linkText: string | null;+ readonly linkType: "wikilink" | "markdown_link" | "tag";+}++export interface KeywordHit {+ readonly chunkId: number;+ readonly documentId: number;+ /** Lower is better (FTS5 returns negative bm25). */+ readonly bm25: number;+}++export interface SemanticHit {+ readonly chunkId: number;+ readonly documentId: number;+ /** L2 distance on unit-normalised vectors. */+ readonly distance: number;+}++export interface HydratedChunk {+ readonly chunkId: number;+ readonly documentId: number;+ readonly path: string;+ readonly title: string | null;+ readonly content: string;+ readonly startLine: number;+ readonly endLine: number;+ readonly mtime: number;+}++export interface StoreCounts {+ readonly documents: number;+ readonly chunks: number;+ readonly embeddings: number;+ /** Embeddings whose `model`/`dimension` no longer match the current config. */+ readonly staleEmbeddings: number;+}++export interface ModelChangeOutcome {+ readonly wasChanged: boolean;+ readonly previousModel: string | null;+ readonly previousDimension: number | null;+ readonly currentModel: string | null;+ readonly currentDimension: number | null;+}++// βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ+// Helpers+// βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++function applyPragmas(db: Database): void {+ db.exec("PRAGMA journal_mode = WAL");+ db.exec("PRAGMA foreign_keys = ON");+ db.exec("PRAGMA synchronous = NORMAL");+}++function ensureFts5(db: Database): void {+ // Probe FTS5 by attempting a benign expression. bun:sqlite ships with+ // FTS5 enabled in the embedded amalgamation; failing here means a+ // custom build without FTS5 and the index is unusable.+ try {+ db.query("SELECT fts5_source('chunk_fts')").get();+ } catch (e) {+ const msg = e instanceof Error ? e.message : String(e);+ // Older SQLite returns "no such function" or "no such table" depending+ // on whether FTS5 is even compiled in. We treat any failure here as+ // FTS5-unavailable rather than guessing the build flag.+ if (!/fts5|chunk_fts/i.test(msg)) {+ throw new SearchError("INDEX_UNREADABLE", `FTS5 probe failed: ${msg}`);+ }+ }+}++function tryLoadVecExtension(db: Database): boolean {+ try {+ // sqlite-vec is an optional dependency. Wrap the import + load so+ // a missing platform package degrades to "extension unavailable"+ // instead of crashing the process.+ // eslint-disable-next-line @typescript-eslint/no-require-imports+ const vec = require("sqlite-vec") as { getLoadablePath(): string };+ db.loadExtension(vec.getLoadablePath());+ // Confirm by calling vec_version() β guards against partial loads.+ db.query("SELECT vec_version()").get();+ return true;+ } catch {+ return false;+ }+}++function nowIso(): string {+ return new Date().toISOString();+}++function vecToBuffer(values: ReadonlyArray<number> | Float32Array): Buffer {+ const arr = values instanceof Float32Array ? values : Float32Array.from(values);+ return Buffer.from(arr.buffer, arr.byteOffset, arr.byteLength);+}++// βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ+// Store+// βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++export class Store {+ private db: Database;+ private readonly config: ResolvedSearchConfig;+ private readonly _vecLoaded: boolean;+ private readonly release: (() => Promise<void>) | null;+ private closed = false;++ private constructor(+ db: Database,+ config: ResolvedSearchConfig,+ vecLoaded: boolean,+ release: (() => Promise<void>) | null,+ ) {+ this.db = db;+ this.config = config;+ this._vecLoaded = vecLoaded;+ this.release = release;+ }++ static async open(config: ResolvedSearchConfig, opts: StoreOpenOptions): Promise<Store> {+ const loadVec = opts.loadVec !== false;++ // Crash recovery: if the main index is missing but `.bak` is present+ // from a failed `reindex` rename window, restore it. Stderr notice+ // (not a thrown error) so existing tooling keeps working.+ if (!existsSync(config.dbPath)) {+ const bak = config.dbPath + ".bak";+ if (existsSync(bak)) {+ try {+ renameSync(bak, config.dbPath);+ // eslint-disable-next-line no-console+ console.error(`restored search index from ${bak} (previous reindex crash)`);+ } catch {+ /* fall through β open path below will report INDEX_MISSING */+ }+ }+ }++ if (opts.mode === "read") {+ if (!existsSync(config.dbPath)) {+ throw new SearchError(+ "INDEX_MISSING",+ `search index not initialised at ${config.dbPath}. Run: o2b search index`,+ );+ }+ let db: Database;+ try {+ db = new Database(config.dbPath);+ } catch (e) {+ const msg = e instanceof Error ? e.message : String(e);+ throw new SearchError("INDEX_UNREADABLE", `cannot open ${config.dbPath}: ${msg}`);+ }+ try {+ applyPragmas(db);+ ensureFts5(db);+ let version: number;+ try {+ version = readSchemaVersion(db);+ } catch (e) {+ // A corrupt or non-OSB sqlite file at the index path can make+ // readSchemaVersion throw raw SQLITE errors (e.g. "no such+ // table: index_state"). Surface those as a typed+ // INDEX_UNREADABLE so callers see a code, not a stray Error.+ if (e instanceof SearchError) throw e;+ const msg = e instanceof Error ? e.message : String(e);+ throw new SearchError(+ "INDEX_UNREADABLE",+ `cannot read schema_version from ${config.dbPath}: ${msg}`,+ );+ }+ if (version !== LATEST_SCHEMA_VERSION) {+ throw new SearchError(+ "SCHEMA_MISMATCH",+ `index schema version ${version} != ${LATEST_SCHEMA_VERSION}. Run: o2b search reindex`,+ );+ }+ const vecLoaded = loadVec && tryLoadVecExtension(db);+ return new Store(db, config, vecLoaded, null);+ } catch (e) {+ db.close();+ throw e;+ }+ }++ // mode === "write"+ mkdirSync(dirname(config.dbPath), { recursive: true });+ if (!existsSync(config.dbPath)) {+ const seed = new Database(config.dbPath);+ seed.close();+ }++ let release: () => Promise<void>;+ try {+ release = await lockfile.lock(config.dbPath, {+ retries: { retries: 3, factor: 1, minTimeout: 1000, maxTimeout: 1000 },+ stale: 60_000,+ realpath: false,+ });+ } catch (e) {+ const msg = e instanceof Error ? e.message : String(e);+ throw new SearchError(+ "INDEX_LOCKED",+ `another writer holds the search index lock: ${msg}`,+ );+ }++ let db: Database;+ try {+ db = new Database(config.dbPath);+ } catch (e) {+ await release();+ const msg = e instanceof Error ? e.message : String(e);+ throw new SearchError("INDEX_UNREADABLE", `cannot open ${config.dbPath}: ${msg}`);+ }++ try {+ applyPragmas(db);+ applyMigrations(db);+ ensureFts5(db);+ const vecLoaded = loadVec && tryLoadVecExtension(db);+ const store = new Store(db, config, vecLoaded, release);+ store.ensureEmbeddingModel(config.semantic.model, config.semantic.dimension);+ return store;+ } catch (e) {+ try {+ db.close();+ } catch {+ /* ignore close errors */+ }+ await release();+ throw e;+ }+ }++ vecLoaded(): boolean {+ return this._vecLoaded;+ }++ schemaVersion(): number {+ return readSchemaVersion(this.db);+ }++ async close(): Promise<void> {+ if (this.closed) return;+ this.closed = true;+ try {+ // Writer mode: consolidate WAL into the main file and switch back to+ // DELETE journal mode so the `-wal`/`-shm` siblings are removed. This+ // matters for `reindexVault`: after the temp-file rename swap, any+ // orphan `*-wal` next to the new main would trigger+ // SQLITE_IOERR_SHORT_READ on the next open.+ if (this.release) {+ try {+ this.db.exec("PRAGMA wal_checkpoint(TRUNCATE)");+ this.db.exec("PRAGMA journal_mode = DELETE");+ } catch (e) {+ // Don't fail the close, but make the failure visible β an+ // unconsolidated WAL is the exact thing that triggers+ // SQLITE_IOERR_SHORT_READ after a `reindexVault` rename swap.+ const msg = e instanceof Error ? e.message : String(e);+ // eslint-disable-next-line no-console+ console.error(`search store: WAL consolidation failed on close: ${msg}`);+ }+ }+ this.db.close();+ } finally {+ if (this.release) await this.release();+ }+ }++ // ββ index_state KV βββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++ getState(key: string): string | null {+ const row = this.db+ .query<{ value: string }, [string]>("SELECT value FROM index_state WHERE key = ?")+ .get(key);+ return row?.value ?? null;+ }++ setState(key: string, value: string): void {+ this.db.run(+ "INSERT INTO index_state(key, value, updated_at) VALUES (?, ?, ?) " ++ "ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at",+ [key, value, nowIso()],+ );+ }++ deleteState(key: string): void {+ this.db.run("DELETE FROM index_state WHERE key = ?", [key]);+ }++ // ββ documents ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++ listDocuments(): Map<string, DocumentSummary> {+ const rows = this.db+ .query<{ id: number; path: string; content_hash: string; mtime: number }, []>(+ "SELECT id, path, content_hash, mtime FROM documents",+ )+ .all();+ const map = new Map<string, DocumentSummary>();+ for (const r of rows) {+ map.set(r.path, { id: r.id, contentHash: r.content_hash, mtime: r.mtime });+ }+ return map;+ }++ getDocumentIdByPath(path: string): number | null {+ const row = this.db+ .query<{ id: number }, [string]>("SELECT id FROM documents WHERE path = ?")+ .get(path);+ return row?.id ?? null;+ }++ upsertDocument(doc: DocumentInput): number {+ const now = nowIso();+ // SQLite RETURNING on INSERT...ON CONFLICT works in 3.35+; bun:sqlite ships modern SQLite.+ const row = this.db+ .query<{ id: number }, [string, string | null, string, number, number, string, string, string]>(+ "INSERT INTO documents(path, title, content_hash, mtime, size, created_at, updated_at, indexed_at) " ++ "VALUES (?, ?, ?, ?, ?, ?, ?, ?) " ++ "ON CONFLICT(path) DO UPDATE SET " ++ " title = excluded.title, " ++ " content_hash = excluded.content_hash, " ++ " mtime = excluded.mtime, " ++ " size = excluded.size, " ++ " updated_at = excluded.updated_at, " ++ " indexed_at = excluded.indexed_at " ++ "RETURNING id",+ )+ .get(doc.path, doc.title, doc.contentHash, doc.mtime, doc.size, now, now, now);+ if (!row) {+ throw new SearchError("INDEX_UNREADABLE", `upsertDocument returned no id for '${doc.path}'`);+ }+ return row.id;+ }++ /**+ * Delete a document and everything that hangs off it. The vec rows+ * are removed first because the FK cascade does not reach the+ * `chunk_vec` virtual table.+ */+ deleteDocument(path: string): void {+ const id = this.getDocumentIdByPath(path);+ if (id === null) return;+ this.purgeVecRowsForDocument(id);+ this.db.run("DELETE FROM documents WHERE id = ?", [id]);+ }++ // ββ chunks βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++ getChunksByDocument(documentId: number): ChunkRow[] {+ const rows = this.db+ .query<+ {+ id: number;+ document_id: number;+ chunk_index: number;+ content: string;+ content_hash: string;+ start_line: number;+ end_line: number;+ token_count: number;+ },+ [number]+ >(+ "SELECT id, document_id, chunk_index, content, content_hash, start_line, end_line, token_count " ++ "FROM chunks WHERE document_id = ? ORDER BY chunk_index",+ )+ .all(documentId);+ return rows.map((r) => ({+ id: r.id,+ documentId: r.document_id,+ chunkIndex: r.chunk_index,+ content: r.content,+ contentHash: r.content_hash,+ startLine: r.start_line,+ endLine: r.end_line,+ tokenCount: r.token_count,+ }));+ }++ /**+ * Atomically replace every chunk for a document. Old vec rows are+ * removed first; FTS5 stays in sync via the chunks_ai/ad/au triggers.+ * Returns the new chunk ids in `chunkIndex` order.+ */+ replaceChunks(documentId: number, chunks: ReadonlyArray<ChunkInput>): number[] {+ const ids: number[] = [];+ this.db.exec("BEGIN");+ try {+ this.purgeVecRowsForDocument(documentId);+ this.db.run("DELETE FROM chunks WHERE document_id = ?", [documentId]);+ const insert = this.db.prepare<+ { id: number },+ [number, number, string, string, number, number, number, string, string]+ >(+ "INSERT INTO chunks(document_id, chunk_index, content, content_hash, start_line, end_line, token_count, created_at, updated_at) " ++ "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id",+ );+ const now = nowIso();+ for (const c of chunks) {+ const row = insert.get(+ documentId,+ c.chunkIndex,+ c.content,+ c.contentHash,+ c.startLine,+ c.endLine,+ c.tokenCount,+ now,+ now,+ );+ if (!row) throw new SearchError("INDEX_UNREADABLE", "chunk insert returned no id");+ ids.push(row.id);+ }+ this.db.exec("COMMIT");+ } catch (e) {+ this.db.exec("ROLLBACK");+ throw e;+ }+ return ids;+ }++ /**+ * Delete a set of chunks by id. Vec rows removed first.+ */+ deleteChunks(chunkIds: ReadonlyArray<number>): void {+ if (chunkIds.length === 0) return;+ this.db.exec("BEGIN");+ try {+ this.purgeVecRowsByChunkIds(chunkIds);+ const placeholders = chunkIds.map(() => "?").join(",");+ this.db.run(`DELETE FROM chunks WHERE id IN (${placeholders})`, chunkIds as number[]);+ this.db.exec("COMMIT");+ } catch (e) {+ this.db.exec("ROLLBACK");+ throw e;+ }+ }++ private purgeVecRowsForDocument(documentId: number): void {+ if (!this._vecLoaded) return;+ const vecRows = this.db+ .query<{ vec_rowid: number }, [number]>(+ "SELECT vec_rowid FROM chunk_vec_map WHERE chunk_id IN (SELECT id FROM chunks WHERE document_id = ?)",+ )+ .all(documentId);+ if (vecRows.length === 0) return;+ this.purgeVecRowidsRaw(vecRows.map((r) => r.vec_rowid));+ }++ private purgeVecRowsByChunkIds(chunkIds: ReadonlyArray<number>): void {+ if (!this._vecLoaded || chunkIds.length === 0) return;+ const placeholders = chunkIds.map(() => "?").join(",");+ const vecRows = this.db+ .query<{ vec_rowid: number }, number[]>(+ `SELECT vec_rowid FROM chunk_vec_map WHERE chunk_id IN (${placeholders})`,+ )+ .all(...(chunkIds as number[]));+ if (vecRows.length === 0) return;+ this.purgeVecRowidsRaw(vecRows.map((r) => r.vec_rowid));+ }++ private purgeVecRowidsRaw(vecRowids: number[]): void {+ if (!this._vecLoaded || vecRowids.length === 0) return;+ const placeholders = vecRowids.map(() => "?").join(",");+ this.db.run(`DELETE FROM chunk_vec WHERE rowid IN (${placeholders})`, vecRowids);+ }++ // ββ embeddings βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++ /**+ * Insert or replace a single embedding. The vec table receives the+ * raw float32 bytes; the metadata row in `embeddings` tracks model /+ * dimension / hash for stale detection.+ *+ * Throws VEC_EXTENSION_UNAVAILABLE if sqlite-vec didn't load. The+ * caller decides whether to surface this (explicit semantic) or warn+ * and skip (implicit semantic).+ */+ vecUpsert(+ chunkId: number,+ vector: ReadonlyArray<number> | Float32Array,+ model: string,+ dimension: number,+ embeddingHash: string,+ ): void {+ if (!this._vecLoaded) {+ throw new SearchError(+ "VEC_EXTENSION_UNAVAILABLE",+ "sqlite-vec extension not loaded; cannot store embeddings",+ );+ }+ const len = vector instanceof Float32Array ? vector.length : vector.length;+ if (len !== dimension) {+ throw new SearchError(+ "EMBEDDING_DIMENSION_MISMATCH",+ `vector dimension ${len} != configured dimension ${dimension}`,+ );+ }+ this.db.exec("BEGIN");+ try {+ const existing = this.db+ .query<{ vec_rowid: number }, [number]>(+ "SELECT vec_rowid FROM chunk_vec_map WHERE chunk_id = ?",+ )+ .get(chunkId);+ const buf = vecToBuffer(vector);+ let vecRowid: number;+ if (existing) {+ this.db.run("UPDATE chunk_vec SET embedding = ? WHERE rowid = ?", [buf, existing.vec_rowid]);+ vecRowid = existing.vec_rowid;+ } else {+ this.db.run("INSERT INTO chunk_vec(embedding) VALUES (?)", [buf]);+ const row = this.db+ .query<{ id: number }, []>("SELECT last_insert_rowid() AS id")+ .get();+ if (!row) throw new SearchError("INDEX_UNREADABLE", "chunk_vec insert returned no rowid");+ vecRowid = row.id;+ this.db.run(+ "INSERT INTO chunk_vec_map(chunk_id, vec_rowid) VALUES (?, ?)",+ [chunkId, vecRowid],+ );+ }+ const now = nowIso();+ this.db.run(+ "INSERT INTO embeddings(chunk_id, model, dimension, embedding_hash, created_at, updated_at) " ++ "VALUES (?, ?, ?, ?, ?, ?) " ++ "ON CONFLICT(chunk_id) DO UPDATE SET " ++ " model = excluded.model, dimension = excluded.dimension, " ++ " embedding_hash = excluded.embedding_hash, updated_at = excluded.updated_at",+ [chunkId, model, dimension, embeddingHash, now, now],+ );+ this.db.exec("COMMIT");+ } catch (e) {+ this.db.exec("ROLLBACK");+ throw e;+ }+ }++ getEmbeddingHash(chunkId: number): string | null {+ const row = this.db+ .query<{ embedding_hash: string }, [number]>(+ "SELECT embedding_hash FROM embeddings WHERE chunk_id = ?",+ )+ .get(chunkId);+ return row?.embedding_hash ?? null;+ }++ /**+ * Drop all embeddings + vec storage. Used when the configured model+ * or dimension changes. `chunks` and `chunk_fts` are preserved.+ */+ clearEmbeddings(): void {+ this.db.exec("BEGIN");+ try {+ this.db.run("DELETE FROM embeddings");+ this.db.run("DELETE FROM chunk_vec_map");+ if (this._vecLoaded) dropVecTable(this.db);+ this.db.exec("COMMIT");+ } catch (e) {+ this.db.exec("ROLLBACK");+ throw e;+ }+ }++ /**+ * Compare the configured embedding model/dimension with what was+ * recorded in `index_state` on the last index run. If they differ+ * and both old + new are non-null, drop embeddings + vec table and+ * log one line per design Β§13. First-time set just records state.+ */+ ensureEmbeddingModel(model: string | null, dimension: number | null): ModelChangeOutcome {+ const prevModel = this.getState("embedding_model");+ const prevDimRaw = this.getState("embedding_dimension");+ const prevDim = prevDimRaw === null ? null : Number(prevDimRaw);++ const modelChanged = prevModel !== null && model !== null && prevModel !== model;+ const dimChanged =+ prevDim !== null && dimension !== null && Number.isFinite(prevDim) && prevDim !== dimension;++ if (modelChanged || dimChanged) {+ this.clearEmbeddings();+ // eslint-disable-next-line no-console+ console.error(+ `embedding model changed from ${prevModel}/${prevDim} to ${model}/${dimension}, embeddings cleared`,+ );+ this.deleteState("embedding_model");+ this.deleteState("embedding_dimension");+ }++ if (model !== null) this.setState("embedding_model", model);+ if (dimension !== null) this.setState("embedding_dimension", String(dimension));++ // (Re)create vec table when we know the dimension and vec is loaded.+ if (this._vecLoaded && dimension !== null) {+ ensureVecTable(this.db, dimension);+ }++ return Object.freeze({+ wasChanged: modelChanged || dimChanged,+ previousModel: prevModel,+ previousDimension: prevDim,+ currentModel: model,+ currentDimension: dimension,+ });+ }++ // ββ links ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++ replaceLinks(sourceDocumentId: number, links: ReadonlyArray<LinkInput>): void {+ this.db.exec("BEGIN");+ try {+ this.db.run("DELETE FROM links WHERE source_document_id = ?", [sourceDocumentId]);+ if (links.length > 0) {+ const insert = this.db.prepare<+ undefined,+ [number, number | null, string | null, string | null, string, string]+ >(+ "INSERT INTO links(source_document_id, source_chunk_id, target_path, link_text, link_type, created_at) " ++ "VALUES (?, ?, ?, ?, ?, ?)",+ );+ const now = nowIso();+ for (const l of links) {+ insert.run(sourceDocumentId, l.sourceChunkId, l.targetPath, l.linkText, l.linkType, now);+ }+ }+ this.db.exec("COMMIT");+ } catch (e) {+ this.db.exec("ROLLBACK");+ throw e;+ }+ }++ resolveLinkTargets(): void {+ this.db.run(+ "UPDATE links SET target_document_id = (SELECT id FROM documents WHERE documents.path = links.target_path) " ++ "WHERE target_path IS NOT NULL",+ );+ }++ // ββ search βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++ /**+ * Top-K BM25 keyword hits. `fts5Query` is an already-escaped FTS5+ * MATCH expression; building it is fts.ts's job.+ */+ keywordTopK(+ fts5Query: string,+ opts: { readonly limit: number; readonly pathPrefix?: string | null },+ ): KeywordHit[] {+ const limit = Math.max(1, opts.limit | 0);+ const prefix = opts.pathPrefix && opts.pathPrefix.length > 0 ? opts.pathPrefix : null;++ if (prefix) {+ const rows = this.db+ .query<+ { chunk_id: number; document_id: number; bm25: number },+ [string, string, string, number]+ >(+ "SELECT c.id AS chunk_id, c.document_id AS document_id, bm25(chunk_fts) AS bm25 " ++ "FROM chunk_fts " ++ "JOIN chunks c ON c.id = chunk_fts.rowid " ++ "JOIN documents d ON d.id = c.document_id " ++ "WHERE chunk_fts MATCH ? AND substr(d.path, 1, length(?)) = ? " ++ "ORDER BY bm25 ASC LIMIT ?",+ )+ .all(fts5Query, prefix, prefix, limit);+ return rows.map((r) => ({ chunkId: r.chunk_id, documentId: r.document_id, bm25: r.bm25 }));+ }++ const rows = this.db+ .query<{ chunk_id: number; document_id: number; bm25: number }, [string, number]>(+ "SELECT c.id AS chunk_id, c.document_id AS document_id, bm25(chunk_fts) AS bm25 " ++ "FROM chunk_fts JOIN chunks c ON c.id = chunk_fts.rowid " ++ "WHERE chunk_fts MATCH ? ORDER BY bm25 ASC LIMIT ?",+ )+ .all(fts5Query, limit);+ return rows.map((r) => ({ chunkId: r.chunk_id, documentId: r.document_id, bm25: r.bm25 }));+ }++ semanticTopK(+ queryVector: ReadonlyArray<number> | Float32Array,+ opts: { readonly limit: number; readonly pathPrefix?: string | null },+ ): SemanticHit[] {+ if (!this._vecLoaded) {+ throw new SearchError(+ "VEC_EXTENSION_UNAVAILABLE",+ "sqlite-vec extension not loaded; semantic search unavailable",+ );+ }+ const limit = Math.max(1, opts.limit | 0);+ const prefix = opts.pathPrefix && opts.pathPrefix.length > 0 ? opts.pathPrefix : null;++ const buf = vecToBuffer(queryVector);+ if (prefix) {+ const rows = this.db+ .query<+ { chunk_id: number; document_id: number; distance: number },+ [Buffer, number, string, string]+ >(+ "SELECT m.chunk_id AS chunk_id, c.document_id AS document_id, v.distance AS distance " ++ "FROM chunk_vec v " ++ "JOIN chunk_vec_map m ON m.vec_rowid = v.rowid " ++ "JOIN chunks c ON c.id = m.chunk_id " ++ "JOIN documents d ON d.id = c.document_id " ++ "WHERE v.embedding MATCH ? AND k = ? AND substr(d.path, 1, length(?)) = ? " ++ "ORDER BY v.distance ASC",+ )+ .all(buf, limit * 4, prefix, prefix);+ return rows.slice(0, limit).map((r) => ({+ chunkId: r.chunk_id,+ documentId: r.document_id,+ distance: r.distance,+ }));+ }++ const rows = this.db+ .query<{ chunk_id: number; document_id: number; distance: number }, [Buffer, number]>(+ "SELECT m.chunk_id AS chunk_id, c.document_id AS document_id, v.distance AS distance " ++ "FROM chunk_vec v " ++ "JOIN chunk_vec_map m ON m.vec_rowid = v.rowid " ++ "JOIN chunks c ON c.id = m.chunk_id " ++ "WHERE v.embedding MATCH ? AND k = ? " ++ "ORDER BY v.distance ASC",+ )+ .all(buf, limit);+ return rows.map((r) => ({ chunkId: r.chunk_id, documentId: r.document_id, distance: r.distance }));+ }++ hydrateChunks(chunkIds: ReadonlyArray<number>): Map<number, HydratedChunk> {+ const out = new Map<number, HydratedChunk>();+ if (chunkIds.length === 0) return out;+ const placeholders = chunkIds.map(() => "?").join(",");+ const rows = this.db+ .query<+ {+ chunk_id: number;+ document_id: number;+ path: string;+ title: string | null;+ content: string;+ start_line: number;+ end_line: number;+ mtime: number;+ },+ number[]+ >(+ "SELECT c.id AS chunk_id, c.document_id, d.path AS path, d.title AS title, " ++ " c.content AS content, c.start_line AS start_line, c.end_line AS end_line, d.mtime AS mtime " ++ "FROM chunks c JOIN documents d ON d.id = c.document_id " ++ `WHERE c.id IN (${placeholders})`,+ )+ .all(...(chunkIds as number[]));+ for (const r of rows) {+ out.set(r.chunk_id, {+ chunkId: r.chunk_id,+ documentId: r.document_id,+ path: r.path,+ title: r.title,+ content: r.content,+ startLine: r.start_line,+ endLine: r.end_line,+ mtime: r.mtime,+ });+ }+ return out;+ }++ /**+ * For each chunk id, return the document ids that link TO that+ * chunk's document via wikilink or markdown_link. Pure data; the+ * ranker decides how to convert this into a boost.+ */+ inboundLinkSources(candidateChunkIds: ReadonlyArray<number>): Map<number, Set<number>> {+ const out = new Map<number, Set<number>>();+ if (candidateChunkIds.length === 0) return out;+ const placeholders = candidateChunkIds.map(() => "?").join(",");+ const rows = this.db+ .query<+ { chunk_id: number; source_document_id: number },+ number[]+ >(+ "SELECT c.id AS chunk_id, l.source_document_id " ++ `FROM chunks c JOIN links l ON l.target_document_id = c.document_id ` ++ `WHERE c.id IN (${placeholders}) AND l.link_type IN ('wikilink','markdown_link') ` ++ ` AND l.source_document_id != c.document_id`,+ )+ .all(...(candidateChunkIds as number[]));+ for (const r of rows) {+ let set = out.get(r.chunk_id);+ if (!set) {+ set = new Set();+ out.set(r.chunk_id, set);+ }+ set.add(r.source_document_id);+ }+ return out;+ }++ /**+ * For each chunk id, the set of tag link_text values associated with+ * its document. Two chunks "share a tag" iff their tag sets intersect.+ */+ tagsByChunkDocument(candidateChunkIds: ReadonlyArray<number>): Map<number, Set<string>> {+ const out = new Map<number, Set<string>>();+ if (candidateChunkIds.length === 0) return out;+ const placeholders = candidateChunkIds.map(() => "?").join(",");+ const rows = this.db+ .query<+ { chunk_id: number; tag: string },+ number[]+ >(+ "SELECT c.id AS chunk_id, l.link_text AS tag " ++ `FROM chunks c JOIN links l ON l.source_document_id = c.document_id ` ++ `WHERE c.id IN (${placeholders}) AND l.link_type = 'tag' AND l.link_text IS NOT NULL`,+ )+ .all(...(candidateChunkIds as number[]));+ for (const r of rows) {+ let set = out.get(r.chunk_id);+ if (!set) {+ set = new Set();+ out.set(r.chunk_id, set);+ }+ set.add(r.tag);+ }+ return out;+ }++ // ββ counts βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++ counts(): StoreCounts {+ const docs = this.db.query<{ c: number }, []>("SELECT count(*) AS c FROM documents").get();+ const chunks = this.db.query<{ c: number }, []>("SELECT count(*) AS c FROM chunks").get();+ const emb = this.db.query<{ c: number }, []>("SELECT count(*) AS c FROM embeddings").get();+ const stale = this.staleEmbeddings();+ return Object.freeze({+ documents: docs?.c ?? 0,+ chunks: chunks?.c ?? 0,+ embeddings: emb?.c ?? 0,+ staleEmbeddings: stale,+ });+ }++ private staleEmbeddings(): number {+ const model = this.config.semantic.model;+ const dimension = this.config.semantic.dimension;+ if (!model || !dimension) {+ // No baseline to compare against: 0 stale by convention.+ return 0;+ }+ const row = this.db+ .query<{ c: number }, [string, number]>(+ "SELECT count(*) AS c FROM embeddings WHERE model != ? OR dimension != ?",+ )+ .get(model, dimension);+ return row?.c ?? 0;+ }++ // ββ direct accessors used by indexer/CLI/status ββββββββββββββββββββββββββββ++ /**+ * Chunks that have no row in `embeddings`. Used by the indexer to+ * populate vectors after a fresh index or after the model-change drop.+ */+ findChunksWithoutEmbeddings(): Array<{ chunkId: number; content: string }> {+ const rows = this.db+ .query<{ id: number; content: string }, []>(+ "SELECT c.id AS id, c.content AS content FROM chunks c " ++ "LEFT JOIN embeddings e ON e.chunk_id = c.id " ++ "WHERE e.chunk_id IS NULL ORDER BY c.id",+ )+ .all();+ return rows.map((r) => ({ chunkId: r.id, content: r.content }));+ }++ /** Escape hatch for status queries that don't fit the typed API. */+ rawQuery<T>(sql: string, params: ReadonlyArray<string | number | null> = []): T[] {+ return this.db.query<T, (string | number | null)[]>(sql).all(...(params as (string | number | null)[]));+ }+}diff --git a/src/core/search/types.ts b/src/core/search/types.tsnew file mode 100644index 00000000..44da1b67--- /dev/null+++ b/src/core/search/types.ts@@ -0,0 +1,135 @@+/**+ * Public types for `src/core/search/*`. Plain data β no behaviour, no I/O.+ *+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§12, Β§14.+ */++export const SEARCH_ERROR_CODES = [+ "INDEX_MISSING",+ "INDEX_UNREADABLE",+ "SCHEMA_MISMATCH",+ "VEC_EXTENSION_UNAVAILABLE",+ "EMBEDDING_DISABLED",+ "EMBEDDING_KEY_MISSING",+ "EMBEDDING_PROVIDER_HTTP",+ "EMBEDDING_PROVIDER_TIMEOUT",+ "EMBEDDING_DIMENSION_MISMATCH",+ "INDEX_LOCKED",+ "INVALID_INPUT",+] as const;+export type SearchErrorCode = (typeof SEARCH_ERROR_CODES)[number];++export class SearchError extends Error {+ readonly code: SearchErrorCode;+ constructor(code: SearchErrorCode, message: string) {+ super(message);+ this.name = "SearchError";+ this.code = code;+ }+}++export interface BrainSearchResult {+ readonly documentId: number;+ readonly chunkId: number;+ readonly path: string;+ readonly title: string | null;+ readonly content: string;+ readonly startLine: number;+ readonly endLine: number;+ readonly score: number;+ readonly keywordScore: number;+ readonly semanticScore: number;+ readonly linkBoost: number;+ readonly recencyBoost: number;+ readonly searchType: "keyword" | "semantic" | "hybrid";+}++export interface IndexStats {+ readonly added: number;+ readonly updated: number;+ readonly unchanged: number;+ readonly deleted: number;+ readonly chunksTotal: number;+ readonly embeddingsComputed: number;+ readonly embeddingsRetries: number;+ readonly errors: ReadonlyArray<{ readonly path: string; readonly message: string }>;+ readonly durationMs: number;+}++/**+ * State of the optional sqlite-vec extension. `not-attempted` covers+ * the diagnostic path where we never tried to load (e.g. `check` on a+ * vault with semantic disabled); `unknown` covers status snapshots+ * taken before any open has happened (index file missing).+ */+export type VecExtensionState = "loaded" | "unavailable" | "unknown" | "not-attempted";++export interface IndexStatusSnapshot {+ readonly indexPath: string;+ readonly exists: boolean;+ readonly schemaVersion: number | null;+ readonly documents: number;+ readonly chunks: number;+ readonly embeddings: number;+ readonly staleEmbeddings: number;+ readonly embeddingModel: string | null;+ readonly embeddingDimension: number | null;+ readonly vecExtension: VecExtensionState;+ readonly semanticEnabled: boolean;+ readonly embeddingKeyPresent: boolean;+ readonly lastIndexedAt: string | null;+ readonly lastFullIndexAt: string | null;+ readonly warnings: ReadonlyArray<string>;+}++export interface IndexCheckReport {+ readonly vaultReadable: boolean;+ readonly indexDirWritable: boolean;+ readonly sqliteOk: boolean;+ readonly fts5Ok: boolean;+ readonly vecExtension: VecExtensionState;+ readonly embeddingKeyResolved: boolean;+ readonly providerReachable: boolean | null;+ readonly providerReason: string | null;+ readonly warnings: ReadonlyArray<string>;+ readonly fatal: ReadonlyArray<string>;+}++export interface SearchOptions {+ readonly query: string;+ readonly limit?: number;+ readonly semantic?: boolean | null;+ readonly keywordOnly?: boolean;+ readonly pathPrefix?: string;+ readonly keywordWeight?: number;+ readonly semanticWeight?: number;+}++export interface SearchOutcome {+ readonly results: ReadonlyArray<BrainSearchResult>;+ readonly warnings: ReadonlyArray<string>;+ readonly total: number;+}++export interface ResolvedEmbeddingConfig {+ readonly enabled: boolean;+ readonly provider: "openai-compat" | "disabled";+ readonly baseUrl: string | null;+ readonly model: string | null;+ readonly apiKey: string | null;+ readonly dimension: number | null;+ readonly timeoutMs: number;+ readonly concurrency: number;+ readonly batchSize: number;+}++export interface ResolvedSearchConfig {+ readonly vault: string;+ readonly dbPath: string;+ readonly ignorePaths: ReadonlyArray<string>;+ readonly chunkSize: number;+ readonly chunkOverlap: number;+ readonly keywordWeight: number;+ readonly semanticWeight: number;+ readonly semantic: ResolvedEmbeddingConfig;+}diff --git a/src/core/search/walker.ts b/src/core/search/walker.tsnew file mode 100644index 00000000..a579ffa2--- /dev/null+++ b/src/core/search/walker.ts@@ -0,0 +1,131 @@+/**+ * Walk the vault, yielding `.md` files that should be indexed.+ *+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§6 edge cases+ * (symlinks, ignore list).+ */++import { readdirSync, statSync, realpathSync, type Dirent, type Stats } from "node:fs";+import { join, relative, sep } from "node:path";++import type { ResolvedSearchConfig } from "./types.ts";++export interface WalkedFile {+ /** Absolute path on disk. */+ readonly absPath: string;+ /** Vault-relative POSIX path. */+ readonly relPath: string;+ readonly stat: Stats;+}++function toPosix(p: string): string {+ return sep === "/" ? p : p.split(sep).join("/");+}++/**+ * Parse the ignore list into two sets:+ *+ * - bare names (no `/`) match a directory whose name equals the entry+ * anywhere in the tree;+ * - relative paths match the vault-relative directory exactly.+ */+function parseIgnore(ignorePaths: ReadonlyArray<string>): {+ names: Set<string>;+ relPaths: Set<string>;+} {+ const names = new Set<string>();+ const relPaths = new Set<string>();+ for (const raw of ignorePaths) {+ const e = raw.trim();+ if (e === "") continue;+ if (e.includes("/")) relPaths.add(e);+ else names.add(e);+ }+ return { names, relPaths };+}++function isInsideVault(absTarget: string, vaultReal: string): boolean {+ const targetReal = (() => {+ try {+ return realpathSync(absTarget);+ } catch {+ return null;+ }+ })();+ if (targetReal === null) return false;+ return targetReal === vaultReal || targetReal.startsWith(vaultReal + sep);+}++/**+ * Synchronous generator yielding every `.md` file under `config.vault`+ * (respecting `config.ignorePaths`). The caller drives the iteration so+ * the indexer can pipeline reads + writes without buffering the whole+ * tree.+ */+export function* walkVault(config: ResolvedSearchConfig): Generator<WalkedFile> {+ const vaultReal = (() => {+ try {+ return realpathSync(config.vault);+ } catch {+ return config.vault;+ }+ })();+ const { names, relPaths } = parseIgnore(config.ignorePaths);+ // Track real (canonical) paths of visited directories so a symlink+ // pointing back at an ancestor (or sibling) cannot send the walker+ // into an infinite loop. `isInsideVault` covers escape outside the+ // vault but is not acyclic on its own.+ const seenDirs = new Set<string>([vaultReal]);++ function* walk(dir: string): Generator<WalkedFile> {+ let entries: Dirent[];+ try {+ entries = readdirSync(dir, { withFileTypes: true, encoding: "utf8" }) as Dirent[];+ } catch {+ return;+ }+ // Sort by name so two identical vaults produce the same traversal+ // order across filesystems and platforms β important for the+ // deterministic-indexing contract and for stable Syncthing peers.+ entries.sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0));+ for (const entry of entries) {+ const absPath = join(dir, entry.name);+ const relPathRaw = relative(vaultReal, absPath);+ if (relPathRaw === "" || relPathRaw.startsWith("..")) continue;+ const relPath = toPosix(relPathRaw);++ const isLinkHint = entry.isSymbolicLink();++ let stat: Stats;+ try {+ stat = statSync(absPath);+ } catch {+ continue;+ }++ if (stat.isDirectory()) {+ if (names.has(entry.name)) continue;+ if (relPaths.has(relPath)) continue;+ let dirReal: string;+ try {+ dirReal = realpathSync(absPath);+ } catch {+ continue;+ }+ if (dirReal !== vaultReal && !dirReal.startsWith(vaultReal + sep)) continue;+ if (seenDirs.has(dirReal)) continue;+ seenDirs.add(dirReal);+ yield* walk(absPath);+ continue;+ }++ if (!stat.isFile()) continue;+ if (!entry.name.toLowerCase().endsWith(".md")) continue;+ if (isLinkHint && !isInsideVault(absPath, vaultReal)) continue;++ yield { absPath, relPath, stat };+ }+ }++ yield* walk(vaultReal);+}diff --git a/src/core/search/with-timeout.ts b/src/core/search/with-timeout.tsnew file mode 100644index 00000000..0dafde1b--- /dev/null+++ b/src/core/search/with-timeout.ts@@ -0,0 +1,35 @@+/**+ * Race `p` against a fixed timeout. The factory builds the rejection+ * value so callers can throw the typed error class they need+ * (`SearchError`, `MCPError`, a plain `Error`, β¦) without each call+ * site reimplementing this utility.+ */++export function withTimeout<T>(+ p: Promise<T>,+ ms: number,+ rejectionFactory: (ms: number) => unknown = (n) => new Error(`timeout after ${n}ms`),+): Promise<T> {+ return new Promise<T>((resolve, reject) => {+ const timer = setTimeout(() => {+ // If a buggy factory throws while building the timeout error, the+ // promise would otherwise hang forever β settle with whatever the+ // factory threw instead.+ try {+ reject(rejectionFactory(ms));+ } catch (e) {+ reject(e);+ }+ }, ms);+ p.then(+ (v) => {+ clearTimeout(timer);+ resolve(v);+ },+ (e) => {+ clearTimeout(timer);+ reject(e);+ },+ );+ });+}diff --git a/src/mcp/search-tools.ts b/src/mcp/search-tools.tsnew file mode 100644index 00000000..3040336b--- /dev/null+++ b/src/mcp/search-tools.ts@@ -0,0 +1,214 @@+/**+ * MCP tool registry slice for Brain Search.+ *+ * Exposes `brain_search` (read-only, agent-facing) and the+ * `search.*` enrichment used by `second_brain_status`. Index management+ * verbs (`index`, `reindex`, `check`) are intentionally NOT exposed+ * over MCP β they are operator business, never agent business+ * (design doc Β§3, principle 5).+ *+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§9.+ */++import {+ indexStatus,+ resolveSearchConfig,+ search,+ SearchError,+} from "../core/search/index.ts";+import type {+ BrainSearchResult,+ SearchOutcome,+} from "../core/search/index.ts";+import { withTimeout } from "../core/search/with-timeout.ts";+import { INTERNAL_ERROR, INVALID_PARAMS, MCPError } from "./protocol.ts";+import type { ServerContext, ToolDefinition } from "./tools.ts";++const MCP_LIMIT_MAX = 50;+const MCP_CONTENT_MAX = 600;+const SEARCH_TIMEOUT_MS = 10_000;++const SEARCH_INPUT_SCHEMA: Record<string, unknown> = {+ type: "object",+ properties: {+ query: { type: "string", minLength: 1, maxLength: 2000 },+ limit: { type: "integer", minimum: 1, maximum: MCP_LIMIT_MAX },+ semantic: { type: "boolean" },+ keyword_only: { type: "boolean" },+ path_prefix: { type: "string", maxLength: 256 },+ },+ required: ["query"],+ additionalProperties: false,+};++function coerceBool(args: Record<string, unknown>, key: string): boolean | undefined {+ if (!(key in args)) return undefined;+ const v = args[key];+ if (v === undefined || v === null) return undefined;+ if (typeof v !== "boolean") {+ throw new MCPError(INVALID_PARAMS, `argument '${key}' must be a boolean`);+ }+ return v;+}++function coerceStringOptional(+ args: Record<string, unknown>,+ key: string,+ maxLen: number,+): string | undefined {+ if (!(key in args)) return undefined;+ const v = args[key];+ if (v === undefined || v === null) return undefined;+ if (typeof v !== "string") {+ throw new MCPError(INVALID_PARAMS, `argument '${key}' must be a string`);+ }+ if (v.length > maxLen) {+ throw new MCPError(INVALID_PARAMS, `argument '${key}' exceeds ${maxLen} characters`);+ }+ return v;+}++function searchTimeoutError(ms: number): MCPError {+ return new MCPError(INTERNAL_ERROR, `search timeout after ${ms}ms`);+}++function truncateContent(c: string, max: number): string {+ if (c.length <= max) return c;+ return c.slice(0, max - 1) + "β¦";+}++function searchErrorToMcp(e: SearchError): MCPError {+ if (e.code === "INVALID_INPUT") return new MCPError(INVALID_PARAMS, e.message);+ if (e.code === "INDEX_MISSING") {+ return new MCPError(INTERNAL_ERROR, "search index not initialised. Run: o2b search index");+ }+ if (e.code === "INDEX_UNREADABLE") {+ return new MCPError(INTERNAL_ERROR, `search index unreadable: ${e.message}`);+ }+ if (e.code === "VEC_EXTENSION_UNAVAILABLE") {+ return new MCPError(+ INTERNAL_ERROR,+ "semantic search unavailable: sqlite-vec extension not loaded",+ );+ }+ if (e.code === "EMBEDDING_KEY_MISSING") {+ return new MCPError(INTERNAL_ERROR, "embedding key not configured");+ }+ if (e.code === "EMBEDDING_PROVIDER_HTTP" || e.code === "EMBEDDING_PROVIDER_TIMEOUT") {+ return new MCPError(INTERNAL_ERROR, `embedding provider unavailable: ${e.message}`);+ }+ return new MCPError(INTERNAL_ERROR, `${e.message} [${e.code}]`);+}++async function toolBrainSearch(+ ctx: ServerContext,+ args: Record<string, unknown>,+): Promise<Record<string, unknown>> {+ const query = args["query"];+ if (typeof query !== "string" || query.trim() === "") {+ throw new MCPError(INVALID_PARAMS, "missing required argument: query");+ }+ if (query.length > 2000) {+ throw new MCPError(INVALID_PARAMS, "argument 'query' exceeds 2000 characters");+ }++ let limit = 10;+ if ("limit" in args && args["limit"] !== undefined && args["limit"] !== null) {+ const raw = args["limit"];+ if (typeof raw !== "number" || !Number.isInteger(raw)) {+ throw new MCPError(INVALID_PARAMS, "argument 'limit' must be an integer");+ }+ if (raw < 1 || raw > MCP_LIMIT_MAX) {+ throw new MCPError(INVALID_PARAMS, `argument 'limit' must be between 1 and ${MCP_LIMIT_MAX}`);+ }+ limit = raw;+ }++ const semantic = coerceBool(args, "semantic");+ const keywordOnly = coerceBool(args, "keyword_only") ?? false;+ const pathPrefix = coerceStringOptional(args, "path_prefix", 256);++ const config = resolveSearchConfig({+ vault: ctx.vault,+ configPath: ctx.configPath ?? undefined,+ });++ let outcome: SearchOutcome;+ try {+ outcome = await withTimeout(+ search(config, {+ query,+ limit,+ semantic: semantic ?? null,+ keywordOnly,+ pathPrefix,+ }),+ SEARCH_TIMEOUT_MS,+ searchTimeoutError,+ );+ } catch (e) {+ if (e instanceof SearchError) throw searchErrorToMcp(e);+ if (e instanceof MCPError) throw e;+ throw new MCPError(INTERNAL_ERROR, e instanceof Error ? e.message : String(e));+ }++ return {+ results: outcome.results.map((r: BrainSearchResult) => ({+ path: r.path,+ title: r.title,+ content: truncateContent(r.content, MCP_CONTENT_MAX),+ score: r.score,+ startLine: r.startLine,+ endLine: r.endLine,+ searchType: r.searchType,+ })),+ warnings: outcome.warnings,+ total: outcome.total,+ };+}++export const SEARCH_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([+ {+ name: "brain_search",+ description:+ "Full-text search across the vault. Optional semantic layer when configured. Read-only.",+ inputSchema: SEARCH_INPUT_SCHEMA,+ handler: toolBrainSearch,+ },+]);++/**+ * `search.*` block for `second_brain_status`. Mirrors design Β§9+ * exactly. Never throws β returns `{ exists: false, hint }` if the+ * index does not exist; surfaces errors as `error: "<message>"`.+ */+export async function buildSearchStatusBlock(ctx: ServerContext): Promise<Record<string, unknown>> {+ try {+ const config = resolveSearchConfig({+ vault: ctx.vault,+ configPath: ctx.configPath ?? undefined,+ });+ const snap = await indexStatus(config);+ if (!snap.exists) {+ return { exists: false, hint: "run: o2b search index" };+ }+ return {+ index_path: snap.indexPath,+ exists: true,+ schema_version: snap.schemaVersion,+ documents: snap.documents,+ chunks: snap.chunks,+ embeddings: snap.embeddings,+ stale_embeddings: snap.staleEmbeddings,+ embedding_model: snap.embeddingModel,+ embedding_dimension: snap.embeddingDimension,+ vec_extension: snap.vecExtension,+ semantic_enabled: snap.semanticEnabled,+ embedding_key_present: snap.embeddingKeyPresent,+ last_indexed_at: snap.lastIndexedAt,+ last_full_index_at: snap.lastFullIndexAt,+ };+ } catch (e) {+ return { exists: false, error: e instanceof Error ? e.message : String(e) };+ }+}diff --git a/src/mcp/tools.ts b/src/mcp/tools.tsindex 66e7c8c4..c1058044 100644--- a/src/mcp/tools.ts+++ b/src/mcp/tools.ts@@ -29,6 +29,7 @@ import { computeBrainStatus } from "../core/brain/status.ts";import { doctor } from "../core/doctor.ts";import { appendEvent, validateEventTime } from "../core/event-log.ts";import { BRAIN_TOOLS } from "./brain-tools.ts";+import { SEARCH_TOOLS, buildSearchStatusBlock } from "./search-tools.ts";import {checkPolicy,consumePendingRequest,@@ -170,6 +171,8 @@ async function toolStatus(ctx: ServerContext): Promise<Record<string, unknown>>// Safe to call on a vault that has no Brain layer yet β returns// `present: false` with zero counts.const brain = vaultExists ? computeBrainStatus(ctx.vault) : null;+ const searchDisabled = discovery.data["search_enabled"] === "false";+ const search = vaultExists && !searchDisabled ? await buildSearchStatusBlock(ctx) : null;return {config_path: String(discovery.path),config_exists: discovery.exists,@@ -178,6 +181,7 @@ async function toolStatus(ctx: ServerContext): Promise<Record<string, unknown>>vault_path: ctx.vault,vault_exists: vaultExists,...(brain ? { brain } : {}),+ ...(search ? { search } : {}),};}@@ -636,6 +640,7 @@ export function buildToolTable(): ToolDefinition[] {// use (`o2b append-event`); only the entries in the exported tool// array are removed β see Β§11.1 of the v0.9.0 design doc....BRAIN_TOOLS,+ ...SEARCH_TOOLS,{name: "vault_health",description: "Run vault, config, and plugin manifest health checks.",diff --git a/tests/cli/search.test.ts b/tests/cli/search.test.tsnew file mode 100644index 00000000..4ef10583--- /dev/null+++ b/tests/cli/search.test.ts@@ -0,0 +1,169 @@+/**+ * CLI tests for `o2b search *`.+ *+ * Each test forks the actual `bun src/cli/main.ts` binary via `runCli`,+ * the same harness `tests/cli/brain.test.ts` uses. We assert exit codes,+ * canonical text patterns, and the JSON shape produced by `--json`.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";+import {+ existsSync,+ mkdirSync,+ mkdtempSync,+ rmSync,+ writeFileSync,+} from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { runCli } from "../helpers/run-cli.ts";++let tmp: string;+let vault: string;+let config: string;++function writeVaultFile(rel: string, content: string): void {+ const abs = join(vault, rel);+ mkdirSync(join(abs, ".."), { recursive: true });+ writeFileSync(abs, content);+}++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-search-cli-"));+ vault = join(tmp, "vault");+ mkdirSync(vault, { recursive: true });+ config = join(tmp, "config.yaml");+ writeFileSync(config, `vault: "${vault}"\n`);+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++test("search index creates the index file under <vault>/.open-second-brain/", async () => {+ writeVaultFile("a.md", "# A\n\nhello world");+ const out = await runCli(["search", "index"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: config },+ });+ expect(out.returncode).toBe(0);+ expect(existsSync(join(vault, ".open-second-brain", "brain.sqlite"))).toBe(true);+});++test("search status without an index reports 'not initialised' and exits 0", async () => {+ const out = await runCli(["search", "status"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: config },+ });+ expect(out.returncode).toBe(0);+ expect(out.stdout).toContain("not initialised");+});++test("search status --json after an index returns documents count", async () => {+ writeVaultFile("a.md", "# A\n\nbody");+ writeVaultFile("b.md", "# B\n\nbody");+ await runCli(["search", "index"], { env: { OPEN_SECOND_BRAIN_CONFIG: config } });+ const out = await runCli(["search", "status", "--json"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: config },+ });+ expect(out.returncode).toBe(0);+ const obj = JSON.parse(out.stdout);+ expect(obj.exists).toBe(true);+ expect(obj.documents).toBe(2);+ expect(obj.schema_version).toBe(1);+});++test("search query returns a human-readable hit for indexed content", async () => {+ writeVaultFile("notes/foo.md", "# Foo\n\nthe quick brown fox jumps over the lazy dog");+ await runCli(["search", "index"], { env: { OPEN_SECOND_BRAIN_CONFIG: config } });+ const out = await runCli(["search", "fox"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: config },+ });+ expect(out.returncode).toBe(0);+ expect(out.stdout).toContain("notes/foo.md");+});++test("search query --json returns structured results", async () => {+ writeVaultFile("notes/foo.md", "# Foo\n\nfox content");+ await runCli(["search", "index"], { env: { OPEN_SECOND_BRAIN_CONFIG: config } });+ const out = await runCli(["search", "fox", "--json", "--limit", "5"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: config },+ });+ expect(out.returncode).toBe(0);+ const obj = JSON.parse(out.stdout);+ expect(Array.isArray(obj.results)).toBe(true);+ expect(obj.results.length).toBeGreaterThan(0);+ expect(obj.results[0].path).toBe("notes/foo.md");+});++test("search query on missing index fails with exit 1", async () => {+ const out = await runCli(["search", "nothing-here"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: config },+ });+ expect(out.returncode).toBe(1);+ expect(out.stderr).toContain("INDEX_MISSING");+});++test("search check reports vault_readable and sqlite_ok on a fresh vault", async () => {+ const out = await runCli(["search", "check", "--json"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: config },+ });+ expect(out.returncode).toBe(0);+ const obj = JSON.parse(out.stdout);+ expect(obj.vault_readable).toBe(true);+ expect(obj.sqlite_ok).toBe(true);+ expect(obj.fts5_ok).toBe(true);+});++test("search reindex rebuilds the index atomically", async () => {+ writeVaultFile("a.md", "# A");+ await runCli(["search", "index"], { env: { OPEN_SECOND_BRAIN_CONFIG: config } });+ writeVaultFile("b.md", "# B");+ const out = await runCli(["search", "reindex"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: config },+ });+ expect(out.returncode).toBe(0);+ expect(existsSync(join(vault, ".open-second-brain", "brain.sqlite"))).toBe(true);+ expect(existsSync(join(vault, ".open-second-brain", "brain.sqlite.bak"))).toBe(true);+});++test("unknown flag exits with code 2", async () => {+ const out = await runCli(["search", "index", "--bogus"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: config },+ });+ expect(out.returncode).toBe(2);+});++test("invalid numeric search flags exit with code 2 before touching the index", async () => {+ let out = await runCli(["search", "fox", "--keyword-weight", "nan"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: config },+ });+ expect(out.returncode).toBe(2);+ expect(out.stderr).toContain("search_keyword_weight");++ out = await runCli(["search", "index", "--concurrency", "0"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: config },+ });+ expect(out.returncode).toBe(2);+ expect(out.stderr).toContain("embedding_concurrency");+});++test("path-prefix escaping returns exit 2 with INVALID_INPUT", async () => {+ writeVaultFile("a.md", "# A");+ await runCli(["search", "index"], { env: { OPEN_SECOND_BRAIN_CONFIG: config } });+ const out = await runCli(["search", "A", "--path", "../etc/"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: config },+ });+ expect(out.returncode).toBe(2);+ expect(out.stderr).toContain("INVALID_INPUT");+});++test("the default verb is `query` when first positional is unknown", async () => {+ writeVaultFile("a.md", "# A\n\nalpha word");+ await runCli(["search", "index"], { env: { OPEN_SECOND_BRAIN_CONFIG: config } });+ // No explicit verb, just a query token:+ const out = await runCli(["search", "alpha"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: config },+ });+ expect(out.returncode).toBe(0);+ expect(out.stdout).toContain("a.md");+});diff --git a/tests/core/search/chunker.test.ts b/tests/core/search/chunker.test.tsnew file mode 100644index 00000000..1231a373--- /dev/null+++ b/tests/core/search/chunker.test.ts@@ -0,0 +1,172 @@+import { test, expect } from "bun:test";+import { chunkMarkdown } from "../../../src/core/search/chunker.ts";++test("empty file yields no chunks but a title from filename", () => {+ const r = chunkMarkdown("", "my_note");+ expect(r.chunks.length).toBe(0);+ expect(r.title).toBe("my note");+});++test("whitespace-only file yields no chunks", () => {+ const r = chunkMarkdown("\n\n \n", "blank-file");+ expect(r.chunks.length).toBe(0);+});++test("single short paragraph becomes one chunk", () => {+ const r = chunkMarkdown("Hello world.", "doc");+ expect(r.chunks.length).toBe(1);+ expect(r.chunks[0]?.content).toContain("Hello world");+ expect(r.chunks[0]?.startLine).toBe(1);+ expect(r.chunks[0]?.endLine).toBe(1);+ expect(r.chunks[0]?.tokenCount).toBe(2);+});++test("frontmatter becomes synthetic chunk 0 with raw text", () => {+ const text = `---+title: My Title+tags:+ - foo+ - bar+---++# Body H1++Some body paragraph.`;+ const r = chunkMarkdown(text, "ignored");+ expect(r.title).toBe("My Title");+ expect(r.chunks.length).toBeGreaterThanOrEqual(2);+ expect(r.chunks[0]?.content).toContain("title: My Title");+ expect(r.chunks[0]?.content).toContain("tags:");+});++test("malformed frontmatter (no closing ---) is dropped with a warning", () => {+ const text = `---+title: Unterminated++# Body+content here`;+ const r = chunkMarkdown(text, "ignored");+ expect(r.warnings.length).toBe(1);+ expect(r.warnings[0]).toContain("frontmatter");+ // Body still chunked.+ expect(r.chunks.length).toBeGreaterThanOrEqual(1);+});++test("title falls back to first H1 when no frontmatter title", () => {+ const text = `# The Heading++Body.`;+ const r = chunkMarkdown(text, "fallback");+ expect(r.title).toBe("The Heading");+});++test("title falls back to filename when no frontmatter and no H1", () => {+ const r = chunkMarkdown("Just a paragraph.", "my-note_file");+ expect(r.title).toBe("my note file");+});++test("code fence is atomic and not split", () => {+ const long = Array.from({ length: 50 }, (_, i) => `line ${i}`).join("\n");+ const text = "Some intro.\n\n```py\n" + long + "\n```\n\nMore prose.";+ const r = chunkMarkdown(text, "doc");+ // Find a chunk containing the opening fence; it should also contain the closing fence.+ const fenceChunks = r.chunks.filter((c) => c.content.includes("```py"));+ expect(fenceChunks.length).toBe(1);+ expect(fenceChunks[0]?.content).toContain("line 0");+ expect(fenceChunks[0]?.content).toContain("line 49");+});++test("oversized code fence becomes its own chunk and is not truncated", () => {+ const huge = Array.from({ length: 2000 }, (_, i) => `word${i}`).join(" ");+ const text = "Intro paragraph.\n\n```\n" + huge + "\n```\n\nOutro.";+ const r = chunkMarkdown(text, "doc", { maxTokens: 100, minTokens: 50, overlapTokens: 0 });+ const fenceChunk = r.chunks.find((c) => c.content.includes("word1999"));+ expect(fenceChunk).toBeDefined();+ expect(fenceChunk?.content).toContain("word0");+ expect(fenceChunk?.tokenCount).toBeGreaterThan(100);+});++test("heading attaches to current chunk when below min_tokens", () => {+ const text = `# Tiny++word.++# Next Heading++# Even Next`;+ const r = chunkMarkdown(text, "doc", { maxTokens: 800, minTokens: 100, overlapTokens: 0 });+ // All very small; expect a single chunk.+ expect(r.chunks.length).toBe(1);+ expect(r.chunks[0]?.content).toContain("# Tiny");+ expect(r.chunks[0]?.content).toContain("# Next Heading");+});++test("heading starts a new chunk when current >= min_tokens", () => {+ // Generate a paragraph with >=100 tokens, then a new heading.+ const body = Array.from({ length: 150 }, (_, i) => `w${i}`).join(" ");+ const text = `# A\n\n${body}\n\n# B\n\nmore.`;+ const r = chunkMarkdown(text, "doc", { maxTokens: 800, minTokens: 100, overlapTokens: 0 });+ // Find chunk starting with "# B"+ expect(r.chunks.length).toBeGreaterThanOrEqual(2);+ const second = r.chunks[1]!;+ expect(second.content).toContain("# B");+ expect(second.content).toContain("more");+});++test("list block stays atomic until first non-list line", () => {+ const text = `- one+- two+- three++Next paragraph.`;+ const r = chunkMarkdown(text, "doc", { maxTokens: 100, minTokens: 50, overlapTokens: 0 });+ const listChunk = r.chunks.find((c) => c.content.includes("- one"));+ expect(listChunk?.content).toContain("- three");+});++test("table is captured atomically", () => {+ const text = `| a | b |+|---|---|+| 1 | 2 |+| 3 | 4 |++After table.`;+ const r = chunkMarkdown(text, "doc", { maxTokens: 100, minTokens: 50, overlapTokens: 0 });+ const tableChunk = r.chunks.find((c) => c.content.includes("| a | b |"));+ expect(tableChunk?.content).toContain("| 3 | 4 |");+});++test("UTF-8 Cyrillic content roundtrips and tokenises sensibly", () => {+ const text = `# ΠΡΠΈΠ²Π΅Ρ++ΠΡΠΎ ΡΠ΅ΡΡΠΎΠ²Π°Ρ ΡΡΡΠΎΠΊΠ° Ρ ΠΊΠΈΡΠΈΠ»Π»ΠΈΡΠ΅ΠΉ. ΠΡΠΈΠ²Π΅Ρ, ΠΌΠΈΡ!`;+ const r = chunkMarkdown(text, "ru-doc");+ expect(r.chunks.length).toBe(1);+ expect(r.chunks[0]?.content).toContain("ΠΡΠΈΠ²Π΅Ρ");+ expect(r.chunks[0]?.content).toContain("ΠΊΠΈΡΠΈΠ»Π»ΠΈΡΠ΅ΠΉ");+ expect(r.chunks[0]?.tokenCount).toBeGreaterThan(5);+});++test("overlap prepends previous chunk tail to next", () => {+ // Force two chunks via small maxTokens; verify second begins with overlap.+ const para1 = Array.from({ length: 50 }, (_, i) => `a${i}`).join(" ");+ const para2 = Array.from({ length: 50 }, (_, i) => `b${i}`).join(" ");+ const text = `${para1}\n\n${para2}`;+ const r = chunkMarkdown(text, "doc", { maxTokens: 50, minTokens: 10, overlapTokens: 10 });+ expect(r.chunks.length).toBeGreaterThanOrEqual(2);+ const second = r.chunks[1]!;+ // The second chunk should contain something from a-range (tail) AND the b-range start.+ expect(second.content).toContain("b0");+ expect(/a4\d/.test(second.content)).toBe(true);+});++test("line numbers in chunks point to original (non-overlap) lines", () => {+ const text = `line A+line B+line C++line D+line E`;+ const r = chunkMarkdown(text, "doc", { maxTokens: 4, minTokens: 1, overlapTokens: 0 });+ expect(r.chunks[0]?.startLine).toBe(1);+});diff --git a/tests/core/search/config.test.ts b/tests/core/search/config.test.tsnew file mode 100644index 00000000..afe25b66--- /dev/null+++ b/tests/core/search/config.test.ts@@ -0,0 +1,158 @@+import { test, expect, beforeEach, afterEach } from "bun:test";+import { mkdtempSync, writeFileSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";+import { resolveSearchConfig } from "../../../src/core/search/index.ts";+import { SearchError } from "../../../src/core/search/types.ts";++let tmp: string;+let configPath: string;+let origEnv: Record<string, string | undefined>;++const ENV_KEYS = [+ "OPEN_SECOND_BRAIN_SEARCH_DB",+ "OPEN_SECOND_BRAIN_SEARCH_SEMANTIC",+ "OPEN_SECOND_BRAIN_SEARCH_CHUNK_SIZE",+ "OPEN_SECOND_BRAIN_SEARCH_CHUNK_OVERLAP",+ "OPEN_SECOND_BRAIN_SEARCH_KW_WEIGHT",+ "OPEN_SECOND_BRAIN_SEARCH_SEM_WEIGHT",+ "OPEN_SECOND_BRAIN_SEARCH_IGNORE",+ "OPEN_SECOND_BRAIN_EMBEDDING_PROVIDER",+ "OPEN_SECOND_BRAIN_EMBEDDING_BASE_URL",+ "OPEN_SECOND_BRAIN_EMBEDDING_MODEL",+ "OPEN_SECOND_BRAIN_EMBEDDING_KEY",+ "OPEN_SECOND_BRAIN_EMBEDDING_DIM",+ "OPEN_SECOND_BRAIN_EMBEDDING_TIMEOUT",+ "OPEN_SECOND_BRAIN_EMBEDDING_CONCURRENCY",+ "OPEN_SECOND_BRAIN_EMBEDDING_BATCH",+];++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "osb-cfg-"));+ configPath = join(tmp, "config.yaml");+ origEnv = {};+ for (const k of ENV_KEYS) {+ origEnv[k] = process.env[k];+ delete process.env[k];+ }+});++afterEach(() => {+ for (const k of ENV_KEYS) {+ if (origEnv[k] === undefined) delete process.env[k];+ else process.env[k] = origEnv[k];+ }+ rmSync(tmp, { recursive: true, force: true });+});++test("defaults are returned when config and env are empty", () => {+ writeFileSync(configPath, `vault: "${tmp}"\n`);+ const cfg = resolveSearchConfig({ vault: tmp, configPath });+ expect(cfg.vault).toBe(tmp);+ expect(cfg.dbPath).toBe(join(tmp, ".open-second-brain", "brain.sqlite"));+ expect(cfg.chunkSize).toBe(800);+ expect(cfg.chunkOverlap).toBe(100);+ expect(cfg.keywordWeight).toBe(0.6);+ expect(cfg.semanticWeight).toBe(0.4);+ expect(cfg.semantic.enabled).toBe(false);+ expect(cfg.semantic.provider).toBe("openai-compat");+ expect(cfg.semantic.apiKey).toBeNull();+ expect(cfg.ignorePaths).toContain(".git");+ expect(cfg.ignorePaths).toContain(".open-second-brain");+});++test("env overrides config which overrides defaults", () => {+ writeFileSync(+ configPath,+ [+ `vault: "${tmp}"`,+ `search_chunk_size: "500"`,+ `search_semantic_enabled: "true"`,+ `embedding_base_url: "https://config.example/v1"`,+ `embedding_model: "config-model"`,+ `embedding_api_key: "cfg-key"`,+ ].join("\n") + "\n",+ );+ process.env["OPEN_SECOND_BRAIN_EMBEDDING_MODEL"] = "env-model";+ const cfg = resolveSearchConfig({ vault: tmp, configPath });+ expect(cfg.chunkSize).toBe(500);+ expect(cfg.semantic.enabled).toBe(true);+ expect(cfg.semantic.baseUrl).toBe("https://config.example/v1");+ expect(cfg.semantic.model).toBe("env-model");+ expect(cfg.semantic.apiKey).toBe("cfg-key");+});++test("overrides win over both env and config", () => {+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_chunk_size: "500"\n`);+ process.env["OPEN_SECOND_BRAIN_SEARCH_SEMANTIC"] = "false";+ const cfg = resolveSearchConfig({+ vault: tmp,+ configPath,+ overrides: { keywordWeight: 0.9, semanticWeight: 0.1 },+ });+ expect(cfg.keywordWeight).toBe(0.9);+ expect(cfg.semanticWeight).toBe(0.1);+});++test("invalid numeric value throws INVALID_INPUT", () => {+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_chunk_size: "not-a-number"\n`);+ let err: SearchError | null = null;+ try {+ resolveSearchConfig({ vault: tmp, configPath });+ } catch (e) {+ err = e as SearchError;+ }+ expect(err).not.toBeNull();+ expect(err?.code).toBe("INVALID_INPUT");+ expect(err?.message).toContain("search_chunk_size");+});++test("weights summing above 1 is rejected", () => {+ writeFileSync(+ configPath,+ `vault: "${tmp}"\nsearch_keyword_weight: "0.7"\nsearch_semantic_weight: "0.5"\n`,+ );+ expect(() => resolveSearchConfig({ vault: tmp, configPath })).toThrow(/sum.*1/);+});++test("non-positive integer knobs are rejected before provider construction", () => {+ for (const key of [+ "search_chunk_size",+ "embedding_dimension",+ "embedding_timeout_ms",+ "embedding_concurrency",+ "embedding_batch_size",+ ]) {+ writeFileSync(configPath, `vault: "${tmp}"\n${key}: "0"\n`);+ expect(() => resolveSearchConfig({ vault: tmp, configPath })).toThrow(key);+ }+});++test("chunk overlap must be non-negative and smaller than chunk size", () => {+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_chunk_overlap: "-1"\n`);+ expect(() => resolveSearchConfig({ vault: tmp, configPath })).toThrow(/search_chunk_overlap/);++ writeFileSync(+ configPath,+ `vault: "${tmp}"\nsearch_chunk_size: "100"\nsearch_chunk_overlap: "100"\n`,+ );+ expect(() => resolveSearchConfig({ vault: tmp, configPath })).toThrow(/smaller than/);+});++test("overrides are validated after merging", () => {+ writeFileSync(configPath, `vault: "${tmp}"\n`);+ expect(() =>+ resolveSearchConfig({+ vault: tmp,+ configPath,+ overrides: { keywordWeight: Number.NaN },+ }),+ ).toThrow(/search_keyword_weight/);+ expect(() =>+ resolveSearchConfig({+ vault: tmp,+ configPath,+ overrides: { semantic: { batchSize: 0 } },+ }),+ ).toThrow(/embedding_batch_size/);+});diff --git a/tests/core/search/embeddings.test.ts b/tests/core/search/embeddings.test.tsnew file mode 100644index 00000000..a8f9ac1e--- /dev/null+++ b/tests/core/search/embeddings.test.ts@@ -0,0 +1,246 @@+import { test, expect, beforeEach, afterEach } from "bun:test";++import { OpenAICompatProvider } from "../../../src/core/search/embeddings/openai-compat.ts";+import { NullProvider } from "../../../src/core/search/embeddings/null-provider.ts";+import { makeProvider } from "../../../src/core/search/embeddings/provider.ts";+import { SearchError } from "../../../src/core/search/types.ts";+import type { ResolvedEmbeddingConfig } from "../../../src/core/search/types.ts";+import { startFakeHttp, type FakeHttp } from "../../helpers/fake-http.ts";++let server: FakeHttp;++function cfg(overrides: Partial<ResolvedEmbeddingConfig> = {}): ResolvedEmbeddingConfig {+ return Object.freeze({+ enabled: true,+ provider: "openai-compat",+ baseUrl: server.url,+ model: "fake-model",+ apiKey: "test-key",+ dimension: null,+ timeoutMs: 5_000,+ concurrency: 2,+ batchSize: 32,+ ...overrides,+ });+}++beforeEach(async () => {+ server = await startFakeHttp();+});++afterEach(async () => {+ await server.close();+});++test("embed() returns unit-normalised vectors in input order", async () => {+ const p = new OpenAICompatProvider(cfg());+ const out = await p.embed(["a", "b", "c"]);+ expect(out.length).toBe(3);+ // The default fake handler echoes [tokens, index, len, 1]. Index differs.+ // After unit-normalisation, ||v|| == 1.+ for (const v of out) {+ const norm = Math.sqrt(v.reduce((s, x) => s + x * x, 0));+ expect(norm).toBeCloseTo(1.0, 6);+ }+});++test("embed() preserves input order across batches", async () => {+ const p = new OpenAICompatProvider(cfg({ batchSize: 2 }));+ const inputs = ["one", "two", "three", "four", "five"];+ const out = await p.embed(inputs);+ expect(out.length).toBe(5);+ // Server returns >2 calls because batchSize=2 β 3 batches.+ expect(server.callCount()).toBe(3);+});++test("embed() sorts response.data by `index` even if server returns shuffled", async () => {+ server.setHandler((req) => {+ const body = (req.body as { input: string[]; model: string });+ const indices = body.input.map((_, i) => i);+ // Shuffle: reverse.+ const shuffled = [...indices].reverse();+ const data = shuffled.map((origIdx) => {+ const text = body.input[origIdx]!;+ const v = [text.length, origIdx, 1, 1];+ return { object: "embedding", embedding: v, index: origIdx };+ });+ return { status: 200, body: { data, model: body.model } };+ });++ const p = new OpenAICompatProvider(cfg({ dimension: 4 }), { backoffMs: [5, 5] });+ const out = await p.embed(["aaaa", "bb"]);+ // First input "aaaa" has length 4 β unit-normalised first component dominates.+ // Second input "bb" has length 2.+ expect(out[0]!.length).toBe(4);+ expect(out[1]!.length).toBe(4);+ // Sanity: the first input's first dimension > second input's first dimension.+ // After normalisation, monotonic relationship preserved because all other+ // components are identical between the two outputs (index, 1, 1).+ // (We don't need exact values; just confirm we received the right vector+ // per the original `index`.)+ expect(out[0]).not.toEqual(out[1]);+});++test("embed() rejects duplicate response indexes that leave a missing vector", async () => {+ server.setHandler((req) => {+ const body = req.body as { input: string[]; model: string };+ return {+ status: 200,+ body: {+ data: [+ { object: "embedding", embedding: [1, 0, 0, 0], index: 0 },+ { object: "embedding", embedding: [0, 1, 0, 0], index: 0 },+ ],+ model: body.model,+ },+ };+ });++ const p = new OpenAICompatProvider(cfg({ dimension: 4 }), { backoffMs: [5, 5] });+ let err: SearchError | null = null;+ try {+ await p.embed(["a", "b"]);+ } catch (e) {+ err = e as SearchError;+ }+ expect(err?.code).toBe("EMBEDDING_PROVIDER_HTTP");+ expect(err?.message).toContain("missing vector");+});++test("embed() retries on 503 then succeeds", async () => {+ let calls = 0;+ server.setHandler((req) => {+ calls++;+ if (calls === 1) return { status: 503, body: { error: "busy" } };+ return {+ status: 200,+ body: {+ data: [{ object: "embedding", embedding: [0.5, 0.5, 0.5, 0.5], index: 0 }],+ model: (req.body as { model: string }).model,+ },+ };+ });++ const p = new OpenAICompatProvider(cfg(), { backoffMs: [10, 20] });+ const out = await p.embed(["once"]);+ expect(out.length).toBe(1);+ expect(calls).toBe(2);+});++test("embed() fails fast on 400 (non-retriable 4xx)", async () => {+ server.setHandler(() => ({ status: 400, body: { error: "bad request" } }));+ const p = new OpenAICompatProvider(cfg(), { backoffMs: [10, 20] });+ let err: SearchError | null = null;+ try {+ await p.embed(["x"]);+ } catch (e) {+ err = e as SearchError;+ }+ expect(err?.code).toBe("EMBEDDING_PROVIDER_HTTP");+ expect(err?.message).toContain("400");+ expect(server.callCount()).toBe(1);+});++test("embed() throws EMBEDDING_PROVIDER_HTTP after exhausting retries on 5xx", async () => {+ server.setHandler(() => ({ status: 503, body: { error: "busy" } }));+ const p = new OpenAICompatProvider(cfg(), { backoffMs: [10, 20] });+ let err: SearchError | null = null;+ try {+ await p.embed(["x"]);+ } catch (e) {+ err = e as SearchError;+ }+ expect(err?.code).toBe("EMBEDDING_PROVIDER_HTTP");+ expect(server.callCount()).toBe(3);+});++test("embed() honours per-request timeout via AbortController", async () => {+ server.setHandler(() => ({ status: 200, body: { data: [] }, delayMs: 200 }));+ const p = new OpenAICompatProvider(cfg({ timeoutMs: 30 }), { backoffMs: [5, 5] });+ let err: SearchError | null = null;+ try {+ await p.embed(["x"]);+ } catch (e) {+ err = e as SearchError;+ }+ // Either the timeout or the shape-mismatch is acceptable as the surfaced+ // error, but it should be a SearchError from this module.+ expect(err?.code).toBe("EMBEDDING_PROVIDER_TIMEOUT");+}, 15_000);++test("embed() rejects EMBEDDING_DIMENSION_MISMATCH across batch", async () => {+ let i = 0;+ server.setHandler((req) => {+ const body = req.body as { input: string[]; model: string };+ const data = body.input.map((_, idx) => ({+ object: "embedding",+ embedding: i++ === 0 ? [1, 0, 0, 0] : [1, 0, 0],+ index: idx,+ }));+ return { status: 200, body: { data, model: body.model } };+ });+ const p = new OpenAICompatProvider(cfg({ batchSize: 1, concurrency: 1 }));+ let err: SearchError | null = null;+ try {+ await p.embed(["a", "b"]);+ } catch (e) {+ err = e as SearchError;+ }+ expect(err?.code).toBe("EMBEDDING_DIMENSION_MISMATCH");+});++test("ping() returns ok with dimension on success", async () => {+ const p = new OpenAICompatProvider(cfg());+ const res = await p.ping();+ if (!res.ok) throw new Error("expected ok");+ expect(res.dimension).toBeGreaterThan(0);+});++test("ping() returns ok:false when server is unreachable", async () => {+ await server.close();+ const p = new OpenAICompatProvider(cfg(), { backoffMs: [10] });+ const res = await p.ping();+ expect(res.ok).toBe(false);+});++test("makeProvider returns NullProvider when semantic is disabled", () => {+ const p = makeProvider({+ enabled: false,+ provider: "disabled",+ baseUrl: null,+ model: null,+ apiKey: null,+ dimension: null,+ timeoutMs: 10_000,+ concurrency: 1,+ batchSize: 1,+ });+ expect(p).toBeInstanceOf(NullProvider);+});++test("makeProvider throws when key is missing under openai-compat", () => {+ expect(() =>+ makeProvider({+ enabled: true,+ provider: "openai-compat",+ baseUrl: "https://x/v1",+ model: "m",+ apiKey: null,+ dimension: null,+ timeoutMs: 10_000,+ concurrency: 1,+ batchSize: 1,+ }),+ ).toThrow(/embedding_api_key/);+});++test("NullProvider.embed() throws EMBEDDING_DISABLED", async () => {+ const p = new NullProvider();+ let err: SearchError | null = null;+ try {+ await p.embed(["x"]);+ } catch (e) {+ err = e as SearchError;+ }+ expect(err?.code).toBe("EMBEDDING_DISABLED");+});diff --git a/tests/core/search/fts.test.ts b/tests/core/search/fts.test.tsnew file mode 100644index 00000000..f84f6127--- /dev/null+++ b/tests/core/search/fts.test.ts@@ -0,0 +1,133 @@+import { test, expect, beforeEach, afterEach } from "bun:test";++import { buildFtsMatch, runFtsQuery } from "../../../src/core/search/fts.ts";+import { Store } from "../../../src/core/search/store.ts";+import { createTempVault, makeConfig } from "../../helpers/search-fixtures.ts";++let vault: string;+let dbPath: string;+let cleanup: () => void;++beforeEach(() => {+ const v = createTempVault("fts");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+});++afterEach(() => {+ cleanup();+});++async function fixture() {+ const store = await Store.open(makeConfig({ vault, dbPath }), { mode: "write", loadVec: false });+ const d1 = store.upsertDocument({+ path: "Notes/alpha.md",+ title: "Alpha",+ contentHash: "h1",+ mtime: 1700000000,+ size: 10,+ });+ store.replaceChunks(d1, [+ {+ chunkIndex: 0,+ content: "the quick brown fox jumps over the lazy dog",+ contentHash: "c1",+ startLine: 1,+ endLine: 1,+ tokenCount: 9,+ },+ {+ chunkIndex: 1,+ content: "Π±ΡΠΎΠΆΡ ΠΏΠΎ ΡΠΈΡ ΠΈΠΌ ΡΠ»ΠΈΡΠ°ΠΌ ΠΏΠ΅ΡΠ΅ΡΠ±ΡΡΠ³Π°. white nights.",+ contentHash: "c2",+ startLine: 2,+ endLine: 2,+ tokenCount: 7,+ },+ ]);+ const d2 = store.upsertDocument({+ path: "Other/beta.md",+ title: "Beta",+ contentHash: "h2",+ mtime: 1700001000,+ size: 10,+ });+ store.replaceChunks(d2, [+ {+ chunkIndex: 0,+ content: "fox in beta document repeats: fox fox fox",+ contentHash: "c3",+ startLine: 1,+ endLine: 1,+ tokenCount: 7,+ },+ ]);+ return { store, d1, d2 };+}++test("buildFtsMatch quotes tokens for safety", () => {+ expect(buildFtsMatch("hello")).toBe('"hello"');+ expect(buildFtsMatch("two words")).toBe('"two" "words"');+});++test("buildFtsMatch escapes embedded double quotes", () => {+ expect(buildFtsMatch('he said "hi"')).toBe('"he" "said" """hi"""');+});++test("buildFtsMatch ignores empty input", () => {+ expect(buildFtsMatch("")).toBe("");+ expect(buildFtsMatch(" ")).toBe("");+});++test("buildFtsMatch defangs FTS5 operator words and metacharacters", () => {+ // Without quoting, "AND" would be a boolean operator and "*" a prefix glob.+ expect(buildFtsMatch("AND foo*")).toBe('"AND" "foo*"');+});++test("runFtsQuery returns BM25-ordered hits across documents", async () => {+ const { store, d2 } = await fixture();+ const hits = runFtsQuery(store, "fox", { limit: 10 });+ expect(hits.length).toBeGreaterThanOrEqual(2);+ // Beta document has the term repeated four times β should rank above alpha.+ expect(hits[0]?.documentId).toBe(d2);+ await store.close();+});++test("runFtsQuery respects path_prefix filter", async () => {+ const { store, d1 } = await fixture();+ const hits = runFtsQuery(store, "fox", { limit: 10, pathPrefix: "Notes/" });+ expect(hits.length).toBe(1);+ expect(hits[0]?.documentId).toBe(d1);+ await store.close();+});++test("runFtsQuery finds Cyrillic content", async () => {+ const { store, d1 } = await fixture();+ const hits = runFtsQuery(store, "ΡΠΈΡ ΠΈΠΌ", { limit: 10 });+ expect(hits.length).toBe(1);+ expect(hits[0]?.documentId).toBe(d1);+ await store.close();+});++test("runFtsQuery returns empty array on empty query", async () => {+ const { store } = await fixture();+ expect(runFtsQuery(store, " ", { limit: 5 })).toEqual([]);+ await store.close();+});++test("runFtsQuery survives FTS5-operator-like queries safely", async () => {+ const { store } = await fixture();+ // Without quoting, the bare token "AND" would be a syntax error (left-hand+ // operand missing). Quoted as a phrase it's just a literal word match.+ // No row contains "AND" literally so the result is empty, but the query+ // must not throw and must not return spurious matches.+ expect(() => runFtsQuery(store, "AND", { limit: 5 })).not.toThrow();+ expect(runFtsQuery(store, "AND", { limit: 5 })).toEqual([]);+ // A query mixing operator-like tokens with a real token returns matches+ // when all tokens are present in the document.+ const hits = runFtsQuery(store, "fox", { limit: 5 });+ expect(hits.length).toBeGreaterThan(0);+ await store.close();+});+diff --git a/tests/core/search/indexer.embeddings.test.ts b/tests/core/search/indexer.embeddings.test.tsnew file mode 100644index 00000000..d1f7b0df--- /dev/null+++ b/tests/core/search/indexer.embeddings.test.ts@@ -0,0 +1,138 @@+import { test, expect, beforeEach, afterEach } from "bun:test";++import { indexVault } from "../../../src/core/search/indexer.ts";+import { Store } from "../../../src/core/search/store.ts";+import { SearchError } from "../../../src/core/search/types.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";+import { startFakeHttp, type FakeHttp } from "../../helpers/fake-http.ts";++let vault: string;+let dbPath: string;+let cleanup: () => void;+let server: FakeHttp;++beforeEach(async () => {+ const v = createTempVault("indexer-emb");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+ server = await startFakeHttp();+});++afterEach(async () => {+ cleanup();+ await server.close();+});++function semanticConfig(model = "fake-model", dim = 4) {+ return makeConfig({+ vault,+ dbPath,+ semantic: {+ enabled: true,+ provider: "openai-compat",+ baseUrl: server.url,+ model,+ apiKey: "test-key",+ dimension: dim,+ timeoutMs: 5_000,+ concurrency: 2,+ batchSize: 8,+ },+ });+}++test("index --embeddings populates chunk_vec for new chunks", async () => {+ writeMd(vault, "a.md", "# A\n\nFirst note about something.");+ writeMd(vault, "b.md", "# B\n\nSecond note discussing things.");+ const cfg = semanticConfig();++ const stats = await indexVault(cfg, { embeddings: true });+ expect(stats.embeddingsComputed).toBeGreaterThanOrEqual(2);++ const store = await Store.open(cfg, { mode: "read" });+ const counts = store.counts();+ expect(counts.embeddings).toBe(counts.chunks);+ await store.close();+});++test("unchanged files do NOT trigger re-embedding on the next run", async () => {+ writeMd(vault, "a.md", "# A\n\nHello.");+ const cfg = semanticConfig();+ await indexVault(cfg, { embeddings: true });+ const before = server.callCount();++ const second = await indexVault(cfg, { embeddings: true });+ expect(second.embeddingsComputed).toBe(0);+ expect(server.callCount()).toBe(before);+});++test("changing the embedding model drops embeddings and the next index repopulates", async () => {+ writeMd(vault, "a.md", "# A\n\nHello.");+ const first = await indexVault(semanticConfig("modelA", 4), { embeddings: true });+ expect(first.embeddingsComputed).toBeGreaterThan(0);++ const before = server.callCount();+ const second = await indexVault(semanticConfig("modelB", 4), { embeddings: true });+ // All chunks re-embedded under the new model.+ expect(second.embeddingsComputed).toBeGreaterThan(0);+ expect(server.callCount()).toBeGreaterThan(before);+});++test("--embeddings without a key throws EMBEDDING_KEY_MISSING", async () => {+ writeMd(vault, "a.md", "# A");+ const cfg = makeConfig({+ vault,+ dbPath,+ semantic: {+ enabled: true,+ provider: "openai-compat",+ baseUrl: server.url,+ model: "fake-model",+ apiKey: null,+ dimension: 4,+ timeoutMs: 5_000,+ concurrency: 1,+ batchSize: 8,+ },+ });+ let err: SearchError | null = null;+ try {+ await indexVault(cfg, { embeddings: true });+ } catch (e) {+ err = e as SearchError;+ }+ expect(err?.code).toBe("EMBEDDING_KEY_MISSING");+});++test("--embeddings with semantic disabled throws EMBEDDING_DISABLED", async () => {+ writeMd(vault, "a.md", "# A");+ const cfg = makeConfig({ vault, dbPath }); // semantic.enabled defaults to false+ let err: SearchError | null = null;+ try {+ await indexVault(cfg, { embeddings: true });+ } catch (e) {+ err = e as SearchError;+ }+ expect(err?.code).toBe("EMBEDDING_DISABLED");+});++test("onFile callback fires for every classification", async () => {+ writeMd(vault, "stay.md", "# A");+ writeMd(vault, "go.md", "# B");+ const cfg = makeConfig({ vault, dbPath });+ await indexVault(cfg);++ // Change one, delete another, add a third.+ writeMd(vault, "stay.md", "# A\n\nupdated body.");+ await Bun.$`rm ${vault}/go.md`.quiet();+ writeMd(vault, "new.md", "# C");++ const events: Array<{ kind: string; path: string }> = [];+ await indexVault(cfg, { onFile: (e) => events.push({ kind: e.kind, path: e.path }) });++ const kinds = new Set(events.map((e) => e.kind));+ expect(kinds.has("added")).toBe(true);+ expect(kinds.has("updated")).toBe(true);+ expect(kinds.has("deleted")).toBe(true);+});diff --git a/tests/core/search/indexer.test.ts b/tests/core/search/indexer.test.tsnew file mode 100644index 00000000..58c5e981--- /dev/null+++ b/tests/core/search/indexer.test.ts@@ -0,0 +1,165 @@+import { test, expect, beforeEach, afterEach } from "bun:test";+import { existsSync, rmSync, utimesSync, writeFileSync } from "node:fs";+import { join } from "node:path";++import { indexVault, reindexVault, indexStatus, indexCheck } from "../../../src/core/search/indexer.ts";+import { Store } from "../../../src/core/search/store.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";++let vault: string;+let dbPath: string;+let cleanup: () => void;++beforeEach(() => {+ const v = createTempVault("indexer");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+});++afterEach(() => {+ cleanup();+});++test("first index adds every file; second run reports unchanged", async () => {+ writeMd(vault, "a.md", "# A\n\nFirst note.");+ writeMd(vault, "sub/b.md", "# B\n\nSecond note.");+ const cfg = makeConfig({ vault, dbPath });++ const first = await indexVault(cfg);+ expect(first.added).toBe(2);+ expect(first.updated).toBe(0);+ expect(first.unchanged).toBe(0);+ expect(first.deleted).toBe(0);+ expect(first.chunksTotal).toBeGreaterThanOrEqual(2);++ const second = await indexVault(cfg);+ expect(second.added).toBe(0);+ expect(second.updated).toBe(0);+ expect(second.unchanged).toBe(2);+ expect(second.embeddingsComputed).toBe(0);+});++test("modifying a file produces an `updated` event and replaces chunks", async () => {+ const abs = writeMd(vault, "x.md", "# Old\n\noriginal content.");+ const cfg = makeConfig({ vault, dbPath });+ await indexVault(cfg);++ // Mutate content and mtime.+ writeFileSync(abs, "# New\n\nnew content that should be found.");+ const t = Date.now() / 1000 + 60;+ utimesSync(abs, t, t);++ const events: Array<{ path: string; kind: string }> = [];+ const stats = await indexVault(cfg, { onFile: (e) => events.push({ path: e.path, kind: e.kind }) });+ expect(stats.updated).toBe(1);+ expect(events.find((e) => e.path === "x.md")?.kind).toBe("updated");++ // The stale text is gone from FTS.+ const store = await Store.open(cfg, { mode: "read", loadVec: false });+ expect(store.keywordTopK('"original"', { limit: 5 }).length).toBe(0);+ expect(store.keywordTopK('"new"', { limit: 5 }).length).toBe(1);+ await store.close();+});++test("removing a file from disk produces a `deleted` event next run", async () => {+ writeMd(vault, "stays.md", "# A");+ const removed = writeMd(vault, "removed.md", "# B");+ const cfg = makeConfig({ vault, dbPath });+ await indexVault(cfg);+ rmSync(removed);++ const events: string[] = [];+ const stats = await indexVault(cfg, { onFile: (e) => events.push(`${e.kind}:${e.path}`) });+ expect(stats.deleted).toBe(1);+ expect(events).toContain("deleted:removed.md");++ const store = await Store.open(cfg, { mode: "read", loadVec: false });+ expect(store.listDocuments().size).toBe(1);+ await store.close();+});++test("indexStatus reports not initialised when index file is absent", async () => {+ const cfg = makeConfig({ vault, dbPath });+ const status = await indexStatus(cfg);+ expect(status.exists).toBe(false);+ expect(status.documents).toBe(0);+});++test("indexStatus reports accurate counts after an index run", async () => {+ writeMd(vault, "a.md", "# A\n\nbody");+ writeMd(vault, "b.md", "# B\n\nbody");+ const cfg = makeConfig({ vault, dbPath });+ await indexVault(cfg);++ const status = await indexStatus(cfg);+ expect(status.exists).toBe(true);+ expect(status.schemaVersion).toBe(1);+ expect(status.documents).toBe(2);+ expect(status.chunks).toBeGreaterThanOrEqual(2);+ expect(status.lastIndexedAt).toBeTruthy();+});++test("indexCheck reports vault readable, sqlite ok, fts ok", async () => {+ const cfg = makeConfig({ vault, dbPath });+ const report = await indexCheck(cfg);+ expect(report.vaultReadable).toBe(true);+ expect(report.indexDirWritable).toBe(true);+ expect(report.sqliteOk).toBe(true);+ expect(report.fts5Ok).toBe(true);+ expect(report.fatal.length).toBe(0);+});++test("reindexVault rebuilds atomically and preserves .bak", async () => {+ writeMd(vault, "a.md", "# A");+ const cfg = makeConfig({ vault, dbPath });+ await indexVault(cfg);++ expect(existsSync(dbPath)).toBe(true);+ expect(existsSync(dbPath + ".bak")).toBe(false);++ writeMd(vault, "b.md", "# B");+ const stats = await reindexVault(cfg);+ expect(stats.added).toBeGreaterThanOrEqual(2);+ expect(existsSync(dbPath)).toBe(true);+ expect(existsSync(dbPath + ".bak")).toBe(true);+ expect(existsSync(dbPath + ".new")).toBe(false);++ // Stats reflect the rebuilt index (2 docs from scratch).+ const status = await indexStatus(cfg);+ expect(status.documents).toBe(2);+});++test("Store.open auto-restores from .bak when main file is missing", async () => {+ writeMd(vault, "a.md", "# A");+ const cfg = makeConfig({ vault, dbPath });+ await indexVault(cfg);+ await reindexVault(cfg);++ // Simulate crash mid-reindex by removing the main file (we still have .bak).+ rmSync(dbPath);+ expect(existsSync(dbPath + ".bak")).toBe(true);++ const status = await indexStatus(cfg);+ expect(status.exists).toBe(true);+});++test("UTF-8 only: invalid bytes record an error, do not crash the run", async () => {+ // Latin-1 high-byte content is not valid UTF-8 sequencing here.+ const bad = join(vault, "broken.md");+ writeFileSync(bad, Buffer.from([0xc3, 0x28])); // 0xc3 0x28 is invalid UTF-8 sequence+ writeMd(vault, "good.md", "valid content");+ const cfg = makeConfig({ vault, dbPath });++ const stats = await indexVault(cfg);+ expect(stats.added).toBe(1); // only good.md+ expect(stats.errors.some((e) => e.path === "broken.md")).toBe(true);+});++test("indexVault tolerates empty vault", async () => {+ const cfg = makeConfig({ vault, dbPath });+ const stats = await indexVault(cfg);+ expect(stats.added).toBe(0);+ expect(stats.chunksTotal).toBe(0);+});+diff --git a/tests/core/search/links.test.ts b/tests/core/search/links.test.tsnew file mode 100644index 00000000..9252af07--- /dev/null+++ b/tests/core/search/links.test.ts@@ -0,0 +1,82 @@+import { test, expect } from "bun:test";+import { extractLinks } from "../../../src/core/search/links.ts";++test("extracts wikilinks with and without alt text", () => {+ const links = extractLinks("See [[foo-note]] and [[bar|the bar]].");+ const wikilinks = links.filter((l) => l.linkType === "wikilink");+ expect(wikilinks.length).toBe(2);+ expect(wikilinks[0]).toEqual({ linkType: "wikilink", targetPath: "foo-note", linkText: null });+ expect(wikilinks[1]).toEqual({ linkType: "wikilink", targetPath: "bar", linkText: "the bar" });+});++test("extracts relative markdown links and skips external URLs", () => {+ const text = "See [the spec](./design.md) and [Google](https://google.com).";+ const links = extractLinks(text);+ const md = links.filter((l) => l.linkType === "markdown_link");+ expect(md.length).toBe(1);+ expect(md[0]?.targetPath).toBe("./design.md");+ expect(md[0]?.linkText).toBe("the spec");+});++test("image embeds (``) are not captured as markdown links", () => {+ const text = " and a real [Spec](./design.md)";+ const links = extractLinks(text);+ const md = links.filter((l) => l.linkType === "markdown_link");+ expect(md.length).toBe(1);+ expect(md[0]?.targetPath).toBe("./design.md");+ expect(md.find((l) => l.targetPath === "assets/cover.png")).toBeUndefined();+});++test("strips anchor fragments from markdown link targets", () => {+ const links = extractLinks("[Anchor](notes/x.md#section)");+ const md = links.filter((l) => l.linkType === "markdown_link");+ expect(md[0]?.targetPath).toBe("notes/x.md");+});++test("extracts Obsidian-style tags including hierarchy", () => {+ const links = extractLinks("Tagged #foo and #bar/baz here. Not a tag: word#nope.");+ const tags = links.filter((l) => l.linkType === "tag").map((l) => l.linkText);+ expect(tags).toContain("foo");+ expect(tags).toContain("bar/baz");+ expect(tags).not.toContain("nope");+});++test("ignores wikilinks and tags inside fenced code blocks", () => {+ const text = `Outside [[real-link]].++\`\`\`+this is code with [[fake-link]] and #not-a-tag+\`\`\`++After fence.`;+ const links = extractLinks(text);+ const wikilinks = links.filter((l) => l.linkType === "wikilink").map((l) => l.targetPath);+ expect(wikilinks).toContain("real-link");+ expect(wikilinks).not.toContain("fake-link");+ const tags = links.filter((l) => l.linkType === "tag").map((l) => l.linkText);+ expect(tags).not.toContain("not-a-tag");+});++test("ignores wikilinks inside inline code spans", () => {+ const links = extractLinks("Inline `[[code-link]]` and real [[outside]].");+ const wikilinks = links.filter((l) => l.linkType === "wikilink").map((l) => l.targetPath);+ expect(wikilinks).toEqual(["outside"]);+});++test("deduplicates identical links from the same chunk", () => {+ const links = extractLinks("[[same]] [[same]] [[same|with alt]] #tag #tag");+ const wikilinks = links.filter((l) => l.linkType === "wikilink");+ // Two distinct: (same, null) and (same, "with alt")+ expect(wikilinks.length).toBe(2);+ const tags = links.filter((l) => l.linkType === "tag");+ expect(tags.length).toBe(1);+});++test("empty content returns empty array", () => {+ expect(extractLinks("")).toEqual([]);+});++test("skips mailto:", () => {+ const links = extractLinks("[email me](mailto:[email protected])");+ expect(links.length).toBe(0);+});diff --git a/tests/core/search/paths.test.ts b/tests/core/search/paths.test.tsnew file mode 100644index 00000000..491612b1--- /dev/null+++ b/tests/core/search/paths.test.ts@@ -0,0 +1,14 @@+import { test, expect } from "bun:test";+import { resolveIndexPath } from "../../../src/core/search/paths.ts";++test("default points under <vault>/.open-second-brain", () => {+ expect(resolveIndexPath("/v", null)).toBe("/v/.open-second-brain/brain.sqlite");+});++test("explicit override wins", () => {+ expect(resolveIndexPath("/v", "/tmp/custom.sqlite")).toBe("/tmp/custom.sqlite");+});++test("blank override falls back to default", () => {+ expect(resolveIndexPath("/v", "")).toBe("/v/.open-second-brain/brain.sqlite");+});diff --git a/tests/core/search/ranker.test.ts b/tests/core/search/ranker.test.tsnew file mode 100644index 00000000..7fd79a30--- /dev/null+++ b/tests/core/search/ranker.test.ts@@ -0,0 +1,238 @@+import { test, expect } from "bun:test";+import { rankResults } from "../../../src/core/search/ranker.ts";+import type { KeywordHit, SemanticHit, HydratedChunk } from "../../../src/core/search/store.ts";++function hyd(chunkId: number, docId: number, mtime: number): HydratedChunk {+ return Object.freeze({+ chunkId,+ documentId: docId,+ path: `doc${docId}.md`,+ title: `Doc ${docId}`,+ content: `chunk ${chunkId}`,+ startLine: 1,+ endLine: 1,+ mtime,+ });+}++const NOW = 1_750_000_000_000; // ms+const RECENT_MTIME = NOW / 1000 - 3 * 24 * 3600; // 3 days ago in unix seconds β 0.05 recency+const OLD_MTIME = NOW / 1000 - 365 * 24 * 3600; // >90d β 0 recency++test("union of keyword + semantic produces hybrid scoring", () => {+ const kw: KeywordHit[] = [+ { chunkId: 1, documentId: 10, bm25: -5 },+ { chunkId: 2, documentId: 11, bm25: -2 },+ ];+ const sem: SemanticHit[] = [+ { chunkId: 2, documentId: 11, distance: 0.5 },+ { chunkId: 3, documentId: 12, distance: 0.1 },+ ];+ const hydrated = new Map<number, HydratedChunk>([+ [1, hyd(1, 10, OLD_MTIME)],+ [2, hyd(2, 11, OLD_MTIME)],+ [3, hyd(3, 12, OLD_MTIME)],+ ]);+ const ranked = rankResults(+ {+ keyword: kw,+ semantic: sem,+ hydrated,+ inboundLinkSources: new Map(),+ tagsByDoc: new Map(),+ },+ { keywordWeight: 0.6, semanticWeight: 0.4, limit: 10, nowMs: NOW },+ );+ const map = new Map(ranked.map((r) => [r.chunkId, r]));+ expect(map.get(1)?.searchType).toBe("keyword");+ expect(map.get(2)?.searchType).toBe("hybrid");+ expect(map.get(3)?.searchType).toBe("semantic");+});++test("score is clamped to [0,1] even with boosts", () => {+ const ranked = rankResults(+ {+ keyword: [{ chunkId: 1, documentId: 10, bm25: -10 }],+ semantic: [{ chunkId: 1, documentId: 10, distance: 0 }],+ hydrated: new Map([[1, hyd(1, 10, RECENT_MTIME)]]),+ inboundLinkSources: new Map(),+ tagsByDoc: new Map(),+ },+ { keywordWeight: 0.6, semanticWeight: 0.4, limit: 10, nowMs: NOW },+ );+ expect(ranked[0]?.score).toBeGreaterThan(0);+ expect(ranked[0]?.score).toBeLessThanOrEqual(1);+});++test("keyword-only fallback ignores semantic when semanticEnabled=false", () => {+ const ranked = rankResults(+ {+ keyword: [{ chunkId: 1, documentId: 10, bm25: -5 }],+ semantic: [{ chunkId: 2, documentId: 11, distance: 0.0 }],+ hydrated: new Map([+ [1, hyd(1, 10, OLD_MTIME)],+ [2, hyd(2, 11, OLD_MTIME)],+ ]),+ inboundLinkSources: new Map(),+ tagsByDoc: new Map(),+ },+ {+ keywordWeight: 1.0,+ semanticWeight: 0,+ limit: 10,+ nowMs: NOW,+ semanticEnabled: false,+ },+ );+ expect(ranked.length).toBe(1);+ expect(ranked[0]?.chunkId).toBe(1);+ expect(ranked[0]?.searchType).toBe("keyword");+});++test("min-max normalisation: best BM25 hit gets 1.0, worst 0.0", () => {+ const ranked = rankResults(+ {+ keyword: [+ { chunkId: 1, documentId: 10, bm25: -10 },+ { chunkId: 2, documentId: 11, bm25: -1 },+ ],+ semantic: [],+ hydrated: new Map([+ [1, hyd(1, 10, OLD_MTIME)],+ [2, hyd(2, 11, OLD_MTIME)],+ ]),+ inboundLinkSources: new Map(),+ tagsByDoc: new Map(),+ },+ { keywordWeight: 1.0, semanticWeight: 0, limit: 10, nowMs: NOW },+ );+ const c1 = ranked.find((r) => r.chunkId === 1)!;+ const c2 = ranked.find((r) => r.chunkId === 2)!;+ expect(c1.keywordScore).toBe(1);+ expect(c2.keywordScore).toBe(0);+});++test("single-result keyword gets keywordScore=1 (max==min branch)", () => {+ const ranked = rankResults(+ {+ keyword: [{ chunkId: 1, documentId: 10, bm25: -5 }],+ semantic: [],+ hydrated: new Map([[1, hyd(1, 10, OLD_MTIME)]]),+ inboundLinkSources: new Map(),+ tagsByDoc: new Map(),+ },+ { keywordWeight: 1, semanticWeight: 0, limit: 10, nowMs: NOW },+ );+ expect(ranked[0]?.keywordScore).toBe(1);+});++test("link_boost adds +0.02 when another candidate doc points to this chunk's doc", () => {+ const inbound = new Map<number, ReadonlySet<number>>([+ // chunk 1 (doc 10) is pointed to by doc 11+ [1, new Set([11])],+ ]);+ const ranked = rankResults(+ {+ keyword: [+ { chunkId: 1, documentId: 10, bm25: -5 },+ { chunkId: 2, documentId: 11, bm25: -5 },+ ],+ semantic: [],+ hydrated: new Map([+ [1, hyd(1, 10, OLD_MTIME)],+ [2, hyd(2, 11, OLD_MTIME)],+ ]),+ inboundLinkSources: inbound,+ tagsByDoc: new Map(),+ },+ { keywordWeight: 1, semanticWeight: 0, limit: 10, nowMs: NOW },+ );+ const c1 = ranked.find((r) => r.chunkId === 1)!;+ expect(c1.linkBoost).toBeCloseTo(0.02, 6);+});++test("tag_boost adds +0.01 when candidates share a tag", () => {+ const tags = new Map<number, ReadonlySet<string>>([+ [1, new Set(["shared"])],+ [2, new Set(["shared", "extra"])],+ ]);+ const ranked = rankResults(+ {+ keyword: [+ { chunkId: 1, documentId: 10, bm25: -5 },+ { chunkId: 2, documentId: 11, bm25: -5 },+ ],+ semantic: [],+ hydrated: new Map([+ [1, hyd(1, 10, OLD_MTIME)],+ [2, hyd(2, 11, OLD_MTIME)],+ ]),+ inboundLinkSources: new Map(),+ tagsByDoc: tags,+ },+ { keywordWeight: 1, semanticWeight: 0, limit: 10, nowMs: NOW },+ );+ expect(ranked[0]?.linkBoost).toBeCloseTo(0.01, 6);+});++test("recency_boost is 0.05 for β€7d, 0.025 for β€30d, 0.01 for β€90d, 0 older", () => {+ const make = (mt: number) =>+ rankResults(+ {+ keyword: [{ chunkId: 1, documentId: 10, bm25: -5 }],+ semantic: [],+ hydrated: new Map([[1, hyd(1, 10, mt)]]),+ inboundLinkSources: new Map(),+ tagsByDoc: new Map(),+ },+ { keywordWeight: 1, semanticWeight: 0, limit: 10, nowMs: NOW },+ )[0]!.recencyBoost;++ const baseSec = NOW / 1000;+ expect(make(baseSec - 3 * 86400)).toBe(0.05);+ expect(make(baseSec - 20 * 86400)).toBe(0.025);+ expect(make(baseSec - 60 * 86400)).toBe(0.01);+ expect(make(baseSec - 200 * 86400)).toBe(0);+});++test("tie-break: equal final_score β higher keywordScore wins; then mtime; then chunkId", () => {+ // Two chunks with identical score after combination: same kw=1, no semantic, no boosts.+ const ranked = rankResults(+ {+ keyword: [+ { chunkId: 2, documentId: 11, bm25: -5 },+ { chunkId: 1, documentId: 10, bm25: -5 },+ ],+ semantic: [],+ hydrated: new Map([+ [1, hyd(1, 10, OLD_MTIME)],+ [2, hyd(2, 11, OLD_MTIME)],+ ]),+ inboundLinkSources: new Map(),+ tagsByDoc: new Map(),+ },+ { keywordWeight: 1, semanticWeight: 0, limit: 10, nowMs: NOW },+ );+ // Same score, same kw, same mtime, lower chunkId first β [1, 2]+ expect(ranked.map((r) => r.chunkId)).toEqual([1, 2]);+});++test("limit truncates after ranking", () => {+ const kw: KeywordHit[] = [];+ const hydrated = new Map<number, HydratedChunk>();+ for (let i = 0; i < 30; i++) {+ kw.push({ chunkId: i + 1, documentId: 100 + i, bm25: -(i + 1) });+ hydrated.set(i + 1, hyd(i + 1, 100 + i, OLD_MTIME));+ }+ const ranked = rankResults(+ {+ keyword: kw,+ semantic: [],+ hydrated,+ inboundLinkSources: new Map(),+ tagsByDoc: new Map(),+ },+ { keywordWeight: 1, semanticWeight: 0, limit: 5, nowMs: NOW },+ );+ expect(ranked.length).toBe(5);+});diff --git a/tests/core/search/schema.test.ts b/tests/core/search/schema.test.tsnew file mode 100644index 00000000..38976114--- /dev/null+++ b/tests/core/search/schema.test.ts@@ -0,0 +1,145 @@+import { test, expect, beforeEach, afterEach } from "bun:test";+import { Database } from "bun:sqlite";+import { mkdtempSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ applyMigrations,+ LATEST_SCHEMA_VERSION,+ readSchemaVersion,+ ensureVecTable,+ dropVecTable,+} from "../../../src/core/search/schema.ts";+import { SearchError } from "../../../src/core/search/types.ts";++let tmp: string;+let dbPath: string;++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "osb-schema-"));+ dbPath = join(tmp, "test.sqlite");+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++function tableNames(db: Database): Set<string> {+ const rows = db+ .query<{ name: string }, []>(+ "SELECT name FROM sqlite_master WHERE type IN ('table','virtual') OR type='table' OR type='trigger' OR type='view'",+ )+ .all();+ return new Set(rows.map((r) => r.name));+}++function objectNames(db: Database, type: string): Set<string> {+ const rows = db+ .query<{ name: string }, [string]>(+ "SELECT name FROM sqlite_master WHERE type = ? AND name NOT LIKE 'sqlite_%'",+ )+ .all(type);+ return new Set(rows.map((r) => r.name));+}++test("applyMigrations on a fresh db creates the v1 schema", () => {+ const db = new Database(dbPath);+ const v = applyMigrations(db);+ expect(v).toBe(LATEST_SCHEMA_VERSION);++ const tables = objectNames(db, "table");+ expect(tables.has("documents")).toBe(true);+ expect(tables.has("chunks")).toBe(true);+ expect(tables.has("embeddings")).toBe(true);+ expect(tables.has("chunk_vec_map")).toBe(true);+ expect(tables.has("links")).toBe(true);+ expect(tables.has("index_state")).toBe(true);++ const triggers = objectNames(db, "trigger");+ expect(triggers.has("chunks_ai")).toBe(true);+ expect(triggers.has("chunks_ad")).toBe(true);+ expect(triggers.has("chunks_au")).toBe(true);++ // FTS5 virtual table is registered as type 'table'.+ expect(tables.has("chunk_fts")).toBe(true);++ expect(readSchemaVersion(db)).toBe(1);+ db.close();+});++test("applyMigrations is idempotent", () => {+ const db = new Database(dbPath);+ applyMigrations(db);+ const first = tableNames(db).size;+ applyMigrations(db);+ applyMigrations(db);+ expect(tableNames(db).size).toBe(first);+ expect(readSchemaVersion(db)).toBe(1);+ db.close();+});++test("readSchemaVersion returns 0 when index_state is missing", () => {+ const db = new Database(dbPath);+ expect(() => readSchemaVersion(db)).toThrow(/no such table/);+ db.close();+});++test("applyMigrations throws SCHEMA_MISMATCH if db is newer than binary", () => {+ const db = new Database(dbPath);+ applyMigrations(db);+ // Simulate a future version.+ db.run(+ "UPDATE index_state SET value = ? WHERE key = 'schema_version'",+ [String(LATEST_SCHEMA_VERSION + 7)],+ );++ let err: SearchError | null = null;+ try {+ applyMigrations(db);+ } catch (e) {+ err = e as SearchError;+ }+ expect(err).not.toBeNull();+ expect(err?.code).toBe("SCHEMA_MISMATCH");+ expect(err?.message).toContain("reindex");+ db.close();+});++test("ensureVecTable + dropVecTable round-trip", () => {+ // sqlite-vec is optional; load it for this test. If not available we+ // skip β vec-specific behaviour is exercised in store.vec.test.ts.+ const db = new Database(dbPath);+ applyMigrations(db);++ try {+ const vec = require("sqlite-vec");+ db.loadExtension(vec.getLoadablePath());+ } catch {+ db.close();+ return;+ }++ ensureVecTable(db, 4);+ ensureVecTable(db, 4); // idempotent++ const tables = objectNames(db, "table");+ expect(tables.has("chunk_vec")).toBe(true);++ // Insert a vector.+ db.run("INSERT INTO chunk_vec(rowid, embedding) VALUES (1, ?)", [JSON.stringify([0.1, 0.2, 0.3, 0.4])]);+ const row = db.query<{ c: number }, []>("SELECT count(*) AS c FROM chunk_vec").get();+ expect(row?.c).toBe(1);++ dropVecTable(db);+ expect(objectNames(db, "table").has("chunk_vec")).toBe(false);+ db.close();+});++test("ensureVecTable rejects non-positive dimension", () => {+ const db = new Database(dbPath);+ applyMigrations(db);+ expect(() => ensureVecTable(db, 0)).toThrow(/positive integer/);+ expect(() => ensureVecTable(db, -1)).toThrow(/positive integer/);+ db.close();+});diff --git a/tests/core/search/search.test.ts b/tests/core/search/search.test.tsnew file mode 100644index 00000000..93dbfa75--- /dev/null+++ b/tests/core/search/search.test.ts@@ -0,0 +1,185 @@+import { test, expect, beforeEach, afterEach } from "bun:test";++import { indexVault } from "../../../src/core/search/indexer.ts";+import { search } from "../../../src/core/search/search.ts";+import { SearchError } from "../../../src/core/search/types.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";+import { startFakeHttp, type FakeHttp } from "../../helpers/fake-http.ts";++let vault: string;+let dbPath: string;+let cleanup: () => void;+let server: FakeHttp;++beforeEach(async () => {+ const v = createTempVault("search");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+ server = await startFakeHttp();+});++afterEach(async () => {+ cleanup();+ await server.close();+});++function semanticConfig() {+ return makeConfig({+ vault,+ dbPath,+ semantic: {+ enabled: true,+ provider: "openai-compat",+ baseUrl: server.url,+ model: "fake-model",+ apiKey: "test-key",+ dimension: 4,+ timeoutMs: 5_000,+ concurrency: 2,+ batchSize: 8,+ },+ });+}++async function seedKeyword() {+ writeMd(+ vault,+ "Notes/foo.md",+ "# Foo\n\nThe quick brown fox jumps over the lazy dog. Foxes are interesting.",+ );+ writeMd(vault, "Other/bar.md", "# Bar\n\nA different note about cats and turtles.");+ writeMd(vault, "Notes/ru.md", "# Ru\n\nΠΠΈΡΠΈΠ»Π»ΠΈΡΠ΅ΡΠΊΠΈΠΉ ΡΠ΅ΠΊΡΡ ΠΏΡΠΎ Π±ΡΠΎΠΆΠ΅Π½ΠΈΠ΅ ΡΠ΅ΡΡΠΎΠ²ΠΎΠ³ΠΎ Π·Π΅ΡΠ½Π°.");+ return makeConfig({ vault, dbPath });+}++test("search returns keyword-only results when semantic is disabled", async () => {+ const cfg = await seedKeyword();+ await indexVault(cfg);++ const out = await search(cfg, { query: "fox", limit: 5 });+ expect(out.results.length).toBe(1);+ expect(out.results[0]?.path).toBe("Notes/foo.md");+ expect(out.results[0]?.searchType).toBe("keyword");+ expect(out.warnings).toEqual([]);+});++test("search returns empty (no error) when no results match", async () => {+ const cfg = await seedKeyword();+ await indexVault(cfg);+ const out = await search(cfg, { query: "nonexistentterm", limit: 5 });+ expect(out.results).toEqual([]);+ expect(out.total).toBe(0);+});++test("search supports Cyrillic query against Cyrillic content", async () => {+ const cfg = await seedKeyword();+ await indexVault(cfg);+ const out = await search(cfg, { query: "Π±ΡΠΎΠΆΠ΅Π½ΠΈΠ΅", limit: 5 });+ expect(out.results.length).toBe(1);+ expect(out.results[0]?.path).toBe("Notes/ru.md");+});++test("path_prefix filter scopes results", async () => {+ const cfg = await seedKeyword();+ await indexVault(cfg);+ const all = await search(cfg, { query: "quick", limit: 5 });+ expect(all.results.length).toBe(1);+ const scoped = await search(cfg, { query: "quick", limit: 5, pathPrefix: "Other/" });+ expect(scoped.results.length).toBe(0);+});++test("path_prefix escaping is rejected with INVALID_INPUT", async () => {+ const cfg = await seedKeyword();+ await indexVault(cfg);+ let err: SearchError | null = null;+ try {+ await search(cfg, { query: "x", pathPrefix: "../etc/" });+ } catch (e) {+ err = e as SearchError;+ }+ expect(err?.code).toBe("INVALID_INPUT");+});++test("empty query throws INVALID_INPUT", async () => {+ const cfg = await seedKeyword();+ await indexVault(cfg);+ let err: SearchError | null = null;+ try {+ await search(cfg, { query: " " });+ } catch (e) {+ err = e as SearchError;+ }+ expect(err?.code).toBe("INVALID_INPUT");+});++test("missing index throws INDEX_MISSING", async () => {+ const cfg = await seedKeyword();+ // skip indexVault+ let err: SearchError | null = null;+ try {+ await search(cfg, { query: "fox" });+ } catch (e) {+ err = e as SearchError;+ }+ expect(err?.code).toBe("INDEX_MISSING");+});++test("hybrid search combines keyword and semantic when both contribute", async () => {+ writeMd(vault, "A/foo.md", "alpha beta gamma");+ writeMd(vault, "A/bar.md", "delta epsilon zeta");+ const cfg = semanticConfig();+ await indexVault(cfg, { embeddings: true });++ const out = await search(cfg, { query: "alpha", limit: 5, semantic: true });+ expect(out.results.length).toBeGreaterThan(0);+ // The keyword hit should be in the top result; searchType may be hybrid.+ const first = out.results[0]!;+ expect(["keyword", "hybrid", "semantic"]).toContain(first.searchType);+});++test("explicit --semantic without vec extension throws (configured-down case)", async () => {+ await seedKeyword();+ const cfg = makeConfig({+ vault,+ dbPath,+ semantic: {+ enabled: true,+ provider: "openai-compat",+ baseUrl: server.url,+ model: "fake-model",+ apiKey: "k",+ dimension: 4,+ timeoutMs: 5_000,+ concurrency: 1,+ batchSize: 8,+ },+ });+ await indexVault(cfg);+ // Now disable vec by closing the index and reopening through a config+ // that uses a fresh dbPath without vec. We can't unload the extension β+ // so emulate by stubbing config.semantic.enabled=true but no embeddings+ // recorded. The store loads vec OK but counts.embeddings===0, which is+ // the data-state warning case (warn + skip even when explicit).+ const out = await search(cfg, { query: "fox", limit: 5, semantic: true });+ expect(out.warnings.some((w) => w.includes("no compatible embeddings"))).toBe(true);+});++test("implicit semantic + no embeddings β keyword-only + warning, no throw", async () => {+ await seedKeyword();+ const cfg = semanticConfig();+ await indexVault(cfg); // no --embeddings+ const out = await search(cfg, { query: "fox", limit: 5 });+ expect(out.warnings.some((w) => w.includes("no compatible embeddings"))).toBe(true);+ expect(out.results.length).toBeGreaterThan(0);+});++test("limit truncates results", async () => {+ for (let i = 0; i < 20; i++) {+ writeMd(vault, `notes/n${i}.md`, `# Note ${i}\n\nrepeated word repeated word repeated word`);+ }+ const cfg = makeConfig({ vault, dbPath });+ await indexVault(cfg);+ const out = await search(cfg, { query: "repeated", limit: 5 });+ expect(out.results.length).toBe(5);+});diff --git a/tests/core/search/store.test.ts b/tests/core/search/store.test.tsnew file mode 100644index 00000000..cb730cbb--- /dev/null+++ b/tests/core/search/store.test.ts@@ -0,0 +1,287 @@+import { test, expect, beforeEach, afterEach } from "bun:test";+import { mkdtempSync, rmSync, existsSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { Store } from "../../../src/core/search/store.ts";+import { SearchError } from "../../../src/core/search/types.ts";+import type { ResolvedSearchConfig, ResolvedEmbeddingConfig } from "../../../src/core/search/types.ts";++let tmp: string;+let dbPath: string;++function makeConfig(overrides?: Partial<ResolvedSearchConfig>): ResolvedSearchConfig {+ const semantic: ResolvedEmbeddingConfig = Object.freeze({+ enabled: false,+ provider: "openai-compat",+ baseUrl: null,+ model: null,+ apiKey: null,+ dimension: null,+ timeoutMs: 10_000,+ concurrency: 4,+ batchSize: 32,+ });+ return Object.freeze({+ vault: tmp,+ dbPath,+ ignorePaths: Object.freeze([".git"]),+ chunkSize: 800,+ chunkOverlap: 100,+ keywordWeight: 0.6,+ semanticWeight: 0.4,+ semantic,+ ...(overrides ?? {}),+ });+}++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "osb-store-"));+ dbPath = join(tmp, "brain.sqlite");+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++test("open in write mode creates file and applies migrations", async () => {+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });+ expect(existsSync(dbPath)).toBe(true);+ expect(store.schemaVersion()).toBe(1);+ await store.close();+});++test("open in read mode throws INDEX_MISSING when file doesn't exist", async () => {+ let err: SearchError | null = null;+ try {+ await Store.open(makeConfig(), { mode: "read", loadVec: false });+ } catch (e) {+ err = e as SearchError;+ }+ expect(err?.code).toBe("INDEX_MISSING");+});++test("open in read mode succeeds after writer creates the file", async () => {+ const w = await Store.open(makeConfig(), { mode: "write", loadVec: false });+ await w.close();+ const r = await Store.open(makeConfig(), { mode: "read", loadVec: false });+ expect(r.schemaVersion()).toBe(1);+ await r.close();+});++test("upsertDocument is unique on path; second call updates same id", async () => {+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });+ const id1 = store.upsertDocument({+ path: "Daily/2026-05-16.md",+ title: "2026-05-16",+ contentHash: "h1",+ mtime: 1700000000,+ size: 12,+ });+ const id2 = store.upsertDocument({+ path: "Daily/2026-05-16.md",+ title: "renamed",+ contentHash: "h2",+ mtime: 1700000099,+ size: 22,+ });+ expect(id2).toBe(id1);++ const docs = store.listDocuments();+ expect(docs.size).toBe(1);+ expect(docs.get("Daily/2026-05-16.md")?.contentHash).toBe("h2");+ await store.close();+});++test("replaceChunks populates FTS via triggers", async () => {+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });+ const docId = store.upsertDocument({+ path: "notes/hello.md",+ title: "Hello",+ contentHash: "h",+ mtime: 1700000000,+ size: 20,+ });+ const ids = store.replaceChunks(docId, [+ {+ chunkIndex: 0,+ content: "hello world from open second brain",+ contentHash: "c0",+ startLine: 1,+ endLine: 1,+ tokenCount: 6,+ },+ {+ chunkIndex: 1,+ content: "second chunk talks about cats and dogs",+ contentHash: "c1",+ startLine: 2,+ endLine: 2,+ tokenCount: 8,+ },+ ]);+ expect(ids.length).toBe(2);++ const hits = store.keywordTopK("brain", { limit: 10 });+ expect(hits.length).toBe(1);+ expect(hits[0]?.chunkId).toBe(ids[0]!);++ const noHits = store.keywordTopK("nonexistentterm", { limit: 10 });+ expect(noHits.length).toBe(0);+ await store.close();+});++test("replaceChunks twice replaces all old chunks", async () => {+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });+ const docId = store.upsertDocument({+ path: "x.md",+ title: null,+ contentHash: "h",+ mtime: 0,+ size: 1,+ });+ store.replaceChunks(docId, [+ { chunkIndex: 0, content: "alpha bravo", contentHash: "c0", startLine: 1, endLine: 1, tokenCount: 2 },+ ]);+ store.replaceChunks(docId, [+ { chunkIndex: 0, content: "charlie delta", contentHash: "c1", startLine: 1, endLine: 1, tokenCount: 2 },+ ]);+ expect(store.keywordTopK("alpha", { limit: 5 }).length).toBe(0);+ expect(store.keywordTopK("charlie", { limit: 5 }).length).toBe(1);+ expect(store.getChunksByDocument(docId).length).toBe(1);+ await store.close();+});++test("deleteDocument cascades to chunks and FTS", async () => {+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });+ const docId = store.upsertDocument({+ path: "doomed.md",+ title: null,+ contentHash: "h",+ mtime: 0,+ size: 1,+ });+ store.replaceChunks(docId, [+ { chunkIndex: 0, content: "doomed content", contentHash: "c", startLine: 1, endLine: 1, tokenCount: 2 },+ ]);+ expect(store.keywordTopK("doomed", { limit: 5 }).length).toBe(1);++ store.deleteDocument("doomed.md");+ expect(store.listDocuments().size).toBe(0);+ expect(store.keywordTopK("doomed", { limit: 5 }).length).toBe(0);+ await store.close();+});++test("path-prefix filter constrains keywordTopK", async () => {+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });+ const d1 = store.upsertDocument({ path: "A/foo.md", title: null, contentHash: "h", mtime: 0, size: 1 });+ const d2 = store.upsertDocument({ path: "B/bar.md", title: null, contentHash: "h", mtime: 0, size: 1 });+ store.replaceChunks(d1, [+ { chunkIndex: 0, content: "alpha alpha alpha", contentHash: "c", startLine: 1, endLine: 1, tokenCount: 3 },+ ]);+ store.replaceChunks(d2, [+ { chunkIndex: 0, content: "alpha bravo", contentHash: "c", startLine: 1, endLine: 1, tokenCount: 2 },+ ]);+ const hitsA = store.keywordTopK("alpha", { limit: 10, pathPrefix: "A/" });+ expect(hitsA.length).toBe(1);+ expect(hitsA[0]?.documentId).toBe(d1);+ await store.close();+});++test("path-prefix filter treats SQL wildcard characters literally", async () => {+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });+ const literal = store.upsertDocument({ path: "A_/foo.md", title: null, contentHash: "h1", mtime: 0, size: 1 });+ const wildcardMatch = store.upsertDocument({ path: "AB/bar.md", title: null, contentHash: "h2", mtime: 0, size: 1 });+ store.replaceChunks(literal, [+ { chunkIndex: 0, content: "alpha literal", contentHash: "c1", startLine: 1, endLine: 1, tokenCount: 2 },+ ]);+ store.replaceChunks(wildcardMatch, [+ { chunkIndex: 0, content: "alpha wildcard", contentHash: "c2", startLine: 1, endLine: 1, tokenCount: 2 },+ ]);+ const hits = store.keywordTopK("alpha", { limit: 10, pathPrefix: "A_/" });+ expect(hits.map((h) => h.documentId)).toEqual([literal]);+ await store.close();+});++test("counts() reflects documents/chunks", async () => {+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });+ const d = store.upsertDocument({ path: "a.md", title: null, contentHash: "h", mtime: 0, size: 1 });+ store.replaceChunks(d, [+ { chunkIndex: 0, content: "one", contentHash: "c0", startLine: 1, endLine: 1, tokenCount: 1 },+ { chunkIndex: 1, content: "two", contentHash: "c1", startLine: 2, endLine: 2, tokenCount: 1 },+ ]);+ const c = store.counts();+ expect(c.documents).toBe(1);+ expect(c.chunks).toBe(2);+ expect(c.embeddings).toBe(0);+ await store.close();+});++test("getState / setState / deleteState round-trip", async () => {+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });+ expect(store.getState("unset")).toBeNull();+ store.setState("foo", "bar");+ expect(store.getState("foo")).toBe("bar");+ store.setState("foo", "bar2");+ expect(store.getState("foo")).toBe("bar2");+ store.deleteState("foo");+ expect(store.getState("foo")).toBeNull();+ await store.close();+});++test("ensureEmbeddingModel sets state on first call and clears on change", async () => {+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });++ const first = store.ensureEmbeddingModel("modelA", 384);+ expect(first.wasChanged).toBe(false);+ expect(store.getState("embedding_model")).toBe("modelA");+ expect(store.getState("embedding_dimension")).toBe("384");++ // Insert a fake embedding row so we can witness clearing.+ // Bypass typed API because there's no chunk; SQLite FK would block, so use unrelated insert.+ // Instead, mimic the change against an existing chunk-less embeddings row:+ // (skip β we verify only state cleanup; full embeddings cascade lives in store.vec.test.ts)++ const second = store.ensureEmbeddingModel("modelB", 768);+ expect(second.wasChanged).toBe(true);+ expect(store.getState("embedding_model")).toBe("modelB");+ expect(store.getState("embedding_dimension")).toBe("768");+ await store.close();+});++test("vecUpsert without sqlite-vec throws VEC_EXTENSION_UNAVAILABLE", async () => {+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });+ expect(() =>+ store.vecUpsert(1, [0.1, 0.2, 0.3], "m", 3, "h"),+ ).toThrow(/sqlite-vec/);+ await store.close();+});++test("schema-mismatch is surfaced on open", async () => {+ // Prepare a db at schema 2 (above LATEST=1)+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });+ store.setState("schema_version", "99");+ await store.close();++ let err: SearchError | null = null;+ try {+ const s = await Store.open(makeConfig(), { mode: "read", loadVec: false });+ await s.close();+ } catch (e) {+ err = e as SearchError;+ }+ expect(err?.code).toBe("SCHEMA_MISMATCH");+});++test("two writers contend β second fails with INDEX_LOCKED", async () => {+ const first = await Store.open(makeConfig(), { mode: "write", loadVec: false });+ let err: SearchError | null = null;+ try {+ const second = await Store.open(makeConfig(), { mode: "write", loadVec: false });+ await second.close();+ } catch (e) {+ err = e as SearchError;+ }+ expect(err?.code).toBe("INDEX_LOCKED");+ await first.close();+}, 30_000);diff --git a/tests/core/search/store.vec.test.ts b/tests/core/search/store.vec.test.tsnew file mode 100644index 00000000..39acf85e--- /dev/null+++ b/tests/core/search/store.vec.test.ts@@ -0,0 +1,162 @@+import { test, expect, beforeEach, afterEach } from "bun:test";+import { mkdtempSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { Store } from "../../../src/core/search/store.ts";+import type { ResolvedSearchConfig, ResolvedEmbeddingConfig } from "../../../src/core/search/types.ts";++let tmp: string;+let dbPath: string;++function semanticConfig(model: string, dim: number, overrides?: Partial<ResolvedSearchConfig>): ResolvedSearchConfig {+ const semantic: ResolvedEmbeddingConfig = Object.freeze({+ enabled: true,+ provider: "openai-compat",+ baseUrl: "https://x/v1",+ model,+ apiKey: "k",+ dimension: dim,+ timeoutMs: 10_000,+ concurrency: 4,+ batchSize: 32,+ });+ return Object.freeze({+ vault: tmp,+ dbPath,+ ignorePaths: Object.freeze([".git"]),+ chunkSize: 800,+ chunkOverlap: 100,+ keywordWeight: 0.6,+ semanticWeight: 0.4,+ semantic,+ ...(overrides ?? {}),+ });+}++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "osb-vec-"));+ dbPath = join(tmp, "brain.sqlite");+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++function vecAvailable(): boolean {+ try {+ require("sqlite-vec");+ return true;+ } catch {+ return false;+ }+}++function unit(values: number[]): number[] {+ const norm = Math.hypot(...values);+ if (norm === 0) return values;+ return values.map((v) => v / norm);+}++test("vec round-trip when sqlite-vec is loaded", async () => {+ if (!vecAvailable()) return;+ const store = await Store.open(semanticConfig("m1", 4), { mode: "write" });+ expect(store.vecLoaded()).toBe(true);++ const docId = store.upsertDocument({+ path: "vec/a.md",+ title: null,+ contentHash: "h",+ mtime: 0,+ size: 1,+ });+ const [c0, c1] = store.replaceChunks(docId, [+ { chunkIndex: 0, content: "alpha", contentHash: "h0", startLine: 1, endLine: 1, tokenCount: 1 },+ { chunkIndex: 1, content: "bravo", contentHash: "h1", startLine: 2, endLine: 2, tokenCount: 1 },+ ]);++ store.vecUpsert(c0!, unit([1, 0, 0, 0]), "m1", 4, "eh0");+ store.vecUpsert(c1!, unit([0, 1, 0, 0]), "m1", 4, "eh1");++ // Query close to first vector β first chunk is nearest.+ const hits = store.semanticTopK(unit([0.9, 0.1, 0, 0]), { limit: 5 });+ expect(hits.length).toBe(2);+ expect(hits[0]?.chunkId).toBe(c0!);+ expect(hits[1]?.chunkId).toBe(c1!);+ expect(hits[0]!.distance).toBeLessThan(hits[1]!.distance);+ await store.close();+});++test("deleting a document leaves zero rows in chunk_vec/chunk_vec_map", async () => {+ if (!vecAvailable()) return;+ const store = await Store.open(semanticConfig("m1", 4), { mode: "write" });+ const docId = store.upsertDocument({+ path: "vec/b.md",+ title: null,+ contentHash: "h",+ mtime: 0,+ size: 1,+ });+ const [c0, c1] = store.replaceChunks(docId, [+ { chunkIndex: 0, content: "x", contentHash: "h0", startLine: 1, endLine: 1, tokenCount: 1 },+ { chunkIndex: 1, content: "y", contentHash: "h1", startLine: 2, endLine: 2, tokenCount: 1 },+ ]);+ store.vecUpsert(c0!, unit([1, 0, 0, 0]), "m1", 4, "eh0");+ store.vecUpsert(c1!, unit([0, 1, 0, 0]), "m1", 4, "eh1");++ const beforeVec = store.rawQuery<{ c: number }>("SELECT count(*) AS c FROM chunk_vec");+ const beforeMap = store.rawQuery<{ c: number }>("SELECT count(*) AS c FROM chunk_vec_map");+ expect(beforeVec[0]?.c).toBe(2);+ expect(beforeMap[0]?.c).toBe(2);++ store.deleteDocument("vec/b.md");++ const afterVec = store.rawQuery<{ c: number }>("SELECT count(*) AS c FROM chunk_vec");+ const afterMap = store.rawQuery<{ c: number }>("SELECT count(*) AS c FROM chunk_vec_map");+ expect(afterVec[0]?.c).toBe(0);+ expect(afterMap[0]?.c).toBe(0);+ await store.close();+});++test("ensureEmbeddingModel drops chunk_vec when dimension changes", async () => {+ if (!vecAvailable()) return;+ const store = await Store.open(semanticConfig("m1", 4), { mode: "write" });++ // Touch the vec table to confirm it exists.+ let tables = store.rawQuery<{ name: string }>(+ "SELECT name FROM sqlite_master WHERE type='table' AND name='chunk_vec'",+ );+ expect(tables.length).toBe(1);++ // Change to a different model+dim.+ const change = store.ensureEmbeddingModel("m2", 8);+ expect(change.wasChanged).toBe(true);+ expect(change.previousModel).toBe("m1");+ expect(change.currentModel).toBe("m2");++ // Vec table recreated at new dim, embeddings empty.+ tables = store.rawQuery<{ name: string }>(+ "SELECT name FROM sqlite_master WHERE type='table' AND name='chunk_vec'",+ );+ expect(tables.length).toBe(1);+ const emb = store.rawQuery<{ c: number }>("SELECT count(*) AS c FROM embeddings");+ expect(emb[0]?.c).toBe(0);+ await store.close();+});++test("dimension mismatch is rejected on vecUpsert", async () => {+ if (!vecAvailable()) return;+ const store = await Store.open(semanticConfig("m1", 4), { mode: "write" });+ const docId = store.upsertDocument({+ path: "z.md",+ title: null,+ contentHash: "h",+ mtime: 0,+ size: 1,+ });+ const [c0] = store.replaceChunks(docId, [+ { chunkIndex: 0, content: "x", contentHash: "h", startLine: 1, endLine: 1, tokenCount: 1 },+ ]);+ expect(() => store.vecUpsert(c0!, [0.1, 0.2, 0.3], "m1", 4, "eh")).toThrow(/dimension/);+ await store.close();+});diff --git a/tests/core/search/types.test.ts b/tests/core/search/types.test.tsnew file mode 100644index 00000000..57c62d1a--- /dev/null+++ b/tests/core/search/types.test.ts@@ -0,0 +1,15 @@+import { test, expect } from "bun:test";+import { SearchError } from "../../../src/core/search/types.ts";++test("SearchError carries a typed code", () => {+ const err = new SearchError("INDEX_MISSING", "no index at /tmp/x");+ expect(err).toBeInstanceOf(Error);+ expect(err.name).toBe("SearchError");+ expect(err.code).toBe("INDEX_MISSING");+ expect(err.message).toBe("no index at /tmp/x");+});++test("SearchError preserves stack trace", () => {+ const err = new SearchError("INVALID_INPUT", "bad");+ expect(err.stack).toContain("SearchError");+});diff --git a/tests/core/search/walker.test.ts b/tests/core/search/walker.test.tsnew file mode 100644index 00000000..2814565f--- /dev/null+++ b/tests/core/search/walker.test.ts@@ -0,0 +1,91 @@+import { test, expect, beforeEach, afterEach } from "bun:test";+import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { walkVault } from "../../../src/core/search/walker.ts";+import { createTempVault, writeMd, writeSymlink, makeConfig } from "../../helpers/search-fixtures.ts";++let vault: string;+let cleanup: () => void;++beforeEach(() => {+ const v = createTempVault("walker");+ vault = v.vault;+ cleanup = v.cleanup;+});++afterEach(() => {+ cleanup();+});++function collect(cfg: ReturnType<typeof makeConfig>): string[] {+ const out: string[] = [];+ for (const f of walkVault(cfg)) out.push(f.relPath);+ return out.sort();+}++test("returns *.md files relative to vault, POSIX paths", () => {+ writeMd(vault, "a.md", "# a");+ writeMd(vault, "b/c.md", "# c");+ writeMd(vault, "b/d/e.md", "# e");+ const cfg = makeConfig({ vault, dbPath: join(vault, "x.sqlite") });+ expect(collect(cfg)).toEqual(["a.md", "b/c.md", "b/d/e.md"]);+});++test("skips non-md files", () => {+ writeMd(vault, "keep.md", "x");+ writeMd(vault, "drop.txt", "x");+ writeMd(vault, "Notes/README", "x");+ const cfg = makeConfig({ vault, dbPath: join(vault, "x.sqlite") });+ expect(collect(cfg)).toEqual(["keep.md"]);+});++test("skips ignored bare directory names anywhere in tree", () => {+ writeMd(vault, "keep.md", "x");+ writeMd(vault, ".git/dropped.md", "x");+ writeMd(vault, "node_modules/lib/index.md", "x");+ writeMd(vault, "deep/.git/also-dropped.md", "x");+ const cfg = makeConfig({ vault, dbPath: join(vault, "x.sqlite") });+ expect(collect(cfg)).toEqual(["keep.md"]);+});++test("skips ignored relative-path directories", () => {+ writeMd(vault, "keep.md", "x");+ writeMd(vault, ".obsidian/cache/cached.md", "x");+ writeMd(vault, ".obsidian/other.md", "x"); // kept (cache is the specific subdir)+ const cfg = makeConfig({ vault, dbPath: join(vault, "x.sqlite") });+ expect(collect(cfg)).toEqual([".obsidian/other.md", "keep.md"]);+});++test("ignores symlinks pointing outside vault", () => {+ const outside = mkdtempSync(join(tmpdir(), "osb-walker-out-"));+ writeFileSync(join(outside, "leak.md"), "secret");+ mkdirSync(join(vault, "inside"), { recursive: true });++ // Symlink file β outside file+ writeSymlink(vault, "evil.md", join(outside, "leak.md"));+ // Symlink dir β outside dir+ writeSymlink(vault, "linked-dir", outside);++ writeMd(vault, "real.md", "x");+ const cfg = makeConfig({ vault, dbPath: join(vault, "x.sqlite") });+ const found = collect(cfg);+ expect(found).toContain("real.md");+ expect(found).not.toContain("evil.md");+ expect(found.find((p) => p.startsWith("linked-dir/"))).toBeUndefined();+ rmSync(outside, { recursive: true, force: true });+});++test("hidden non-ignored dirs are still walked", () => {+ writeMd(vault, ".hiddendir/note.md", "x");+ writeMd(vault, "normal.md", "x");+ const cfg = makeConfig({ vault, dbPath: join(vault, "x.sqlite"), ignorePaths: [".git"] });+ expect(collect(cfg)).toEqual([".hiddendir/note.md", "normal.md"]);+});++test("empty directories produce zero files", () => {+ mkdirSync(join(vault, "empty"), { recursive: true });+ const cfg = makeConfig({ vault, dbPath: join(vault, "x.sqlite") });+ expect(collect(cfg)).toEqual([]);+});diff --git a/tests/helpers/fake-http.ts b/tests/helpers/fake-http.tsnew file mode 100644index 00000000..4c0069dc--- /dev/null+++ b/tests/helpers/fake-http.ts@@ -0,0 +1,94 @@+/**+ * Tiny Bun-native HTTP server stub for embedding-provider tests.+ *+ * Each test can install a handler that decides what to return for any+ * incoming request. The default handler returns OpenAI-shaped vectors+ * deterministically derived from the input texts (no randomness β so+ * `toBeCloseTo` checks stay stable).+ */+++export interface FakeRequest {+ readonly method: string;+ readonly path: string;+ readonly headers: Record<string, string>;+ readonly body: unknown;+}++export interface FakeResponseSpec {+ status?: number;+ headers?: Record<string, string>;+ body?: unknown;+ delayMs?: number;+}++type Handler = (req: FakeRequest, callIndex: number) => FakeResponseSpec | Promise<FakeResponseSpec>;++export interface FakeHttp {+ url: string;+ close: () => Promise<void>;+ setHandler: (h: Handler) => void;+ /** Number of requests received since creation. */+ callCount: () => number;+}++function defaultHandler(req: FakeRequest): FakeResponseSpec {+ if (req.path.endsWith("/embeddings") && req.method === "POST") {+ const body = (req.body ?? {}) as { input?: string[]; model?: string };+ const inputs = Array.isArray(body.input) ? body.input : [];+ const data = inputs.map((text, index) => {+ // Deterministic 4-dim vector from token count + position.+ const tokens = text.split(/\s+/).filter(Boolean).length;+ const v = [tokens, index, text.length, 1];+ return { object: "embedding", embedding: v, index };+ });+ return { status: 200, body: { data, model: body.model ?? "fake-model" } };+ }+ return { status: 404, body: { error: "not_found" } };+}++export async function startFakeHttp(): Promise<FakeHttp> {+ let handler: Handler = defaultHandler;+ let count = 0;+ const server = Bun.serve({+ port: 0,+ fetch: async (req) => {+ const url = new URL(req.url);+ let body: unknown = null;+ const ctype = req.headers.get("content-type") ?? "";+ if (ctype.startsWith("application/json")) {+ try {+ body = await req.json();+ } catch {+ body = null;+ }+ }+ const headers: Record<string, string> = {};+ req.headers.forEach((v, k) => {+ headers[k] = v;+ });+ const idx = count++;+ const resp = await handler({ method: req.method, path: url.pathname, headers, body }, idx);+ if (resp.delayMs && resp.delayMs > 0) {+ await new Promise<void>((r) => setTimeout(r, resp.delayMs));+ }+ return new Response(+ resp.body === undefined ? "" : JSON.stringify(resp.body),+ {+ status: resp.status ?? 200,+ headers: { "content-type": "application/json", ...(resp.headers ?? {}) },+ },+ );+ },+ });++ const url = `http://127.0.0.1:${server.port}/v1`;+ return {+ url,+ close: () => server.stop(true) as unknown as Promise<void>,+ setHandler: (h: Handler) => {+ handler = h;+ },+ callCount: () => count,+ };+}diff --git a/tests/helpers/mock-embedding.ts b/tests/helpers/mock-embedding.tsnew file mode 100644index 00000000..4cb1eb95--- /dev/null+++ b/tests/helpers/mock-embedding.ts@@ -0,0 +1,54 @@+/**+ * Deterministic in-process embedding provider for indexer tests.+ *+ * Vectors are derived from sha256 of the input text: every 4 bytes of+ * the digest become one signed float in [-1, 1], padded/truncated to+ * the requested dimension, then unit-normalised. Two equal inputs+ * always produce the same vector β so assertions on chunk identity+ * stay stable across runs.+ */++import { createHash } from "node:crypto";++import type { EmbeddingProvider } from "../../src/core/search/embeddings/provider.ts";++export class MockEmbeddingProvider implements EmbeddingProvider {+ readonly name = "mock";+ readonly model: string;+ readonly dimension: number;+ callCount = 0;++ constructor(opts?: { model?: string; dimension?: number }) {+ this.model = opts?.model ?? "mock-embed";+ this.dimension = opts?.dimension ?? 8;+ }++ embed(texts: ReadonlyArray<string>): Promise<number[][]> {+ this.callCount++;+ return Promise.resolve(texts.map((t) => this.vectorFor(t)));+ }++ async ping(): Promise<{ ok: true; dimension: number }> {+ return { ok: true, dimension: this.dimension };+ }++ private vectorFor(text: string): number[] {+ const v = new Array<number>(this.dimension).fill(0);+ let acc = createHash("sha256").update(text).digest();+ let bytes: number[] = Array.from(acc);+ while (bytes.length < this.dimension * 2) {+ acc = createHash("sha256").update(acc).digest();+ bytes = bytes.concat(Array.from(acc));+ }+ for (let i = 0; i < this.dimension; i++) {+ const b1 = bytes[i * 2] ?? 0;+ const b2 = bytes[i * 2 + 1] ?? 0;+ v[i] = (b1 - 128) / 128 + ((b2 - 128) / 128) * 0.5;+ }+ let s = 0;+ for (const x of v) s += x * x;+ const norm = Math.sqrt(s);+ if (norm === 0) return v;+ return v.map((x) => x / norm);+ }+}diff --git a/tests/helpers/search-fixtures.ts b/tests/helpers/search-fixtures.tsnew file mode 100644index 00000000..c364846e--- /dev/null+++ b/tests/helpers/search-fixtures.ts@@ -0,0 +1,76 @@+/**+ * Helpers for building short-lived vault directories for search tests.+ */++import { mkdtempSync, mkdirSync, writeFileSync, symlinkSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { dirname, join } from "node:path";++import type { ResolvedSearchConfig, ResolvedEmbeddingConfig } from "../../src/core/search/types.ts";++export function createTempVault(prefix: string): {+ vault: string;+ dbPath: string;+ cleanup: () => void;+} {+ const vault = mkdtempSync(join(tmpdir(), `osb-${prefix}-`));+ const dbPath = join(vault, ".open-second-brain", "brain.sqlite");+ return {+ vault,+ dbPath,+ cleanup: () => rmSync(vault, { recursive: true, force: true }),+ };+}++export function writeMd(vault: string, relPath: string, content: string): string {+ const abs = join(vault, relPath);+ mkdirSync(dirname(abs), { recursive: true });+ writeFileSync(abs, content);+ return abs;+}++export function writeSymlink(vault: string, relLink: string, absTarget: string): string {+ const abs = join(vault, relLink);+ mkdirSync(dirname(abs), { recursive: true });+ symlinkSync(absTarget, abs);+ return abs;+}++export function makeConfig(opts: {+ vault: string;+ dbPath: string;+ ignorePaths?: ReadonlyArray<string>;+ semantic?: Partial<ResolvedEmbeddingConfig>;+}): ResolvedSearchConfig {+ const baseSemantic: ResolvedEmbeddingConfig = Object.freeze({+ enabled: false,+ provider: "openai-compat",+ baseUrl: null,+ model: null,+ apiKey: null,+ dimension: null,+ timeoutMs: 10_000,+ concurrency: 4,+ batchSize: 32,+ });+ const semantic: ResolvedEmbeddingConfig = Object.freeze({ ...baseSemantic, ...(opts.semantic ?? {}) });+ return Object.freeze({+ vault: opts.vault,+ dbPath: opts.dbPath,+ ignorePaths: Object.freeze([+ ...(opts.ignorePaths ?? [+ ".git",+ "node_modules",+ ".open-second-brain",+ ".obsidian/cache",+ ".trash",+ ".stversions",+ ]),+ ]),+ chunkSize: 800,+ chunkOverlap: 100,+ keywordWeight: 0.6,+ semanticWeight: 0.4,+ semantic,+ });+}diff --git a/tests/mcp/mcp.test.ts b/tests/mcp/mcp.test.tsindex 41260491..f7a38431 100644--- a/tests/mcp/mcp.test.ts+++ b/tests/mcp/mcp.test.ts@@ -165,6 +165,8 @@ describe("tool listing", () => {"payment_request_approval","payment_request_status","payment_request_consume",+ // Search (added in v0.10.0).+ "brain_search",]),);// Explicit grep: legacy writable tools are no longer advertised.@@ -285,8 +287,8 @@ describe("stdio loop", () => {const list = JSON.parse(lines[1]!);expect(init.id).toBe(1);expect(list.id).toBe(2);- // v0.9.0: 3 core (status/query/health) + 6 Brain + 8 Pay Memory = 17.- expect(list.result.tools.length).toBe(18);+ // v0.10.0: 3 core (status/query/health) + 7 Brain + 8 Pay Memory + 1 Search = 19.+ expect(list.result.tools.length).toBe(19);});test("returns parse error for invalid JSON", async () => {diff --git a/tests/mcp/search.test.ts b/tests/mcp/search.test.tsnew file mode 100644index 00000000..1c4a5eb8--- /dev/null+++ b/tests/mcp/search.test.ts@@ -0,0 +1,187 @@+/**+ * MCP integration tests for `brain_search` and the `search.*`+ * enrichment on `second_brain_status`.+ *+ * Like the Brain MCP suite, these go through `MCPServer.handleRequest`+ * end-to-end β so registration, schema validation, error mapping, and+ * the actual search query are all exercised together.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { JSONRPC_VERSION, MCPServer, PROTOCOL_VERSION } from "../../src/mcp/index.ts";+import { indexVault } from "../../src/core/search/indexer.ts";+import { resolveSearchConfig } from "../../src/core/search/index.ts";+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";++let tmp: string;+let vault: string;+let configPath: string;+const savedEnv: Record<string, string | undefined> = {};++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-mcp-search-"));+ vault = join(tmp, "vault");+ mkdirSync(vault, { recursive: true });+ configPath = join(tmp, "config.yaml");+ for (const k of ["VAULT_AGENT_NAME", "VAULT_TIMEZONE", "VAULT_DIR", "OPEN_SECOND_BRAIN_CONFIG"]) {+ savedEnv[k] = process.env[k];+ delete process.env[k];+ }+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = configPath;+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+ for (const [k, v] of Object.entries(savedEnv)) {+ if (v === undefined) delete process.env[k];+ else process.env[k] = v;+ }+});++async function initialize(server: MCPServer): Promise<void> {+ await server.handleRequest({+ jsonrpc: JSONRPC_VERSION,+ id: 1,+ method: "initialize",+ params: {+ protocolVersion: PROTOCOL_VERSION,+ capabilities: {},+ clientInfo: { name: "search-test", version: "0" },+ },+ });+ await server.handleRequest({+ jsonrpc: JSONRPC_VERSION,+ method: "notifications/initialized",+ });+}++function call(server: MCPServer, name: string, args: Record<string, unknown> = {}): Promise<unknown> {+ return server.handleRequest({+ jsonrpc: JSONRPC_VERSION,+ id: 99,+ method: "tools/call",+ params: { name, arguments: args },+ }) as Promise<unknown>;+}++function makeServer(): MCPServer {+ return new MCPServer({ vault, configPath });+}++function writeMd(rel: string, content: string): void {+ const abs = join(vault, rel);+ mkdirSync(join(abs, ".."), { recursive: true });+ writeFileSync(abs, content);+}++function extractToolResult(resp: any): Record<string, unknown> {+ // MCP tool/call wraps the handler return value in { content: [{ type:"text", text: "<json>" }] }.+ const content = resp?.result?.content?.[0];+ if (content?.type === "text" && typeof content.text === "string") {+ return JSON.parse(content.text);+ }+ // Some servers also expose `structuredContent` directly; accept both.+ if (resp?.result?.structuredContent) return resp.result.structuredContent;+ return resp?.result ?? {};+}++test("brain_search is advertised by tools/list", async () => {+ const server = makeServer();+ await initialize(server);+ const r = (await server.handleRequest({+ jsonrpc: JSONRPC_VERSION,+ id: 10,+ method: "tools/list",+ })) as any;+ const tools = r.result.tools as Array<{ name: string }>;+ expect(tools.find((t) => t.name === "brain_search")).toBeDefined();+});++test("brain_search happy path returns paths and respects 600-char content cap", async () => {+ writeMd("notes/foo.md", "# Foo\n\n" + "fox ".repeat(400));+ const cfg = resolveSearchConfig({ vault, configPath });+ await indexVault(cfg);++ const server = makeServer();+ await initialize(server);+ const resp = await call(server, "brain_search", { query: "fox", limit: 5 });+ const body = extractToolResult(resp);+ const results = body["results"] as Array<{ content: string; path: string; searchType: string; score: number }>;+ expect(Array.isArray(results)).toBe(true);+ expect(results.length).toBeGreaterThan(0);+ expect(results[0]?.path).toBe("notes/foo.md");+ expect(results[0]?.content.length).toBeLessThanOrEqual(600);+ // The MCP shape does NOT expose diagnostic score components.+ expect((results[0] as Record<string, unknown>)["keywordScore"]).toBeUndefined();+});++test("brain_search rejects missing query with INVALID_PARAMS", async () => {+ const server = makeServer();+ await initialize(server);+ const resp = (await call(server, "brain_search", {})) as any;+ expect(resp?.error).toBeDefined();+ expect(resp.error.code).toBe(-32602); // INVALID_PARAMS+});++test("brain_search rejects path_prefix that escapes vault", async () => {+ writeMd("a.md", "# A");+ const cfg = resolveSearchConfig({ vault, configPath });+ await indexVault(cfg);++ const server = makeServer();+ await initialize(server);+ const resp = (await call(server, "brain_search", { query: "A", path_prefix: "../etc/" })) as any;+ expect(resp?.error?.code).toBe(-32602);+ expect(resp.error.message).toContain("path_prefix");+});++test("brain_search on missing index surfaces INTERNAL_ERROR with hint", async () => {+ const server = makeServer();+ await initialize(server);+ const resp = (await call(server, "brain_search", { query: "fox" })) as any;+ expect(resp?.error?.code).toBe(-32603); // INTERNAL_ERROR+ expect(resp.error.message).toMatch(/not initialised|index/i);+});++test("brain_search rejects limit > 50", async () => {+ writeMd("a.md", "# A");+ const cfg = resolveSearchConfig({ vault, configPath });+ await indexVault(cfg);++ const server = makeServer();+ await initialize(server);+ const resp = (await call(server, "brain_search", { query: "A", limit: 100 })) as any;+ expect(resp?.error?.code).toBe(-32602);+});++test("second_brain_status includes search block after indexing", async () => {+ writeMd("a.md", "# A\n\nbody");+ const cfg = resolveSearchConfig({ vault, configPath });+ await indexVault(cfg);++ const server = makeServer();+ await initialize(server);+ const resp = await call(server, "second_brain_status", {});+ const body = extractToolResult(resp);+ const search = body["search"] as Record<string, unknown> | undefined;+ expect(search).toBeDefined();+ expect(search!["exists"]).toBe(true);+ expect(search!["documents"]).toBe(1);+ expect(search!["schema_version"]).toBe(1);+});++test("second_brain_status reports search.exists=false when index missing", async () => {+ const server = makeServer();+ await initialize(server);+ const resp = await call(server, "second_brain_status", {});+ const body = extractToolResult(resp);+ const search = body["search"] as Record<string, unknown> | undefined;+ expect(search).toBeDefined();+ expect(search!["exists"]).toBe(false);+ expect(search!["hint"]).toMatch(/o2b search index/);+});