Local-first π§ memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly π΄ dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
feat: Indexer Durability & Resilience Suite - cooperative abort, graceful watch shutdown, resumable reindex (v1.8.0) (#97)
Sol Aitken7cdbfc036ef1
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.jsonindex 000c12b5..10bb18e6 100644--- a/.claude-plugin/plugin.json+++ b/.claude-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.7.0",+ "version": "1.8.0","description": "Plugin-first second brain package for AI agents and humans.","author": {"name": "Open Second Brain contributors"diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.jsonindex 73378d78..7c89f9dc 100644--- a/.codex-plugin/plugin.json+++ b/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.7.0",+ "version": "1.8.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","skills": "./skills","hooks": "./hooks/hooks.json",diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7b5e1e47..de610a4e 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,6 +5,50 @@ All notable changes to this project will be documented in this file.The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).+## [1.8.0] - 2026-06-13++### Added++- **Indexer Durability & Resilience Suite.** Interrupting a long index run no+ longer risks losing work or wedging the index. The suite makes cancellation+ cooperative, the watcher shutdown graceful, and a full rebuild resumable -+ reusing the existing `Safeguard` and per-vault lock rather than adding a new+ lifecycle subsystem. Every behavioural change defaults to today's behaviour;+ a vault that sets nothing is byte-identical.+ - **Cooperative abort (`Safeguard` + `AbortSignal`).** The cooperative+ deadline gains an optional `AbortSignal`: one `checkpoint()` now trips on+ either an aborted signal (a new `SafeguardAbortError`, checked first) or the+ existing timeout. The signal threads through `indexVault` and+ `populateEmbeddings`, checked at the same boundaries the deadline uses -+ between files and between embed batches, never mid-write - so a run can be+ cancelled on demand. Bun's SQLite is synchronous, so abort stays+ cooperative; the deletion sweep runs only on full completion, so an aborted+ run leaves a consistent, partially-refreshed index.+ - **Graceful `o2b search watch` shutdown.** SIGINT/SIGTERM now stops accepting+ new flushes, aborts the in-flight pass, and awaits it to settle at a+ cooperative boundary before exiting - bounded by `search_shutdown_grace_seconds`+ (default 5). A signal no longer kills a pass mid-write. The flush/shutdown+ coordination lives in a testable `IndexWatchRunner`; a second signal falls+ back to the default terminate. `0` grace exits immediately after signalling.+ - **Opt-in resumable reindex (`search_resume_reindex`).** An interrupted full+ `reindexVault` rebuild no longer discards all progress: a compatible+ in-progress `brain.sqlite.new` staging build is resumed via the incremental+ fastpath instead of rebuilt from scratch. Resume is gated on a signature+ marker (schema version + chunk parameters + embedding signature) stored in+ the staging DB's `index_state` KV - no schema migration - so a drifted or+ unreadable staging DB is discarded and rebuilt, never trusted. The marker is+ cleared before the atomic swap, so the live index never carries staging+ state. Default off keeps the always-fresh rebuild.+ - **Writer-lock heartbeat + WAL-flush-on-exit.** The async writer lock refreshes+ its mtime mid-run (explicit heartbeat below the 60s stale window) so a long+ index is never mistaken for a stale lock. A process-exit registry consolidates+ each open writer's WAL on a bypassed `close()`, mirroring the existing+ sync-lock cleanup hook.+- Honest multi-instance story (no new daemon): the MCP server is stdio-only, so+ isolation comes from the per-`dbPath` writer lock. Two instances on different+ vaults run conflict-free; a second writer on the same vault gets a typed+ `INDEX_LOCKED`. Pinned by tests; no `--port`/`--instance` model was fabricated.+## [1.7.0] - 2026-06-13### Added@@ -5344,6 +5388,7 @@ plugin config (vault field)`, and exits with a clear- Sandbox vault and plugin manifest fixtures for tests.- GitHub release workflow for tag-based and manually dispatched releases.+[1.8.0]: https://github.com/itechmeat/open-second-brain/compare/v1.7.0...v1.8.0[1.7.0]: https://github.com/itechmeat/open-second-brain/compare/v1.6.0...v1.7.0[1.6.0]: https://github.com/itechmeat/open-second-brain/compare/v1.5.0...v1.6.0[1.5.0]: https://github.com/itechmeat/open-second-brain/compare/v1.4.0...v1.5.0diff --git a/README.md b/README.mdindex ec408437..46fc5196 100644--- a/README.md+++ b/README.md@@ -87,6 +87,7 @@ Each non-Hermes target writes a sidecar manifest under `<vault>/.open-second-bra- **Memory that cleans itself, on your terms.** `o2b brain hygiene scan` surfaces contested facts, near-duplicate rules, stale derived pages, and never-recalled memories; `apply` executes only the findings you select, and stale pages recompile from their recorded sources with a dry-run preview.- **A vault that stays fresh, consistent, and scoped.** `o2b search watch` keeps the index live as you edit, debounced and incremental; note identity is Unicode-normalized so the same file is one entry across macOS and Linux devices instead of a phantom cross-device duplicate; and recall accepts an opt-in `agent_scope` so a page marked with an `owner:` is reachable only to its owner while shared pages stay open to all.- **Knowledge that knows where it came from.** Drop a source document and the agent's extraction becomes cross-referenced entity and concept pages plus a summary page that backlinks the source and lists its connections; N sources become one dated report whose every finding cites the source that flagged it; a derived fact carries a `deduced`/`inferred` provenance level and links back to its premises, and recall trusts an operator-stated rule above a machine-derived one. A fact can declare an `owner:` so multi-agent brains keep separate truth spaces, and a standing-query attention flow surfaces the open loops you declare. The plugin never runs a model itself - the agent owns generation, the vault owns the durable, provenanced record - and every behaviour is opt-in.+- **An index that survives interruption.** Stopping `o2b search watch` mid-sync finishes the in-flight pass at a file boundary before exiting (within `search_shutdown_grace_seconds`) instead of killing it mid-write; an incremental run is already resumable through the unchanged-file fastpath, and `search_resume_reindex` extends that to a full rebuild - an interrupted reindex resumes its staging build instead of starting over, guarded by a signature so a drifted build is discarded rather than trusted. Both default off, so behaviour is unchanged unless you ask for it.That is the day-to-day picture. The full capability surface, every CLI verb, and the mental model live in the [documentation](#documentation) below.diff --git a/docs/brainstorm/indexer-durability/cli-output/claude.md b/docs/brainstorm/indexer-durability/cli-output/claude.mdnew file mode 100644index 00000000..8c2f4d05--- /dev/null+++ b/docs/brainstorm/indexer-durability/cli-output/claude.md@@ -0,0 +1,35 @@+### Variant 1: Fix the real gaps in place (distributed wiring)+- **Approach**: Extend `Safeguard` with an optional `AbortSignal` so `checkpoint()` throws either the existing `SafeguardTimeoutError` or a new typed `SafeguardAbortError` - one cooperative primitive, two trip conditions. Fix the `o2b search watch` signal handler to await the in-flight `indexVault` pass within a configurable grace window, then fire the abort to unwind cleanly; guarantee a WAL-consolidating, lock-releasing close on every exit path by routing through the existing `sync-lockfile` process-exit hook. Add a lock-heartbeat that refreshes the proper-lockfile mtime during long runs (so a genuine long index never looks stale at 60s) and treat lock-busy as a skip, plus honest attempt accounting in the embed retry path (no retry for the non-idempotent batch commit). `reindexVault` stays non-resumable but becomes abort-honoring, leaving the original DB intact via the temp file as today.+- **Trade-offs**:+ - Pro: smallest surface that closes every OSB-applicable gap across all three tasks; reuses `Safeguard`, `lock.ts`, `sync-lockfile.ts` (DRY, no new abstraction).+ - Pro: no new schema/table, so it cannot become a "redundant resumable mechanism" for the already-resumable incremental path; byte-identical when grace window/heartbeat flags are off.+ - Pro: fixes the genuine watch-handler-kills-mid-flush bug and the orphan `-wal`/held-lock-on-SIGTERM bug directly.+ - Con: leaves `reindexVault` restarting full rebuilds from scratch - the one genuinely non-resumable path is unaddressed.+ - Con: abort/grace/heartbeat concerns are wired at several call sites rather than one object, so the lifecycle is spread out and easy to forget to thread on a future code path.+- **Complexity**: medium+- **Risk**: low++### Variant 2: Resumable staged reindex (carry-forward temp DB)+- **Approach**: Build on Variant 1's shutdown/abort/heartbeat core, then make the genuinely non-resumable `reindexVault` resumable by treating `brain.sqlite.new` as a durable staging DB instead of discarding it on interruption. On the next reindex, detect a compatible in-progress staging build (matching `LATEST_SCHEMA_VERSION` plus a vault fingerprint) and resume it - reusing the existing `indexInto` mtime/size fastpath and `findChunksWithoutEmbeddings()` logic on the temp file - before the unchanged atomic same-file rename swap. A small staging marker records fingerprint and schema version so a stale or schema-drifted temp build is invalidated and rebuilt rather than silently resumed.+- **Trade-offs**:+ - Pro: addresses the only genuinely non-resumable path honestly, without duplicating the incremental path's resume logic (it reuses it on the temp DB).+ - Pro: atomic swap and `.bak` crash-recovery window are untouched; gated behind a flag that defaults to today's discard-on-interrupt behaviour.+ - Con: the staging marker is a quasi-checkpoint that must be rigorously invalidated; a subtle fingerprint bug could resume onto a stale build and produce a wrong index - the highest-consequence failure mode in the suite.+ - Con: significantly larger validation surface (schema drift, vault mutation mid-build, partial-batch embed state) for a path that runs rarely.+ - Con: closest of the three to brushing against the "no misleading/duplicate mechanism" rule, so it needs the most justification.+- **Complexity**: large+- **Risk**: medium-high++### Variant 3: Centralized RunController supervisor+- **Approach**: Introduce one queue-scoped supervisor object (the OSB-faithful reading of upstream's "DB supervisor singleton") that owns the entire long-run lifecycle: an `AbortController` composed into `Safeguard`, the signal handlers with grace window and in-flight await, the SIGTERM-safe WAL-consolidate-and-release close, the lock heartbeat with lock-busy-as-skip, and honest attempt accounting with per-handler wall-clock budgets. Every long-running operation (`indexVault`, `populateEmbeddings`, `reindexVault`, the `watch` loop) receives this one controller instead of threading deadline, signal, and lock concerns separately. `reindexVault` stays non-resumable but abort-honoring.+- **Trade-offs**:+ - Pro: single composition point makes abort/shutdown/heartbeat/accounting impossible to forget on new code paths; most faithful adaptation of all three upstream intents to OSB's actual architecture.+ - Pro: centralizes honest accounting and budgets, which are otherwise scattered.+ - Con: a new singleton lifecycle abstraction is heavy for a codebase whose only long-running process is `watch`; real risk of over-engineering relative to the actual surface.+ - Con: a singleton invites accidentally re-importing multi-instance/daemon framing that OSB's stdio + per-vault locking explicitly forbids.+ - Con: larger refactor touching every long-op call site, raising regression risk against the "byte-identical when off" guarantee.+- **Complexity**: large+- **Risk**: medium++### Recommended: Variant 1+**Rationale**: It closes every gap that actually exists in the OSB codebase - the watch-handler mid-flush bug, the orphan-WAL/held-lock-on-SIGTERM bug, the missing on-demand abort, the stale-lock-during-long-run, and dishonest retry - across all three kanban tasks, while honoring the hard constraints: it composes with `Safeguard` rather than replacing it, adds no table that would duplicate the already-resumable incremental path, and stays byte-identical when its flags are off. Variant 2's staged-reindex resume carries the suite's only wrong-index failure mode for a rarely-run path and can be added later if reindex restart cost becomes real; Variant 3's supervisor is the cleanest design in the abstract but over-engineers a lifecycle abstraction for a tool with a single long-running process and risks smuggling back the daemon framing the grounding facts forbid.diff --git a/docs/brainstorm/indexer-durability/cli-output/prompt.md b/docs/brainstorm/indexer-durability/cli-output/prompt.mdnew file mode 100644index 00000000..c5045ec2--- /dev/null+++ b/docs/brainstorm/indexer-durability/cli-output/prompt.md@@ -0,0 +1,54 @@+You are brainstorming architectural variants for the following task. Do not write code. Do not write a final design. Only produce variants and a recommendation.++# Task++Build an "Indexer Durability & Resilience" suite for Open Second Brain (OSB), a TypeScript/Bun knowledge-vault tool whose search index is a SQLite database (FTS5 + sqlite-vec) derived from a Markdown vault. The suite bundles three kanban tasks that the board's validator clustered as the "indexer-durability family":++1. t_ea80ddb5 - Graceful stop/restart with index flush and multi-instance support. Upstream (agentmemory) inverted stop order: worker first with a SIGTERM grace window for index flush, engine second; added a --instance N flag for multi-daemon support with --port. The OSB-side concern: an operator stopping OSB mid-index (crash, SIGTERM, timeout) can lose uncommitted changes; and running multiple instances on one host.++2. t_672c751e - Resumable durable append-only sync checkpoints with lock heartbeat. Upstream (gbrain) added an append-only checkpoint delta table, guaranteed final flush on every exit path including SIGTERM, lock heartbeat-aware takeover, and treated lock-busy as a skip rather than a phase failure. The OSB-side concern: if a large index run is interrupted it restarts from scratch.++3. t_79e773be - Cooperative abort-honoring with honest attempt accounting. Upstream (gbrain) threaded an AbortSignal through embed phases, checking on both the stale and full paths and between batches; added honest attempt accounting (no retry for non-idempotent work), per-handler wall-clock budgets, and a queue-scoped DB supervisor singleton. The OSB-side concern: long embed jobs can wedge a cycle with no way to abort cooperatively.++# Critical grounding facts about the ACTUAL OSB codebase (verified in code, must shape the design)++- The INCREMENTAL index path (`indexVault`) is ALREADY resumable. `indexInto` walks the vault and skips unchanged files via an mtime+size fastpath (re-running after an interruption skips everything already committed). `populateEmbeddings` only processes `findChunksWithoutEmbeddings()` and commits each batch before the next, so an interrupted embed phase resumes by computing only the still-missing embeddings. => A new resumable-checkpoint table for the INCREMENTAL path would be REDUNDANT and is forbidden by project rules (no misleading or duplicate mechanisms).+- The genuinely NON-resumable path is `reindexVault`: a full rebuild (force=true) that writes a fresh temp DB (`brain.sqlite.new`), then does an atomic same-file rename swap with `.bak` retention. An interruption discards the whole temp build; there is no intermediate resume. Crash-recovery for the rename window already exists (Store.open restores `.bak` if the main file is missing).+- There is already a cooperative deadline primitive: `Safeguard` (src/core/brain/safeguard.ts). An operation receives a `Safeguard` and calls `checkpoint()` at iteration boundaries (per file, per embed batch); past the deadline the next checkpoint throws `SafeguardTimeoutError`. Bun runs SQLite synchronously, so preemptive cancellation is impossible - the honest contract is cooperative. The gap: Safeguard is a TIME deadline only; there is no on-demand AbortSignal to cancel a run that is still within its time budget (e.g. from a SIGTERM in the watcher, or a parent cancellation).+- The only long-running process is `o2b search watch` (src/cli/search.ts): a debounced fs.watch loop that calls `indexVault`. Its SIGINT/SIGTERM handler currently resolves immediately WITHOUT awaiting an in-flight `indexVault` pass - so a signal during a flush kills the run mid-write. This is a real bug.+- The MCP server is stdio-only (no HTTP, no port). Multiple OSB instances against DIFFERENT vaults already work: the writer path in `Store.open` acquires a per-dbPath `proper-lockfile` lock (stale 60s) and a second writer on the same vault gets a typed `INDEX_LOCKED` error. => A `--port`/`--instance` daemon model from the upstream is NOT applicable to OSB's architecture and must NOT be fabricated.+- The SQLite store uses WAL; `Store.close()` consolidates the WAL (`PRAGMA wal_checkpoint(TRUNCATE)` then `journal_mode=DELETE`) and releases the lock, but only on an orderly close - a SIGTERM that bypasses close leaves an orphan `-wal` and a held lock.+- Schema migrations live in src/core/search/schema.ts: `LATEST_SCHEMA_VERSION` (currently 7) plus an ordered `MIGRATIONS` array of `{version, up}`. Bumping the version auto-migrates on the next write-mode open.+- Existing reusable primitives: `src/core/reliability/lock.ts` (async proper-lockfile wrapper), `src/core/brain/sync-lockfile.ts` (sync lock with a process-exit cleanup hook that unlinks held locks).++# Project context++- Open Second Brain - TypeScript, Bun runtime, SQLite (bun:sqlite), MCP server, `o2b` CLI, Obsidian/Markdown vault under `Brain/`.+- Recent suites shipped as cohesive versioned releases (v1.4 Grok, v1.5 Search & Recall Quality, v1.6 Vault Integrity & Trust, v1.7 Knowledge Provenance).+- Conventions: opt-in guardrail flags in config so behaviour is byte-identical when a feature is off; DRY shared primitives; strict TypeScript with NO `as` cast crutches (build values with the correct type via conditional spreads / narrowing guards); language-agnostic (no hardcoded natural-language word lists); provider-agnostic kernel (OSB never calls an LLM itself); no new heavy or ML dependencies; no misleading fallbacks (a feature that would silently no-op when its substrate is absent is forbidden - fail loud and typed instead).++# Constraints++- Do NOT add a redundant resumable-checkpoint mechanism for the already-resumable incremental path.+- Do NOT invent a --port/--instance daemon model; OSB is stdio and per-vault locking already isolates instances.+- No new external dependencies; reuse proper-lockfile and bun:sqlite.+- Cancellation must be cooperative (Bun SQLite is synchronous); never claim preemptive abort.+- The abort/deadline mechanism should compose with the existing `Safeguard`, not replace it (DRY).+- Resumable behaviour and graceful-shutdown grace windows should be controllable and default to current behaviour where they would otherwise change output shape.++# Required output format++Produce exactly 3 distinct architectural variants. For each variant:++### Variant N: <short name>+- **Approach**: 2-3 sentences describing the variant.+- **Trade-offs**: bullet list of pros and cons.+- **Complexity**: small | medium | large+- **Risk**: low | medium | high++After the three variants, add exactly one recommendation:++### Recommended: Variant N+**Rationale**: 2-3 sentences explaining why this variant over the others, considering the project context and constraints above.++Output nothing outside of these sections.diff --git a/docs/brainstorm/indexer-durability/design.md b/docs/brainstorm/indexer-durability/design.mdnew file mode 100644index 00000000..d40ddc32--- /dev/null+++ b/docs/brainstorm/indexer-durability/design.md@@ -0,0 +1,156 @@+# Indexer Durability & Resilience - making interrupted index runs safe and resumable++**Status:** draft+**Author:** claude-dev-agent (via feature-release-playbook)+**Audience:** implementation++## Problem statement++Open Second Brain's search index is a SQLite database derived from the vault.+Long index runs can be interrupted (SIGTERM, crash, deadline). Three real gaps+exist today: (1) the `o2b search watch` shutdown handler resolves immediately+and kills an in-flight `indexVault` pass mid-write; (2) the only way to stop a+running index that is still within its time budget is to kill the process -+there is no cooperative, on-demand abort; (3) a full `reindexVault` rebuild that+is interrupted discards all progress and restarts from scratch. This suite+closes those gaps without duplicating mechanisms that already work.++## Grounding: what is already durable (do NOT rebuild)++- The incremental `indexVault` path is already resumable. `indexInto` skips+ unchanged files via an mtime+size fastpath, so a re-run after an interruption+ re-walks and skips everything already committed. `populateEmbeddings` only+ processes `findChunksWithoutEmbeddings()` and commits each batch, so an+ interrupted embed phase resumes by computing only the still-missing vectors.+ A new resumable-checkpoint table for this path would be redundant and is+ explicitly out of scope.+- The reindex rename window is already crash-safe: `Store.open` restores the+ `.bak` if the main DB is missing.+- A cooperative time deadline already exists: `Safeguard.checkpoint()` trips+ past a budget at file / batch boundaries. We compose with it, not replace it.++## Scope++- A cooperative `AbortSignal`, composed into the existing `Safeguard`, threaded+ through `indexVault` / `indexInto` / `populateEmbeddings` / `reindexVault`,+ checked at the same boundaries the deadline already uses (between files,+ between embed batches - never mid-write).+- A graceful `o2b search watch` shutdown: on SIGINT/SIGTERM, stop accepting new+ flushes, abort the in-flight pass, await it to settle at the next boundary+ within a bounded grace window, then close the store (WAL consolidate + lock+ release) and exit. A second signal forces immediate exit.+- A best-effort synchronous WAL checkpoint on process exit for any open writer+ store, so a bypassed `close()` does not leave an orphan `-wal` that trips+ `SQLITE_IOERR_SHORT_READ` on the next open.+- A lock heartbeat: the writer's `proper-lockfile` is refreshed during long runs+ so a legitimate long index is never mistaken for a stale lock; lock-busy is+ surfaced as a clean skip in the watch loop rather than crashing the watcher.+- An opt-in resumable `reindexVault`: a compatible in-progress `brain.sqlite.new`+ staging build is detected by a signature marker and resumed (via the existing+ incremental fastpath over the temp DB) instead of being discarded.+- Two config keys (`search_shutdown_grace_seconds`, `search_resume_reindex`)+ with env mirrors, defaulting to today's behaviour.++## Out of scope++- No `--port` / `--instance` daemon model. OSB's MCP server is stdio-only;+ multiple instances against different vaults already run conflict-free because+ the writer path takes a per-`dbPath` lock, and a second writer on the same+ vault already gets a typed `INDEX_LOCKED`. We verify and document this; we do+ not fabricate a daemon. (The honest reading of t_ea80ddb5's "multi-instance".)+- No resumable-checkpoint table for the incremental path (already resumable).+- No "attempt accounting / dead-letter" job-queue accounting from the upstream:+ the index is not a retrying job queue, so importing that vocabulary would be+ misleading. The honest slice of t_79e773be is the cooperative AbortSignal.+- No preemptive cancellation: Bun runs SQLite synchronously, so abort is+ cooperative only. We never claim otherwise.++## Chosen approach++Variant 2 (see variants.md): the Variant 1 "fix the real gaps in place" core -+abort-composed Safeguard, graceful watch shutdown, exit-path WAL flush, lock+heartbeat - PLUS the genuinely non-resumable path made resumable: opt-in staged+reindex resume. The CLI consultant recommended Variant 1 (skip resumable+reindex) on risk grounds. We override, because:++1. Without resumable reindex, the chosen task t_672c751e delivers almost nothing+ new - everything else it asks for (resume, final flush) is either already+ present (incremental resume) or is the same shutdown/flush work as t_ea80ddb5.+2. The operator's brief for this cycle is an explicitly larger, substantive+ change; the resumable reindex is the only honest, non-redundant home for real+ volume here.+3. The consultant's one serious objection - resuming onto a stale staging build+ could yield a wrong index - is removed by design: resume is gated on a+ signature marker (embedding signature + schema version + chunk params); on+ any drift the staging DB is discarded and rebuilt; and resume itself is the+ ordinary incremental walk over the temp DB (fastpath re-reads any changed+ file, and the final deletion sweep removes vanished files), so it is+ self-healing rather than a blind trust of partial state. The whole behaviour+ is behind a flag that defaults to today's discard-on-interrupt.++## Design decisions++- **Compose, don't replace.** The abort lives inside `Safeguard`: an optional+ `signal` makes `checkpoint()` throw a typed `SafeguardAbortError` when the+ signal is aborted, in priority over the timeout. Callers that already hold a+ Safeguard get abort for free; there is one trip point, not two.+- **Throw, don't silently truncate.** An aborted run throws `SafeguardAbortError`+ exactly as a timed-out run throws `SafeguardTimeoutError`. The deletion sweep+ runs only after the walk completes, so a thrown abort leaves a consistent,+ partially-refreshed index (identical to today's timeout behaviour). The watch+ shutdown coordinator catches the abort and treats it as a clean stop.+- **Reuse the KV table, no migration.** The reindex staging marker is a row in+ the existing `index_state` KV table of the temp DB. No new table, no schema+ version bump, so existing indices are untouched and reads never break with a+ spurious `SCHEMA_MISMATCH`. Byte-identical when `search_resume_reindex` is off.+- **Testable coordinator, thin CLI.** The watch shutdown logic (in-flight await,+ grace window, refuse-after-stop) moves into a `IndexWatchRunner` with an+ injected clock and index function, so it is unit-tested without real signals+ or timers. The CLI wires `fs.watch` + signal handlers + store lifecycle to it.+- **Exit-hook mirrors the existing pattern.** The sync WAL-checkpoint-on-exit+ registry follows `sync-lockfile.ts`'s `process.on("exit", ...)` shape:+ best-effort, synchronous (bun:sqlite is sync), never throws.+- **Fail loud on lock contention.** Lock-busy stays a typed `INDEX_LOCKED`; the+ watch loop treats it as a skip (report + keep watching) rather than crashing,+ but a one-shot `o2b search index` still surfaces it. No silent no-op.++## File changes++New:+- `src/core/search/watch-runner.ts` - testable watch flush/shutdown coordinator.+- `src/core/search/store-exit.ts` - sync WAL-checkpoint-on-exit registry.+- tests: `tests/core/brain/safeguard-abort.test.ts`,+ `tests/core/search/indexer-abort.test.ts`,+ `tests/core/search/watch-runner.test.ts`,+ `tests/core/search/store-exit.test.ts`,+ `tests/core/search/reindex-resume.test.ts`,+ `tests/core/search/lock-heartbeat.test.ts`,+ `tests/core/search/multi-instance.test.ts`,+ plus a config-resolution test extension.++Modified:+- `src/core/brain/safeguard.ts` - optional `signal`, `SafeguardAbortError`,+ `throwIfAborted` helper.+- `src/core/search/indexer.ts` - thread `signal` through options; resumable+ `reindexVault`; staging signature marker.+- `src/core/search/store.ts` - proper-lockfile `update` heartbeat; register with+ the exit-flush registry on write open / unregister on close.+- `src/core/search/types.ts` - `shutdownGraceMs`, `resumeReindex` on+ `ResolvedSearchConfig`.+- `src/core/search/index.ts` - resolve the two new keys + env mirrors + DEFAULTS.+- `src/cli/search.ts` - `cmdSearchWatch` uses `IndexWatchRunner`; graceful close.+- `README.md`, `CHANGELOG.md`, `package.json` (+ manifest sync), config docs.++## Risks and open questions++- **Stale staging resume (highest consequence).** Mitigated by the signature+ marker + self-healing incremental walk + opt-in flag (see Chosen approach).+ Tests assert that signature drift discards the staging DB.+- **Lock heartbeat interval.** Must be well under `stale` (e.g. stale/2) and must+ not leak a timer when the store closes. The `update` option of proper-lockfile+ handles the refresh; `close()`/release cancels it.+- **Exit-hook reentrancy.** The sync checkpoint must be idempotent and tolerate+ an already-closed DB; it never throws out of the exit handler.+- **Grace window default.** 5 seconds (matches the upstream agentmemory grace and+ is ample for a single-file boundary). Configurable; 0 means "abort and exit+ immediately without awaiting", which we treat as a valid explicit choice.diff --git a/docs/brainstorm/indexer-durability/plan.md b/docs/brainstorm/indexer-durability/plan.mdnew file mode 100644index 00000000..99e8cd2a--- /dev/null+++ b/docs/brainstorm/indexer-durability/plan.md@@ -0,0 +1,105 @@+# Indexer Durability & Resilience - implementation plan++Each task is one atomic unit = one TDD cycle = one conventional commit on+`feat/indexer-durability`. Order respects dependencies: the shared abort+primitive first, then the consumers.++## Task 1: Abort-composed Safeguard (shared primitive)+- **Files**: `src/core/brain/safeguard.ts`; `tests/core/brain/safeguard-abort.test.ts`.+- **Change**: add `SafeguardAbortError`; `CreateSafeguardOptions.signal?: AbortSignal`;+ `checkpoint()` throws `SafeguardAbortError` when `signal?.aborted`, in priority+ over the timeout check. `noopSafeguard` unchanged. Export a tiny+ `throwIfAborted(signal?: AbortSignal, operation?: string)` helper. No `as`+ casts; `signal` read with optional chaining.+- **Acceptance**: aborted signal trips `checkpoint()` with `SafeguardAbortError`+ even before the deadline; unset signal + live deadline behaves exactly as today;+ a no-deadline guard with an aborted signal still trips.+- **Depends on**: none.++## Task 2: AbortSignal threaded through the indexer+- **Files**: `src/core/search/indexer.ts`; `tests/core/search/indexer-abort.test.ts`.+- **Change**: `IndexVaultOptions.signal?: AbortSignal`. In `indexInto` and+ `populateEmbeddings`, after the existing `safeguard?.checkpoint()`, call+ `throwIfAborted(opts?.signal)` at the same boundaries (between files, between+ embed batches). The deletion sweep still runs only on full completion, so an+ abort leaves a consistent partial index. `reindexVault` forwards `signal`.+- **Acceptance**: a signal aborted before the loop throws `SafeguardAbortError`+ with nothing partial deleted; aborting between embed batches stops further+ embedding but keeps already-committed vectors; a run with no signal is+ byte-identical to today.+- **Depends on**: Task 1.++## Task 3: Writer lock heartbeat + lock-busy skip+- **Files**: `src/core/search/store.ts`; `tests/core/search/lock-heartbeat.test.ts`,+ `tests/core/search/multi-instance.test.ts`.+- **Change**: pass `update` (refresh interval = floor(stale/2)) to the+ `proper-lockfile.lock` call in write mode so a long run keeps the lock fresh.+ Confirm release cancels the refresh. `multi-instance.test.ts` asserts: two+ writers on different `dbPath`s both succeed; a second writer on the same+ `dbPath` gets `INDEX_LOCKED` (documents the honest multi-instance story).+- **Acceptance**: lock options include a positive `update` below `stale`; same-vault+ second writer is rejected with the typed error; different-vault writers coexist.+- **Depends on**: none.++## Task 4: Sync WAL-checkpoint-on-exit registry+- **Files**: `src/core/search/store-exit.ts` (new); `src/core/search/store.ts`;+ `tests/core/search/store-exit.test.ts`.+- **Change**: registry of open writer `Database` handles. `Store.open(write)`+ registers; `close()` unregisters. A `process.on("exit", ...)` hook (installed+ once, mirroring `sync-lockfile.ts`) runs `PRAGMA wal_checkpoint(TRUNCATE)`+ best-effort on each registered handle, swallowing errors. Idempotent; never+ throws.+- **Acceptance**: a registered handle is checkpointed by the hook function;+ unregister removes it; running the hook with a closed handle is a no-op, not a+ throw.+- **Depends on**: none.++## Task 5: Config keys (grace window + resume flag)+- **Files**: `src/core/search/types.ts`, `src/core/search/index.ts`;+ extend the existing search-config resolution test.+- **Change**: `ResolvedSearchConfig.shutdownGraceMs: number` and+ `resumeReindex: boolean`. DEFAULTS: `shutdownGraceMs` from 5s, `resumeReindex`+ false. Resolve `search_shutdown_grace_seconds`+ (env `OPEN_SECOND_BRAIN_SEARCH_SHUTDOWN_GRACE`, integer seconds >= 0, *1000)+ and `search_resume_reindex` (env `OPEN_SECOND_BRAIN_SEARCH_RESUME_REINDEX`,+ bool). Both threaded into the frozen `base` config.+- **Acceptance**: defaults present when keys absent; values parse from config and+ env; invalid values fail soft to default (matches existing parse helpers).+- **Depends on**: none.++## Task 6: Graceful watch shutdown (testable runner)+- **Files**: `src/core/search/watch-runner.ts` (new); `src/cli/search.ts`;+ `tests/core/search/watch-runner.test.ts`.+- **Change**: `IndexWatchRunner` owns flush single-flight + shutdown. Injected+ `now()`, `index()` function, and `AbortController` factory. `shutdown(graceMs)`+ sets stopped, aborts the in-flight controller, awaits the running flush up to+ `graceMs` (then returns even if still running - process exit follows), refuses+ new flushes after stop. `cmdSearchWatch` refactors to wire `fs.watch` ++ `SIGINT`/`SIGTERM` -> `runner.shutdown(cfg.shutdownGraceMs)` -> `store`/watcher+ close. A second signal forces immediate `resolve`.+- **Acceptance**: shutdown awaits an in-flight flush before resolving; if the+ flush exceeds the grace window shutdown still returns; a flush requested after+ stop is refused; a normal flush path is unchanged.+- **Depends on**: Tasks 1, 2, 5.++## Task 7: Opt-in resumable staged reindex+- **Files**: `src/core/search/indexer.ts`; `tests/core/search/reindex-resume.test.ts`.+- **Change**: in `reindexVault`, when `config.resumeReindex` is true, compute a+ staging signature (`embeddingSignature` + `LATEST_SCHEMA_VERSION` + chunkSize ++ chunkOverlap). If `brain.sqlite.new` exists and its `index_state`+ `reindex_signature` matches, resume by building with `force: false` (incremental+ fastpath completes the partial build); otherwise unlink the stale `.new` and+ start fresh with `force: true`. Write the signature marker at build start;+ delete it on success before the rename swap. When `resumeReindex` is false,+ keep today's unconditional `tryUnlink(newPath)` + `force: true`.+- **Acceptance**: with the flag on, a partial `.new` (some docs committed, marker+ matching) is completed without reprocessing the committed docs and the final+ index is identical to a fresh reindex; a marker signature mismatch discards the+ `.new` and rebuilds; with the flag off, behaviour is byte-identical to today+ (always fresh). The swapped main DB carries no `reindex_signature` row.+- **Depends on**: Tasks 1, 2, 5.++## Cross-cutting (done within the tasks above, no separate commit)+- No new dependencies; reuse `proper-lockfile`, `bun:sqlite`, Node `AbortSignal`.+- No `as` cast crutches; no natural-language word lists; provider-agnostic.+- Version bump + CHANGELOG + README land in the docs commit (Phase 5), per CLAUDE.md.diff --git a/docs/brainstorm/indexer-durability/variants.md b/docs/brainstorm/indexer-durability/variants.mdnew file mode 100644index 00000000..cefd90d7--- /dev/null+++ b/docs/brainstorm/indexer-durability/variants.md@@ -0,0 +1,71 @@+# Indexer Durability & Resilience - variant audit trail++Primary consultant: Claude Code (`claude -p`), output captured verbatim in+`cli-output/claude.md`. Fallback (Codex) not invoked - the primary returned+three parseable variants and a recommendation. Below: the consultant's variants+verbatim, then the orchestrator's final decision and rationale.++---++## Consultant variants (verbatim)++### Variant 1: Fix the real gaps in place (distributed wiring)+- **Approach**: Extend `Safeguard` with an optional `AbortSignal` so `checkpoint()` throws either the existing `SafeguardTimeoutError` or a new typed `SafeguardAbortError` - one cooperative primitive, two trip conditions. Fix the `o2b search watch` signal handler to await the in-flight `indexVault` pass within a configurable grace window, then fire the abort to unwind cleanly; guarantee a WAL-consolidating, lock-releasing close on every exit path by routing through the existing `sync-lockfile` process-exit hook. Add a lock-heartbeat that refreshes the proper-lockfile mtime during long runs (so a genuine long index never looks stale at 60s) and treat lock-busy as a skip, plus honest attempt accounting in the embed retry path (no retry for the non-idempotent batch commit). `reindexVault` stays non-resumable but becomes abort-honoring, leaving the original DB intact via the temp file as today.+- **Trade-offs**: Pro: smallest surface that closes every OSB-applicable gap across all three tasks; reuses `Safeguard`, `lock.ts`, `sync-lockfile.ts` (DRY, no new abstraction). Pro: no new schema/table, so it cannot become a "redundant resumable mechanism" for the already-resumable incremental path; byte-identical when grace window/heartbeat flags are off. Pro: fixes the genuine watch-handler-kills-mid-flush bug and the orphan `-wal`/held-lock-on-SIGTERM bug directly. Con: leaves `reindexVault` restarting full rebuilds from scratch - the one genuinely non-resumable path is unaddressed. Con: abort/grace/heartbeat concerns are wired at several call sites rather than one object.+- **Complexity**: medium+- **Risk**: low++### Variant 2: Resumable staged reindex (carry-forward temp DB)+- **Approach**: Build on Variant 1's shutdown/abort/heartbeat core, then make the genuinely non-resumable `reindexVault` resumable by treating `brain.sqlite.new` as a durable staging DB instead of discarding it on interruption. On the next reindex, detect a compatible in-progress staging build (matching `LATEST_SCHEMA_VERSION` plus a vault fingerprint) and resume it - reusing the existing `indexInto` mtime/size fastpath and `findChunksWithoutEmbeddings()` logic on the temp file - before the unchanged atomic same-file rename swap. A small staging marker records fingerprint and schema version so a stale or schema-drifted temp build is invalidated and rebuilt rather than silently resumed.+- **Trade-offs**: Pro: addresses the only genuinely non-resumable path honestly, without duplicating the incremental path's resume logic (it reuses it on the temp DB). Pro: atomic swap and `.bak` crash-recovery window are untouched; gated behind a flag that defaults to today's discard-on-interrupt behaviour. Con: the staging marker is a quasi-checkpoint that must be rigorously invalidated; a subtle fingerprint bug could resume onto a stale build and produce a wrong index. Con: significantly larger validation surface for a path that runs rarely. Con: closest of the three to brushing against the "no misleading/duplicate mechanism" rule.+- **Complexity**: large+- **Risk**: medium-high++### Variant 3: Centralized RunController supervisor+- **Approach**: Introduce one queue-scoped supervisor object that owns the entire long-run lifecycle: an `AbortController` composed into `Safeguard`, the signal handlers with grace window and in-flight await, the SIGTERM-safe WAL-consolidate-and-release close, the lock heartbeat with lock-busy-as-skip, and honest attempt accounting with per-handler wall-clock budgets. Every long-running operation receives this one controller. `reindexVault` stays non-resumable but abort-honoring.+- **Trade-offs**: Pro: single composition point makes abort/shutdown/heartbeat/accounting impossible to forget on new code paths. Pro: centralizes honest accounting and budgets. Con: a new singleton lifecycle abstraction is heavy for a codebase whose only long-running process is `watch`; real risk of over-engineering. Con: a singleton invites accidentally re-importing multi-instance/daemon framing that OSB's stdio + per-vault locking forbids. Con: larger refactor touching every long-op call site.+- **Complexity**: large+- **Risk**: medium++### Consultant's recommendation: Variant 1+Rationale (verbatim summary): closes every gap that actually exists while honoring the hard constraints; Variant 2 carries the suite's only wrong-index failure mode for a rarely-run path; Variant 3 over-engineers a lifecycle abstraction for a tool with a single long-running process.++---++## Orchestrator decision: Variant 2 (override, documented)++**Chosen: Variant 2** = Variant 1's full core (abort-composed Safeguard, graceful+watch shutdown, exit-path WAL flush, lock heartbeat, lock-busy skip) PLUS the+opt-in resumable staged reindex.++**Why override the consultant's Variant 1:**++1. **Task coverage.** t_672c751e ("resumable durable checkpoints") is one of the+ three chosen tasks. Under Variant 1 it delivers nothing new: the incremental+ path is already resumable, and the "final flush" it asks for is the same work+ as t_ea80ddb5's shutdown. Variant 2 is the only honest, non-redundant way to+ actually satisfy t_672c751e - by making the one genuinely non-resumable path+ (`reindexVault`) resumable.++2. **Cycle intent.** The operator's brief for this cycle is a deliberately larger,+ substantive change. Resumable reindex is the only place real volume can come+ from without padding or duplicating existing mechanisms.++3. **The risk is designed out.** The consultant's sole serious objection -+ resuming onto a stale staging build yields a wrong index - is removed:+ - resume is gated on a signature marker (embedding signature + schema version+ + chunk params); any drift discards `.new` and rebuilds from scratch;+ - resume is the ordinary incremental walk over the temp DB, so the fastpath+ re-reads any file whose mtime/size changed and the final deletion sweep+ removes vanished files - it is self-healing, not blind trust;+ - the whole behaviour sits behind `search_resume_reindex`, default false, so+ when off the path is byte-identical to today (always discard + fresh build);+ - the marker lives in the existing `index_state` KV table, so there is no+ schema migration and no risk of breaking existing indices.++**Why not Variant 3:** the supervisor singleton over-engineers a lifecycle object+for a codebase whose only long-running process is `watch`, and a singleton risks+smuggling back the daemon/multi-instance framing that OSB's stdio + per-vault+locking explicitly forbids. Variant 2 keeps the abort inside the existing+`Safeguard` (one trip point) and the shutdown inside a small testable runner -+the DRY benefit Variant 3 promised, without the abstraction weight.diff --git a/openclaw.plugin.json b/openclaw.plugin.jsonindex bafad69f..26b5fdef 100644--- a/openclaw.plugin.json+++ b/openclaw.plugin.json@@ -2,7 +2,7 @@"id": "open-second-brain","name": "Open Second Brain","description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",- "version": "1.7.0",+ "version": "1.8.0","activation": { "onStartup": true },"skills": ["./skills"],"contracts": {diff --git a/package.json b/package.jsonindex 8a342e48..7a430b22 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.7.0",+ "version": "1.8.0","private": false,"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.","keywords": [diff --git a/plugin.yaml b/plugin.yamlindex db9e616e..63d84707 100644--- a/plugin.yaml+++ b/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.7.0"+version: "1.8.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.jsonindex 34461995..ed8f4126 100644--- a/plugins/codex/.codex-plugin/plugin.json+++ b/plugins/codex/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.7.0",+ "version": "1.8.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yamlindex db9e616e..63d84707 100644--- a/plugins/hermes/plugin.yaml+++ b/plugins/hermes/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.7.0"+version: "1.8.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/pyproject.toml b/pyproject.tomlindex 79863cd7..9891e71e 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"# CLI entry points (those moved to `package.json` `bin`).[project]name = "open-second-brain"-version = "1.7.0"+version = "1.8.0"description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."readme = "README.md"requires-python = ">=3.11"diff --git a/src/cli/search.ts b/src/cli/search.tsindex 97728ea4..5cc4c6a4 100644--- a/src/cli/search.ts+++ b/src/cli/search.ts@@ -56,6 +56,8 @@ import type {} from "../core/search/index.ts";import { searchAcrossVaults } from "../core/search/cross-vault.ts";import { IndexWatchPlanner } from "../core/search/index-watch.ts";+import { IndexWatchRunner } from "../core/search/watch-runner.ts";+import { SafeguardAbortError } from "../core/brain/safeguard.ts";import { canonicalNotePath } from "../core/path-safety.ts";import { watch, type FSWatcher } from "node:fs";import { CliError, parseFlags } from "./argparse.ts";@@ -744,28 +746,32 @@ async function cmdSearchWatch(argv: ReadonlyArray<string>): Promise<number> {);}- let running = false;- let stopped = false;-- const flush = async (): Promise<void> => {- if (running || stopped) return;- const due = planner.take(Date.now());- if (due.length === 0) return;- running = true;- try {- const stats = await indexVault(cfg, { embeddings: flags["embeddings"] === true });- process.stderr.write(- `synced ${due.length} change(s): +${stats.added} ~${stats.updated} =${stats.unchanged}` +- (stats.errors.length > 0 ? ` (${stats.errors.length} error(s))` : "") +- "\n",- );- } catch (e) {- // A failed pass must not kill the watcher; report and keep watching.- process.stderr.write(`index sync failed: ${e instanceof Error ? e.message : String(e)}\n`);- } finally {- running = false;- }- };+ // Single-flight + graceful-shutdown coordinator (Indexer Durability+ // suite). A SIGINT/SIGTERM aborts the in-flight pass at its next file+ // boundary and waits for it to settle (bounded by the configured+ // grace window) before exiting, so a signal never kills a run+ // mid-write. indexInto closes its store in a finally, so the aborted+ // pass still consolidates the WAL and releases the writer lock.+ const runner = new IndexWatchRunner({+ graceMs: cfg.shutdownGraceMs,+ index: async (signal): Promise<void> => {+ const due = planner.take(Date.now());+ if (due.length === 0) return;+ try {+ const stats = await indexVault(cfg, { embeddings: flags["embeddings"] === true, signal });+ process.stderr.write(+ `synced ${due.length} change(s): +${stats.added} ~${stats.updated} =${stats.unchanged}` ++ (stats.errors.length > 0 ? ` (${stats.errors.length} error(s))` : "") ++ "\n",+ );+ } catch (e) {+ // An abort is the expected stop on shutdown - let the runner+ // observe it. Any other failure must not kill the watcher.+ if (e instanceof SafeguardAbortError) throw e;+ process.stderr.write(`index sync failed: ${e instanceof Error ? e.message : String(e)}\n`);+ }+ },+ });watcher.on("error", (e) => {process.stderr.write(`watch error: ${e instanceof Error ? e.message : String(e)}\n`);@@ -778,22 +784,25 @@ async function cmdSearchWatch(argv: ReadonlyArray<string>): Promise<number> {const timer = setInterval(() => {- void flush();+ void runner.flush();},Math.max(50, debounceMs),);return await new Promise<number>((resolve) => {- const shutdown = (): void => {- if (stopped) return;- stopped = true;+ const shutdown = async (): Promise<void> => {+ if (runner.isStopped) return;clearInterval(timer);watcher.close();+ // Drain the in-flight pass (aborted) within the grace window.+ // A second SIGINT/SIGTERM bypasses this (process.once consumed+ // the first), falling back to the default terminate = force exit.+ await runner.shutdown();process.stderr.write("watch stopped\n");resolve(0);};- process.once("SIGINT", shutdown);- process.once("SIGTERM", shutdown);+ process.once("SIGINT", () => void shutdown());+ process.once("SIGTERM", () => void shutdown());process.stderr.write(`watching ${cfg.vault} for .md changes (debounce ${debounceMs}ms); Ctrl-C to stop\n`,);diff --git a/src/core/brain/safeguard.ts b/src/core/brain/safeguard.tsindex 1b8edb26..30486ba8 100644--- a/src/core/brain/safeguard.ts+++ b/src/core/brain/safeguard.ts@@ -42,6 +42,31 @@ export class SafeguardTimeoutError extends Error {}}+/**+ * Thrown when an operation's `AbortSignal` is aborted on demand (not a+ * deadline). Distinct from `SafeguardTimeoutError` so callers - notably+ * the watch shutdown coordinator - can treat an intentional abort as a+ * clean stop rather than a failure.+ */+export class SafeguardAbortError extends Error {+ readonly operation: string;++ constructor(operation: string) {+ super(`${operation} was aborted at a checkpoint`);+ this.name = "SafeguardAbortError";+ this.operation = operation;+ }+}++/**+ * Cooperative abort check for hot loops that hold a signal but not a+ * full `Safeguard`. Throws `SafeguardAbortError` when the signal is+ * aborted; a no-op for a live or absent signal.+ */+export function throwIfAborted(signal?: AbortSignal, operation = "operation"): void {+ if (signal?.aborted === true) throw new SafeguardAbortError(operation);+}+export interface Safeguard {/** Throws `SafeguardTimeoutError` when the deadline has passed. */checkpoint(): void;@@ -57,6 +82,13 @@ export interface CreateSafeguardOptions {readonly timeoutMs?: number | null;/** Injected clock (tests). Defaults to `Date.now`. */readonly now?: () => number;+ /**+ * Optional on-demand cancellation. When aborted, the next+ * `checkpoint()` throws `SafeguardAbortError` - checked in priority+ * over the deadline. A signal with no deadline still produces a live+ * guard (no downgrade to noop).+ */+ readonly signal?: AbortSignal;}/** A no-deadline guard for callers that need a Safeguard-shaped arg. */@@ -73,14 +105,21 @@ export function createSafeguard(opts: CreateSafeguardOptions): Safeguard {opts.timeoutMs === undefined || opts.timeoutMs === null || opts.timeoutMs <= 0? null: opts.timeoutMs;- if (budget === null) return noopSafeguard(opts.operation);+ const signal = opts.signal;+ // No deadline AND no signal: nothing to guard.+ if (budget === null && signal === undefined) return noopSafeguard(opts.operation);const now = opts.now ?? Date.now;- const deadline = now() + budget;+ const deadline = budget === null ? null : now() + budget;return Object.freeze({operation: opts.operation,timeoutMs: budget,checkpoint: () => {- if (now() > deadline) throw new SafeguardTimeoutError(opts.operation, budget);+ // Abort wins over the deadline: an intentional cancellation is+ // reported as such, not masked as a timeout.+ if (signal?.aborted === true) throw new SafeguardAbortError(opts.operation);+ if (budget !== null && deadline !== null && now() > deadline) {+ throw new SafeguardTimeoutError(opts.operation, budget);+ }},});}diff --git a/src/core/search/index.ts b/src/core/search/index.tsindex 33393298..2fca57d0 100644--- a/src/core/search/index.ts+++ b/src/core/search/index.ts@@ -115,6 +115,8 @@ const DEFAULTS = {cacheTtlSeconds: 300,fusionMode: "linear" as const,rrfK: DEFAULT_RRF_K,+ shutdownGraceSeconds: 5,+ resumeReindex: false,};function parseFusionMode(raw: string | null): "linear" | "rrf" {@@ -508,6 +510,22 @@ export function resolveSearchConfig(opts: {false,"search_self_tuning_enabled",);+ const shutdownGraceSeconds = parseInteger(+ envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_SEARCH_SHUTDOWN_GRACE",+ "search_shutdown_grace_seconds",+ ),+ DEFAULTS.shutdownGraceSeconds,+ "search_shutdown_grace_seconds",+ { min: 0 },+ );+ const resumeReindex = parseBool(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_RESUME_REINDEX", "search_resume_reindex"),+ DEFAULTS.resumeReindex,+ "search_resume_reindex",+ );const recall: ResolvedRecallConfig = Object.freeze({mmrLambda,maxHops,@@ -541,6 +559,8 @@ export function resolveSearchConfig(opts: {rrfK,semantic,recall,+ shutdownGraceMs: shutdownGraceSeconds * 1000,+ resumeReindex,});if (!opts.overrides) {diff --git a/src/core/search/indexer.ts b/src/core/search/indexer.tsindex bec85177..d44aa803 100644--- a/src/core/search/indexer.ts+++ b/src/core/search/indexer.ts@@ -17,6 +17,7 @@ import { createHash } from "node:crypto";import {accessSync,constants,+ existsSync,mkdirSync,readFileSync,renameSync,@@ -43,8 +44,10 @@ import { tieredFieldsForKind } from "../brain/frontmatter-tiers.ts";import { normalizeSchemaToken } from "../brain/schema-vocab.ts";import { parseFrontmatterText } from "../vault.ts";import { appendMetric } from "../brain/metrics.ts";+import { throwIfAborted } from "../brain/safeguard.ts";import { extractEntities } from "./entities.ts";import { Store } from "./store.ts";+import { LATEST_SCHEMA_VERSION } from "./schema.ts";import { SearchError } from "./types.ts";import { walkVault } from "./walker.ts";import { withTimeout } from "./with-timeout.ts";@@ -76,6 +79,14 @@ export interface IndexVaultOptions {* file, so a tripped guard aborts between files - never mid-write.*/readonly safeguard?: import("../brain/safeguard.ts").Safeguard;+ /**+ * On-demand cancellation (Indexer Durability suite). Checked at the+ * same boundaries the deadline uses - between files and between embed+ * batches, never mid-write. An aborted signal throws+ * `SafeguardAbortError`; the deletion sweep runs only on full+ * completion, so an aborted run leaves a consistent partial index.+ */+ readonly signal?: AbortSignal;}interface MutableStats {@@ -210,6 +221,8 @@ async function indexInto(// Per-file document upserts are transactional, so a tripped// guard leaves a consistent (partially refreshed) index.opts?.safeguard?.checkpoint();+ // On-demand cancellation, same boundary as the deadline.+ throwIfAborted(opts?.signal, "index");// Mark seen FIRST. If anything downstream throws (read fault,// chunker bug, transient FS error), the file must not look// "missing" to the deletion sweep below β that would wipe a@@ -415,7 +428,14 @@ async function indexInto(}if (opts?.embeddings) {- await populateEmbeddings(store, config, stats, opts?.forceCost === true, opts?.safeguard);+ await populateEmbeddings(+ store,+ config,+ stats,+ opts?.forceCost === true,+ opts?.safeguard,+ opts?.signal,+ );}const now = new Date().toISOString();@@ -480,6 +500,7 @@ async function populateEmbeddings(stats: MutableStats,forceCost: boolean,safeguard?: import("../brain/safeguard.ts").Safeguard,+ signal?: AbortSignal,): Promise<void> {if (!config.semantic.enabled) {throw new SearchError(@@ -535,6 +556,7 @@ async function populateEmbeddings(// Cooperative deadline: embedding batches are the other long// phase of an index run - abort between batches, never mid-batch.safeguard?.checkpoint();+ throwIfAborted(signal, "index");const batch = pending.slice(i, i + superBatch);const texts = batch.map((p) => p.content);const vectors = await provider.embed(texts);@@ -576,14 +598,42 @@ export async function reindexVault(const bakPath = config.dbPath + ".bak";mkdirSync(dirname(config.dbPath), { recursive: true });- tryUnlink(newPath);// Build into the temp file with an override config.const tempConfig: ResolvedSearchConfig = Object.freeze({...config,dbPath: newPath,});- const stats = await indexVault(tempConfig, { ...opts, force: true });++ // Resumable staging (opt-in). A compatible in-progress `.new` from an+ // interrupted run is resumed via the incremental fastpath instead of+ // rebuilt from scratch; an incompatible or unreadable one is+ // discarded. With the flag OFF, the temp file is always rebuilt fresh+ // and no staging marker is written or read - byte-identical to the+ // pre-suite path (no extra staging-DB open/close cycles).+ let resume = false;+ if (config.resumeReindex) {+ const signature = reindexStagingSignature(config, opts?.embeddings === true);+ if (existsSync(newPath)) {+ resume = await stagingSignatureMatches(tempConfig, signature);+ }+ if (!resume) tryUnlink(newPath);+ // Stamp the signature so an interruption leaves a build a later run+ // can recognise and resume.+ await withStagingStore(tempConfig, (store) => store.setState(REINDEX_SIGNATURE_KEY, signature));+ } else {+ tryUnlink(newPath);+ }++ // `force: false` on resume lets the fastpath skip the files the+ // partial build already committed; a fresh build forces every file.+ const stats = await indexVault(tempConfig, { ...opts, force: !resume });++ // Clear the marker so the swapped-in live index carries no staging+ // state. Only present when resume was enabled.+ if (config.resumeReindex) {+ await withStagingStore(tempConfig, (store) => store.deleteState(REINDEX_SIGNATURE_KEY));+ }// Same-directory rename swap. The two renames are each atomic on// POSIX; the gap between them is the only crash window, which the@@ -594,6 +644,56 @@ export async function reindexVault(return stats;}+/** index_state key carrying the staging-build compatibility signature. */+const REINDEX_SIGNATURE_KEY = "reindex_signature";++/**+ * Compatibility signature for a staging rebuild: a resume is safe only+ * when the schema version, chunk parameters, and (when embeddings are+ * computed) the active embedding signature all match the partial build.+ * Any drift invalidates the staging DB and forces a fresh rebuild.+ */+function reindexStagingSignature(config: ResolvedSearchConfig, embeddings: boolean): string {+ const embedding = embeddings ? (activeEmbeddingSignature(config) ?? "active") : "off";+ return JSON.stringify({+ schema: LATEST_SCHEMA_VERSION,+ chunkSize: config.chunkSize,+ chunkOverlap: config.chunkOverlap,+ embedding,+ });+}++/** Read the staging signature from an existing `.new`; false on any+ * read error (a corrupt or partial staging DB is rebuilt, not trusted). */+async function stagingSignatureMatches(+ tempConfig: ResolvedSearchConfig,+ signature: string,+): Promise<boolean> {+ try {+ const store = await Store.open(tempConfig, { mode: "write", loadVec: false });+ try {+ return store.getState(REINDEX_SIGNATURE_KEY) === signature;+ } finally {+ await store.close();+ }+ } catch {+ return false;+ }+}++/** Run a short write against the staging DB (set/clear the marker). */+async function withStagingStore(+ tempConfig: ResolvedSearchConfig,+ fn: (store: Store) => void,+): Promise<void> {+ const store = await Store.open(tempConfig, { mode: "write", loadVec: false });+ try {+ fn(store);+ } finally {+ await store.close();+ }+}+/** `unlinkSync` that tolerates ENOENT (file already absent). */function tryUnlink(p: string): void {try {diff --git a/src/core/search/store-exit.ts b/src/core/search/store-exit.tsnew file mode 100644index 00000000..2292652c--- /dev/null+++ b/src/core/search/store-exit.ts@@ -0,0 +1,67 @@+/**+ * Writer WAL-flush-on-exit registry (Indexer Durability suite,+ * t_672c751e).+ *+ * The search store runs in WAL mode. An orderly `Store.close()`+ * consolidates the WAL (`wal_checkpoint(TRUNCATE)` then+ * `journal_mode=DELETE`) and releases the lock. When `close()` is+ * bypassed - an error path that calls `process.exit`, or a signal+ * handler that lets the process exit normally - an orphan `-wal` is+ * left next to the main DB. SQLite replays it on the next open, so no+ * data is lost; this registry is a belt-and-suspenders that+ * synchronously consolidates the WAL for every still-open writer on the+ * process `exit` event, mirroring `sync-lockfile.ts`'s cleanup hook.+ *+ * It is best-effort and synchronous (bun:sqlite is synchronous and the+ * `exit` event cannot await async work): every operation is wrapped so+ * the hook can never throw. A hard kill (SIGKILL) fires no `exit`+ * event; that case relies on SQLite's own WAL replay on the next open.+ *+ * The registry holds a strong reference until `Store.close()`+ * unregisters it. Every code path that opens a writer closes it in a+ * `finally`, so handles do not accumulate over a long-lived process;+ * the set holds only the writers currently open.+ */++import type { Database } from "bun:sqlite";++const openWriters = new Set<Database>();+let exitHookInstalled = false;++function ensureExitHook(): void {+ if (exitHookInstalled) return;+ exitHookInstalled = true;+ process.on("exit", flushRegisteredWriters);+}++/** Register an open writer DB so its WAL is consolidated on exit. */+export function registerWriterDb(db: Database): void {+ ensureExitHook();+ openWriters.add(db);+}++/** Unregister a writer DB (called from an orderly `Store.close()`). */+export function unregisterWriterDb(db: Database): void {+ openWriters.delete(db);+}++/**+ * Best-effort synchronous WAL consolidation for every registered+ * writer. Never throws: a closed, locked, or mid-write handle is+ * skipped silently. Safe to call more than once.+ */+export function flushRegisteredWriters(): void {+ for (const db of openWriters) {+ try {+ db.exec("PRAGMA wal_checkpoint(TRUNCATE)");+ } catch {+ // already closed / locked / mid-write β consolidation is a+ // best-effort safety net, not a correctness requirement.+ }+ }+}++/** Test-only: clear the registry without flushing. */+export function _resetWriterRegistryForTests(): void {+ openWriters.clear();+}diff --git a/src/core/search/store.ts b/src/core/search/store.tsindex dc8ef26d..0f06ca58 100644--- a/src/core/search/store.ts+++ b/src/core/search/store.ts@@ -18,6 +18,8 @@ import { existsSync, mkdirSync, renameSync } from "node:fs";import { dirname } from "node:path";import lockfile from "proper-lockfile";+import { registerWriterDb, unregisterWriterDb } from "./store-exit.ts";+import { computeCorpusGeneration } from "./corpus-generation.ts";import { SearchError } from "./types.ts";import type { ResolvedSearchConfig } from "./types.ts";@@ -204,13 +206,32 @@ function nowIso(): string {return new Date().toISOString();}+/**+ * Stale window for the writer lock (ms). A lock whose mtime is older+ * than this is treated as abandoned (crashed holder) and taken over by+ * the next writer, so a SIGKILL never wedges the index for longer than+ * this window.+ */+export const WRITER_LOCK_STALE_MS = 60_000;++/**+ * Heartbeat interval (ms): the async writer lock refreshes its mtime+ * this often so a legitimate long-running index is never mistaken for+ * a stale lock. Must stay below {@link WRITER_LOCK_STALE_MS}. NOTE: the+ * per-file document walk is synchronous, so this timer only fires+ * across the await points (embed batches); a multi-minute fully+ * synchronous walk still relies on the stale window, which 60s amply+ * covers for real vaults.+ */+export const WRITER_LOCK_HEARTBEAT_MS = 30_000;+function acquireWriterLockSync(path: string): () => void {const maxAttempts = 10;const sleepMs = 50;let lastError: unknown;for (let attempt = 0; attempt < maxAttempts; attempt++) {try {- return lockfile.lockSync(path, { stale: 60_000, realpath: false });+ return lockfile.lockSync(path, { stale: WRITER_LOCK_STALE_MS, realpath: false });} catch (err) {const code = (err as NodeJS.ErrnoException).code;if (code !== "ELOCKED") throw err;@@ -336,7 +357,10 @@ export class Store {try {release = await lockfile.lock(config.dbPath, {retries: { retries: 3, factor: 1, minTimeout: 1000, maxTimeout: 1000 },- stale: 60_000,+ stale: WRITER_LOCK_STALE_MS,+ // Explicit heartbeat: refresh the lock mtime mid-run so a long+ // index is never mistaken for a stale lock and taken over.+ update: WRITER_LOCK_HEARTBEAT_MS,realpath: false,});} catch (e) {@@ -360,6 +384,9 @@ export class Store {const vecLoaded = loadVec && tryLoadVecExtension(db);const store = new Store(db, config, vecLoaded, release);store.ensureEmbeddingModel(config.semantic.model, config.semantic.dimension);+ // Belt-and-suspenders: consolidate this writer's WAL on a+ // bypassed close (process.exit / signal-driven exit).+ registerWriterDb(db);return store;} catch (e) {try {@@ -383,6 +410,8 @@ export class Store {async close(): Promise<void> {if (this.closed) return;this.closed = true;+ // Orderly close owns consolidation below; drop the exit-hook entry.+ unregisterWriterDb(this.db);try {// Writer mode: consolidate WAL into the main file and switch back to// DELETE journal mode so the `-wal`/`-shm` siblings are removed. Thisdiff --git a/src/core/search/types.ts b/src/core/search/types.tsindex 8957ac93..2bcb39a7 100644--- a/src/core/search/types.ts+++ b/src/core/search/types.ts@@ -568,4 +568,18 @@ export interface ResolvedSearchConfig {readonly rrfK: number;readonly semantic: ResolvedEmbeddingConfig;readonly recall: ResolvedRecallConfig;+ /**+ * Grace window (ms) the `o2b search watch` shutdown waits for an+ * in-flight index pass to settle at a cooperative boundary before+ * exiting (Indexer Durability suite). `0` exits immediately after+ * signalling the abort, without awaiting. Default 5000.+ */+ readonly shutdownGraceMs: number;+ /**+ * When true, an interrupted full `reindexVault` rebuild resumes a+ * compatible `brain.sqlite.new` staging build instead of discarding+ * it. Opt-in; default false keeps the always-fresh rebuild. Resume is+ * gated on a signature marker, so a drifted staging DB is rebuilt.+ */+ readonly resumeReindex: boolean;}diff --git a/src/core/search/watch-runner.ts b/src/core/search/watch-runner.tsnew file mode 100644index 00000000..de64b869--- /dev/null+++ b/src/core/search/watch-runner.ts@@ -0,0 +1,103 @@+/**+ * IndexWatchRunner (Indexer Durability suite, t_ea80ddb5).+ *+ * The testable core of `o2b search watch`'s lifecycle. Two+ * responsibilities, both deliberately free of `fs.watch`, timers, and+ * OS signals so they can be unit-tested directly:+ *+ * - Single-flight flush: one index pass at a time; concurrent+ * `flush()` calls share the in-flight promise instead of stacking+ * overlapping passes.+ * - Graceful shutdown: stop accepting new flushes, abort the+ * in-flight pass via an AbortSignal (which the indexer honours at+ * its next file / embed-batch boundary), and await it to settle -+ * bounded by a grace window so a wedged pass cannot block exit+ * forever. `graceMs <= 0` aborts and returns without awaiting.+ *+ * The CLI provides the `index` callback (which runs `indexVault` with+ * the supplied signal and reports stats) and wires the watcher, the+ * debounce planner, and SIGINT/SIGTERM to this runner.+ */++export interface IndexWatchRunnerDeps {+ /**+ * Run exactly one index pass, honouring the abort signal. It owns its+ * own reporting and swallows its own recoverable errors; an abort may+ * surface as a rejection, which the runner treats as an expected+ * stop.+ */+ readonly index: (signal: AbortSignal) => Promise<unknown>;+ /** Milliseconds the shutdown awaits an aborted in-flight pass. */+ readonly graceMs: number;+ /** Resolves after `ms`; injectable for tests. Defaults to setTimeout. */+ readonly graceWaiter?: (ms: number) => Promise<void>;+}++function defaultGraceWaiter(ms: number): Promise<void> {+ return new Promise((resolve) => {+ const timer = setTimeout(resolve, ms);+ // Do not keep the event loop alive solely for the grace timer.+ if (typeof timer === "object" && timer !== null && "unref" in timer) timer.unref();+ });+}++export class IndexWatchRunner {+ private readonly deps: IndexWatchRunnerDeps;+ private running: Promise<void> | null = null;+ private controller: AbortController | null = null;+ private stopped = false;++ constructor(deps: IndexWatchRunnerDeps) {+ this.deps = deps;+ }++ /** True once shutdown has begun; no further flushes are accepted. */+ get isStopped(): boolean {+ return this.stopped;+ }++ /** True while an index pass is in flight. */+ get isFlushing(): boolean {+ return this.running !== null;+ }++ /**+ * Run one index pass unless shutdown began or a pass is already in+ * flight (single-flight: the in-flight promise is returned instead).+ */+ flush(): Promise<void> {+ if (this.stopped) return Promise.resolve();+ if (this.running) return this.running;+ const controller = new AbortController();+ this.controller = controller;+ const run = (async () => {+ try {+ await this.deps.index(controller.signal);+ } catch {+ // The index callback reports its own recoverable failures; an+ // abort surfaces here on shutdown and is the expected stop.+ }+ })().finally(() => {+ this.running = null;+ this.controller = null;+ });+ this.running = run;+ return run;+ }++ /**+ * Stop accepting flushes, abort any in-flight pass, and await it to+ * settle at a cooperative boundary - bounded by the grace window.+ * Idempotent: a second call is a no-op.+ */+ async shutdown(): Promise<void> {+ if (this.stopped) return;+ this.stopped = true;+ const inflight = this.running;+ if (inflight === null) return;+ this.controller?.abort();+ if (this.deps.graceMs <= 0) return;+ const wait = this.deps.graceWaiter ?? defaultGraceWaiter;+ await Promise.race([inflight, wait(this.deps.graceMs)]);+ }+}diff --git a/tests/core/brain/safeguard-abort.test.ts b/tests/core/brain/safeguard-abort.test.tsnew file mode 100644index 00000000..5df361d2--- /dev/null+++ b/tests/core/brain/safeguard-abort.test.ts@@ -0,0 +1,94 @@+/**+ * Safeguard abort composition (Indexer Durability suite, t_79e773be):+ * the cooperative deadline gains an optional AbortSignal so one+ * checkpoint() has two trip conditions - an aborted signal (priority)+ * or a passed deadline. Bun + synchronous SQLite still cannot be+ * preempted, so abort is cooperative: the NEXT checkpoint after an+ * abort throws SafeguardAbortError at a natural iteration boundary.+ */++import { describe, expect, test } from "bun:test";++import {+ createSafeguard,+ SafeguardAbortError,+ SafeguardTimeoutError,+ throwIfAborted,+} from "../../../src/core/brain/safeguard.ts";++describe("createSafeguard with an AbortSignal", () => {+ test("an aborted signal trips checkpoint() with SafeguardAbortError", () => {+ const ac = new AbortController();+ const guard = createSafeguard({ operation: "reindex", timeoutMs: 0, signal: ac.signal });+ guard.checkpoint();+ ac.abort();+ expect(() => guard.checkpoint()).toThrow(SafeguardAbortError);+ });++ test("abort wins over a live deadline (checked first)", () => {+ let clock = 1_000;+ const ac = new AbortController();+ const guard = createSafeguard({+ operation: "reindex",+ timeoutMs: 100_000,+ now: () => clock,+ signal: ac.signal,+ });+ clock += 10; // well within the deadline+ guard.checkpoint();+ ac.abort();+ try {+ guard.checkpoint();+ throw new Error("expected SafeguardAbortError");+ } catch (exc) {+ expect(exc).toBeInstanceOf(SafeguardAbortError);+ expect(exc).not.toBeInstanceOf(SafeguardTimeoutError);+ const e = exc as SafeguardAbortError;+ expect(e.operation).toBe("reindex");+ expect(e.message).toContain("reindex");+ }+ });++ test("a signal with no deadline still trips on abort (no noop downgrade)", () => {+ const ac = new AbortController();+ const guard = createSafeguard({ operation: "index", timeoutMs: null, signal: ac.signal });+ expect(() => guard.checkpoint()).not.toThrow();+ ac.abort();+ expect(() => guard.checkpoint()).toThrow(SafeguardAbortError);+ });++ test("the deadline still trips when the signal is never aborted", () => {+ let clock = 0;+ const ac = new AbortController();+ const guard = createSafeguard({+ operation: "reindex",+ timeoutMs: 10,+ now: () => clock,+ signal: ac.signal,+ });+ clock = 11;+ expect(() => guard.checkpoint()).toThrow(SafeguardTimeoutError);+ });++ test("no signal + live deadline behaves exactly as before", () => {+ let clock = 1_000;+ const guard = createSafeguard({ operation: "dream", timeoutMs: 50, now: () => clock });+ guard.checkpoint();+ clock += 51;+ expect(() => guard.checkpoint()).toThrow(SafeguardTimeoutError);+ });+});++describe("throwIfAborted", () => {+ test("throws SafeguardAbortError for an aborted signal", () => {+ const ac = new AbortController();+ ac.abort();+ expect(() => throwIfAborted(ac.signal, "index")).toThrow(SafeguardAbortError);+ });++ test("is a no-op for a live or absent signal", () => {+ const ac = new AbortController();+ expect(() => throwIfAborted(ac.signal, "index")).not.toThrow();+ expect(() => throwIfAborted(undefined, "index")).not.toThrow();+ });+});diff --git a/tests/core/search/config.test.ts b/tests/core/search/config.test.tsindex 16cc1ef7..e84039e1 100644--- a/tests/core/search/config.test.ts+++ b/tests/core/search/config.test.ts@@ -33,6 +33,8 @@ const ENV_KEYS = ["OPEN_SECOND_BRAIN_SEARCH_SYNONYM_MAX_TERMS","OPEN_SECOND_BRAIN_SEARCH_CACHE_ENABLED","OPEN_SECOND_BRAIN_SEARCH_CACHE_TTL",+ "OPEN_SECOND_BRAIN_SEARCH_SHUTDOWN_GRACE",+ "OPEN_SECOND_BRAIN_SEARCH_RESUME_REINDEX",];beforeEach(() => {@@ -280,3 +282,44 @@ test("overrides are validated after merging", () => {}),).toThrow(/embedding_batch_size/);});++// ββ Indexer Durability suite: shutdown grace + resume flag ββββββββββββββββββ++test("durability keys default to a 5s grace and no resume", () => {+ writeFileSync(configPath, `vault: "${tmp}"\n`);+ const cfg = resolveSearchConfig({ vault: tmp, configPath });+ expect(cfg.shutdownGraceMs).toBe(5_000);+ expect(cfg.resumeReindex).toBe(false);+});++test("config keys override the durability defaults", () => {+ writeFileSync(+ configPath,+ `vault: "${tmp}"\nsearch_shutdown_grace_seconds: 12\nsearch_resume_reindex: true\n`,+ );+ const cfg = resolveSearchConfig({ vault: tmp, configPath });+ expect(cfg.shutdownGraceMs).toBe(12_000);+ expect(cfg.resumeReindex).toBe(true);+});++test("env mirrors override the durability config", () => {+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_shutdown_grace_seconds: 12\n`);+ process.env["OPEN_SECOND_BRAIN_SEARCH_SHUTDOWN_GRACE"] = "3";+ process.env["OPEN_SECOND_BRAIN_SEARCH_RESUME_REINDEX"] = "true";+ const cfg = resolveSearchConfig({ vault: tmp, configPath });+ expect(cfg.shutdownGraceMs).toBe(3_000);+ expect(cfg.resumeReindex).toBe(true);+});++test("an invalid grace is rejected loudly (no silent fallback)", () => {+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_shutdown_grace_seconds: soon\n`);+ expect(() => resolveSearchConfig({ vault: tmp, configPath })).toThrow(+ /search_shutdown_grace_seconds/,+ );+});++test("grace of 0 disables the await window", () => {+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_shutdown_grace_seconds: 0\n`);+ const cfg = resolveSearchConfig({ vault: tmp, configPath });+ expect(cfg.shutdownGraceMs).toBe(0);+});diff --git a/tests/core/search/indexer-abort.test.ts b/tests/core/search/indexer-abort.test.tsnew file mode 100644index 00000000..e3fe458e--- /dev/null+++ b/tests/core/search/indexer-abort.test.ts@@ -0,0 +1,163 @@+/**+ * AbortSignal threading through the indexer (Indexer Durability suite,+ * t_79e773be). indexVault accepts an optional signal checked at the+ * same cooperative boundaries the deadline already uses - between files+ * in indexInto, between embed batches in populateEmbeddings - so a run+ * can be cancelled on demand without a mid-write kill. The deletion+ * sweep runs only on full completion, so an aborted run leaves a+ * consistent, partially-refreshed index. A run with no signal is+ * unchanged.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";++import { SafeguardAbortError } from "../../../src/core/brain/safeguard.ts";+import { indexVault } from "../../../src/core/search/indexer.ts";+import { Store } from "../../../src/core/search/store.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";+import { startFakeHttp, type FakeHttp } from "../../helpers/fake-http.ts";+import { sqliteVecLoadable } from "../../helpers/sqlite-vec.ts";++let vault: string;+let dbPath: string;+let cleanup: () => void;++beforeEach(() => {+ const v = createTempVault("indexer-abort");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+});++afterEach(() => {+ cleanup();+});++test("a signal aborted before the walk throws and indexes nothing", async () => {+ writeMd(vault, "a.md", "# A\n\nAlpha.");+ writeMd(vault, "b.md", "# B\n\nBeta.");+ const cfg = makeConfig({ vault, dbPath });+ const ac = new AbortController();+ ac.abort();++ await expect(indexVault(cfg, { signal: ac.signal })).rejects.toBeInstanceOf(SafeguardAbortError);++ const store = await Store.open(cfg, { mode: "write" });+ expect(store.listDocuments().size).toBe(0);+ await store.close();+});++test("aborting mid-walk commits files seen before the abort, drops the rest", async () => {+ for (let i = 0; i < 5; i++) writeMd(vault, `n${i}.md`, `# N${i}\n\nbody ${i}`);+ const cfg = makeConfig({ vault, dbPath });+ const ac = new AbortController();+ let processed = 0;++ // Abort right after the first file is committed; the next loop-top+ // checkpoint must trip and stop the walk.+ await expect(+ indexVault(cfg, {+ signal: ac.signal,+ onFile: () => {+ processed++;+ if (processed === 1) ac.abort();+ },+ }),+ ).rejects.toBeInstanceOf(SafeguardAbortError);++ const store = await Store.open(cfg, { mode: "write" });+ const count = store.listDocuments().size;+ await store.close();+ // Exactly the pre-abort file(s) survived; not all five, not zero.+ expect(count).toBeGreaterThanOrEqual(1);+ expect(count).toBeLessThan(5);+});++test("an interrupted run is resumable: a clean re-run completes the index", async () => {+ for (let i = 0; i < 5; i++) writeMd(vault, `n${i}.md`, `# N${i}\n\nbody ${i}`);+ const cfg = makeConfig({ vault, dbPath });+ const ac = new AbortController();+ let processed = 0;+ await expect(+ indexVault(cfg, {+ signal: ac.signal,+ onFile: () => {+ if (++processed === 1) ac.abort();+ },+ }),+ ).rejects.toBeInstanceOf(SafeguardAbortError);++ // No signal: the fastpath skips the already-committed file and+ // finishes the rest. The index ends complete.+ const stats = await indexVault(cfg);+ const store = await Store.open(cfg, { mode: "write" });+ expect(store.listDocuments().size).toBe(5);+ await store.close();+ expect(stats.added + stats.unchanged).toBe(5);+});++test("no signal indexes the whole vault unchanged", async () => {+ writeMd(vault, "a.md", "# A\n\nAlpha.");+ writeMd(vault, "b.md", "# B\n\nBeta.");+ const cfg = makeConfig({ vault, dbPath });+ const stats = await indexVault(cfg);+ expect(stats.added).toBe(2);+ const store = await Store.open(cfg, { mode: "write" });+ expect(store.listDocuments().size).toBe(2);+ await store.close();+});++test("aborting between embed batches stops further embedding but keeps committed vectors", async () => {+ if (!sqliteVecLoadable()) return;+ const server: FakeHttp = await startFakeHttp();+ try {+ for (let i = 0; i < 6; i++) writeMd(vault, `e${i}.md`, `# E${i}\n\nembed body number ${i}`);+ const cfg = makeConfig({+ vault,+ dbPath,+ semantic: {+ enabled: true,+ provider: "openai-compat",+ baseUrl: server.url,+ model: "fake-model",+ apiKey: "test-key",+ dimension: 4,+ timeoutMs: 5_000,+ concurrency: 1,+ batchSize: 1,+ costGateUsd: 0,+ },+ });+ // Index documents first (no embeddings), so the abort isolates the+ // embed phase: each batch is one chunk -> one HTTP call.+ await indexVault(cfg);++ const ac = new AbortController();+ // Serve the first batch normally, then abort - the populateEmbeddings+ // loop trips at the NEXT batch boundary, after batch 0 is committed.+ server.setHandler((req, callIndex) => {+ const body = (req.body ?? {}) as { input?: string[]; model?: string };+ const inputs = Array.isArray(body.input) ? body.input : [];+ const data = inputs.map((text, index) => ({+ object: "embedding",+ embedding: [text.length, index, 1, 1],+ index,+ }));+ if (callIndex === 0) ac.abort();+ return { status: 200, body: { data, model: body.model ?? "fake-model" } };+ });++ await expect(indexVault(cfg, { embeddings: true, signal: ac.signal })).rejects.toBeInstanceOf(+ SafeguardAbortError,+ );++ // The first batch committed; the rest did not. Partial, not all-or-nothing.+ const store = await Store.open(cfg, { mode: "write" });+ const counts = store.counts();+ await store.close();+ expect(counts.embeddings).toBeGreaterThanOrEqual(1);+ expect(counts.embeddings).toBeLessThan(counts.chunks);+ } finally {+ await server.close();+ }+});diff --git a/tests/core/search/multi-instance.test.ts b/tests/core/search/multi-instance.test.tsnew file mode 100644index 00000000..3eb29ef5--- /dev/null+++ b/tests/core/search/multi-instance.test.ts@@ -0,0 +1,77 @@+/**+ * Multi-instance isolation + writer-lock heartbeat (Indexer Durability+ * suite, t_ea80ddb5 / t_672c751e). The honest OSB reading of upstream's+ * "multi-instance" support: the MCP server is stdio-only, so there is+ * no port to multiplex. Isolation comes from the per-dbPath writer+ * lock - two instances on DIFFERENT vaults run conflict-free, and a+ * second writer on the SAME vault gets a typed INDEX_LOCKED rather than+ * corrupting the index. No --port / --instance daemon is fabricated.+ *+ * The heartbeat constants are pinned here so a future edit cannot+ * silently drop the lock refresh below the stale window.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";++import {+ Store,+ WRITER_LOCK_HEARTBEAT_MS,+ WRITER_LOCK_STALE_MS,+} from "../../../src/core/search/store.ts";+import { SearchError } from "../../../src/core/search/types.ts";+import { createTempVault, makeConfig } from "../../helpers/search-fixtures.ts";++const vaults: Array<() => void> = [];++beforeEach(() => {+ vaults.length = 0;+});++afterEach(() => {+ for (const c of vaults) c();+});++function freshConfig(prefix: string) {+ const v = createTempVault(prefix);+ vaults.push(v.cleanup);+ return makeConfig({ vault: v.vault, dbPath: v.dbPath });+}++test("the heartbeat refreshes well within the stale window", () => {+ expect(WRITER_LOCK_HEARTBEAT_MS).toBeGreaterThan(0);+ expect(WRITER_LOCK_HEARTBEAT_MS).toBeLessThan(WRITER_LOCK_STALE_MS);+});++test("writers on different vaults coexist", async () => {+ const a = freshConfig("multi-a");+ const b = freshConfig("multi-b");+ const sa = await Store.open(a, { mode: "write" });+ const sb = await Store.open(b, { mode: "write" });+ expect(sa).toBeInstanceOf(Store);+ expect(sb).toBeInstanceOf(Store);+ await sa.close();+ await sb.close();+});++test("a second writer on the same vault gets INDEX_LOCKED", async () => {+ const cfg = freshConfig("multi-same");+ const first = await Store.open(cfg, { mode: "write" });+ try {+ await Store.open(cfg, { mode: "write" });+ throw new Error("expected the second writer to be rejected");+ } catch (e) {+ expect(e).toBeInstanceOf(SearchError);+ expect(e).toHaveProperty("code", "INDEX_LOCKED");+ } finally {+ await first.close();+ }+});++test("the lock is released on close: a later writer succeeds", async () => {+ const cfg = freshConfig("multi-reopen");+ const first = await Store.open(cfg, { mode: "write" });+ await first.close();+ const second = await Store.open(cfg, { mode: "write" });+ expect(second).toBeInstanceOf(Store);+ await second.close();+});diff --git a/tests/core/search/reindex-resume.test.ts b/tests/core/search/reindex-resume.test.tsnew file mode 100644index 00000000..cbca7957--- /dev/null+++ b/tests/core/search/reindex-resume.test.ts@@ -0,0 +1,116 @@+/**+ * Opt-in resumable staged reindex (Indexer Durability suite,+ * t_672c751e). reindexVault builds into brain.sqlite.new then atomically+ * swaps. Today an interrupted rebuild discards all progress. With+ * search_resume_reindex on, a compatible in-progress .new is resumed+ * via the incremental fastpath instead of rebuilt from scratch; a+ * signature mismatch (schema / chunk params / embedding signature)+ * discards it. The signature marker lives in the staging DB's+ * index_state KV (no schema migration) and is cleared before the swap,+ * so the live index never carries staging state. Flag off keeps the+ * always-fresh rebuild byte-for-byte.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";+import { existsSync, mkdirSync, writeFileSync } from "node:fs";+import { dirname } from "node:path";++import { SafeguardAbortError } from "../../../src/core/brain/safeguard.ts";+import { indexVault, reindexVault } from "../../../src/core/search/indexer.ts";+import { Store } from "../../../src/core/search/store.ts";+import type { ResolvedSearchConfig } from "../../../src/core/search/types.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";++let vault: string;+let dbPath: string;+let cleanup: () => void;++beforeEach(() => {+ const v = createTempVault("reindex-resume");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+ for (let i = 0; i < 6; i++) writeMd(vault, `n${i}.md`, `# N${i}\n\nbody number ${i}`);+});++afterEach(() => {+ cleanup();+});++function cfgWith(overrides: Partial<ResolvedSearchConfig>): ResolvedSearchConfig {+ return Object.freeze({ ...makeConfig({ vault, dbPath }), ...overrides });+}++async function docCount(cfg: ResolvedSearchConfig): Promise<number> {+ const store = await Store.open(cfg, { mode: "write" });+ const n = store.listDocuments().size;+ await store.close();+ return n;+}++/** Drive a reindex that aborts after the first committed file, leaving+ * a partial staging .new behind. */+async function partialReindex(cfg: ResolvedSearchConfig): Promise<void> {+ const ac = new AbortController();+ let processed = 0;+ await expect(+ reindexVault(cfg, {+ signal: ac.signal,+ onFile: () => {+ if (++processed === 1) ac.abort();+ },+ }),+ ).rejects.toBeInstanceOf(SafeguardAbortError);+ expect(existsSync(`${dbPath}.new`)).toBe(true);+}++test("flag off: an interrupted rebuild leaves no resumable state and a clean rebuild succeeds", async () => {+ const cfg = cfgWith({ resumeReindex: false });+ // A stray .new from a prior crash must be discarded, not trusted.+ mkdirSync(dirname(dbPath), { recursive: true });+ writeFileSync(`${dbPath}.new`, "garbage-not-a-sqlite-db");+ const stats = await reindexVault(cfg);+ expect(stats.added).toBe(6);+ expect(await docCount(cfg)).toBe(6);+});++test("flag on: resume completes the partial build without reprocessing committed docs", async () => {+ const cfg = cfgWith({ resumeReindex: true });+ await partialReindex(cfg);++ // Resume: the fastpath skips the already-committed doc(s) (counted as+ // unchanged), so a from-scratch rebuild (all 'added') is ruled out.+ const stats = await reindexVault(cfg);+ expect(stats.unchanged).toBeGreaterThanOrEqual(1);+ expect(stats.added + stats.unchanged).toBe(6);+ expect(await docCount(cfg)).toBe(6);+ // Staging is gone after the swap.+ expect(existsSync(`${dbPath}.new`)).toBe(false);+});++test("flag on: a signature mismatch discards the staging build and rebuilds fresh", async () => {+ await partialReindex(cfgWith({ resumeReindex: true }));+ // Change a chunk parameter: the staging signature no longer matches.+ const drifted = cfgWith({ resumeReindex: true, chunkSize: 64, chunkOverlap: 8 });+ const stats = await reindexVault(drifted);+ // No resume: every file is processed fresh.+ expect(stats.added).toBe(6);+ expect(stats.unchanged).toBe(0);+ expect(await docCount(drifted)).toBe(6);+});++test("the swapped live index carries no staging signature marker", async () => {+ const cfg = cfgWith({ resumeReindex: true });+ await reindexVault(cfg);+ const store = await Store.open(cfg, { mode: "write" });+ expect(store.getState("reindex_signature")).toBeNull();+ await store.close();+});++test("flag on with no prior staging behaves like a normal full reindex", async () => {+ const cfg = cfgWith({ resumeReindex: true });+ await indexVault(cfg); // seed a live index first+ const stats = await reindexVault(cfg);+ expect(stats.added).toBe(6);+ expect(await docCount(cfg)).toBe(6);+});diff --git a/tests/core/search/store-exit.test.ts b/tests/core/search/store-exit.test.tsnew file mode 100644index 00000000..9ed24072--- /dev/null+++ b/tests/core/search/store-exit.test.ts@@ -0,0 +1,77 @@+/**+ * Writer WAL-flush-on-exit registry (Indexer Durability suite,+ * t_672c751e). Mirrors sync-lockfile's process-exit hook: open writer+ * DB handles register themselves, and a synchronous best-effort+ * wal_checkpoint(TRUNCATE) runs for each on process exit / process.exit+ * paths where close() was bypassed. SQLite already replays an orphan+ * -wal on the next open, so this is a belt-and-suspenders consolidation+ * that must never throw out of the exit hook.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";+import { Database } from "bun:sqlite";+import { mkdtempSync, rmSync, statSync, existsSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ flushRegisteredWriters,+ registerWriterDb,+ unregisterWriterDb,+ _resetWriterRegistryForTests,+} from "../../../src/core/search/store-exit.ts";++let tmp: string;++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-store-exit-"));+ _resetWriterRegistryForTests();+});++afterEach(() => {+ _resetWriterRegistryForTests();+ rmSync(tmp, { recursive: true, force: true });+});++function walDb(name: string): { db: Database; path: string } {+ const path = join(tmp, name);+ const db = new Database(path);+ db.exec("PRAGMA journal_mode = WAL");+ db.exec("CREATE TABLE t(x INTEGER)");+ db.exec("INSERT INTO t(x) VALUES (1), (2), (3)");+ return { db, path };+}++test("flush consolidates the WAL of a registered writer", () => {+ const { db, path } = walDb("a.sqlite");+ registerWriterDb(db);+ // A populated WAL exists before the flush.+ expect(existsSync(`${path}-wal`)).toBe(true);+ flushRegisteredWriters();+ // TRUNCATE checkpoint leaves the -wal at zero bytes (or removed).+ if (existsSync(`${path}-wal`)) {+ expect(statSync(`${path}-wal`).size).toBe(0);+ }+ db.close();+});++test("an unregistered writer is left untouched by the flush", () => {+ const { db, path } = walDb("b.sqlite");+ registerWriterDb(db);+ unregisterWriterDb(db);+ expect(existsSync(`${path}-wal`)).toBe(true);+ const before = statSync(`${path}-wal`).size;+ expect(before).toBeGreaterThan(0);+ flushRegisteredWriters();+ // The flush must NOT consolidate a DB that unregistered: its WAL is+ // still populated (no checkpoint ran against it).+ expect(statSync(`${path}-wal`).size).toBe(before);+ db.close();+});++test("the flush never throws on an already-closed handle", () => {+ const { db } = walDb("c.sqlite");+ registerWriterDb(db);+ db.close();+ expect(() => flushRegisteredWriters()).not.toThrow();+});diff --git a/tests/core/search/store.test.ts b/tests/core/search/store.test.tsindex 9820df48..0565dbb4 100644--- a/tests/core/search/store.test.ts+++ b/tests/core/search/store.test.ts@@ -58,6 +58,8 @@ function makeConfig(overrides?: Partial<ResolvedSearchConfig>): ResolvedSearchCopoolMultiplier: 3,selfTuningEnabled: false,}),+ shutdownGraceMs: 5_000,+ resumeReindex: false,...overrides,});}diff --git a/tests/core/search/store.vec.test.ts b/tests/core/search/store.vec.test.tsindex 8f0e45bc..79dfbeea 100644--- a/tests/core/search/store.vec.test.ts+++ b/tests/core/search/store.vec.test.ts@@ -61,6 +61,8 @@ function semanticConfig(poolMultiplier: 3,selfTuningEnabled: false,}),+ shutdownGraceMs: 5_000,+ resumeReindex: false,...overrides,});}diff --git a/tests/core/search/watch-runner.test.ts b/tests/core/search/watch-runner.test.tsnew file mode 100644index 00000000..0dcddb39--- /dev/null+++ b/tests/core/search/watch-runner.test.ts@@ -0,0 +1,104 @@+/**+ * IndexWatchRunner (Indexer Durability suite, t_ea80ddb5): the testable+ * core of `o2b search watch`'s flush + shutdown coordination. Single-+ * flight flushes; a shutdown aborts the in-flight pass and awaits it to+ * settle at a cooperative boundary, bounded by a grace window, so a+ * SIGTERM never kills a run mid-write. Clock-free and signal-free here -+ * the CLI wires fs.watch and the OS signals to it.+ */++import { expect, test } from "bun:test";++import { IndexWatchRunner } from "../../../src/core/search/watch-runner.ts";++const delay = (ms: number): Promise<void> => new Promise((r) => setTimeout(r, ms));++test("shutdown awaits an in-flight flush to completion within the grace window", async () => {+ let done = false;+ const runner = new IndexWatchRunner({+ graceMs: 5_000,+ index: async () => {+ await delay(5);+ done = true;+ },+ });+ void runner.flush();+ await runner.shutdown();+ expect(done).toBe(true);+ expect(runner.isStopped).toBe(true);+});++test("shutdown aborts the in-flight pass at its cooperative boundary", async () => {+ let abortedSeen = false;+ let release = (): void => {};+ const gate = new Promise<void>((r) => {+ release = r;+ });+ const runner = new IndexWatchRunner({+ graceMs: 5_000,+ index: async (signal) => {+ signal.addEventListener("abort", () => {+ abortedSeen = true;+ release();+ });+ await gate;+ },+ });+ void runner.flush();+ await Promise.resolve();+ await runner.shutdown();+ expect(abortedSeen).toBe(true);+});++test("a flush requested after shutdown is refused", async () => {+ let calls = 0;+ const runner = new IndexWatchRunner({+ graceMs: 0,+ index: async () => {+ calls++;+ },+ });+ await runner.shutdown();+ await runner.flush();+ expect(calls).toBe(0);+ expect(runner.isStopped).toBe(true);+});++test("a flush that outlasts the grace window still lets shutdown return", async () => {+ let waited = -1;+ const runner = new IndexWatchRunner({+ graceMs: 5_000,+ index: () => new Promise<void>(() => {}), // never resolves+ graceWaiter: (ms) => {+ waited = ms;+ return Promise.resolve();+ },+ });+ void runner.flush();+ await Promise.resolve();+ await runner.shutdown();+ expect(waited).toBe(5_000);+ // The pass is still pending; shutdown returned via the grace waiter.+ expect(runner.isFlushing).toBe(true);+});++test("concurrent flush calls coalesce into a single pass", async () => {+ let calls = 0;+ let release = (): void => {};+ const gate = new Promise<void>((r) => {+ release = r;+ });+ const runner = new IndexWatchRunner({+ graceMs: 0,+ index: async () => {+ calls++;+ await gate;+ },+ });+ const a = runner.flush();+ const b = runner.flush();+ expect(runner.isFlushing).toBe(true);+ release();+ await Promise.all([a, b]);+ expect(calls).toBe(1);+});diff --git a/tests/helpers/search-fixtures.ts b/tests/helpers/search-fixtures.tsindex a3b5fa18..b4adad9c 100644--- a/tests/helpers/search-fixtures.ts+++ b/tests/helpers/search-fixtures.ts@@ -127,5 +127,7 @@ export function makeConfig(opts: {poolMultiplier: 3,selfTuningEnabled: opts.selfTuningEnabled ?? false,}),+ shutdownGraceMs: 5_000,+ resumeReindex: false,});}