Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript

feat: Indexer Durability & Resilience Suite - cooperative abort, graceful watch shutdown, resumable reindex (v1.8.0) (#97)

Sol Aitken7cdbfc036ef1
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json
index 000c12b5..10bb18e6 100644
--- a/.claude-plugin/plugin.json
+++ b/.claude-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.7.0",
+ "version": "1.8.0",
"description": "Plugin-first second brain package for AI agents and humans.",
"author": {
"name": "Open Second Brain contributors"
diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json
index 73378d78..7c89f9dc 100644
--- a/.codex-plugin/plugin.json
+++ b/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.7.0",
+ "version": "1.8.0",
"description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
"skills": "./skills",
"hooks": "./hooks/hooks.json",
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7b5e1e47..de610a4e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,50 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+## [1.8.0] - 2026-06-13
+
+### Added
+
+- **Indexer Durability & Resilience Suite.** Interrupting a long index run no
+ longer risks losing work or wedging the index. The suite makes cancellation
+ cooperative, the watcher shutdown graceful, and a full rebuild resumable -
+ reusing the existing `Safeguard` and per-vault lock rather than adding a new
+ lifecycle subsystem. Every behavioural change defaults to today's behaviour;
+ a vault that sets nothing is byte-identical.
+ - **Cooperative abort (`Safeguard` + `AbortSignal`).** The cooperative
+ deadline gains an optional `AbortSignal`: one `checkpoint()` now trips on
+ either an aborted signal (a new `SafeguardAbortError`, checked first) or the
+ existing timeout. The signal threads through `indexVault` and
+ `populateEmbeddings`, checked at the same boundaries the deadline uses -
+ between files and between embed batches, never mid-write - so a run can be
+ cancelled on demand. Bun's SQLite is synchronous, so abort stays
+ cooperative; the deletion sweep runs only on full completion, so an aborted
+ run leaves a consistent, partially-refreshed index.
+ - **Graceful `o2b search watch` shutdown.** SIGINT/SIGTERM now stops accepting
+ new flushes, aborts the in-flight pass, and awaits it to settle at a
+ cooperative boundary before exiting - bounded by `search_shutdown_grace_seconds`
+ (default 5). A signal no longer kills a pass mid-write. The flush/shutdown
+ coordination lives in a testable `IndexWatchRunner`; a second signal falls
+ back to the default terminate. `0` grace exits immediately after signalling.
+ - **Opt-in resumable reindex (`search_resume_reindex`).** An interrupted full
+ `reindexVault` rebuild no longer discards all progress: a compatible
+ in-progress `brain.sqlite.new` staging build is resumed via the incremental
+ fastpath instead of rebuilt from scratch. Resume is gated on a signature
+ marker (schema version + chunk parameters + embedding signature) stored in
+ the staging DB's `index_state` KV - no schema migration - so a drifted or
+ unreadable staging DB is discarded and rebuilt, never trusted. The marker is
+ cleared before the atomic swap, so the live index never carries staging
+ state. Default off keeps the always-fresh rebuild.
+ - **Writer-lock heartbeat + WAL-flush-on-exit.** The async writer lock refreshes
+ its mtime mid-run (explicit heartbeat below the 60s stale window) so a long
+ index is never mistaken for a stale lock. A process-exit registry consolidates
+ each open writer's WAL on a bypassed `close()`, mirroring the existing
+ sync-lock cleanup hook.
+- Honest multi-instance story (no new daemon): the MCP server is stdio-only, so
+ isolation comes from the per-`dbPath` writer lock. Two instances on different
+ vaults run conflict-free; a second writer on the same vault gets a typed
+ `INDEX_LOCKED`. Pinned by tests; no `--port`/`--instance` model was fabricated.
+
## [1.7.0] - 2026-06-13
### Added
@@ -5344,6 +5388,7 @@ plugin config (vault field)`, and exits with a clear
- Sandbox vault and plugin manifest fixtures for tests.
- GitHub release workflow for tag-based and manually dispatched releases.
+[1.8.0]: https://github.com/itechmeat/open-second-brain/compare/v1.7.0...v1.8.0
[1.7.0]: https://github.com/itechmeat/open-second-brain/compare/v1.6.0...v1.7.0
[1.6.0]: https://github.com/itechmeat/open-second-brain/compare/v1.5.0...v1.6.0
[1.5.0]: https://github.com/itechmeat/open-second-brain/compare/v1.4.0...v1.5.0
diff --git a/README.md b/README.md
index ec408437..46fc5196 100644
--- a/README.md
+++ b/README.md
@@ -87,6 +87,7 @@ Each non-Hermes target writes a sidecar manifest under `<vault>/.open-second-bra
- **Memory that cleans itself, on your terms.** `o2b brain hygiene scan` surfaces contested facts, near-duplicate rules, stale derived pages, and never-recalled memories; `apply` executes only the findings you select, and stale pages recompile from their recorded sources with a dry-run preview.
- **A vault that stays fresh, consistent, and scoped.** `o2b search watch` keeps the index live as you edit, debounced and incremental; note identity is Unicode-normalized so the same file is one entry across macOS and Linux devices instead of a phantom cross-device duplicate; and recall accepts an opt-in `agent_scope` so a page marked with an `owner:` is reachable only to its owner while shared pages stay open to all.
- **Knowledge that knows where it came from.** Drop a source document and the agent's extraction becomes cross-referenced entity and concept pages plus a summary page that backlinks the source and lists its connections; N sources become one dated report whose every finding cites the source that flagged it; a derived fact carries a `deduced`/`inferred` provenance level and links back to its premises, and recall trusts an operator-stated rule above a machine-derived one. A fact can declare an `owner:` so multi-agent brains keep separate truth spaces, and a standing-query attention flow surfaces the open loops you declare. The plugin never runs a model itself - the agent owns generation, the vault owns the durable, provenanced record - and every behaviour is opt-in.
+- **An index that survives interruption.** Stopping `o2b search watch` mid-sync finishes the in-flight pass at a file boundary before exiting (within `search_shutdown_grace_seconds`) instead of killing it mid-write; an incremental run is already resumable through the unchanged-file fastpath, and `search_resume_reindex` extends that to a full rebuild - an interrupted reindex resumes its staging build instead of starting over, guarded by a signature so a drifted build is discarded rather than trusted. Both default off, so behaviour is unchanged unless you ask for it.
That is the day-to-day picture. The full capability surface, every CLI verb, and the mental model live in the [documentation](#documentation) below.
diff --git a/docs/brainstorm/indexer-durability/cli-output/claude.md b/docs/brainstorm/indexer-durability/cli-output/claude.md
new file mode 100644
index 00000000..8c2f4d05
--- /dev/null
+++ b/docs/brainstorm/indexer-durability/cli-output/claude.md
@@ -0,0 +1,35 @@
+### Variant 1: Fix the real gaps in place (distributed wiring)
+- **Approach**: Extend `Safeguard` with an optional `AbortSignal` so `checkpoint()` throws either the existing `SafeguardTimeoutError` or a new typed `SafeguardAbortError` - one cooperative primitive, two trip conditions. Fix the `o2b search watch` signal handler to await the in-flight `indexVault` pass within a configurable grace window, then fire the abort to unwind cleanly; guarantee a WAL-consolidating, lock-releasing close on every exit path by routing through the existing `sync-lockfile` process-exit hook. Add a lock-heartbeat that refreshes the proper-lockfile mtime during long runs (so a genuine long index never looks stale at 60s) and treat lock-busy as a skip, plus honest attempt accounting in the embed retry path (no retry for the non-idempotent batch commit). `reindexVault` stays non-resumable but becomes abort-honoring, leaving the original DB intact via the temp file as today.
+- **Trade-offs**:
+ - Pro: smallest surface that closes every OSB-applicable gap across all three tasks; reuses `Safeguard`, `lock.ts`, `sync-lockfile.ts` (DRY, no new abstraction).
+ - Pro: no new schema/table, so it cannot become a "redundant resumable mechanism" for the already-resumable incremental path; byte-identical when grace window/heartbeat flags are off.
+ - Pro: fixes the genuine watch-handler-kills-mid-flush bug and the orphan `-wal`/held-lock-on-SIGTERM bug directly.
+ - Con: leaves `reindexVault` restarting full rebuilds from scratch - the one genuinely non-resumable path is unaddressed.
+ - Con: abort/grace/heartbeat concerns are wired at several call sites rather than one object, so the lifecycle is spread out and easy to forget to thread on a future code path.
+- **Complexity**: medium
+- **Risk**: low
+
+### Variant 2: Resumable staged reindex (carry-forward temp DB)
+- **Approach**: Build on Variant 1's shutdown/abort/heartbeat core, then make the genuinely non-resumable `reindexVault` resumable by treating `brain.sqlite.new` as a durable staging DB instead of discarding it on interruption. On the next reindex, detect a compatible in-progress staging build (matching `LATEST_SCHEMA_VERSION` plus a vault fingerprint) and resume it - reusing the existing `indexInto` mtime/size fastpath and `findChunksWithoutEmbeddings()` logic on the temp file - before the unchanged atomic same-file rename swap. A small staging marker records fingerprint and schema version so a stale or schema-drifted temp build is invalidated and rebuilt rather than silently resumed.
+- **Trade-offs**:
+ - Pro: addresses the only genuinely non-resumable path honestly, without duplicating the incremental path's resume logic (it reuses it on the temp DB).
+ - Pro: atomic swap and `.bak` crash-recovery window are untouched; gated behind a flag that defaults to today's discard-on-interrupt behaviour.
+ - Con: the staging marker is a quasi-checkpoint that must be rigorously invalidated; a subtle fingerprint bug could resume onto a stale build and produce a wrong index - the highest-consequence failure mode in the suite.
+ - Con: significantly larger validation surface (schema drift, vault mutation mid-build, partial-batch embed state) for a path that runs rarely.
+ - Con: closest of the three to brushing against the "no misleading/duplicate mechanism" rule, so it needs the most justification.
+- **Complexity**: large
+- **Risk**: medium-high
+
+### Variant 3: Centralized RunController supervisor
+- **Approach**: Introduce one queue-scoped supervisor object (the OSB-faithful reading of upstream's "DB supervisor singleton") that owns the entire long-run lifecycle: an `AbortController` composed into `Safeguard`, the signal handlers with grace window and in-flight await, the SIGTERM-safe WAL-consolidate-and-release close, the lock heartbeat with lock-busy-as-skip, and honest attempt accounting with per-handler wall-clock budgets. Every long-running operation (`indexVault`, `populateEmbeddings`, `reindexVault`, the `watch` loop) receives this one controller instead of threading deadline, signal, and lock concerns separately. `reindexVault` stays non-resumable but abort-honoring.
+- **Trade-offs**:
+ - Pro: single composition point makes abort/shutdown/heartbeat/accounting impossible to forget on new code paths; most faithful adaptation of all three upstream intents to OSB's actual architecture.
+ - Pro: centralizes honest accounting and budgets, which are otherwise scattered.
+ - Con: a new singleton lifecycle abstraction is heavy for a codebase whose only long-running process is `watch`; real risk of over-engineering relative to the actual surface.
+ - Con: a singleton invites accidentally re-importing multi-instance/daemon framing that OSB's stdio + per-vault locking explicitly forbids.
+ - Con: larger refactor touching every long-op call site, raising regression risk against the "byte-identical when off" guarantee.
+- **Complexity**: large
+- **Risk**: medium
+
+### Recommended: Variant 1
+**Rationale**: It closes every gap that actually exists in the OSB codebase - the watch-handler mid-flush bug, the orphan-WAL/held-lock-on-SIGTERM bug, the missing on-demand abort, the stale-lock-during-long-run, and dishonest retry - across all three kanban tasks, while honoring the hard constraints: it composes with `Safeguard` rather than replacing it, adds no table that would duplicate the already-resumable incremental path, and stays byte-identical when its flags are off. Variant 2's staged-reindex resume carries the suite's only wrong-index failure mode for a rarely-run path and can be added later if reindex restart cost becomes real; Variant 3's supervisor is the cleanest design in the abstract but over-engineers a lifecycle abstraction for a tool with a single long-running process and risks smuggling back the daemon framing the grounding facts forbid.
diff --git a/docs/brainstorm/indexer-durability/cli-output/prompt.md b/docs/brainstorm/indexer-durability/cli-output/prompt.md
new file mode 100644
index 00000000..c5045ec2
--- /dev/null
+++ b/docs/brainstorm/indexer-durability/cli-output/prompt.md
@@ -0,0 +1,54 @@
+You are brainstorming architectural variants for the following task. Do not write code. Do not write a final design. Only produce variants and a recommendation.
+
+# Task
+
+Build an "Indexer Durability & Resilience" suite for Open Second Brain (OSB), a TypeScript/Bun knowledge-vault tool whose search index is a SQLite database (FTS5 + sqlite-vec) derived from a Markdown vault. The suite bundles three kanban tasks that the board's validator clustered as the "indexer-durability family":
+
+1. t_ea80ddb5 - Graceful stop/restart with index flush and multi-instance support. Upstream (agentmemory) inverted stop order: worker first with a SIGTERM grace window for index flush, engine second; added a --instance N flag for multi-daemon support with --port. The OSB-side concern: an operator stopping OSB mid-index (crash, SIGTERM, timeout) can lose uncommitted changes; and running multiple instances on one host.
+
+2. t_672c751e - Resumable durable append-only sync checkpoints with lock heartbeat. Upstream (gbrain) added an append-only checkpoint delta table, guaranteed final flush on every exit path including SIGTERM, lock heartbeat-aware takeover, and treated lock-busy as a skip rather than a phase failure. The OSB-side concern: if a large index run is interrupted it restarts from scratch.
+
+3. t_79e773be - Cooperative abort-honoring with honest attempt accounting. Upstream (gbrain) threaded an AbortSignal through embed phases, checking on both the stale and full paths and between batches; added honest attempt accounting (no retry for non-idempotent work), per-handler wall-clock budgets, and a queue-scoped DB supervisor singleton. The OSB-side concern: long embed jobs can wedge a cycle with no way to abort cooperatively.
+
+# Critical grounding facts about the ACTUAL OSB codebase (verified in code, must shape the design)
+
+- The INCREMENTAL index path (`indexVault`) is ALREADY resumable. `indexInto` walks the vault and skips unchanged files via an mtime+size fastpath (re-running after an interruption skips everything already committed). `populateEmbeddings` only processes `findChunksWithoutEmbeddings()` and commits each batch before the next, so an interrupted embed phase resumes by computing only the still-missing embeddings. => A new resumable-checkpoint table for the INCREMENTAL path would be REDUNDANT and is forbidden by project rules (no misleading or duplicate mechanisms).
+- The genuinely NON-resumable path is `reindexVault`: a full rebuild (force=true) that writes a fresh temp DB (`brain.sqlite.new`), then does an atomic same-file rename swap with `.bak` retention. An interruption discards the whole temp build; there is no intermediate resume. Crash-recovery for the rename window already exists (Store.open restores `.bak` if the main file is missing).
+- There is already a cooperative deadline primitive: `Safeguard` (src/core/brain/safeguard.ts). An operation receives a `Safeguard` and calls `checkpoint()` at iteration boundaries (per file, per embed batch); past the deadline the next checkpoint throws `SafeguardTimeoutError`. Bun runs SQLite synchronously, so preemptive cancellation is impossible - the honest contract is cooperative. The gap: Safeguard is a TIME deadline only; there is no on-demand AbortSignal to cancel a run that is still within its time budget (e.g. from a SIGTERM in the watcher, or a parent cancellation).
+- The only long-running process is `o2b search watch` (src/cli/search.ts): a debounced fs.watch loop that calls `indexVault`. Its SIGINT/SIGTERM handler currently resolves immediately WITHOUT awaiting an in-flight `indexVault` pass - so a signal during a flush kills the run mid-write. This is a real bug.
+- The MCP server is stdio-only (no HTTP, no port). Multiple OSB instances against DIFFERENT vaults already work: the writer path in `Store.open` acquires a per-dbPath `proper-lockfile` lock (stale 60s) and a second writer on the same vault gets a typed `INDEX_LOCKED` error. => A `--port`/`--instance` daemon model from the upstream is NOT applicable to OSB's architecture and must NOT be fabricated.
+- The SQLite store uses WAL; `Store.close()` consolidates the WAL (`PRAGMA wal_checkpoint(TRUNCATE)` then `journal_mode=DELETE`) and releases the lock, but only on an orderly close - a SIGTERM that bypasses close leaves an orphan `-wal` and a held lock.
+- Schema migrations live in src/core/search/schema.ts: `LATEST_SCHEMA_VERSION` (currently 7) plus an ordered `MIGRATIONS` array of `{version, up}`. Bumping the version auto-migrates on the next write-mode open.
+- Existing reusable primitives: `src/core/reliability/lock.ts` (async proper-lockfile wrapper), `src/core/brain/sync-lockfile.ts` (sync lock with a process-exit cleanup hook that unlinks held locks).
+
+# Project context
+
+- Open Second Brain - TypeScript, Bun runtime, SQLite (bun:sqlite), MCP server, `o2b` CLI, Obsidian/Markdown vault under `Brain/`.
+- Recent suites shipped as cohesive versioned releases (v1.4 Grok, v1.5 Search & Recall Quality, v1.6 Vault Integrity & Trust, v1.7 Knowledge Provenance).
+- Conventions: opt-in guardrail flags in config so behaviour is byte-identical when a feature is off; DRY shared primitives; strict TypeScript with NO `as` cast crutches (build values with the correct type via conditional spreads / narrowing guards); language-agnostic (no hardcoded natural-language word lists); provider-agnostic kernel (OSB never calls an LLM itself); no new heavy or ML dependencies; no misleading fallbacks (a feature that would silently no-op when its substrate is absent is forbidden - fail loud and typed instead).
+
+# Constraints
+
+- Do NOT add a redundant resumable-checkpoint mechanism for the already-resumable incremental path.
+- Do NOT invent a --port/--instance daemon model; OSB is stdio and per-vault locking already isolates instances.
+- No new external dependencies; reuse proper-lockfile and bun:sqlite.
+- Cancellation must be cooperative (Bun SQLite is synchronous); never claim preemptive abort.
+- The abort/deadline mechanism should compose with the existing `Safeguard`, not replace it (DRY).
+- Resumable behaviour and graceful-shutdown grace windows should be controllable and default to current behaviour where they would otherwise change output shape.
+
+# Required output format
+
+Produce exactly 3 distinct architectural variants. For each variant:
+
+### Variant N: <short name>
+- **Approach**: 2-3 sentences describing the variant.
+- **Trade-offs**: bullet list of pros and cons.
+- **Complexity**: small | medium | large
+- **Risk**: low | medium | high
+
+After the three variants, add exactly one recommendation:
+
+### Recommended: Variant N
+**Rationale**: 2-3 sentences explaining why this variant over the others, considering the project context and constraints above.
+
+Output nothing outside of these sections.
diff --git a/docs/brainstorm/indexer-durability/design.md b/docs/brainstorm/indexer-durability/design.md
new file mode 100644
index 00000000..d40ddc32
--- /dev/null
+++ b/docs/brainstorm/indexer-durability/design.md
@@ -0,0 +1,156 @@
+# Indexer Durability & Resilience - making interrupted index runs safe and resumable
+
+**Status:** draft
+**Author:** claude-dev-agent (via feature-release-playbook)
+**Audience:** implementation
+
+## Problem statement
+
+Open Second Brain's search index is a SQLite database derived from the vault.
+Long index runs can be interrupted (SIGTERM, crash, deadline). Three real gaps
+exist today: (1) the `o2b search watch` shutdown handler resolves immediately
+and kills an in-flight `indexVault` pass mid-write; (2) the only way to stop a
+running index that is still within its time budget is to kill the process -
+there is no cooperative, on-demand abort; (3) a full `reindexVault` rebuild that
+is interrupted discards all progress and restarts from scratch. This suite
+closes those gaps without duplicating mechanisms that already work.
+
+## Grounding: what is already durable (do NOT rebuild)
+
+- The incremental `indexVault` path is already resumable. `indexInto` skips
+ unchanged files via an mtime+size fastpath, so a re-run after an interruption
+ re-walks and skips everything already committed. `populateEmbeddings` only
+ processes `findChunksWithoutEmbeddings()` and commits each batch, so an
+ interrupted embed phase resumes by computing only the still-missing vectors.
+ A new resumable-checkpoint table for this path would be redundant and is
+ explicitly out of scope.
+- The reindex rename window is already crash-safe: `Store.open` restores the
+ `.bak` if the main DB is missing.
+- A cooperative time deadline already exists: `Safeguard.checkpoint()` trips
+ past a budget at file / batch boundaries. We compose with it, not replace it.
+
+## Scope
+
+- A cooperative `AbortSignal`, composed into the existing `Safeguard`, threaded
+ through `indexVault` / `indexInto` / `populateEmbeddings` / `reindexVault`,
+ checked at the same boundaries the deadline already uses (between files,
+ between embed batches - never mid-write).
+- A graceful `o2b search watch` shutdown: on SIGINT/SIGTERM, stop accepting new
+ flushes, abort the in-flight pass, await it to settle at the next boundary
+ within a bounded grace window, then close the store (WAL consolidate + lock
+ release) and exit. A second signal forces immediate exit.
+- A best-effort synchronous WAL checkpoint on process exit for any open writer
+ store, so a bypassed `close()` does not leave an orphan `-wal` that trips
+ `SQLITE_IOERR_SHORT_READ` on the next open.
+- A lock heartbeat: the writer's `proper-lockfile` is refreshed during long runs
+ so a legitimate long index is never mistaken for a stale lock; lock-busy is
+ surfaced as a clean skip in the watch loop rather than crashing the watcher.
+- An opt-in resumable `reindexVault`: a compatible in-progress `brain.sqlite.new`
+ staging build is detected by a signature marker and resumed (via the existing
+ incremental fastpath over the temp DB) instead of being discarded.
+- Two config keys (`search_shutdown_grace_seconds`, `search_resume_reindex`)
+ with env mirrors, defaulting to today's behaviour.
+
+## Out of scope
+
+- No `--port` / `--instance` daemon model. OSB's MCP server is stdio-only;
+ multiple instances against different vaults already run conflict-free because
+ the writer path takes a per-`dbPath` lock, and a second writer on the same
+ vault already gets a typed `INDEX_LOCKED`. We verify and document this; we do
+ not fabricate a daemon. (The honest reading of t_ea80ddb5's "multi-instance".)
+- No resumable-checkpoint table for the incremental path (already resumable).
+- No "attempt accounting / dead-letter" job-queue accounting from the upstream:
+ the index is not a retrying job queue, so importing that vocabulary would be
+ misleading. The honest slice of t_79e773be is the cooperative AbortSignal.
+- No preemptive cancellation: Bun runs SQLite synchronously, so abort is
+ cooperative only. We never claim otherwise.
+
+## Chosen approach
+
+Variant 2 (see variants.md): the Variant 1 "fix the real gaps in place" core -
+abort-composed Safeguard, graceful watch shutdown, exit-path WAL flush, lock
+heartbeat - PLUS the genuinely non-resumable path made resumable: opt-in staged
+reindex resume. The CLI consultant recommended Variant 1 (skip resumable
+reindex) on risk grounds. We override, because:
+
+1. Without resumable reindex, the chosen task t_672c751e delivers almost nothing
+ new - everything else it asks for (resume, final flush) is either already
+ present (incremental resume) or is the same shutdown/flush work as t_ea80ddb5.
+2. The operator's brief for this cycle is an explicitly larger, substantive
+ change; the resumable reindex is the only honest, non-redundant home for real
+ volume here.
+3. The consultant's one serious objection - resuming onto a stale staging build
+ could yield a wrong index - is removed by design: resume is gated on a
+ signature marker (embedding signature + schema version + chunk params); on
+ any drift the staging DB is discarded and rebuilt; and resume itself is the
+ ordinary incremental walk over the temp DB (fastpath re-reads any changed
+ file, and the final deletion sweep removes vanished files), so it is
+ self-healing rather than a blind trust of partial state. The whole behaviour
+ is behind a flag that defaults to today's discard-on-interrupt.
+
+## Design decisions
+
+- **Compose, don't replace.** The abort lives inside `Safeguard`: an optional
+ `signal` makes `checkpoint()` throw a typed `SafeguardAbortError` when the
+ signal is aborted, in priority over the timeout. Callers that already hold a
+ Safeguard get abort for free; there is one trip point, not two.
+- **Throw, don't silently truncate.** An aborted run throws `SafeguardAbortError`
+ exactly as a timed-out run throws `SafeguardTimeoutError`. The deletion sweep
+ runs only after the walk completes, so a thrown abort leaves a consistent,
+ partially-refreshed index (identical to today's timeout behaviour). The watch
+ shutdown coordinator catches the abort and treats it as a clean stop.
+- **Reuse the KV table, no migration.** The reindex staging marker is a row in
+ the existing `index_state` KV table of the temp DB. No new table, no schema
+ version bump, so existing indices are untouched and reads never break with a
+ spurious `SCHEMA_MISMATCH`. Byte-identical when `search_resume_reindex` is off.
+- **Testable coordinator, thin CLI.** The watch shutdown logic (in-flight await,
+ grace window, refuse-after-stop) moves into a `IndexWatchRunner` with an
+ injected clock and index function, so it is unit-tested without real signals
+ or timers. The CLI wires `fs.watch` + signal handlers + store lifecycle to it.
+- **Exit-hook mirrors the existing pattern.** The sync WAL-checkpoint-on-exit
+ registry follows `sync-lockfile.ts`'s `process.on("exit", ...)` shape:
+ best-effort, synchronous (bun:sqlite is sync), never throws.
+- **Fail loud on lock contention.** Lock-busy stays a typed `INDEX_LOCKED`; the
+ watch loop treats it as a skip (report + keep watching) rather than crashing,
+ but a one-shot `o2b search index` still surfaces it. No silent no-op.
+
+## File changes
+
+New:
+- `src/core/search/watch-runner.ts` - testable watch flush/shutdown coordinator.
+- `src/core/search/store-exit.ts` - sync WAL-checkpoint-on-exit registry.
+- tests: `tests/core/brain/safeguard-abort.test.ts`,
+ `tests/core/search/indexer-abort.test.ts`,
+ `tests/core/search/watch-runner.test.ts`,
+ `tests/core/search/store-exit.test.ts`,
+ `tests/core/search/reindex-resume.test.ts`,
+ `tests/core/search/lock-heartbeat.test.ts`,
+ `tests/core/search/multi-instance.test.ts`,
+ plus a config-resolution test extension.
+
+Modified:
+- `src/core/brain/safeguard.ts` - optional `signal`, `SafeguardAbortError`,
+ `throwIfAborted` helper.
+- `src/core/search/indexer.ts` - thread `signal` through options; resumable
+ `reindexVault`; staging signature marker.
+- `src/core/search/store.ts` - proper-lockfile `update` heartbeat; register with
+ the exit-flush registry on write open / unregister on close.
+- `src/core/search/types.ts` - `shutdownGraceMs`, `resumeReindex` on
+ `ResolvedSearchConfig`.
+- `src/core/search/index.ts` - resolve the two new keys + env mirrors + DEFAULTS.
+- `src/cli/search.ts` - `cmdSearchWatch` uses `IndexWatchRunner`; graceful close.
+- `README.md`, `CHANGELOG.md`, `package.json` (+ manifest sync), config docs.
+
+## Risks and open questions
+
+- **Stale staging resume (highest consequence).** Mitigated by the signature
+ marker + self-healing incremental walk + opt-in flag (see Chosen approach).
+ Tests assert that signature drift discards the staging DB.
+- **Lock heartbeat interval.** Must be well under `stale` (e.g. stale/2) and must
+ not leak a timer when the store closes. The `update` option of proper-lockfile
+ handles the refresh; `close()`/release cancels it.
+- **Exit-hook reentrancy.** The sync checkpoint must be idempotent and tolerate
+ an already-closed DB; it never throws out of the exit handler.
+- **Grace window default.** 5 seconds (matches the upstream agentmemory grace and
+ is ample for a single-file boundary). Configurable; 0 means "abort and exit
+ immediately without awaiting", which we treat as a valid explicit choice.
diff --git a/docs/brainstorm/indexer-durability/plan.md b/docs/brainstorm/indexer-durability/plan.md
new file mode 100644
index 00000000..99e8cd2a
--- /dev/null
+++ b/docs/brainstorm/indexer-durability/plan.md
@@ -0,0 +1,105 @@
+# Indexer Durability & Resilience - implementation plan
+
+Each task is one atomic unit = one TDD cycle = one conventional commit on
+`feat/indexer-durability`. Order respects dependencies: the shared abort
+primitive first, then the consumers.
+
+## Task 1: Abort-composed Safeguard (shared primitive)
+- **Files**: `src/core/brain/safeguard.ts`; `tests/core/brain/safeguard-abort.test.ts`.
+- **Change**: add `SafeguardAbortError`; `CreateSafeguardOptions.signal?: AbortSignal`;
+ `checkpoint()` throws `SafeguardAbortError` when `signal?.aborted`, in priority
+ over the timeout check. `noopSafeguard` unchanged. Export a tiny
+ `throwIfAborted(signal?: AbortSignal, operation?: string)` helper. No `as`
+ casts; `signal` read with optional chaining.
+- **Acceptance**: aborted signal trips `checkpoint()` with `SafeguardAbortError`
+ even before the deadline; unset signal + live deadline behaves exactly as today;
+ a no-deadline guard with an aborted signal still trips.
+- **Depends on**: none.
+
+## Task 2: AbortSignal threaded through the indexer
+- **Files**: `src/core/search/indexer.ts`; `tests/core/search/indexer-abort.test.ts`.
+- **Change**: `IndexVaultOptions.signal?: AbortSignal`. In `indexInto` and
+ `populateEmbeddings`, after the existing `safeguard?.checkpoint()`, call
+ `throwIfAborted(opts?.signal)` at the same boundaries (between files, between
+ embed batches). The deletion sweep still runs only on full completion, so an
+ abort leaves a consistent partial index. `reindexVault` forwards `signal`.
+- **Acceptance**: a signal aborted before the loop throws `SafeguardAbortError`
+ with nothing partial deleted; aborting between embed batches stops further
+ embedding but keeps already-committed vectors; a run with no signal is
+ byte-identical to today.
+- **Depends on**: Task 1.
+
+## Task 3: Writer lock heartbeat + lock-busy skip
+- **Files**: `src/core/search/store.ts`; `tests/core/search/lock-heartbeat.test.ts`,
+ `tests/core/search/multi-instance.test.ts`.
+- **Change**: pass `update` (refresh interval = floor(stale/2)) to the
+ `proper-lockfile.lock` call in write mode so a long run keeps the lock fresh.
+ Confirm release cancels the refresh. `multi-instance.test.ts` asserts: two
+ writers on different `dbPath`s both succeed; a second writer on the same
+ `dbPath` gets `INDEX_LOCKED` (documents the honest multi-instance story).
+- **Acceptance**: lock options include a positive `update` below `stale`; same-vault
+ second writer is rejected with the typed error; different-vault writers coexist.
+- **Depends on**: none.
+
+## Task 4: Sync WAL-checkpoint-on-exit registry
+- **Files**: `src/core/search/store-exit.ts` (new); `src/core/search/store.ts`;
+ `tests/core/search/store-exit.test.ts`.
+- **Change**: registry of open writer `Database` handles. `Store.open(write)`
+ registers; `close()` unregisters. A `process.on("exit", ...)` hook (installed
+ once, mirroring `sync-lockfile.ts`) runs `PRAGMA wal_checkpoint(TRUNCATE)`
+ best-effort on each registered handle, swallowing errors. Idempotent; never
+ throws.
+- **Acceptance**: a registered handle is checkpointed by the hook function;
+ unregister removes it; running the hook with a closed handle is a no-op, not a
+ throw.
+- **Depends on**: none.
+
+## Task 5: Config keys (grace window + resume flag)
+- **Files**: `src/core/search/types.ts`, `src/core/search/index.ts`;
+ extend the existing search-config resolution test.
+- **Change**: `ResolvedSearchConfig.shutdownGraceMs: number` and
+ `resumeReindex: boolean`. DEFAULTS: `shutdownGraceMs` from 5s, `resumeReindex`
+ false. Resolve `search_shutdown_grace_seconds`
+ (env `OPEN_SECOND_BRAIN_SEARCH_SHUTDOWN_GRACE`, integer seconds >= 0, *1000)
+ and `search_resume_reindex` (env `OPEN_SECOND_BRAIN_SEARCH_RESUME_REINDEX`,
+ bool). Both threaded into the frozen `base` config.
+- **Acceptance**: defaults present when keys absent; values parse from config and
+ env; invalid values fail soft to default (matches existing parse helpers).
+- **Depends on**: none.
+
+## Task 6: Graceful watch shutdown (testable runner)
+- **Files**: `src/core/search/watch-runner.ts` (new); `src/cli/search.ts`;
+ `tests/core/search/watch-runner.test.ts`.
+- **Change**: `IndexWatchRunner` owns flush single-flight + shutdown. Injected
+ `now()`, `index()` function, and `AbortController` factory. `shutdown(graceMs)`
+ sets stopped, aborts the in-flight controller, awaits the running flush up to
+ `graceMs` (then returns even if still running - process exit follows), refuses
+ new flushes after stop. `cmdSearchWatch` refactors to wire `fs.watch` +
+ `SIGINT`/`SIGTERM` -> `runner.shutdown(cfg.shutdownGraceMs)` -> `store`/watcher
+ close. A second signal forces immediate `resolve`.
+- **Acceptance**: shutdown awaits an in-flight flush before resolving; if the
+ flush exceeds the grace window shutdown still returns; a flush requested after
+ stop is refused; a normal flush path is unchanged.
+- **Depends on**: Tasks 1, 2, 5.
+
+## Task 7: Opt-in resumable staged reindex
+- **Files**: `src/core/search/indexer.ts`; `tests/core/search/reindex-resume.test.ts`.
+- **Change**: in `reindexVault`, when `config.resumeReindex` is true, compute a
+ staging signature (`embeddingSignature` + `LATEST_SCHEMA_VERSION` + chunkSize +
+ chunkOverlap). If `brain.sqlite.new` exists and its `index_state`
+ `reindex_signature` matches, resume by building with `force: false` (incremental
+ fastpath completes the partial build); otherwise unlink the stale `.new` and
+ start fresh with `force: true`. Write the signature marker at build start;
+ delete it on success before the rename swap. When `resumeReindex` is false,
+ keep today's unconditional `tryUnlink(newPath)` + `force: true`.
+- **Acceptance**: with the flag on, a partial `.new` (some docs committed, marker
+ matching) is completed without reprocessing the committed docs and the final
+ index is identical to a fresh reindex; a marker signature mismatch discards the
+ `.new` and rebuilds; with the flag off, behaviour is byte-identical to today
+ (always fresh). The swapped main DB carries no `reindex_signature` row.
+- **Depends on**: Tasks 1, 2, 5.
+
+## Cross-cutting (done within the tasks above, no separate commit)
+- No new dependencies; reuse `proper-lockfile`, `bun:sqlite`, Node `AbortSignal`.
+- No `as` cast crutches; no natural-language word lists; provider-agnostic.
+- Version bump + CHANGELOG + README land in the docs commit (Phase 5), per CLAUDE.md.
diff --git a/docs/brainstorm/indexer-durability/variants.md b/docs/brainstorm/indexer-durability/variants.md
new file mode 100644
index 00000000..cefd90d7
--- /dev/null
+++ b/docs/brainstorm/indexer-durability/variants.md
@@ -0,0 +1,71 @@
+# Indexer Durability & Resilience - variant audit trail
+
+Primary consultant: Claude Code (`claude -p`), output captured verbatim in
+`cli-output/claude.md`. Fallback (Codex) not invoked - the primary returned
+three parseable variants and a recommendation. Below: the consultant's variants
+verbatim, then the orchestrator's final decision and rationale.
+
+---
+
+## Consultant variants (verbatim)
+
+### Variant 1: Fix the real gaps in place (distributed wiring)
+- **Approach**: Extend `Safeguard` with an optional `AbortSignal` so `checkpoint()` throws either the existing `SafeguardTimeoutError` or a new typed `SafeguardAbortError` - one cooperative primitive, two trip conditions. Fix the `o2b search watch` signal handler to await the in-flight `indexVault` pass within a configurable grace window, then fire the abort to unwind cleanly; guarantee a WAL-consolidating, lock-releasing close on every exit path by routing through the existing `sync-lockfile` process-exit hook. Add a lock-heartbeat that refreshes the proper-lockfile mtime during long runs (so a genuine long index never looks stale at 60s) and treat lock-busy as a skip, plus honest attempt accounting in the embed retry path (no retry for the non-idempotent batch commit). `reindexVault` stays non-resumable but becomes abort-honoring, leaving the original DB intact via the temp file as today.
+- **Trade-offs**: Pro: smallest surface that closes every OSB-applicable gap across all three tasks; reuses `Safeguard`, `lock.ts`, `sync-lockfile.ts` (DRY, no new abstraction). Pro: no new schema/table, so it cannot become a "redundant resumable mechanism" for the already-resumable incremental path; byte-identical when grace window/heartbeat flags are off. Pro: fixes the genuine watch-handler-kills-mid-flush bug and the orphan `-wal`/held-lock-on-SIGTERM bug directly. Con: leaves `reindexVault` restarting full rebuilds from scratch - the one genuinely non-resumable path is unaddressed. Con: abort/grace/heartbeat concerns are wired at several call sites rather than one object.
+- **Complexity**: medium
+- **Risk**: low
+
+### Variant 2: Resumable staged reindex (carry-forward temp DB)
+- **Approach**: Build on Variant 1's shutdown/abort/heartbeat core, then make the genuinely non-resumable `reindexVault` resumable by treating `brain.sqlite.new` as a durable staging DB instead of discarding it on interruption. On the next reindex, detect a compatible in-progress staging build (matching `LATEST_SCHEMA_VERSION` plus a vault fingerprint) and resume it - reusing the existing `indexInto` mtime/size fastpath and `findChunksWithoutEmbeddings()` logic on the temp file - before the unchanged atomic same-file rename swap. A small staging marker records fingerprint and schema version so a stale or schema-drifted temp build is invalidated and rebuilt rather than silently resumed.
+- **Trade-offs**: Pro: addresses the only genuinely non-resumable path honestly, without duplicating the incremental path's resume logic (it reuses it on the temp DB). Pro: atomic swap and `.bak` crash-recovery window are untouched; gated behind a flag that defaults to today's discard-on-interrupt behaviour. Con: the staging marker is a quasi-checkpoint that must be rigorously invalidated; a subtle fingerprint bug could resume onto a stale build and produce a wrong index. Con: significantly larger validation surface for a path that runs rarely. Con: closest of the three to brushing against the "no misleading/duplicate mechanism" rule.
+- **Complexity**: large
+- **Risk**: medium-high
+
+### Variant 3: Centralized RunController supervisor
+- **Approach**: Introduce one queue-scoped supervisor object that owns the entire long-run lifecycle: an `AbortController` composed into `Safeguard`, the signal handlers with grace window and in-flight await, the SIGTERM-safe WAL-consolidate-and-release close, the lock heartbeat with lock-busy-as-skip, and honest attempt accounting with per-handler wall-clock budgets. Every long-running operation receives this one controller. `reindexVault` stays non-resumable but abort-honoring.
+- **Trade-offs**: Pro: single composition point makes abort/shutdown/heartbeat/accounting impossible to forget on new code paths. Pro: centralizes honest accounting and budgets. Con: a new singleton lifecycle abstraction is heavy for a codebase whose only long-running process is `watch`; real risk of over-engineering. Con: a singleton invites accidentally re-importing multi-instance/daemon framing that OSB's stdio + per-vault locking forbids. Con: larger refactor touching every long-op call site.
+- **Complexity**: large
+- **Risk**: medium
+
+### Consultant's recommendation: Variant 1
+Rationale (verbatim summary): closes every gap that actually exists while honoring the hard constraints; Variant 2 carries the suite's only wrong-index failure mode for a rarely-run path; Variant 3 over-engineers a lifecycle abstraction for a tool with a single long-running process.
+
+---
+
+## Orchestrator decision: Variant 2 (override, documented)
+
+**Chosen: Variant 2** = Variant 1's full core (abort-composed Safeguard, graceful
+watch shutdown, exit-path WAL flush, lock heartbeat, lock-busy skip) PLUS the
+opt-in resumable staged reindex.
+
+**Why override the consultant's Variant 1:**
+
+1. **Task coverage.** t_672c751e ("resumable durable checkpoints") is one of the
+ three chosen tasks. Under Variant 1 it delivers nothing new: the incremental
+ path is already resumable, and the "final flush" it asks for is the same work
+ as t_ea80ddb5's shutdown. Variant 2 is the only honest, non-redundant way to
+ actually satisfy t_672c751e - by making the one genuinely non-resumable path
+ (`reindexVault`) resumable.
+
+2. **Cycle intent.** The operator's brief for this cycle is a deliberately larger,
+ substantive change. Resumable reindex is the only place real volume can come
+ from without padding or duplicating existing mechanisms.
+
+3. **The risk is designed out.** The consultant's sole serious objection -
+ resuming onto a stale staging build yields a wrong index - is removed:
+ - resume is gated on a signature marker (embedding signature + schema version
+ + chunk params); any drift discards `.new` and rebuilds from scratch;
+ - resume is the ordinary incremental walk over the temp DB, so the fastpath
+ re-reads any file whose mtime/size changed and the final deletion sweep
+ removes vanished files - it is self-healing, not blind trust;
+ - the whole behaviour sits behind `search_resume_reindex`, default false, so
+ when off the path is byte-identical to today (always discard + fresh build);
+ - the marker lives in the existing `index_state` KV table, so there is no
+ schema migration and no risk of breaking existing indices.
+
+**Why not Variant 3:** the supervisor singleton over-engineers a lifecycle object
+for a codebase whose only long-running process is `watch`, and a singleton risks
+smuggling back the daemon/multi-instance framing that OSB's stdio + per-vault
+locking explicitly forbids. Variant 2 keeps the abort inside the existing
+`Safeguard` (one trip point) and the shutdown inside a small testable runner -
+the DRY benefit Variant 3 promised, without the abstraction weight.
diff --git a/openclaw.plugin.json b/openclaw.plugin.json
index bafad69f..26b5fdef 100644
--- a/openclaw.plugin.json
+++ b/openclaw.plugin.json
@@ -2,7 +2,7 @@
"id": "open-second-brain",
"name": "Open Second Brain",
"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",
- "version": "1.7.0",
+ "version": "1.8.0",
"activation": { "onStartup": true },
"skills": ["./skills"],
"contracts": {
diff --git a/package.json b/package.json
index 8a342e48..7a430b22 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.7.0",
+ "version": "1.8.0",
"private": false,
"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.",
"keywords": [
diff --git a/plugin.yaml b/plugin.yaml
index db9e616e..63d84707 100644
--- a/plugin.yaml
+++ b/plugin.yaml
@@ -1,5 +1,5 @@
name: open-second-brain
-version: "1.7.0"
+version: "1.8.0"
description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."
author: "Open Second Brain contributors"
memory_provider: true
diff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.json
index 34461995..ed8f4126 100644
--- a/plugins/codex/.codex-plugin/plugin.json
+++ b/plugins/codex/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.7.0",
+ "version": "1.8.0",
"description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
"author": {
"name": "Open Second Brain contributors",
diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yaml
index db9e616e..63d84707 100644
--- a/plugins/hermes/plugin.yaml
+++ b/plugins/hermes/plugin.yaml
@@ -1,5 +1,5 @@
name: open-second-brain
-version: "1.7.0"
+version: "1.8.0"
description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."
author: "Open Second Brain contributors"
memory_provider: true
diff --git a/pyproject.toml b/pyproject.toml
index 79863cd7..9891e71e 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"
# CLI entry points (those moved to `package.json` `bin`).
[project]
name = "open-second-brain"
-version = "1.7.0"
+version = "1.8.0"
description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."
readme = "README.md"
requires-python = ">=3.11"
diff --git a/src/cli/search.ts b/src/cli/search.ts
index 97728ea4..5cc4c6a4 100644
--- a/src/cli/search.ts
+++ b/src/cli/search.ts
@@ -56,6 +56,8 @@ import type {
} from "../core/search/index.ts";
import { searchAcrossVaults } from "../core/search/cross-vault.ts";
import { IndexWatchPlanner } from "../core/search/index-watch.ts";
+import { IndexWatchRunner } from "../core/search/watch-runner.ts";
+import { SafeguardAbortError } from "../core/brain/safeguard.ts";
import { canonicalNotePath } from "../core/path-safety.ts";
import { watch, type FSWatcher } from "node:fs";
import { CliError, parseFlags } from "./argparse.ts";
@@ -744,28 +746,32 @@ async function cmdSearchWatch(argv: ReadonlyArray<string>): Promise<number> {
);
}
- let running = false;
- let stopped = false;
-
- const flush = async (): Promise<void> => {
- if (running || stopped) return;
- const due = planner.take(Date.now());
- if (due.length === 0) return;
- running = true;
- try {
- const stats = await indexVault(cfg, { embeddings: flags["embeddings"] === true });
- process.stderr.write(
- `synced ${due.length} change(s): +${stats.added} ~${stats.updated} =${stats.unchanged}` +
- (stats.errors.length > 0 ? ` (${stats.errors.length} error(s))` : "") +
- "\n",
- );
- } catch (e) {
- // A failed pass must not kill the watcher; report and keep watching.
- process.stderr.write(`index sync failed: ${e instanceof Error ? e.message : String(e)}\n`);
- } finally {
- running = false;
- }
- };
+ // Single-flight + graceful-shutdown coordinator (Indexer Durability
+ // suite). A SIGINT/SIGTERM aborts the in-flight pass at its next file
+ // boundary and waits for it to settle (bounded by the configured
+ // grace window) before exiting, so a signal never kills a run
+ // mid-write. indexInto closes its store in a finally, so the aborted
+ // pass still consolidates the WAL and releases the writer lock.
+ const runner = new IndexWatchRunner({
+ graceMs: cfg.shutdownGraceMs,
+ index: async (signal): Promise<void> => {
+ const due = planner.take(Date.now());
+ if (due.length === 0) return;
+ try {
+ const stats = await indexVault(cfg, { embeddings: flags["embeddings"] === true, signal });
+ process.stderr.write(
+ `synced ${due.length} change(s): +${stats.added} ~${stats.updated} =${stats.unchanged}` +
+ (stats.errors.length > 0 ? ` (${stats.errors.length} error(s))` : "") +
+ "\n",
+ );
+ } catch (e) {
+ // An abort is the expected stop on shutdown - let the runner
+ // observe it. Any other failure must not kill the watcher.
+ if (e instanceof SafeguardAbortError) throw e;
+ process.stderr.write(`index sync failed: ${e instanceof Error ? e.message : String(e)}\n`);
+ }
+ },
+ });
watcher.on("error", (e) => {
process.stderr.write(`watch error: ${e instanceof Error ? e.message : String(e)}\n`);
@@ -778,22 +784,25 @@ async function cmdSearchWatch(argv: ReadonlyArray<string>): Promise<number> {
const timer = setInterval(
() => {
- void flush();
+ void runner.flush();
},
Math.max(50, debounceMs),
);
return await new Promise<number>((resolve) => {
- const shutdown = (): void => {
- if (stopped) return;
- stopped = true;
+ const shutdown = async (): Promise<void> => {
+ if (runner.isStopped) return;
clearInterval(timer);
watcher.close();
+ // Drain the in-flight pass (aborted) within the grace window.
+ // A second SIGINT/SIGTERM bypasses this (process.once consumed
+ // the first), falling back to the default terminate = force exit.
+ await runner.shutdown();
process.stderr.write("watch stopped\n");
resolve(0);
};
- process.once("SIGINT", shutdown);
- process.once("SIGTERM", shutdown);
+ process.once("SIGINT", () => void shutdown());
+ process.once("SIGTERM", () => void shutdown());
process.stderr.write(
`watching ${cfg.vault} for .md changes (debounce ${debounceMs}ms); Ctrl-C to stop\n`,
);
diff --git a/src/core/brain/safeguard.ts b/src/core/brain/safeguard.ts
index 1b8edb26..30486ba8 100644
--- a/src/core/brain/safeguard.ts
+++ b/src/core/brain/safeguard.ts
@@ -42,6 +42,31 @@ export class SafeguardTimeoutError extends Error {
}
}
+/**
+ * Thrown when an operation's `AbortSignal` is aborted on demand (not a
+ * deadline). Distinct from `SafeguardTimeoutError` so callers - notably
+ * the watch shutdown coordinator - can treat an intentional abort as a
+ * clean stop rather than a failure.
+ */
+export class SafeguardAbortError extends Error {
+ readonly operation: string;
+
+ constructor(operation: string) {
+ super(`${operation} was aborted at a checkpoint`);
+ this.name = "SafeguardAbortError";
+ this.operation = operation;
+ }
+}
+
+/**
+ * Cooperative abort check for hot loops that hold a signal but not a
+ * full `Safeguard`. Throws `SafeguardAbortError` when the signal is
+ * aborted; a no-op for a live or absent signal.
+ */
+export function throwIfAborted(signal?: AbortSignal, operation = "operation"): void {
+ if (signal?.aborted === true) throw new SafeguardAbortError(operation);
+}
+
export interface Safeguard {
/** Throws `SafeguardTimeoutError` when the deadline has passed. */
checkpoint(): void;
@@ -57,6 +82,13 @@ export interface CreateSafeguardOptions {
readonly timeoutMs?: number | null;
/** Injected clock (tests). Defaults to `Date.now`. */
readonly now?: () => number;
+ /**
+ * Optional on-demand cancellation. When aborted, the next
+ * `checkpoint()` throws `SafeguardAbortError` - checked in priority
+ * over the deadline. A signal with no deadline still produces a live
+ * guard (no downgrade to noop).
+ */
+ readonly signal?: AbortSignal;
}
/** A no-deadline guard for callers that need a Safeguard-shaped arg. */
@@ -73,14 +105,21 @@ export function createSafeguard(opts: CreateSafeguardOptions): Safeguard {
opts.timeoutMs === undefined || opts.timeoutMs === null || opts.timeoutMs <= 0
? null
: opts.timeoutMs;
- if (budget === null) return noopSafeguard(opts.operation);
+ const signal = opts.signal;
+ // No deadline AND no signal: nothing to guard.
+ if (budget === null && signal === undefined) return noopSafeguard(opts.operation);
const now = opts.now ?? Date.now;
- const deadline = now() + budget;
+ const deadline = budget === null ? null : now() + budget;
return Object.freeze({
operation: opts.operation,
timeoutMs: budget,
checkpoint: () => {
- if (now() > deadline) throw new SafeguardTimeoutError(opts.operation, budget);
+ // Abort wins over the deadline: an intentional cancellation is
+ // reported as such, not masked as a timeout.
+ if (signal?.aborted === true) throw new SafeguardAbortError(opts.operation);
+ if (budget !== null && deadline !== null && now() > deadline) {
+ throw new SafeguardTimeoutError(opts.operation, budget);
+ }
},
});
}
diff --git a/src/core/search/index.ts b/src/core/search/index.ts
index 33393298..2fca57d0 100644
--- a/src/core/search/index.ts
+++ b/src/core/search/index.ts
@@ -115,6 +115,8 @@ const DEFAULTS = {
cacheTtlSeconds: 300,
fusionMode: "linear" as const,
rrfK: DEFAULT_RRF_K,
+ shutdownGraceSeconds: 5,
+ resumeReindex: false,
};
function parseFusionMode(raw: string | null): "linear" | "rrf" {
@@ -508,6 +510,22 @@ export function resolveSearchConfig(opts: {
false,
"search_self_tuning_enabled",
);
+ const shutdownGraceSeconds = parseInteger(
+ envOrConfig(
+ env,
+ config,
+ "OPEN_SECOND_BRAIN_SEARCH_SHUTDOWN_GRACE",
+ "search_shutdown_grace_seconds",
+ ),
+ DEFAULTS.shutdownGraceSeconds,
+ "search_shutdown_grace_seconds",
+ { min: 0 },
+ );
+ const resumeReindex = parseBool(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_RESUME_REINDEX", "search_resume_reindex"),
+ DEFAULTS.resumeReindex,
+ "search_resume_reindex",
+ );
const recall: ResolvedRecallConfig = Object.freeze({
mmrLambda,
maxHops,
@@ -541,6 +559,8 @@ export function resolveSearchConfig(opts: {
rrfK,
semantic,
recall,
+ shutdownGraceMs: shutdownGraceSeconds * 1000,
+ resumeReindex,
});
if (!opts.overrides) {
diff --git a/src/core/search/indexer.ts b/src/core/search/indexer.ts
index bec85177..d44aa803 100644
--- a/src/core/search/indexer.ts
+++ b/src/core/search/indexer.ts
@@ -17,6 +17,7 @@ import { createHash } from "node:crypto";
import {
accessSync,
constants,
+ existsSync,
mkdirSync,
readFileSync,
renameSync,
@@ -43,8 +44,10 @@ import { tieredFieldsForKind } from "../brain/frontmatter-tiers.ts";
import { normalizeSchemaToken } from "../brain/schema-vocab.ts";
import { parseFrontmatterText } from "../vault.ts";
import { appendMetric } from "../brain/metrics.ts";
+import { throwIfAborted } from "../brain/safeguard.ts";
import { extractEntities } from "./entities.ts";
import { Store } from "./store.ts";
+import { LATEST_SCHEMA_VERSION } from "./schema.ts";
import { SearchError } from "./types.ts";
import { walkVault } from "./walker.ts";
import { withTimeout } from "./with-timeout.ts";
@@ -76,6 +79,14 @@ export interface IndexVaultOptions {
* file, so a tripped guard aborts between files - never mid-write.
*/
readonly safeguard?: import("../brain/safeguard.ts").Safeguard;
+ /**
+ * On-demand cancellation (Indexer Durability suite). Checked at the
+ * same boundaries the deadline uses - between files and between embed
+ * batches, never mid-write. An aborted signal throws
+ * `SafeguardAbortError`; the deletion sweep runs only on full
+ * completion, so an aborted run leaves a consistent partial index.
+ */
+ readonly signal?: AbortSignal;
}
interface MutableStats {
@@ -210,6 +221,8 @@ async function indexInto(
// Per-file document upserts are transactional, so a tripped
// guard leaves a consistent (partially refreshed) index.
opts?.safeguard?.checkpoint();
+ // On-demand cancellation, same boundary as the deadline.
+ throwIfAborted(opts?.signal, "index");
// Mark seen FIRST. If anything downstream throws (read fault,
// chunker bug, transient FS error), the file must not look
// "missing" to the deletion sweep below β€” that would wipe a
@@ -415,7 +428,14 @@ async function indexInto(
}
if (opts?.embeddings) {
- await populateEmbeddings(store, config, stats, opts?.forceCost === true, opts?.safeguard);
+ await populateEmbeddings(
+ store,
+ config,
+ stats,
+ opts?.forceCost === true,
+ opts?.safeguard,
+ opts?.signal,
+ );
}
const now = new Date().toISOString();
@@ -480,6 +500,7 @@ async function populateEmbeddings(
stats: MutableStats,
forceCost: boolean,
safeguard?: import("../brain/safeguard.ts").Safeguard,
+ signal?: AbortSignal,
): Promise<void> {
if (!config.semantic.enabled) {
throw new SearchError(
@@ -535,6 +556,7 @@ async function populateEmbeddings(
// Cooperative deadline: embedding batches are the other long
// phase of an index run - abort between batches, never mid-batch.
safeguard?.checkpoint();
+ throwIfAborted(signal, "index");
const batch = pending.slice(i, i + superBatch);
const texts = batch.map((p) => p.content);
const vectors = await provider.embed(texts);
@@ -576,14 +598,42 @@ export async function reindexVault(
const bakPath = config.dbPath + ".bak";
mkdirSync(dirname(config.dbPath), { recursive: true });
- tryUnlink(newPath);
// Build into the temp file with an override config.
const tempConfig: ResolvedSearchConfig = Object.freeze({
...config,
dbPath: newPath,
});
- const stats = await indexVault(tempConfig, { ...opts, force: true });
+
+ // Resumable staging (opt-in). A compatible in-progress `.new` from an
+ // interrupted run is resumed via the incremental fastpath instead of
+ // rebuilt from scratch; an incompatible or unreadable one is
+ // discarded. With the flag OFF, the temp file is always rebuilt fresh
+ // and no staging marker is written or read - byte-identical to the
+ // pre-suite path (no extra staging-DB open/close cycles).
+ let resume = false;
+ if (config.resumeReindex) {
+ const signature = reindexStagingSignature(config, opts?.embeddings === true);
+ if (existsSync(newPath)) {
+ resume = await stagingSignatureMatches(tempConfig, signature);
+ }
+ if (!resume) tryUnlink(newPath);
+ // Stamp the signature so an interruption leaves a build a later run
+ // can recognise and resume.
+ await withStagingStore(tempConfig, (store) => store.setState(REINDEX_SIGNATURE_KEY, signature));
+ } else {
+ tryUnlink(newPath);
+ }
+
+ // `force: false` on resume lets the fastpath skip the files the
+ // partial build already committed; a fresh build forces every file.
+ const stats = await indexVault(tempConfig, { ...opts, force: !resume });
+
+ // Clear the marker so the swapped-in live index carries no staging
+ // state. Only present when resume was enabled.
+ if (config.resumeReindex) {
+ await withStagingStore(tempConfig, (store) => store.deleteState(REINDEX_SIGNATURE_KEY));
+ }
// Same-directory rename swap. The two renames are each atomic on
// POSIX; the gap between them is the only crash window, which the
@@ -594,6 +644,56 @@ export async function reindexVault(
return stats;
}
+/** index_state key carrying the staging-build compatibility signature. */
+const REINDEX_SIGNATURE_KEY = "reindex_signature";
+
+/**
+ * Compatibility signature for a staging rebuild: a resume is safe only
+ * when the schema version, chunk parameters, and (when embeddings are
+ * computed) the active embedding signature all match the partial build.
+ * Any drift invalidates the staging DB and forces a fresh rebuild.
+ */
+function reindexStagingSignature(config: ResolvedSearchConfig, embeddings: boolean): string {
+ const embedding = embeddings ? (activeEmbeddingSignature(config) ?? "active") : "off";
+ return JSON.stringify({
+ schema: LATEST_SCHEMA_VERSION,
+ chunkSize: config.chunkSize,
+ chunkOverlap: config.chunkOverlap,
+ embedding,
+ });
+}
+
+/** Read the staging signature from an existing `.new`; false on any
+ * read error (a corrupt or partial staging DB is rebuilt, not trusted). */
+async function stagingSignatureMatches(
+ tempConfig: ResolvedSearchConfig,
+ signature: string,
+): Promise<boolean> {
+ try {
+ const store = await Store.open(tempConfig, { mode: "write", loadVec: false });
+ try {
+ return store.getState(REINDEX_SIGNATURE_KEY) === signature;
+ } finally {
+ await store.close();
+ }
+ } catch {
+ return false;
+ }
+}
+
+/** Run a short write against the staging DB (set/clear the marker). */
+async function withStagingStore(
+ tempConfig: ResolvedSearchConfig,
+ fn: (store: Store) => void,
+): Promise<void> {
+ const store = await Store.open(tempConfig, { mode: "write", loadVec: false });
+ try {
+ fn(store);
+ } finally {
+ await store.close();
+ }
+}
+
/** `unlinkSync` that tolerates ENOENT (file already absent). */
function tryUnlink(p: string): void {
try {
diff --git a/src/core/search/store-exit.ts b/src/core/search/store-exit.ts
new file mode 100644
index 00000000..2292652c
--- /dev/null
+++ b/src/core/search/store-exit.ts
@@ -0,0 +1,67 @@
+/**
+ * Writer WAL-flush-on-exit registry (Indexer Durability suite,
+ * t_672c751e).
+ *
+ * The search store runs in WAL mode. An orderly `Store.close()`
+ * consolidates the WAL (`wal_checkpoint(TRUNCATE)` then
+ * `journal_mode=DELETE`) and releases the lock. When `close()` is
+ * bypassed - an error path that calls `process.exit`, or a signal
+ * handler that lets the process exit normally - an orphan `-wal` is
+ * left next to the main DB. SQLite replays it on the next open, so no
+ * data is lost; this registry is a belt-and-suspenders that
+ * synchronously consolidates the WAL for every still-open writer on the
+ * process `exit` event, mirroring `sync-lockfile.ts`'s cleanup hook.
+ *
+ * It is best-effort and synchronous (bun:sqlite is synchronous and the
+ * `exit` event cannot await async work): every operation is wrapped so
+ * the hook can never throw. A hard kill (SIGKILL) fires no `exit`
+ * event; that case relies on SQLite's own WAL replay on the next open.
+ *
+ * The registry holds a strong reference until `Store.close()`
+ * unregisters it. Every code path that opens a writer closes it in a
+ * `finally`, so handles do not accumulate over a long-lived process;
+ * the set holds only the writers currently open.
+ */
+
+import type { Database } from "bun:sqlite";
+
+const openWriters = new Set<Database>();
+let exitHookInstalled = false;
+
+function ensureExitHook(): void {
+ if (exitHookInstalled) return;
+ exitHookInstalled = true;
+ process.on("exit", flushRegisteredWriters);
+}
+
+/** Register an open writer DB so its WAL is consolidated on exit. */
+export function registerWriterDb(db: Database): void {
+ ensureExitHook();
+ openWriters.add(db);
+}
+
+/** Unregister a writer DB (called from an orderly `Store.close()`). */
+export function unregisterWriterDb(db: Database): void {
+ openWriters.delete(db);
+}
+
+/**
+ * Best-effort synchronous WAL consolidation for every registered
+ * writer. Never throws: a closed, locked, or mid-write handle is
+ * skipped silently. Safe to call more than once.
+ */
+export function flushRegisteredWriters(): void {
+ for (const db of openWriters) {
+ try {
+ db.exec("PRAGMA wal_checkpoint(TRUNCATE)");
+ } catch {
+ // already closed / locked / mid-write β€” consolidation is a
+ // best-effort safety net, not a correctness requirement.
+ }
+ }
+}
+
+/** Test-only: clear the registry without flushing. */
+export function _resetWriterRegistryForTests(): void {
+ openWriters.clear();
+}
diff --git a/src/core/search/store.ts b/src/core/search/store.ts
index dc8ef26d..0f06ca58 100644
--- a/src/core/search/store.ts
+++ b/src/core/search/store.ts
@@ -18,6 +18,8 @@ import { existsSync, mkdirSync, renameSync } from "node:fs";
import { dirname } from "node:path";
import lockfile from "proper-lockfile";
+import { registerWriterDb, unregisterWriterDb } from "./store-exit.ts";
+
import { computeCorpusGeneration } from "./corpus-generation.ts";
import { SearchError } from "./types.ts";
import type { ResolvedSearchConfig } from "./types.ts";
@@ -204,13 +206,32 @@ function nowIso(): string {
return new Date().toISOString();
}
+/**
+ * Stale window for the writer lock (ms). A lock whose mtime is older
+ * than this is treated as abandoned (crashed holder) and taken over by
+ * the next writer, so a SIGKILL never wedges the index for longer than
+ * this window.
+ */
+export const WRITER_LOCK_STALE_MS = 60_000;
+
+/**
+ * Heartbeat interval (ms): the async writer lock refreshes its mtime
+ * this often so a legitimate long-running index is never mistaken for
+ * a stale lock. Must stay below {@link WRITER_LOCK_STALE_MS}. NOTE: the
+ * per-file document walk is synchronous, so this timer only fires
+ * across the await points (embed batches); a multi-minute fully
+ * synchronous walk still relies on the stale window, which 60s amply
+ * covers for real vaults.
+ */
+export const WRITER_LOCK_HEARTBEAT_MS = 30_000;
+
function acquireWriterLockSync(path: string): () => void {
const maxAttempts = 10;
const sleepMs = 50;
let lastError: unknown;
for (let attempt = 0; attempt < maxAttempts; attempt++) {
try {
- return lockfile.lockSync(path, { stale: 60_000, realpath: false });
+ return lockfile.lockSync(path, { stale: WRITER_LOCK_STALE_MS, realpath: false });
} catch (err) {
const code = (err as NodeJS.ErrnoException).code;
if (code !== "ELOCKED") throw err;
@@ -336,7 +357,10 @@ export class Store {
try {
release = await lockfile.lock(config.dbPath, {
retries: { retries: 3, factor: 1, minTimeout: 1000, maxTimeout: 1000 },
- stale: 60_000,
+ stale: WRITER_LOCK_STALE_MS,
+ // Explicit heartbeat: refresh the lock mtime mid-run so a long
+ // index is never mistaken for a stale lock and taken over.
+ update: WRITER_LOCK_HEARTBEAT_MS,
realpath: false,
});
} catch (e) {
@@ -360,6 +384,9 @@ export class Store {
const vecLoaded = loadVec && tryLoadVecExtension(db);
const store = new Store(db, config, vecLoaded, release);
store.ensureEmbeddingModel(config.semantic.model, config.semantic.dimension);
+ // Belt-and-suspenders: consolidate this writer's WAL on a
+ // bypassed close (process.exit / signal-driven exit).
+ registerWriterDb(db);
return store;
} catch (e) {
try {
@@ -383,6 +410,8 @@ export class Store {
async close(): Promise<void> {
if (this.closed) return;
this.closed = true;
+ // Orderly close owns consolidation below; drop the exit-hook entry.
+ unregisterWriterDb(this.db);
try {
// Writer mode: consolidate WAL into the main file and switch back to
// DELETE journal mode so the `-wal`/`-shm` siblings are removed. This
diff --git a/src/core/search/types.ts b/src/core/search/types.ts
index 8957ac93..2bcb39a7 100644
--- a/src/core/search/types.ts
+++ b/src/core/search/types.ts
@@ -568,4 +568,18 @@ export interface ResolvedSearchConfig {
readonly rrfK: number;
readonly semantic: ResolvedEmbeddingConfig;
readonly recall: ResolvedRecallConfig;
+ /**
+ * Grace window (ms) the `o2b search watch` shutdown waits for an
+ * in-flight index pass to settle at a cooperative boundary before
+ * exiting (Indexer Durability suite). `0` exits immediately after
+ * signalling the abort, without awaiting. Default 5000.
+ */
+ readonly shutdownGraceMs: number;
+ /**
+ * When true, an interrupted full `reindexVault` rebuild resumes a
+ * compatible `brain.sqlite.new` staging build instead of discarding
+ * it. Opt-in; default false keeps the always-fresh rebuild. Resume is
+ * gated on a signature marker, so a drifted staging DB is rebuilt.
+ */
+ readonly resumeReindex: boolean;
}
diff --git a/src/core/search/watch-runner.ts b/src/core/search/watch-runner.ts
new file mode 100644
index 00000000..de64b869
--- /dev/null
+++ b/src/core/search/watch-runner.ts
@@ -0,0 +1,103 @@
+/**
+ * IndexWatchRunner (Indexer Durability suite, t_ea80ddb5).
+ *
+ * The testable core of `o2b search watch`'s lifecycle. Two
+ * responsibilities, both deliberately free of `fs.watch`, timers, and
+ * OS signals so they can be unit-tested directly:
+ *
+ * - Single-flight flush: one index pass at a time; concurrent
+ * `flush()` calls share the in-flight promise instead of stacking
+ * overlapping passes.
+ * - Graceful shutdown: stop accepting new flushes, abort the
+ * in-flight pass via an AbortSignal (which the indexer honours at
+ * its next file / embed-batch boundary), and await it to settle -
+ * bounded by a grace window so a wedged pass cannot block exit
+ * forever. `graceMs <= 0` aborts and returns without awaiting.
+ *
+ * The CLI provides the `index` callback (which runs `indexVault` with
+ * the supplied signal and reports stats) and wires the watcher, the
+ * debounce planner, and SIGINT/SIGTERM to this runner.
+ */
+
+export interface IndexWatchRunnerDeps {
+ /**
+ * Run exactly one index pass, honouring the abort signal. It owns its
+ * own reporting and swallows its own recoverable errors; an abort may
+ * surface as a rejection, which the runner treats as an expected
+ * stop.
+ */
+ readonly index: (signal: AbortSignal) => Promise<unknown>;
+ /** Milliseconds the shutdown awaits an aborted in-flight pass. */
+ readonly graceMs: number;
+ /** Resolves after `ms`; injectable for tests. Defaults to setTimeout. */
+ readonly graceWaiter?: (ms: number) => Promise<void>;
+}
+
+function defaultGraceWaiter(ms: number): Promise<void> {
+ return new Promise((resolve) => {
+ const timer = setTimeout(resolve, ms);
+ // Do not keep the event loop alive solely for the grace timer.
+ if (typeof timer === "object" && timer !== null && "unref" in timer) timer.unref();
+ });
+}
+
+export class IndexWatchRunner {
+ private readonly deps: IndexWatchRunnerDeps;
+ private running: Promise<void> | null = null;
+ private controller: AbortController | null = null;
+ private stopped = false;
+
+ constructor(deps: IndexWatchRunnerDeps) {
+ this.deps = deps;
+ }
+
+ /** True once shutdown has begun; no further flushes are accepted. */
+ get isStopped(): boolean {
+ return this.stopped;
+ }
+
+ /** True while an index pass is in flight. */
+ get isFlushing(): boolean {
+ return this.running !== null;
+ }
+
+ /**
+ * Run one index pass unless shutdown began or a pass is already in
+ * flight (single-flight: the in-flight promise is returned instead).
+ */
+ flush(): Promise<void> {
+ if (this.stopped) return Promise.resolve();
+ if (this.running) return this.running;
+ const controller = new AbortController();
+ this.controller = controller;
+ const run = (async () => {
+ try {
+ await this.deps.index(controller.signal);
+ } catch {
+ // The index callback reports its own recoverable failures; an
+ // abort surfaces here on shutdown and is the expected stop.
+ }
+ })().finally(() => {
+ this.running = null;
+ this.controller = null;
+ });
+ this.running = run;
+ return run;
+ }
+
+ /**
+ * Stop accepting flushes, abort any in-flight pass, and await it to
+ * settle at a cooperative boundary - bounded by the grace window.
+ * Idempotent: a second call is a no-op.
+ */
+ async shutdown(): Promise<void> {
+ if (this.stopped) return;
+ this.stopped = true;
+ const inflight = this.running;
+ if (inflight === null) return;
+ this.controller?.abort();
+ if (this.deps.graceMs <= 0) return;
+ const wait = this.deps.graceWaiter ?? defaultGraceWaiter;
+ await Promise.race([inflight, wait(this.deps.graceMs)]);
+ }
+}
diff --git a/tests/core/brain/safeguard-abort.test.ts b/tests/core/brain/safeguard-abort.test.ts
new file mode 100644
index 00000000..5df361d2
--- /dev/null
+++ b/tests/core/brain/safeguard-abort.test.ts
@@ -0,0 +1,94 @@
+/**
+ * Safeguard abort composition (Indexer Durability suite, t_79e773be):
+ * the cooperative deadline gains an optional AbortSignal so one
+ * checkpoint() has two trip conditions - an aborted signal (priority)
+ * or a passed deadline. Bun + synchronous SQLite still cannot be
+ * preempted, so abort is cooperative: the NEXT checkpoint after an
+ * abort throws SafeguardAbortError at a natural iteration boundary.
+ */
+
+import { describe, expect, test } from "bun:test";
+
+import {
+ createSafeguard,
+ SafeguardAbortError,
+ SafeguardTimeoutError,
+ throwIfAborted,
+} from "../../../src/core/brain/safeguard.ts";
+
+describe("createSafeguard with an AbortSignal", () => {
+ test("an aborted signal trips checkpoint() with SafeguardAbortError", () => {
+ const ac = new AbortController();
+ const guard = createSafeguard({ operation: "reindex", timeoutMs: 0, signal: ac.signal });
+ guard.checkpoint();
+ ac.abort();
+ expect(() => guard.checkpoint()).toThrow(SafeguardAbortError);
+ });
+
+ test("abort wins over a live deadline (checked first)", () => {
+ let clock = 1_000;
+ const ac = new AbortController();
+ const guard = createSafeguard({
+ operation: "reindex",
+ timeoutMs: 100_000,
+ now: () => clock,
+ signal: ac.signal,
+ });
+ clock += 10; // well within the deadline
+ guard.checkpoint();
+ ac.abort();
+ try {
+ guard.checkpoint();
+ throw new Error("expected SafeguardAbortError");
+ } catch (exc) {
+ expect(exc).toBeInstanceOf(SafeguardAbortError);
+ expect(exc).not.toBeInstanceOf(SafeguardTimeoutError);
+ const e = exc as SafeguardAbortError;
+ expect(e.operation).toBe("reindex");
+ expect(e.message).toContain("reindex");
+ }
+ });
+
+ test("a signal with no deadline still trips on abort (no noop downgrade)", () => {
+ const ac = new AbortController();
+ const guard = createSafeguard({ operation: "index", timeoutMs: null, signal: ac.signal });
+ expect(() => guard.checkpoint()).not.toThrow();
+ ac.abort();
+ expect(() => guard.checkpoint()).toThrow(SafeguardAbortError);
+ });
+
+ test("the deadline still trips when the signal is never aborted", () => {
+ let clock = 0;
+ const ac = new AbortController();
+ const guard = createSafeguard({
+ operation: "reindex",
+ timeoutMs: 10,
+ now: () => clock,
+ signal: ac.signal,
+ });
+ clock = 11;
+ expect(() => guard.checkpoint()).toThrow(SafeguardTimeoutError);
+ });
+
+ test("no signal + live deadline behaves exactly as before", () => {
+ let clock = 1_000;
+ const guard = createSafeguard({ operation: "dream", timeoutMs: 50, now: () => clock });
+ guard.checkpoint();
+ clock += 51;
+ expect(() => guard.checkpoint()).toThrow(SafeguardTimeoutError);
+ });
+});
+
+describe("throwIfAborted", () => {
+ test("throws SafeguardAbortError for an aborted signal", () => {
+ const ac = new AbortController();
+ ac.abort();
+ expect(() => throwIfAborted(ac.signal, "index")).toThrow(SafeguardAbortError);
+ });
+
+ test("is a no-op for a live or absent signal", () => {
+ const ac = new AbortController();
+ expect(() => throwIfAborted(ac.signal, "index")).not.toThrow();
+ expect(() => throwIfAborted(undefined, "index")).not.toThrow();
+ });
+});
diff --git a/tests/core/search/config.test.ts b/tests/core/search/config.test.ts
index 16cc1ef7..e84039e1 100644
--- a/tests/core/search/config.test.ts
+++ b/tests/core/search/config.test.ts
@@ -33,6 +33,8 @@ const ENV_KEYS = [
"OPEN_SECOND_BRAIN_SEARCH_SYNONYM_MAX_TERMS",
"OPEN_SECOND_BRAIN_SEARCH_CACHE_ENABLED",
"OPEN_SECOND_BRAIN_SEARCH_CACHE_TTL",
+ "OPEN_SECOND_BRAIN_SEARCH_SHUTDOWN_GRACE",
+ "OPEN_SECOND_BRAIN_SEARCH_RESUME_REINDEX",
];
beforeEach(() => {
@@ -280,3 +282,44 @@ test("overrides are validated after merging", () => {
}),
).toThrow(/embedding_batch_size/);
});
+
+// ── Indexer Durability suite: shutdown grace + resume flag ──────────────────
+
+test("durability keys default to a 5s grace and no resume", () => {
+ writeFileSync(configPath, `vault: "${tmp}"\n`);
+ const cfg = resolveSearchConfig({ vault: tmp, configPath });
+ expect(cfg.shutdownGraceMs).toBe(5_000);
+ expect(cfg.resumeReindex).toBe(false);
+});
+
+test("config keys override the durability defaults", () => {
+ writeFileSync(
+ configPath,
+ `vault: "${tmp}"\nsearch_shutdown_grace_seconds: 12\nsearch_resume_reindex: true\n`,
+ );
+ const cfg = resolveSearchConfig({ vault: tmp, configPath });
+ expect(cfg.shutdownGraceMs).toBe(12_000);
+ expect(cfg.resumeReindex).toBe(true);
+});
+
+test("env mirrors override the durability config", () => {
+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_shutdown_grace_seconds: 12\n`);
+ process.env["OPEN_SECOND_BRAIN_SEARCH_SHUTDOWN_GRACE"] = "3";
+ process.env["OPEN_SECOND_BRAIN_SEARCH_RESUME_REINDEX"] = "true";
+ const cfg = resolveSearchConfig({ vault: tmp, configPath });
+ expect(cfg.shutdownGraceMs).toBe(3_000);
+ expect(cfg.resumeReindex).toBe(true);
+});
+
+test("an invalid grace is rejected loudly (no silent fallback)", () => {
+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_shutdown_grace_seconds: soon\n`);
+ expect(() => resolveSearchConfig({ vault: tmp, configPath })).toThrow(
+ /search_shutdown_grace_seconds/,
+ );
+});
+
+test("grace of 0 disables the await window", () => {
+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_shutdown_grace_seconds: 0\n`);
+ const cfg = resolveSearchConfig({ vault: tmp, configPath });
+ expect(cfg.shutdownGraceMs).toBe(0);
+});
diff --git a/tests/core/search/indexer-abort.test.ts b/tests/core/search/indexer-abort.test.ts
new file mode 100644
index 00000000..e3fe458e
--- /dev/null
+++ b/tests/core/search/indexer-abort.test.ts
@@ -0,0 +1,163 @@
+/**
+ * AbortSignal threading through the indexer (Indexer Durability suite,
+ * t_79e773be). indexVault accepts an optional signal checked at the
+ * same cooperative boundaries the deadline already uses - between files
+ * in indexInto, between embed batches in populateEmbeddings - so a run
+ * can be cancelled on demand without a mid-write kill. The deletion
+ * sweep runs only on full completion, so an aborted run leaves a
+ * consistent, partially-refreshed index. A run with no signal is
+ * unchanged.
+ */
+
+import { afterEach, beforeEach, expect, test } from "bun:test";
+
+import { SafeguardAbortError } from "../../../src/core/brain/safeguard.ts";
+import { indexVault } from "../../../src/core/search/indexer.ts";
+import { Store } from "../../../src/core/search/store.ts";
+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";
+import { startFakeHttp, type FakeHttp } from "../../helpers/fake-http.ts";
+import { sqliteVecLoadable } from "../../helpers/sqlite-vec.ts";
+
+let vault: string;
+let dbPath: string;
+let cleanup: () => void;
+
+beforeEach(() => {
+ const v = createTempVault("indexer-abort");
+ vault = v.vault;
+ dbPath = v.dbPath;
+ cleanup = v.cleanup;
+});
+
+afterEach(() => {
+ cleanup();
+});
+
+test("a signal aborted before the walk throws and indexes nothing", async () => {
+ writeMd(vault, "a.md", "# A\n\nAlpha.");
+ writeMd(vault, "b.md", "# B\n\nBeta.");
+ const cfg = makeConfig({ vault, dbPath });
+ const ac = new AbortController();
+ ac.abort();
+
+ await expect(indexVault(cfg, { signal: ac.signal })).rejects.toBeInstanceOf(SafeguardAbortError);
+
+ const store = await Store.open(cfg, { mode: "write" });
+ expect(store.listDocuments().size).toBe(0);
+ await store.close();
+});
+
+test("aborting mid-walk commits files seen before the abort, drops the rest", async () => {
+ for (let i = 0; i < 5; i++) writeMd(vault, `n${i}.md`, `# N${i}\n\nbody ${i}`);
+ const cfg = makeConfig({ vault, dbPath });
+ const ac = new AbortController();
+ let processed = 0;
+
+ // Abort right after the first file is committed; the next loop-top
+ // checkpoint must trip and stop the walk.
+ await expect(
+ indexVault(cfg, {
+ signal: ac.signal,
+ onFile: () => {
+ processed++;
+ if (processed === 1) ac.abort();
+ },
+ }),
+ ).rejects.toBeInstanceOf(SafeguardAbortError);
+
+ const store = await Store.open(cfg, { mode: "write" });
+ const count = store.listDocuments().size;
+ await store.close();
+ // Exactly the pre-abort file(s) survived; not all five, not zero.
+ expect(count).toBeGreaterThanOrEqual(1);
+ expect(count).toBeLessThan(5);
+});
+
+test("an interrupted run is resumable: a clean re-run completes the index", async () => {
+ for (let i = 0; i < 5; i++) writeMd(vault, `n${i}.md`, `# N${i}\n\nbody ${i}`);
+ const cfg = makeConfig({ vault, dbPath });
+ const ac = new AbortController();
+ let processed = 0;
+ await expect(
+ indexVault(cfg, {
+ signal: ac.signal,
+ onFile: () => {
+ if (++processed === 1) ac.abort();
+ },
+ }),
+ ).rejects.toBeInstanceOf(SafeguardAbortError);
+
+ // No signal: the fastpath skips the already-committed file and
+ // finishes the rest. The index ends complete.
+ const stats = await indexVault(cfg);
+ const store = await Store.open(cfg, { mode: "write" });
+ expect(store.listDocuments().size).toBe(5);
+ await store.close();
+ expect(stats.added + stats.unchanged).toBe(5);
+});
+
+test("no signal indexes the whole vault unchanged", async () => {
+ writeMd(vault, "a.md", "# A\n\nAlpha.");
+ writeMd(vault, "b.md", "# B\n\nBeta.");
+ const cfg = makeConfig({ vault, dbPath });
+ const stats = await indexVault(cfg);
+ expect(stats.added).toBe(2);
+ const store = await Store.open(cfg, { mode: "write" });
+ expect(store.listDocuments().size).toBe(2);
+ await store.close();
+});
+
+test("aborting between embed batches stops further embedding but keeps committed vectors", async () => {
+ if (!sqliteVecLoadable()) return;
+ const server: FakeHttp = await startFakeHttp();
+ try {
+ for (let i = 0; i < 6; i++) writeMd(vault, `e${i}.md`, `# E${i}\n\nembed body number ${i}`);
+ const cfg = makeConfig({
+ vault,
+ dbPath,
+ semantic: {
+ enabled: true,
+ provider: "openai-compat",
+ baseUrl: server.url,
+ model: "fake-model",
+ apiKey: "test-key",
+ dimension: 4,
+ timeoutMs: 5_000,
+ concurrency: 1,
+ batchSize: 1,
+ costGateUsd: 0,
+ },
+ });
+ // Index documents first (no embeddings), so the abort isolates the
+ // embed phase: each batch is one chunk -> one HTTP call.
+ await indexVault(cfg);
+
+ const ac = new AbortController();
+ // Serve the first batch normally, then abort - the populateEmbeddings
+ // loop trips at the NEXT batch boundary, after batch 0 is committed.
+ server.setHandler((req, callIndex) => {
+ const body = (req.body ?? {}) as { input?: string[]; model?: string };
+ const inputs = Array.isArray(body.input) ? body.input : [];
+ const data = inputs.map((text, index) => ({
+ object: "embedding",
+ embedding: [text.length, index, 1, 1],
+ index,
+ }));
+ if (callIndex === 0) ac.abort();
+ return { status: 200, body: { data, model: body.model ?? "fake-model" } };
+ });
+
+ await expect(indexVault(cfg, { embeddings: true, signal: ac.signal })).rejects.toBeInstanceOf(
+ SafeguardAbortError,
+ );
+
+ // The first batch committed; the rest did not. Partial, not all-or-nothing.
+ const store = await Store.open(cfg, { mode: "write" });
+ const counts = store.counts();
+ await store.close();
+ expect(counts.embeddings).toBeGreaterThanOrEqual(1);
+ expect(counts.embeddings).toBeLessThan(counts.chunks);
+ } finally {
+ await server.close();
+ }
+});
diff --git a/tests/core/search/multi-instance.test.ts b/tests/core/search/multi-instance.test.ts
new file mode 100644
index 00000000..3eb29ef5
--- /dev/null
+++ b/tests/core/search/multi-instance.test.ts
@@ -0,0 +1,77 @@
+/**
+ * Multi-instance isolation + writer-lock heartbeat (Indexer Durability
+ * suite, t_ea80ddb5 / t_672c751e). The honest OSB reading of upstream's
+ * "multi-instance" support: the MCP server is stdio-only, so there is
+ * no port to multiplex. Isolation comes from the per-dbPath writer
+ * lock - two instances on DIFFERENT vaults run conflict-free, and a
+ * second writer on the SAME vault gets a typed INDEX_LOCKED rather than
+ * corrupting the index. No --port / --instance daemon is fabricated.
+ *
+ * The heartbeat constants are pinned here so a future edit cannot
+ * silently drop the lock refresh below the stale window.
+ */
+
+import { afterEach, beforeEach, expect, test } from "bun:test";
+
+import {
+ Store,
+ WRITER_LOCK_HEARTBEAT_MS,
+ WRITER_LOCK_STALE_MS,
+} from "../../../src/core/search/store.ts";
+import { SearchError } from "../../../src/core/search/types.ts";
+import { createTempVault, makeConfig } from "../../helpers/search-fixtures.ts";
+
+const vaults: Array<() => void> = [];
+
+beforeEach(() => {
+ vaults.length = 0;
+});
+
+afterEach(() => {
+ for (const c of vaults) c();
+});
+
+function freshConfig(prefix: string) {
+ const v = createTempVault(prefix);
+ vaults.push(v.cleanup);
+ return makeConfig({ vault: v.vault, dbPath: v.dbPath });
+}
+
+test("the heartbeat refreshes well within the stale window", () => {
+ expect(WRITER_LOCK_HEARTBEAT_MS).toBeGreaterThan(0);
+ expect(WRITER_LOCK_HEARTBEAT_MS).toBeLessThan(WRITER_LOCK_STALE_MS);
+});
+
+test("writers on different vaults coexist", async () => {
+ const a = freshConfig("multi-a");
+ const b = freshConfig("multi-b");
+ const sa = await Store.open(a, { mode: "write" });
+ const sb = await Store.open(b, { mode: "write" });
+ expect(sa).toBeInstanceOf(Store);
+ expect(sb).toBeInstanceOf(Store);
+ await sa.close();
+ await sb.close();
+});
+
+test("a second writer on the same vault gets INDEX_LOCKED", async () => {
+ const cfg = freshConfig("multi-same");
+ const first = await Store.open(cfg, { mode: "write" });
+ try {
+ await Store.open(cfg, { mode: "write" });
+ throw new Error("expected the second writer to be rejected");
+ } catch (e) {
+ expect(e).toBeInstanceOf(SearchError);
+ expect(e).toHaveProperty("code", "INDEX_LOCKED");
+ } finally {
+ await first.close();
+ }
+});
+
+test("the lock is released on close: a later writer succeeds", async () => {
+ const cfg = freshConfig("multi-reopen");
+ const first = await Store.open(cfg, { mode: "write" });
+ await first.close();
+ const second = await Store.open(cfg, { mode: "write" });
+ expect(second).toBeInstanceOf(Store);
+ await second.close();
+});
diff --git a/tests/core/search/reindex-resume.test.ts b/tests/core/search/reindex-resume.test.ts
new file mode 100644
index 00000000..cbca7957
--- /dev/null
+++ b/tests/core/search/reindex-resume.test.ts
@@ -0,0 +1,116 @@
+/**
+ * Opt-in resumable staged reindex (Indexer Durability suite,
+ * t_672c751e). reindexVault builds into brain.sqlite.new then atomically
+ * swaps. Today an interrupted rebuild discards all progress. With
+ * search_resume_reindex on, a compatible in-progress .new is resumed
+ * via the incremental fastpath instead of rebuilt from scratch; a
+ * signature mismatch (schema / chunk params / embedding signature)
+ * discards it. The signature marker lives in the staging DB's
+ * index_state KV (no schema migration) and is cleared before the swap,
+ * so the live index never carries staging state. Flag off keeps the
+ * always-fresh rebuild byte-for-byte.
+ */
+
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import { existsSync, mkdirSync, writeFileSync } from "node:fs";
+import { dirname } from "node:path";
+
+import { SafeguardAbortError } from "../../../src/core/brain/safeguard.ts";
+import { indexVault, reindexVault } from "../../../src/core/search/indexer.ts";
+import { Store } from "../../../src/core/search/store.ts";
+import type { ResolvedSearchConfig } from "../../../src/core/search/types.ts";
+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";
+
+let vault: string;
+let dbPath: string;
+let cleanup: () => void;
+
+beforeEach(() => {
+ const v = createTempVault("reindex-resume");
+ vault = v.vault;
+ dbPath = v.dbPath;
+ cleanup = v.cleanup;
+ for (let i = 0; i < 6; i++) writeMd(vault, `n${i}.md`, `# N${i}\n\nbody number ${i}`);
+});
+
+afterEach(() => {
+ cleanup();
+});
+
+function cfgWith(overrides: Partial<ResolvedSearchConfig>): ResolvedSearchConfig {
+ return Object.freeze({ ...makeConfig({ vault, dbPath }), ...overrides });
+}
+
+async function docCount(cfg: ResolvedSearchConfig): Promise<number> {
+ const store = await Store.open(cfg, { mode: "write" });
+ const n = store.listDocuments().size;
+ await store.close();
+ return n;
+}
+
+/** Drive a reindex that aborts after the first committed file, leaving
+ * a partial staging .new behind. */
+async function partialReindex(cfg: ResolvedSearchConfig): Promise<void> {
+ const ac = new AbortController();
+ let processed = 0;
+ await expect(
+ reindexVault(cfg, {
+ signal: ac.signal,
+ onFile: () => {
+ if (++processed === 1) ac.abort();
+ },
+ }),
+ ).rejects.toBeInstanceOf(SafeguardAbortError);
+ expect(existsSync(`${dbPath}.new`)).toBe(true);
+}
+
+test("flag off: an interrupted rebuild leaves no resumable state and a clean rebuild succeeds", async () => {
+ const cfg = cfgWith({ resumeReindex: false });
+ // A stray .new from a prior crash must be discarded, not trusted.
+ mkdirSync(dirname(dbPath), { recursive: true });
+ writeFileSync(`${dbPath}.new`, "garbage-not-a-sqlite-db");
+ const stats = await reindexVault(cfg);
+ expect(stats.added).toBe(6);
+ expect(await docCount(cfg)).toBe(6);
+});
+
+test("flag on: resume completes the partial build without reprocessing committed docs", async () => {
+ const cfg = cfgWith({ resumeReindex: true });
+ await partialReindex(cfg);
+
+ // Resume: the fastpath skips the already-committed doc(s) (counted as
+ // unchanged), so a from-scratch rebuild (all 'added') is ruled out.
+ const stats = await reindexVault(cfg);
+ expect(stats.unchanged).toBeGreaterThanOrEqual(1);
+ expect(stats.added + stats.unchanged).toBe(6);
+ expect(await docCount(cfg)).toBe(6);
+ // Staging is gone after the swap.
+ expect(existsSync(`${dbPath}.new`)).toBe(false);
+});
+
+test("flag on: a signature mismatch discards the staging build and rebuilds fresh", async () => {
+ await partialReindex(cfgWith({ resumeReindex: true }));
+ // Change a chunk parameter: the staging signature no longer matches.
+ const drifted = cfgWith({ resumeReindex: true, chunkSize: 64, chunkOverlap: 8 });
+ const stats = await reindexVault(drifted);
+ // No resume: every file is processed fresh.
+ expect(stats.added).toBe(6);
+ expect(stats.unchanged).toBe(0);
+ expect(await docCount(drifted)).toBe(6);
+});
+
+test("the swapped live index carries no staging signature marker", async () => {
+ const cfg = cfgWith({ resumeReindex: true });
+ await reindexVault(cfg);
+ const store = await Store.open(cfg, { mode: "write" });
+ expect(store.getState("reindex_signature")).toBeNull();
+ await store.close();
+});
+
+test("flag on with no prior staging behaves like a normal full reindex", async () => {
+ const cfg = cfgWith({ resumeReindex: true });
+ await indexVault(cfg); // seed a live index first
+ const stats = await reindexVault(cfg);
+ expect(stats.added).toBe(6);
+ expect(await docCount(cfg)).toBe(6);
+});
diff --git a/tests/core/search/store-exit.test.ts b/tests/core/search/store-exit.test.ts
new file mode 100644
index 00000000..9ed24072
--- /dev/null
+++ b/tests/core/search/store-exit.test.ts
@@ -0,0 +1,77 @@
+/**
+ * Writer WAL-flush-on-exit registry (Indexer Durability suite,
+ * t_672c751e). Mirrors sync-lockfile's process-exit hook: open writer
+ * DB handles register themselves, and a synchronous best-effort
+ * wal_checkpoint(TRUNCATE) runs for each on process exit / process.exit
+ * paths where close() was bypassed. SQLite already replays an orphan
+ * -wal on the next open, so this is a belt-and-suspenders consolidation
+ * that must never throw out of the exit hook.
+ */
+
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import { Database } from "bun:sqlite";
+import { mkdtempSync, rmSync, statSync, existsSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import {
+ flushRegisteredWriters,
+ registerWriterDb,
+ unregisterWriterDb,
+ _resetWriterRegistryForTests,
+} from "../../../src/core/search/store-exit.ts";
+
+let tmp: string;
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "o2b-store-exit-"));
+ _resetWriterRegistryForTests();
+});
+
+afterEach(() => {
+ _resetWriterRegistryForTests();
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+function walDb(name: string): { db: Database; path: string } {
+ const path = join(tmp, name);
+ const db = new Database(path);
+ db.exec("PRAGMA journal_mode = WAL");
+ db.exec("CREATE TABLE t(x INTEGER)");
+ db.exec("INSERT INTO t(x) VALUES (1), (2), (3)");
+ return { db, path };
+}
+
+test("flush consolidates the WAL of a registered writer", () => {
+ const { db, path } = walDb("a.sqlite");
+ registerWriterDb(db);
+ // A populated WAL exists before the flush.
+ expect(existsSync(`${path}-wal`)).toBe(true);
+ flushRegisteredWriters();
+ // TRUNCATE checkpoint leaves the -wal at zero bytes (or removed).
+ if (existsSync(`${path}-wal`)) {
+ expect(statSync(`${path}-wal`).size).toBe(0);
+ }
+ db.close();
+});
+
+test("an unregistered writer is left untouched by the flush", () => {
+ const { db, path } = walDb("b.sqlite");
+ registerWriterDb(db);
+ unregisterWriterDb(db);
+ expect(existsSync(`${path}-wal`)).toBe(true);
+ const before = statSync(`${path}-wal`).size;
+ expect(before).toBeGreaterThan(0);
+ flushRegisteredWriters();
+ // The flush must NOT consolidate a DB that unregistered: its WAL is
+ // still populated (no checkpoint ran against it).
+ expect(statSync(`${path}-wal`).size).toBe(before);
+ db.close();
+});
+
+test("the flush never throws on an already-closed handle", () => {
+ const { db } = walDb("c.sqlite");
+ registerWriterDb(db);
+ db.close();
+ expect(() => flushRegisteredWriters()).not.toThrow();
+});
diff --git a/tests/core/search/store.test.ts b/tests/core/search/store.test.ts
index 9820df48..0565dbb4 100644
--- a/tests/core/search/store.test.ts
+++ b/tests/core/search/store.test.ts
@@ -58,6 +58,8 @@ function makeConfig(overrides?: Partial<ResolvedSearchConfig>): ResolvedSearchCo
poolMultiplier: 3,
selfTuningEnabled: false,
}),
+ shutdownGraceMs: 5_000,
+ resumeReindex: false,
...overrides,
});
}
diff --git a/tests/core/search/store.vec.test.ts b/tests/core/search/store.vec.test.ts
index 8f0e45bc..79dfbeea 100644
--- a/tests/core/search/store.vec.test.ts
+++ b/tests/core/search/store.vec.test.ts
@@ -61,6 +61,8 @@ function semanticConfig(
poolMultiplier: 3,
selfTuningEnabled: false,
}),
+ shutdownGraceMs: 5_000,
+ resumeReindex: false,
...overrides,
});
}
diff --git a/tests/core/search/watch-runner.test.ts b/tests/core/search/watch-runner.test.ts
new file mode 100644
index 00000000..0dcddb39
--- /dev/null
+++ b/tests/core/search/watch-runner.test.ts
@@ -0,0 +1,104 @@
+/**
+ * IndexWatchRunner (Indexer Durability suite, t_ea80ddb5): the testable
+ * core of `o2b search watch`'s flush + shutdown coordination. Single-
+ * flight flushes; a shutdown aborts the in-flight pass and awaits it to
+ * settle at a cooperative boundary, bounded by a grace window, so a
+ * SIGTERM never kills a run mid-write. Clock-free and signal-free here -
+ * the CLI wires fs.watch and the OS signals to it.
+ */
+
+import { expect, test } from "bun:test";
+
+import { IndexWatchRunner } from "../../../src/core/search/watch-runner.ts";
+
+const delay = (ms: number): Promise<void> => new Promise((r) => setTimeout(r, ms));
+
+test("shutdown awaits an in-flight flush to completion within the grace window", async () => {
+ let done = false;
+ const runner = new IndexWatchRunner({
+ graceMs: 5_000,
+ index: async () => {
+ await delay(5);
+ done = true;
+ },
+ });
+ void runner.flush();
+ await runner.shutdown();
+ expect(done).toBe(true);
+ expect(runner.isStopped).toBe(true);
+});
+
+test("shutdown aborts the in-flight pass at its cooperative boundary", async () => {
+ let abortedSeen = false;
+ let release = (): void => {};
+ const gate = new Promise<void>((r) => {
+ release = r;
+ });
+ const runner = new IndexWatchRunner({
+ graceMs: 5_000,
+ index: async (signal) => {
+ signal.addEventListener("abort", () => {
+ abortedSeen = true;
+ release();
+ });
+ await gate;
+ },
+ });
+ void runner.flush();
+ await Promise.resolve();
+ await runner.shutdown();
+ expect(abortedSeen).toBe(true);
+});
+
+test("a flush requested after shutdown is refused", async () => {
+ let calls = 0;
+ const runner = new IndexWatchRunner({
+ graceMs: 0,
+ index: async () => {
+ calls++;
+ },
+ });
+ await runner.shutdown();
+ await runner.flush();
+ expect(calls).toBe(0);
+ expect(runner.isStopped).toBe(true);
+});
+
+test("a flush that outlasts the grace window still lets shutdown return", async () => {
+ let waited = -1;
+ const runner = new IndexWatchRunner({
+ graceMs: 5_000,
+ index: () => new Promise<void>(() => {}), // never resolves
+ graceWaiter: (ms) => {
+ waited = ms;
+ return Promise.resolve();
+ },
+ });
+ void runner.flush();
+ await Promise.resolve();
+ await runner.shutdown();
+ expect(waited).toBe(5_000);
+ // The pass is still pending; shutdown returned via the grace waiter.
+ expect(runner.isFlushing).toBe(true);
+});
+
+test("concurrent flush calls coalesce into a single pass", async () => {
+ let calls = 0;
+ let release = (): void => {};
+ const gate = new Promise<void>((r) => {
+ release = r;
+ });
+ const runner = new IndexWatchRunner({
+ graceMs: 0,
+ index: async () => {
+ calls++;
+ await gate;
+ },
+ });
+ const a = runner.flush();
+ const b = runner.flush();
+ expect(runner.isFlushing).toBe(true);
+ release();
+ await Promise.all([a, b]);
+ expect(calls).toBe(1);
+});
diff --git a/tests/helpers/search-fixtures.ts b/tests/helpers/search-fixtures.ts
index a3b5fa18..b4adad9c 100644
--- a/tests/helpers/search-fixtures.ts
+++ b/tests/helpers/search-fixtures.ts
@@ -127,5 +127,7 @@ export function makeConfig(opts: {
poolMultiplier: 3,
selfTuningEnabled: opts.selfTuningEnabled ?? false,
}),
+ shutdownGraceMs: 5_000,
+ resumeReindex: false,
});
}