Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript

feat: freshen the search index on read, rewriting only changed chunks (v1.76.0) (#239)

Serg Eroshe646a49df5e61
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json
index be2dad1a..5d7ef4ce 100644
--- a/.claude-plugin/plugin.json
+++ b/.claude-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.75.0",
+ "version": "1.76.0",
"description": "Plugin-first second brain package for AI agents and humans.",
"author": {
"name": "Open Second Brain contributors"
diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json
index 95b8fb6c..062e7ba9 100644
--- a/.codex-plugin/plugin.json
+++ b/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.75.0",
+ "version": "1.76.0",
"description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
"author": {
"name": "Open Second Brain contributors",
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 83f4e7f2..97220ec5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,23 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+## [1.76.0] - 2026-10-08
+
+Open Second Brain now keeps its search index current without a scheduler: a search or a session start that finds the index more than a minute old answers from it as it is and starts one low-priority background incremental run, so the next read sees the current vault. Nothing stays resident and nothing runs while no agent works. Two indexer changes make those frequent runs cheap: a run that changed no document writes almost nothing, and a changed document rewrites only the chunks that changed, so an edit to a 1 MB daily log costs about 0.2 MB of writes instead of about 9 MB.
+
+### Added
+
+- **Freshen on read.** `search()` (every reading MCP tool, the recall-inject hook, `o2b search query`) and `ensureVaultCurrent` (SessionStart and MCP start) start a detached `o2b search index --freshen <token>` when `last_indexed_at` is older than `search_freshen_interval_s` (default `60`, `0` turns it off, env `OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S`). The reader never waits for the run. An exclusive claim next to the index (`freshen.claim`) keeps concurrent readers from starting a run each; a dead run's claim is taken over after ten minutes by exactly one reader, under an exclusive takeover lock. A held writer lock or an index directory that refuses the claim (read-only mount, no permission) skips the spawn, a run that still meets another writer ends as a skip rather than a failure, and the child is told where its state lives (`--freshen-state`), so even a failure before it resolves its config is recorded and releases the claim; and a read-only open (cross-vault, recall sources) never starts one. The child lowers its CPU priority and runs under `ionice -c3` on Linux or `taskpolicy -b` on macOS when the tool exists. It is keyword-only unless `search_freshen_embeddings` is on.
+- **Backoff and visibility.** The child records its outcome in the per-device `freshen-state.json` beside the index (a new state surface that follows `OPEN_SECOND_BRAIN_SEARCH_DB` / `search_db_path`, so `o2b state status` and `o2b state migrate` report and leave it where a relocated index keeps it); a failed run backs off one minute, doubling to an hour. `o2b search status` prints `freshen`, `index_age`, `last_freshen` and an active `freshen_backoff: until <time>` (`freshen` object under `--json`); `o2b doctor` warns with the new code `freshen-failing` after three failures in a row; runs that changed documents or failed append one row to the new `index_freshen` metrics surface.
+- **Trail code `index-stale`.** A search over an index more than ten minutes old names it on the retrieval trail with `detail.ageSeconds`. The code is added at read time, also to a query-cache hit, and never stored in the cache, so a cached answer reports the index age it is served at and an operator who reindexes on a schedule keeps the cache.
+
+### Changed
+
+- **A no-change index run skips link and alias resolution.** Both are a pure function of the documents and their links; rewriting every link row on every run was the only write a no-change run made. The relation-constraint and tier passes still run, so a schema-pack edit alone still takes effect. `IndexStats.linkResolutionSkipped` reports it. A run killed between its document writes and link resolution leaves a pending marker (`link_resolution_pending`), so the next run resolves even if it changes nothing itself.
+- **Re-indexing a changed document keeps its unchanged chunks in place.** A chunk at the same position with the same content hash, lines, token count, heading and stored full-text form keeps its row, id, full-text entry, entities and vector; only changed chunks are deleted and inserted, and moved ones still carry their vectors. Measured on a 5,812-note vault: one edit to a 1 MB daily log went from 9.3 MB written and 2.1 s to 0.2 MB and 0.17 s; 200 edited notes from 57 MB and 1.8 s to 28 MB and 0.58 s.
+- **The doctor's repair plan sees the index.** `o2b brain doctor --repair` and `brain_doctor` with `repair` now hand the doctor the resolved index path, so `freshen-failing` and `tier-drift` are listed as not auto-repairable, as the report already listed them. `tier-drift` is counted over the pages the caller may read, so below local reach a drifted page the caller cannot read moves the count no more than an absent one, in the report and the repair plan alike.
+- **Docs:** `docs/cli-reference.md` gains "Freshen on read", the two keys, the `--freshen` flag, the status fields and the `index-stale` row; `docs/how-it-works.md` "Keeping the index current"; `docs/architecture.md` states that background work starts only from agent activity; `docs/hermes-cron.md` and `docs/updating.md` say a scheduled reindex is now only needed for embeddings; `docs/metrics.md` lists `index_freshen`. The README names this release.
+
## [1.75.0] - 2026-10-07
Open Second Brain now reaches the two moments the session-start injection never could: the delegated subagent that performs the writes, and the end of the turn. A carrier hands the operator's context to every delegated subagent once, on the SubagentStart event, before the subagent's first prompt, with the write-shaped PostToolUse registration kept as the fallback for runtimes without that event; an opt-in Stop-hook digest surfaces pending Brain hygiene findings as one line, once per change, on the runtimes whose Stop channel is non-blocking and stays silent elsewhere; and an injection body over its budget degrades its sections to headline tiers - the top three ranked bullets each - least important section first, until the body fits, before it drops a whole section.
@@ -8292,6 +8309,7 @@ plugin config (vault field)`, and exits with a clear
- Sandbox vault and plugin manifest fixtures for tests.
- GitHub release workflow for tag-based and manually dispatched releases.
+[1.76.0]: https://github.com/itechmeat/open-second-brain/compare/v1.75.0...v1.76.0
[1.75.0]: https://github.com/itechmeat/open-second-brain/compare/v1.74.1...v1.75.0
[1.74.1]: https://github.com/itechmeat/open-second-brain/compare/v1.74.0...v1.74.1
[1.74.0]: https://github.com/itechmeat/open-second-brain/compare/v1.73.1...v1.74.0
diff --git a/README.md b/README.md
index 19deaa56..b57a6ca8 100644
--- a/README.md
+++ b/README.md
@@ -113,7 +113,7 @@ The full router with readiness criteria is [`install.md`](install.md); native Wi
## What is new
-1.73.0 makes the query embed honest and upgrades safe. Every query embed, from the search lane behind the MCP tools and hooks to the `brain_context_pack` semantic belief order, passes one gate: under a positive `embedding_cost_gate_usd` an unpriced model is refused for any caller that is not local (a hybrid search falls back to keyword-only and says so by trail code), and a query longer than the model's input window is cut to it and disclosed. `embedding_input_window_tokens` declares the window of a model the curated table does not list, and `embedding_extra_body` sends extra request fields to an OpenAI-compatible endpoint. `brain_recall_feedback` and the benchmark, eval and tune reports name the degradations their searches met, and `brain_tune` no longer saves a winner measured without the semantic lane. `o2b brain upgrade` and the automatic upgrade worker apply exactly the plan they computed and refuse a managed file edited in between instead of overwriting it. Every release is described in the [CHANGELOG](CHANGELOG.md).
+1.76.0 keeps the search index current without a scheduler. A search or a session start that finds the index more than a minute old answers from it as it is and starts one low-priority background run, so the next read sees the current vault; nothing stays resident, and nothing runs while no agent works. A changed note now rewrites only the parts of the index that changed, so appending to a large daily log costs a fraction of what it did, and `o2b search status` shows how fresh the index is. Every release is described in the [CHANGELOG](CHANGELOG.md).
## Documentation
diff --git a/docs/architecture.md b/docs/architecture.md
index 9de5bfa5..a31befef 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -265,7 +265,7 @@ writes it, and they act only on a path the caller named.
Three architectural invariants:
-- **Filesystem-first.** No database, no daemon. Every artifact is plain Markdown with YAML frontmatter; backup is `cp -r` or `tar`.
+- **Filesystem-first.** No database, no daemon. Every artifact is plain Markdown with YAML frontmatter; backup is `cp -r` or `tar`. Background work is started only by agent activity and exits when done: the search index is refreshed by a short detached run a stale read starts (freshen on read), never by a resident process.
- **Deterministic core.** The `dream` algorithm is a pure function of inputs (signals, preferences, log, configuration, current time). No LLM calls inside the core. Semantic merging, if needed, is delegated to external agents via the same CLI / MCP surface. External judgment (the bench judge, the hygiene conflict resolver) goes through one sanctioned fail-open boundary: the operator-configured command bridge (`src/core/reliability/command-bridge.ts`).
- **Pre-run snapshot + atomic per-file writes.** Each `dream` run takes a `.snapshots/<run_id>.tar.zst` before any state change (streamed through a staging file, and published under its final name only once complete); per-file writes go through `fs-atomic` (temp + rename). Combined with retention of the most-recent N snapshots, this gives reversible, audit-friendly mutation.
diff --git a/docs/brainstorm/index-freshness/design.md b/docs/brainstorm/index-freshness/design.md
new file mode 100644
index 00000000..77c6a5bd
--- /dev/null
+++ b/docs/brainstorm/index-freshness/design.md
@@ -0,0 +1,101 @@
+# Index freshness on read - the search index keeps itself current while agents work
+
+**Status:** approved design, not implemented
+**Audience:** implementation
+
+## Problem statement
+
+Nothing keeps the search index current. No write path updates it: the MCP writers, ingest, dream and the hooks all write Markdown and leave `brain.sqlite` as it was. The only automatic rebuild (`ensureVaultCurrent`) fires on a missing index or an old schema or chunker version, never on changed content. Freshness therefore depends on the operator running `o2b search index`, keeping `o2b search watch` in the foreground, or installing one of the printed cron recipes. On the reference machine the index went six days without a run: 223 new or changed notes were invisible to every search, and nothing reported that the index was behind.
+
+## Measurements (reference vault, 5,811 Markdown files, 16 cores, NVMe)
+
+| Run | Wall | CPU | Peak RSS | Disk write |
+|---|---|---|---|---|
+| no change | 0.30 s | 0.35 s | ~110 MB | ~0.6 MB |
+| one note added or deleted | 0.30 s | 0.35 s | ~110 MB | ~1 MB |
+| six days of drift (215 added, 8 updated) | 8.4 s | 5.1 s (60% of one core) | ~160 MB | ~1 GB |
+
+The wall times include a bash start. An incremental run is cheap enough to run often. The catch-up run is cheap in CPU but writes about 1 GB, from two sources: the indexer works in WAL and consolidates to `journal_mode=DELETE` on close (a full checkpoint), and every run, an empty one included, repeats the whole-index post-passes (link and alias resolution, constraint flags, tier guard).
+
+## Goals
+
+1. A note written by an agent, or synced in while an agent works, becomes findable without anyone running a command.
+2. Indexing never blocks the caller and does not get in the way of other work on the machine.
+3. Only agent activity triggers indexing. No resident process, no OS scheduler, no Hermes cron.
+4. How far the index is behind is visible to the operator and, when it matters, to the agent.
+
+## Out of scope
+
+- Any daemon, file watcher or timer that runs while no agent works. `o2b search watch` stays as the opt-in foreground watcher.
+- Installing scheduler entries. The print-never-install rule for cron recipes stands.
+- Embedding computation in the background by default (it costs money; see Configuration).
+- Freshness of other derived state (clusters, bridges, dream). The maintenance lane keeps owning those.
+
+## Chosen approach: refresh on read, plus catch-up at session start
+
+A reader that finds the index older than a threshold answers from the index it has and starts one detached incremental index run in the background; the next read sees the result. `ensureVaultCurrent` (SessionStart hook and MCP server start) applies the same check, so a session starts from a caught-up index. Freshness is paid for where it is needed, at read time, and one seam covers agent writes and external changes (Syncthing, Obsidian) alike.
+
+Rejected alternatives:
+
+- **Catch-up at session start only.** Smallest change, but a session can live for days, and notes written during it stay invisible until the next one.
+- **Index after each write.** The agent finds what it wrote, but changes made outside the agent stay invisible, and the trigger has to be threaded through every write path.
+
+## Design
+
+### 1. Trigger and entry points
+
+- New module `src/core/search/freshen.ts` exporting `maybeFreshenIndex(...)`. It decides and, when due, spawns; it never throws and never awaits the run.
+- Call sites:
+ 1. `search()` in `src/core/search/search.ts`, right after the store is opened. Every reading surface goes through it (`brain_search`, the recall tools, `brain_context_pack`, the recall-inject hook, the CLI).
+ 2. `ensureVaultCurrent` in `src/core/maintenance/ensure-current.ts`, so session start and MCP start catch up before the first read.
+- A read-only open (`opts.selfHeal === false`: cross-vault and recall-source search) never freshens: it would write an index into a vault this device does not own. Same rule as self-heal today.
+- Due when `last_indexed_at` in `index_state` is older than `search_freshen_interval_s` (default 60). One read of `index_state`; the vault is not walked by the caller.
+- The run: a detached `o2b search index --vault <v> --config <c> --freshen <runId>`, spawned like the self-heal reindex (streams ignored, `unref`, `detached` on Windows, `env: { ...process.env }`). Keyword-only unless `search_freshen_embeddings` is on.
+
+### 2. Single flight, throttling and priority
+
+- **Claim.** Before spawning, the caller makes an exclusive create of `<vault>/.open-second-brain/freshen.claim` (`wx`), as the self-heal upgrade lock does. The winner spawns and hands the claim to the child, which removes it when it ends. A claim older than 10 minutes is treated as abandoned and taken over. A held writer lock (`isWriterLockHeld`) skips the spawn: some indexer (manual, self-heal, maintenance) is already running.
+- **Throttle.** A successful run stamps `last_indexed_at`, so for the next interval every session's reads find the index fresh and spawn nothing. Under active use that is at most one run per interval.
+- **Backoff.** A failed run writes a failure marker next to the claim. Spawns then wait 1 min, doubling up to 1 h, so a broken vault does not spawn a failing process per query. `doctor` reports the streak.
+- **Priority.** The child lowers its CPU priority at start with `os.setPriority` (all platforms). On Linux the command is prefixed with `ionice -c3` and on macOS with `taskpolicy -b` when the tool exists; without it the run proceeds at normal I/O priority.
+- **Caller cost.** One `index_state` read, one lock probe and one exclusive create: single-digit milliseconds. The run is always in the background.
+
+### 3. Write cost and visibility
+
+- **A no-change run writes nothing heavy.** When a run added, updated and deleted no document, the whole-index post-passes are skipped and only `last_indexed_at` is stamped, in one small transaction. Frequent runs must not wear the disk.
+- **Catch-up write volume, measured then fixed.** Per-phase written bytes are recorded during a run (from `/proc/self/io` on Linux; the measurement is skipped elsewhere). The dominant phase is fixed: candidates are the WAL-plus-checkpoint double write, whole-index link and alias resolution where only changed documents need it, and FTS segment merges. Target: a 200-file catch-up writes at most ~100 MB instead of ~1 GB. If the target needs a larger rework than this feature warrants, the work stops and the numbers go back to the owner before going further.
+- **Visibility:**
+ - `o2b search status` shows the index age, the outcome of the last background run and any active backoff.
+ - `o2b doctor --readiness` gains a freshness line and warns on a failure streak.
+ - A search whose index is more than 10 minutes old carries the new trail code `index-stale` with `detail.ageSeconds`; a normal lag of under a minute adds nothing.
+ - Every spawn decision and outcome is recorded like `self_heal_reindex` (skipped with reason, spawned, succeeded, failed with reason).
+
+### 4. Configuration
+
+Machine config, each key with an `OPEN_SECOND_BRAIN_*` environment override:
+
+| Key | Meaning | Default |
+|---|---|---|
+| `search_freshen_interval_s` | how old the index may get before a read refreshes it; `0` turns the feature off | `60` |
+| `search_freshen_embeddings` | also compute embeddings in the background run | `false` |
+
+On by default: a feature nobody turns on fixes nothing. The CHANGELOG says that a search may now start a short background process and how to turn it off.
+
+## Compatibility
+
+- `o2b search watch`, `o2b search query --auto-refresh` and the cron recipes are unchanged.
+- The docs of `o2b search reindex --cron-template` say it is no longer needed for keyword freshness, only for a periodic embedding pass.
+- `docs/architecture.md` "no daemon": a short background process started in response to agent work is not a daemon; nothing stays resident.
+- `docs/hermes-cron.md`: a Hermes reindex job becomes optional.
+
+## Testing
+
+1. `maybeFreshenIndex` in isolation, with the spawner injected: index age below and above the interval, `0` disables, own and foreign claim, abandoned claim takeover, writer lock held, backoff active and expired.
+2. `search()`: a stale index triggers one spawn, a fresh one none, a read-only open never.
+3. Indexer: a no-change run skips the post-passes and stamps only `last_indexed_at`.
+4. One end-to-end test with a real child: a new note becomes findable after the background run. Runs on Linux and in the Windows shards.
+5. Write volume: a measurement script, not a unit test; before and after numbers go into the PR.
+
+## Release
+
+One PR, minor version (next minor after the current `main`), with the bump, the CHANGELOG entry and its compare link, `docs/cli-reference.md`, `docs/how-it-works.md`, `docs/architecture.md`, `docs/hermes-cron.md`, the README release line, and regenerated Codex mirrors if a skill changes.
diff --git a/docs/brainstorm/index-freshness/plan.md b/docs/brainstorm/index-freshness/plan.md
new file mode 100644
index 00000000..81720ae3
--- /dev/null
+++ b/docs/brainstorm/index-freshness/plan.md
@@ -0,0 +1,172 @@
+# Index Freshness on Read Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** A read that finds the search index older than a threshold starts one low-priority background incremental index run, so the index stays current while agents work, with no daemon and no scheduler.
+
+**Architecture:** A new `src/core/search/freshen.ts` owns the decision (pure), the per-device state file and claim, and the detached spawn. `search()` and `ensureVaultCurrent` call it. The child is `o2b search index --freshen <runId>`. The indexer skips its whole-index post-passes on a no-change run, and the catch-up write volume is measured per phase and its dominant cause fixed.
+
+**Tech Stack:** Bun 1.4 + TypeScript, `bun:sqlite`, `bun:test`.
+
+**Spec:** `docs/brainstorm/index-freshness/design.md`
+
+## Global Constraints
+
+- No resident process, timer, watcher or scheduler entry; indexing starts only from an agent-driven read or session start.
+- A read never waits for indexing; `maybeFreshenIndex` never throws.
+- A read-only open (`opts.selfHeal === false`) never freshens.
+- Config keys: `search_freshen_interval_s` (default `60`, `0` = off, env `OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S`), `search_freshen_embeddings` (default `false`, env `OPEN_SECOND_BRAIN_SEARCH_FRESHEN_EMBEDDINGS`).
+- Claim file `<dirname(dbPath)>/freshen.claim`, abandoned after 10 minutes; state file `<dirname(dbPath)>/freshen-state.json` (per device, never in `Brain/`).
+- Backoff after a failed run: 60 s doubling, capped at 3600 s.
+- Trail code `index-stale` with `detail.ageSeconds` when the index is older than 600 s.
+- Background child: CPU priority lowered with `os.setPriority`; `ionice -c3` (Linux) or `taskpolicy -b` (macOS) prefix when on PATH.
+- Catch-up write target: a ~200-file catch-up writes at most ~100 MB; if that needs a larger rework, stop and report numbers.
+- Tests never spawn a real child except the one end-to-end test; `tests/setup.ts` pins the interval to `0`.
+- Version: next minor after `main`, bump inside the PR; English only; no AI co-author trailers.
+
+## Review Focus
+
+1. Many sessions searching at once on a stale index: exactly one child runs; the others skip without error.
+2. A child killed mid-run (reboot, OOM): the claim is taken over after 10 minutes and no failure streak is invented.
+3. A vault whose index cannot be written (read-only disk, corrupt file): the child fails, backoff grows, searches keep working on the old index, `doctor` names the streak.
+4. Cross-vault (`global`) and recall-source searches: never spawn into a foreign vault.
+5. A schema-pack edit with no document change: the no-change fast path must still recompute relation-constraint flags.
+
+Each item has a test in the owning task (Tasks 2, 2, 3, 4, 5 respectively).
+
+---
+
+### Task 1: Freshen configuration
+
+**Files:**
+- Modify: `src/core/search/types.ts` (add `ResolvedFreshenConfig`, optional `freshen` on `ResolvedSearchConfig`)
+- Modify: `src/core/search/index.ts` (`resolveSearchConfig`)
+- Modify: `tests/setup.ts` (pin `OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S=0` unless set)
+- Test: `tests/core/search/freshen-config.test.ts`
+
+**Interfaces:**
+- Produces: `interface ResolvedFreshenConfig { readonly intervalSeconds: number; readonly embeddings: boolean; readonly configPath: string | null }`; `ResolvedSearchConfig.freshen?: ResolvedFreshenConfig` (absent = off, so hand-built configs in tests stay off).
+
+- [ ] **Step 1: Failing tests** β€” default resolves `{ intervalSeconds: 60, embeddings: false, configPath }`; env `OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S=0` gives `0`; config key `search_freshen_interval_s: "300"` gives `300`; `"-1"` and `"abc"` throw naming the key; `search_freshen_embeddings: "true"` gives `true`. Tests save and restore the env var because the preload pins it.
+- [ ] **Step 2: Run** `bun test tests/core/search/freshen-config.test.ts` β€” FAIL (`freshen` undefined).
+- [ ] **Step 3: Implement** with the existing `parseInteger(envOrConfig(...), 60, "search_freshen_interval_s", { min: 0 })` and `parseBool` helpers; `configPath: opts.configPath ?? null`.
+- [ ] **Step 4: Preload** β€” in `tests/setup.ts`, next to `O2B_DEVICE_ID`: `if (process.env["OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S"] === undefined) process.env["OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S"] = "0";` with a comment that the suite must never spawn background indexers.
+- [ ] **Step 5: Run** the new test and `bun run typecheck` β€” PASS.
+- [ ] **Step 6: Commit** `feat(search): freshen interval and embeddings config`.
+
+### Task 2: Decision, claim, state and spawn (`freshen.ts`)
+
+**Files:**
+- Create: `src/core/search/freshen.ts`
+- Test: `tests/core/search/freshen.test.ts`
+
+**Interfaces:**
+- Consumes: `ResolvedFreshenConfig` (Task 1), `isWriterLockHeld(dbPath)` from `store/writer-lock.ts`.
+- Produces:
+ - `FRESHEN_SKIP = { off, fresh, backoff, writerLock, claimed, readOnly, noIndex }` (string values `off`, `fresh`, `backoff`, `writer_lock`, `claimed`, `read_only`, `no_index`).
+ - `decideFreshen(input: { nowMs: number; lastIndexedAt: string | null; intervalSeconds: number; backoffUntilMs: number | null }): { action: "spawn" } | { action: "skip"; reason: FreshenSkip }` (pure; lock and claim are checked after it).
+ - `claimFreshen(dir: string, nowMs: number): string | null` β€” exclusive create of `freshen.claim` holding `{ token, at }`; takes over a claim older than 600 s; returns the token or `null`.
+ - `releaseFreshen(dir: string, token: string): void` β€” removes the claim only if it still holds `token`.
+ - `readFreshenState(dir) / writeFreshenState(dir, state)` with `FreshenState { failures: number; backoffUntil: string | null; lastOutcome: "completed" | "failed" | null; lastRunAt: string | null; lastDurationMs: number | null; lastError: string | null; lastChanged: number | null }`; a missing or torn file reads as the empty state.
+ - `nextBackoffMs(failures: number): number` β€” `min(60_000 * 2 ** (failures - 1), 3_600_000)`.
+ - `freshenCommand(base: string[], has: (tool: string) => boolean, platform: NodeJS.Platform): string[]` β€” prefixes `ionice -c3` on linux / `taskpolicy -b` on darwin when `has(tool)`.
+ - `maybeFreshenIndex(config: ResolvedSearchConfig, opts: { lastIndexedAt: string | null; readOnly?: boolean; nowMs?: number; spawn?: (argv: string[]) => void }): FreshenDecision` β€” never throws; returns `"spawned"` or the skip reason.
+
+- [ ] **Step 1: Failing tests** for `decideFreshen`: `intervalSeconds 0` β†’ `off`; `lastIndexedAt null` β†’ `no_index` (self-heal owns a missing index); age 30 s / interval 60 β†’ `fresh`; age 61 s β†’ `spawn`; `backoffUntilMs > now` β†’ `backoff`.
+- [ ] **Step 2: Failing tests** for the claim: first claim returns a token; a second returns `null` (`claimed`); a claim file with `at` 601 s ago is taken over; `releaseFreshen` with a foreign token leaves the file (Review Focus 1, 2).
+- [ ] **Step 3: Failing tests** for state and backoff: round-trip; torn JSON reads as empty; `nextBackoffMs(1) === 60_000`, `(2) === 120_000`, `(10) === 3_600_000`.
+- [ ] **Step 4: Failing tests** for `freshenCommand`: linux with `ionice` β†’ `["ionice", "-c3", ...base]`; linux without β†’ `base`; darwin with `taskpolicy` β†’ `["taskpolicy", "-b", ...base]`; win32 β†’ `base`.
+- [ ] **Step 5: Failing tests** for `maybeFreshenIndex` with an injected `spawn` and a temp dir: stale index β†’ spawn called once with `search index --vault <v> --freshen <id>` (plus `--config <p>` when set); fresh β†’ not called; `readOnly: true` β†’ `read_only`, not called (Review Focus 4); held writer lock (create the lock dir the way `writer-lock.ts` does) β†’ `writer_lock`; a `spawn` that throws β†’ returns a skip, does not throw.
+- [ ] **Step 6: Run** `bun test tests/core/search/freshen.test.ts` β€” FAIL.
+- [ ] **Step 7: Implement** `freshen.ts`. The real spawner reuses the self-heal pattern from `src/core/maintenance/ensure-current.ts` (`Bun.spawn`, streams `ignore`, `windowsHide`, `detached` on win32, `env: { ...process.env }`, `unref`) and the `o2bCommand()` resolution (move it to a small shared helper `src/core/maintenance/o2b-command.ts` and import it from both). The claim token is passed as `--freshen <token>`.
+- [ ] **Step 8: Run** tests and typecheck β€” PASS.
+- [ ] **Step 9: Commit** `feat(search): freshen decision, claim, state and spawn`.
+
+### Task 3: The background child (`o2b search index --freshen`)
+
+**Files:**
+- Modify: `src/cli/search/verbs/indexing.ts` (`cmdSearchIndex`)
+- Modify: `src/core/brain/metrics.ts` only if surfaces are registered there (add `index_freshen`)
+- Test: `tests/cli/search-index-freshen.test.ts`
+
+**Interfaces:**
+- Consumes: `releaseFreshen`, `readFreshenState`, `writeFreshenState`, `nextBackoffMs` (Task 2).
+- Produces: flag `--freshen <token>`; on success the state gets `failures: 0, lastOutcome: "completed", lastChanged: added+updated+deleted`; on failure `failures+1`, `backoffUntil = now + nextBackoffMs(failures)`, `lastError`; a metric row on surface `index_freshen` only on failure or when `lastChanged > 0`; the claim is released in `finally`.
+
+- [ ] **Step 1: Failing tests** (in-process `cmdSearchIndex` with a temp vault): `--freshen <token>` after `claimFreshen` β†’ exit 0, claim gone, state `completed`; a vault whose db path is unwritable (a directory squatting `brain.sqlite`) β†’ state `failed`, `failures 1`, `backoffUntil` set, claim gone (Review Focus 3); a second failure β†’ `failures 2` and a doubled backoff.
+- [ ] **Step 2: Run** β€” FAIL.
+- [ ] **Step 3: Implement**: parse `freshen`; when present, `os.setPriority(process.pid, 10)` in a try; `embeddings: cfg.freshen?.embeddings === true` unless `--embeddings` was given; wrap the run in try/catch/finally writing state, the metric row and releasing the claim; the exit code stays as today.
+- [ ] **Step 4: Run** β€” PASS. **Step 5: Commit** `feat(search): background freshen child records its outcome`.
+
+### Task 4: Call sites and the `index-stale` trail code
+
+**Files:**
+- Modify: `src/core/search/search.ts` (after the store opens)
+- Modify: `src/core/maintenance/ensure-current.ts` (after the rebuild check)
+- Modify: `src/core/search/retrieval-trail.ts` (code, membership, description)
+- Modify: `docs/cli-reference.md` (trail table row; required by `tests/docs/retrieval-trail-doc.test.ts`)
+- Test: `tests/core/search/freshen-search.test.ts`, extend `tests/core/maintenance/ensure-current*.test.ts`
+
+**Interfaces:**
+- Consumes: `maybeFreshenIndex` (Task 2). A test seam: `SearchOptions.freshenSpawn?: (argv: string[]) => void` passed through to `maybeFreshenIndex({ spawn })`.
+- Produces: `RETRIEVAL_DEGRADATION.indexStale = "index-stale"` with `detail: { ageSeconds }`; `EnsureCurrentResult.freshen: FreshenDecision | null`.
+
+- [ ] **Step 1: Failing tests**: a search over an index whose `last_indexed_at` is 5 min old with `freshen.intervalSeconds 60` calls the injected spawn once and returns results; `selfHeal: false` never calls it; age 11 min adds `{ code: "index-stale", detail: { ageSeconds } }` to `degraded`; age 2 min does not; interval `0` with age 11 min still reports `index-stale` but never spawns. `ensureVaultCurrent` on a current-schema stale index returns `freshen: "spawned"` (spawn injected through `EnsureCurrentOptions.freshenSpawn`), and on a rebuild-needed index leaves `freshen: null`.
+- [ ] **Step 2: Run** β€” FAIL.
+- [ ] **Step 3: Implement**: in `search()` read `store.getState(LAST_INDEXED_AT_STATE_KEY)` once; call `maybeFreshenIndex(effectiveConfig, { lastIndexedAt, readOnly: opts.selfHeal === false, spawn: opts.freshenSpawn })`; push `index-stale` when the age exceeds 600 s. In `ensureVaultCurrent`, when no rebuild is needed, open the index read-only, read the stamp, call `maybeFreshenIndex`.
+- [ ] **Step 4: Add the trail code** to `RETRIEVAL_DEGRADATION`, `RETRIEVAL_DEGRADATION_CODES` (after the keyword codes) and `describeRetrievalDegradation`; add the `docs/cli-reference.md` row: "the index was last updated more than ten minutes ago, so notes changed since then may be missing; a background run has been started unless freshening is off or backing off".
+- [ ] **Step 5: Run** the new tests, `tests/docs/retrieval-trail-doc.test.ts`, `tests/core/search/` β€” PASS. **Step 6: Commit** `feat(search): reads refresh a stale index in the background`.
+
+### Task 5: No-change run skips the whole-index post-passes
+
+**Files:**
+- Modify: `src/core/search/indexer.ts` (around the post-passes after the delete loop)
+- Modify: `src/core/search/store/state.ts` (new key `schema_pack_digest`)
+- Test: `tests/core/search/indexer-noop.test.ts`
+
+**Interfaces:**
+- Produces: state key `SCHEMA_PACK_DIGEST_STATE_KEY = "schema_pack_digest"`; `IndexStats.postPassesSkipped: boolean`.
+
+- [ ] **Step 1: Failing tests**: index a small vault, index again with no change β†’ `postPassesSkipped: true` and `last_indexed_at` advanced; change a note β†’ `false`; change only `Brain/_brain.yaml` `link_constraints` β†’ `false` and the blocked flag recomputed (Review Focus 5); a forced run β†’ `false`.
+- [ ] **Step 2: Run** β€” FAIL.
+- [ ] **Step 3: Implement**: compute a SHA-256 of the raw `_brain.yaml` bytes (empty string when absent); skip `resolveLinkTargets`, `resolveAliasTargets`, `recomputeRelationConstraintFlags` and the tier pass when `added + updated + deleted === 0`, not forced, and the digest equals the stored one; store the digest after the post-passes run. Keep the stamp, the census and the deferred-reason reads.
+- [ ] **Step 4: Run** the new test and `tests/core/search/indexer*.test.ts` β€” PASS. **Step 5: Commit** `perf(search): a no-change index run skips the whole-index post-passes`.
+
+### Task 6: Catch-up write volume, measured then fixed
+
+**Files:**
+- Create: `scripts/measure-index-writes.ts` (Linux; reads `/proc/self/io` `write_bytes` around phases; prints a table)
+- Modify: `src/core/search/indexer.ts` (optional `onPhase` hook in `IndexOptions`, no behaviour change)
+- Modify: the module of the dominant phase (decided by the measurement)
+
+- [ ] **Step 1:** Add `onPhase?: (phase: "walk" | "post" | "embed" | "stamp" | "close") => void` to the index options, called at phase boundaries. Commit.
+- [ ] **Step 2:** The script copies the reference vault to a temp dir, indexes it, rewinds the copy's `last_indexed_at` and touches 200 notes with new content, then runs a catch-up index and prints bytes per phase plus the close (WAL checkpoint). Run it; record the table in the PR description.
+- [ ] **Step 3:** Fix the dominant phase. Expected candidates and fixes: per-file transactions β†’ one transaction per batch of files; full-index link/alias resolution β†’ restrict to links touching changed documents; WAL-to-DELETE checkpoint on every close β†’ keep WAL between runs where the backing allows it. Write a regression test for the behaviour changed (not for byte counts).
+- [ ] **Step 4:** Re-run the script; target ≀ ~100 MB. If not reachable without a broader rework, stop, keep what helped, and report the numbers. Commit `perf(search): <what was fixed>`.
+
+### Task 7: Freshness in `search status` and `doctor`
+
+**Files:**
+- Modify: `src/cli/search/verbs/status.ts`
+- Modify: the readiness check module under `src/core/brain/doctor/` that prints index lines
+- Test: extend the status and readiness tests
+
+- [ ] **Step 1: Failing tests**: `o2b search status` prints `freshen: every 60s` (or `off`), `index_age: <n>s`, `last_freshen: completed <ts> (<n> changed)` / `failed <ts>: <error>`, and `freshen_backoff_until: <ts>` when active; `--json` carries the same fields under `freshen`. Readiness warns when `failures >= 3`.
+- [ ] **Step 2: Run** β€” FAIL. **Step 3: Implement** from `readFreshenState(dirname(dbPath))`. **Step 4: Run** β€” PASS. **Step 5: Commit** `feat(search): show index freshness in status and doctor`.
+
+### Task 8: End-to-end with a real child
+
+**Files:**
+- Test: `tests/e2e/index-freshen.test.ts`
+
+- [ ] **Step 1:** Temp vault + config, index it, write a new note, set `OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S=1` for a spawned `o2b search query` (via `runCli` helpers), wait up to 20 s polling the state file for `lastOutcome: "completed"`, then search finds the new note. Timeout generous for Windows.
+- [ ] **Step 2: Run** on Linux β€” PASS. **Step 3: Commit** `test(search): background freshen end to end`.
+
+### Task 9: Docs, version, gates, PR
+
+**Files:** `docs/cli-reference.md` (keys, `--freshen`, status fields), `docs/how-it-works.md`, `docs/architecture.md` ("no daemon" clarification), `docs/hermes-cron.md`, the reindex `--cron-template` help text, `README.md` release line, `CHANGELOG.md` (+ compare link), `package.json` + `bun run scripts/sync-version.ts`, `bun run sync-plugin-mirrors` if a skill changed.
+
+- [ ] **Step 1:** Write the docs and CHANGELOG (Added: freshen on read, trail code, status fields; Changed: no-change runs, catch-up writes with the measured numbers; note that a search may start a short background process and `search_freshen_interval_s: 0` turns it off).
+- [ ] **Step 2:** Bump the minor version, sync, `--check`.
+- [ ] **Step 3:** Gates: `bun run typecheck`, `bun run lint`, `bun run fmt:check`, `bun run link-ratchet:check`, `bun run sync-plugin-mirrors:check`, the OpenClaw bundle diff with Bun 1.4.0, `bun test --parallel`.
+- [ ] **Step 4:** Commit, push the branch, open the PR with the measurement tables; wait for CI and review bots; fix findings.
diff --git a/docs/cli-reference.md b/docs/cli-reference.md
index 0188422d..fd52a8da 100644
--- a/docs/cli-reference.md
+++ b/docs/cli-reference.md
@@ -1827,6 +1827,9 @@ o2b search reindex Rebuild the SQLite + FTS5 index from scratch
machine-local pid
o2b search index Incrementally update the index; --embeddings computes vectors, --progress watches it
--force-cost bypasses the embedding cost gate (since v0.36.0)
+ --freshen <token> [--freshen-state <dir>] is the background run freshen on
+ read starts (since v1.76.0): lowered priority, no output, outcome in
+ <dir>/freshen-state.json; meeting another index writer is a skip
o2b search vector-backfill Run the vector phase ALONE for indexed chunks that have no vector -
no vault walk, no re-chunking, no frontmatter pass (since v1.43.0).
Dry-run by DEFAULT: it counts the pending chunks, reports the
@@ -1875,6 +1878,11 @@ o2b search status Index status; since v0.36.0 also reports the activ
<n> in the last 30 days); --json carries it as the event_time object
(documents, with_event_time, earliest, latest, recent_window_days,
in_recent_window)
+ Since v1.76.0 it prints freshen (every <n>s or off), index_age, last_freshen
+ (completed <ts> (<n> changed) / failed <ts> (<n> in a row): <error> /
+ (none)) and freshen_backoff (until <ts>) while a backoff is active; --json
+ carries them as the freshen object (interval_s, index_age_s, last_outcome,
+ last_run_at, last_changed, last_error, failures, backoff_until)
o2b search check Pre-flight diagnostics: vault, index directory, SQLite/FTS5, the
vector extension, the embedding key, the provider, the vector ABI stamp
--integrity additionally runs a full PRAGMA quick_check over the index
@@ -2027,6 +2035,7 @@ so there is no separate lane field that could drift from it:
| Code | The narrowing it reports |
| ---- | ------------------------ |
+| `index-stale` | the index was last updated more than ten minutes ago (`detail.ageSeconds`), so notes changed since then may be missing; the code reports the index age only and does not say a refresh is running: freshen on read starts a background run when it can, and starts none when it is off, a failed run is backing off, another run or writer holds the index, the index is read-only for this reader, or the run cannot be spawned |
| `keyword-fts-match-empty` | the query tokenised to an empty FTS match, so the keyword lane never ran |
| `keyword-trigram-lane-fault` | the trigram candidate lane could not be read; `detail.fault` carries that lane's own classification |
| `semantic-embeddings-absent` | the index holds no compatible embedding |
@@ -2363,11 +2372,57 @@ Two opt-in ranking guards and one deadline join the suite:
| `search_relational_rerank_pin` | `OPEN_SECOND_BRAIN_SEARCH_RELATIONAL_RERANK_PIN` | `false` | Rerank may promote a relational-origin candidate but never sink it below its pre-rerank order (the relevance floor is untouched) |
| `search_metadata_boost_gate` | `OPEN_SECOND_BRAIN_SEARCH_METADATA_BOOST_GATE` | `false` | A query whose keyword lane returned no hits contributes zero from every additive metadata/structural boost layer |
| `search_hybrid_deadline_ms` | `OPEN_SECOND_BRAIN_SEARCH_HYBRID_DEADLINE` | `15000` | Wall-clock budget over the whole composite hybrid path (embed, semantic top-k, rerank, second pass); on expiry the abandoned embed and rerank requests are aborted, the search completes with what it has (keyword-only, or the pre-rerank order with exclusions, trust gate and the other post-rank phases still applied), reports `hybridDeadlineExceeded` and is not cached; `0` disables |
+| `search_freshen_interval_s` | `OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S` | `60` | Freshen on read (since v1.76.0): a search or a session start that finds the index older than this many seconds starts one background incremental `o2b search index` at low CPU and I/O priority, and answers from the index it has; `0` turns it off. See "Freshen on read" below |
+| `search_freshen_embeddings` | `OPEN_SECOND_BRAIN_SEARCH_FRESHEN_EMBEDDINGS` | `false` | Let that background run compute embeddings too (it costs money); off, it indexes keyword-only and leaves vectors to an explicit `--embeddings` run |
Both guards are off by default and leave every score byte-identical;
the deadline is on by default because its lane budgets already summed
to more, and it bounds exactly the phases with no budget of their own.
+### Freshen on read (since v1.76.0)
+
+Nothing schedules index runs: no daemon, no OS timer, no Hermes job. A
+search (every reading MCP tool, the recall-inject hook, `o2b search
+query`) and a session start (`ensureVaultCurrent`) read the index's
+`last_indexed_at`; when it is older than `search_freshen_interval_s`
+(default 60 s, `0` turns it off), the reader answers from the index it
+has and starts one detached `o2b search index --freshen <token>`, so the
+next read sees the current vault. Only agent activity triggers it.
+
+- **One run at a time.** The reader takes an exclusive claim,
+ `<index dir>/freshen.claim`, and skips when another run holds it, an
+ indexer holds the writer lock, or the index directory refuses the claim
+ (a read-only mount, no permission). A claim older than ten minutes belongs
+ to a run that died and is taken over by exactly one reader, under an
+ exclusive takeover lock. A run that still meets another writer ends as
+ a skip, not a failure.
+- **Out of the way.** The child lowers its CPU priority and runs under
+ `ionice -c3` on Linux or `taskpolicy -b` on macOS when the tool exists.
+ It indexes keyword-only unless `search_freshen_embeddings` is on.
+- **Cheap when nothing changed.** A run that added, updated and deleted
+ no document skips link and alias resolution and writes almost nothing.
+ A changed document rewrites only the chunks that changed, so a daily
+ log that grows all day costs its appended tail per run.
+- **Failures back off.** A failed run records its error in
+ `<index dir>/freshen-state.json` and the next run waits 1 minute,
+ doubling to an hour. `o2b search status` shows the last outcome and
+ the backoff; `o2b doctor` warns with `freshen-failing` after three
+ failures in a row. Runs that changed something or failed append one
+ row to the `index_freshen` metrics surface.
+- **Staleness is named.** A search over an index more than ten minutes
+ old carries the trail code `index-stale` with `detail.ageSeconds`.
+ The code names the age only; whether a background run started is not
+ part of it (see the cases above where freshen on read starts none).
+ The code is added when the answer is served, also on a query-cache
+ hit, and is never stored in the cache.
+- **Never a foreign vault.** Cross-vault and recall-source reads open
+ other indexes read-only and never start a run there.
+
+`o2b search watch` stays available for a foreground watcher, and the
+cron recipes of `o2b search reindex --cron-template` and the maintenance
+lane are still printed, but keyword freshness no longer needs them; a
+scheduled run remains the way to compute embeddings periodically.
+
### The reserved visibility token (since v1.54.0)
```text
diff --git a/docs/hermes-cron.md b/docs/hermes-cron.md
index da2841ad..b31c8f07 100644
--- a/docs/hermes-cron.md
+++ b/docs/hermes-cron.md
@@ -78,7 +78,7 @@ When `--silent-if-empty` is set and the window has no changes, the command write
`dream` is itself idempotent β€” running it on top of an unchanged Brain is a no-op (no log entry written, no snapshot taken). The chained form is safe at any cadence.
-The alternative to a hand-chained dream job is the maintenance lane, which runs dream, reindex, bridges and clusters under one lease and only inside its window: `o2b brain maintenance run --vault /path/to/vault --cron-template` prints a script and the `hermes cron create` line that schedules it (hourly by default; `--interval`, `--window H-H --tz ZONE` narrow it). The script is silent when the lane succeeds or a gate skips, and prints the lane's JSON report when a task failed, timed out or was refused, so Hermes delivers only the runs that need attention. Every recipe this CLI prints - the maintenance lane, `o2b search reindex --cron-template` and `o2b partner codegraph resync --cron-template` - also prints a systemd user timer with `--format systemd`, for a host where Hermes does not own the schedule.
+The alternative to a hand-chained dream job is the maintenance lane, which runs dream, reindex, bridges and clusters under one lease and only inside its window: `o2b brain maintenance run --vault /path/to/vault --cron-template` prints a script and the `hermes cron create` line that schedules it (hourly by default; `--interval`, `--window H-H --tz ZONE` narrow it). The script is silent when the lane succeeds or a gate skips, and prints the lane's JSON report when a task failed, timed out or was refused, so Hermes delivers only the runs that need attention. Since v1.76.0 the keyword index needs no scheduled reindex: freshen on read keeps it current while agents work (see "Freshen on read" in `docs/cli-reference.md`), so a scheduled reindex is only worth keeping to compute embeddings periodically. Every recipe this CLI prints - the maintenance lane, `o2b search reindex --cron-template` and `o2b partner codegraph resync --cron-template` - also prints a systemd user timer with `--format systemd`, for a host where Hermes does not own the schedule.
## Discipline report (daily logging-discipline sanity-check)
diff --git a/docs/how-it-works.md b/docs/how-it-works.md
index 7e8962bb..5f8062bc 100644
--- a/docs/how-it-works.md
+++ b/docs/how-it-works.md
@@ -1342,6 +1342,22 @@ Full design and migration notes:
and the matching implementation plan
[`docs/plans/2026-05-16-brain-search-impl.md`](plans/2026-05-16-brain-search-impl.md).
+### Keeping the index current (since v1.76.0)
+
+The index is kept current by the reads that need it. When a search or a
+session start finds `last_indexed_at` older than
+`search_freshen_interval_s` (60 s by default), it answers from the
+index as it is and starts one detached, low-priority incremental run
+(`o2b search index --freshen`); an exclusive claim next to the index
+keeps concurrent readers from starting one each, and a failed run backs
+off from one minute to an hour. Nothing stays resident: with no agent
+working, nothing indexes. Two indexer changes make frequent runs cheap:
+a run that changed no document skips link and alias resolution, and a
+changed document keeps every chunk whose position and content did not
+change in place (row, full-text entry, entities, vector), so appending
+to a 1 MB daily log costs about 0.2 MB of writes instead of about 9 MB.
+Details: "Freshen on read" in `docs/cli-reference.md`.
+
## Workspace reach and proactive insight (since v0.38.0)
Two suites extend where the Brain can be reached and what it
diff --git a/docs/metrics.md b/docs/metrics.md
index 7051f663..e17254d0 100644
--- a/docs/metrics.md
+++ b/docs/metrics.md
@@ -82,6 +82,7 @@ writes, and a metric record is a summary, not a report.
| `vault_vitals` | `o2b brain vitals` | `preferences_scanned`, `domain_diversity`, `connectivity_index`, `orphan_count`, `gap_pressure` |
| `self_heal_upgrade` | `o2b brain upgrade --self-heal` (the detached worker `ensureVaultCurrent` starts) | `outcome` (`applied` / `failed`), `duration_ms`, `files` (rewritten, or pending on `failed`), `error` (on `failed`). An attempt that finds nothing pending writes no row |
| `self_heal_reindex` | `ensureVaultCurrent` (parent) and `o2b search reindex --self-heal <run-id>` (child) | parent row: `decision` (`spawned` / `skipped_writer_lock`), `run_id` (on `spawned`). Child row: `outcome` (`completed` / `failed`), `run_id`, `duration_ms`, `error` (on `failed`) |
+| `index_freshen` (since v1.76.0) | `o2b search index --freshen <token>` (the background run freshen on read starts) | `outcome` (`completed` / `failed`), `duration_ms`, `changed` (on `completed`), `error` and `failures` (on `failed`). A row is written only when the run changed documents or failed |
| `maintenance_spend` (since v1.64.0) | `o2b brain maintenance run`, MCP `brain_maintenance` (`run`) | `task` (`reindex`), `model` (null when the config names none), `tokens`, `estimated_usd`, `forced` (`force-cost` overrode a positive gate), `lane` (always true: both front doors are the lane) |
Payload fields marked "(lane runs)" appear only on maintenance-lane
diff --git a/docs/updating.md b/docs/updating.md
index 06e47d90..312d6791 100644
--- a/docs/updating.md
+++ b/docs/updating.md
@@ -47,6 +47,15 @@ instruction files such as `CLAUDE.md`/`AGENTS.md`, installed
`.claude/skills/`) and warns with the exact replacement for any stale
reference it finds (`removed-tool-reference`).
+## Upgrading to 1.76.0
+
+No step is required. Searches and session starts now keep the search
+index current on their own: an index older than a minute gets one
+background incremental run. If you scheduled `o2b search index` or
+`o2b search reindex` only to keep keyword search fresh, that job can go;
+keep it if it computes embeddings. To turn the background runs off, set
+`search_freshen_interval_s: "0"`.
+
## Upgrading to 1.73.0
No step is required unless you set a positive `embedding_cost_gate_usd`
diff --git a/openclaw.plugin.json b/openclaw.plugin.json
index 8bd04bc2..fcb8a5fc 100644
--- a/openclaw.plugin.json
+++ b/openclaw.plugin.json
@@ -2,7 +2,7 @@
"id": "open-second-brain",
"name": "Open Second Brain",
"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",
- "version": "1.75.0",
+ "version": "1.76.0",
"activation": { "onStartup": true },
"skills": ["./skills"],
"contracts": {
diff --git a/package.json b/package.json
index 0cdd443e..3be6572f 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.75.0",
+ "version": "1.76.0",
"private": false,
"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.",
"keywords": [
diff --git a/plugin.yaml b/plugin.yaml
index 75c93f72..cd8f1d6a 100644
--- a/plugin.yaml
+++ b/plugin.yaml
@@ -1,5 +1,5 @@
name: open-second-brain
-version: "1.75.0"
+version: "1.76.0"
description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."
author: "Open Second Brain contributors"
memory_provider: true
diff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.json
index 47ed2f3e..42e7b65c 100644
--- a/plugins/codex/.codex-plugin/plugin.json
+++ b/plugins/codex/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.75.0",
+ "version": "1.76.0",
"description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
"author": {
"name": "Open Second Brain contributors",
diff --git a/plugins/codex/README.md b/plugins/codex/README.md
index d92cbfc8..4e3b51fd 100644
--- a/plugins/codex/README.md
+++ b/plugins/codex/README.md
@@ -113,7 +113,7 @@ The full router with readiness criteria is [`install.md`](https://github.com/ite
## What is new
-1.73.0 makes the query embed honest and upgrades safe. Every query embed, from the search lane behind the MCP tools and hooks to the `brain_context_pack` semantic belief order, passes one gate: under a positive `embedding_cost_gate_usd` an unpriced model is refused for any caller that is not local (a hybrid search falls back to keyword-only and says so by trail code), and a query longer than the model's input window is cut to it and disclosed. `embedding_input_window_tokens` declares the window of a model the curated table does not list, and `embedding_extra_body` sends extra request fields to an OpenAI-compatible endpoint. `brain_recall_feedback` and the benchmark, eval and tune reports name the degradations their searches met, and `brain_tune` no longer saves a winner measured without the semantic lane. `o2b brain upgrade` and the automatic upgrade worker apply exactly the plan they computed and refuse a managed file edited in between instead of overwriting it. Every release is described in the [CHANGELOG](https://github.com/itechmeat/open-second-brain/blob/main/CHANGELOG.md).
+1.76.0 keeps the search index current without a scheduler. A search or a session start that finds the index more than a minute old answers from it as it is and starts one low-priority background run, so the next read sees the current vault; nothing stays resident, and nothing runs while no agent works. A changed note now rewrites only the parts of the index that changed, so appending to a large daily log costs a fraction of what it did, and `o2b search status` shows how fresh the index is. Every release is described in the [CHANGELOG](https://github.com/itechmeat/open-second-brain/blob/main/CHANGELOG.md).
## Documentation
diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yaml
index 75c93f72..cd8f1d6a 100644
--- a/plugins/hermes/plugin.yaml
+++ b/plugins/hermes/plugin.yaml
@@ -1,5 +1,5 @@
name: open-second-brain
-version: "1.75.0"
+version: "1.76.0"
description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."
author: "Open Second Brain contributors"
memory_provider: true
diff --git a/pyproject.toml b/pyproject.toml
index 1707c93c..a4818511 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"
# CLI entry points (those moved to `package.json` `bin`).
[project]
name = "open-second-brain"
-version = "1.75.0"
+version = "1.76.0"
description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."
readme = "README.md"
requires-python = ">=3.11"
diff --git a/src/cli/brain/verbs/doctor.ts b/src/cli/brain/verbs/doctor.ts
index e55014ee..492636b4 100644
--- a/src/cli/brain/verbs/doctor.ts
+++ b/src/cli/brain/verbs/doctor.ts
@@ -105,6 +105,7 @@ export async function cmdBrainDoctor(argv: string[]): Promise<number> {
dryRun,
agent: resolveBrainAgent(flags, config),
configPath: config,
+ dbPath: resolveSearchConfig({ vault, configPath: config ?? undefined }).dbPath,
});
return renderRepair(outcome, Boolean(flags["json"]));
} catch (exc) {
diff --git a/src/cli/command-manifest.ts b/src/cli/command-manifest.ts
index 7b8deedf..3c5368d9 100644
--- a/src/cli/command-manifest.ts
+++ b/src/cli/command-manifest.ts
@@ -813,6 +813,8 @@ export const CLI_COMMAND_MANIFEST: CliRootManifest = Object.freeze({
flag("concurrency", "string"),
flag("verbose", "boolean"),
flag("progress", "boolean"),
+ flag("freshen", "string"),
+ flag("freshen-state", "string"),
]),
command("reindex", "Rebuild the search index", [
flag("vault", "string"),
diff --git a/src/cli/search/verbs/indexing.ts b/src/cli/search/verbs/indexing.ts
index 6598fbc6..123c3682 100644
--- a/src/cli/search/verbs/indexing.ts
+++ b/src/cli/search/verbs/indexing.ts
@@ -6,7 +6,11 @@
* name once the index covers every document.
*/
+import { setPriority } from "node:os";
+import { dirname } from "node:path";
+
import { formatDegradationNotice } from "../../../core/integrity/degradation.ts";
+import { recordFreshenOutcome, releaseFreshen } from "../../../core/search/freshen.ts";
import {
createSafeguard,
resolveSafeguardTimeoutMs,
@@ -176,7 +180,13 @@ export async function cmdSearchIndex(argv: ReadonlyArray<string>): Promise<numbe
verbose: { type: "boolean" },
progress: { type: "boolean" },
json: { type: "boolean" },
+ freshen: { type: "string" },
+ "freshen-state": { type: "string" },
});
+ const freshenToken = flagString(flags, "freshen");
+ if (freshenToken !== undefined) {
+ return runFreshenChild(flags, freshenToken, flagString(flags, "freshen-state"));
+ }
const cfg = resolveConfig(flags);
const verbose = flagBoolean(flags, "verbose");
@@ -214,6 +224,56 @@ export async function cmdSearchIndex(argv: ReadonlyArray<string>): Promise<numbe
return 0;
}
+/**
+ * `--freshen <token>`: the background run a stale read started (see
+ * `src/core/search/freshen.ts`). It runs at lowered CPU priority, indexes
+ * incrementally (embeddings only when `search_freshen_embeddings` is on),
+ * records its outcome and backoff in the per-device state file, and
+ * releases the claim it was handed whatever happens. Nothing is printed:
+ * every stream of a detached child is ignored.
+ */
+async function runFreshenChild(
+ flags: SearchVerbFlags,
+ token: string,
+ stateDirFlag: string | undefined,
+): Promise<number> {
+ try {
+ setPriority(process.pid, 10);
+ } catch {
+ // Not permitted here (or not supported): run at normal priority.
+ }
+ const startedAt = Date.now();
+ let cfg: ResolvedSearchConfig | undefined;
+ let stateDir = stateDirFlag;
+ try {
+ cfg = resolveConfig(flags);
+ stateDir ??= dirname(cfg.dbPath);
+ const stats = await indexVault(cfg, {
+ embeddings: cfg.freshen?.embeddings === true,
+ });
+ recordFreshenOutcome(cfg.vault, stateDir, {
+ outcome: "completed",
+ durationMs: Date.now() - startedAt,
+ changed: stats.added + stats.updated + stats.deleted,
+ });
+ return 0;
+ } catch (e) {
+ // Another writer holds the index (a manual run, the watcher, the
+ // maintenance lane): nothing is wrong, the next stale read retries.
+ if (e instanceof SearchError && e.code === "INDEX_LOCKED") return 0;
+ if (stateDir !== undefined) {
+ recordFreshenOutcome(cfg?.vault ?? null, stateDir, {
+ outcome: "failed",
+ durationMs: Date.now() - startedAt,
+ error: e instanceof Error ? e.message : String(e),
+ });
+ }
+ return 1;
+ } finally {
+ if (stateDir !== undefined) releaseFreshen(stateDir, token);
+ }
+}
+
/**
* Rebuild the index from scratch.
*
diff --git a/src/cli/search/verbs/status.ts b/src/cli/search/verbs/status.ts
index b977662f..59e7a529 100644
--- a/src/cli/search/verbs/status.ts
+++ b/src/cli/search/verbs/status.ts
@@ -4,6 +4,12 @@
*/
import { formatEstimatedUsd } from "../../../core/search/embedding-spend.ts";
+import {
+ freshenStatus,
+ renderFreshenStatus,
+ serializeFreshenStatus,
+ type FreshenStatus,
+} from "../../../core/search/freshen.ts";
import {
eventTimeStatus,
indexStatus,
@@ -31,11 +37,13 @@ export async function cmdSearchStatus(argv: ReadonlyArray<string>): Promise<numb
const status = await indexStatus(cfg);
// The event-time summary reads the index, so it exists only with one.
const eventTime = status.exists ? await eventTimeStatus(cfg) : null;
+ const freshen = status.exists ? freshenStatus(cfg, status.lastIndexedAt) : null;
if (flagBoolean(flags, "json")) {
process.stdout.write(
JSON.stringify({
...(serializeIndexStatus(status) as Record<string, unknown>),
...(eventTime !== null ? { event_time: serializeEventTimeStatus(eventTime) } : {}),
+ ...(freshen !== null ? { freshen: serializeFreshenStatus(freshen) } : {}),
// Same polarity as the human twin below, so the two conditions
// read as one rule rather than as each other's negation.
...(!status.exists ? nextCommandField("search-index-missing") : {}),
@@ -43,7 +51,7 @@ export async function cmdSearchStatus(argv: ReadonlyArray<string>): Promise<numb
);
return 0;
}
- process.stdout.write(renderStatusHuman(status, eventTime));
+ process.stdout.write(renderStatusHuman(status, eventTime, freshen));
// no-dead-ends, phase 3: the pointer used to be spliced into the
// renderer's first line, which is a second emission mechanism AND
// beyond the reach of a scan over writer call sites. The rail decides
@@ -56,7 +64,11 @@ export async function cmdSearchStatus(argv: ReadonlyArray<string>): Promise<numb
return 0;
}
-function renderStatusHuman(s: IndexStatusSnapshot, eventTime: EventTimeStatus | null): string {
+function renderStatusHuman(
+ s: IndexStatusSnapshot,
+ eventTime: EventTimeStatus | null,
+ freshen: FreshenStatus | null,
+): string {
if (!s.exists) {
return `index: not initialised\n path: ${s.indexPath}\n`;
}
@@ -77,6 +89,7 @@ function renderStatusHuman(s: IndexStatusSnapshot, eventTime: EventTimeStatus |
lines.push(`embedding_key: ${s.embeddingKeyPresent ? "present" : "missing"}`);
lines.push(`last_indexed_at: ${s.lastIndexedAt ?? "(never)"}`);
lines.push(`last_full_index_at: ${s.lastFullIndexAt ?? "(never)"}`);
+ if (freshen !== null) lines.push(...renderFreshenStatus(freshen));
if (eventTime !== null) lines.push(`event_time: ${renderEventTimeStatus(eventTime)}`);
// The oversize-chunk census when it could NOT run. Above the warnings
// and not among them, deliberately: this line reports a check that did
diff --git a/src/core/brain/diagnostics.ts b/src/core/brain/diagnostics.ts
index 52ec8222..c4d11248 100644
--- a/src/core/brain/diagnostics.ts
+++ b/src/core/brain/diagnostics.ts
@@ -715,6 +715,14 @@ export const DIAGNOSTIC_SIGNALS: ReadonlyMap<string, DiagnosticSignal> = new Map
nextCommand: "o2b brain tiers check",
autoRepairable: false,
},
+ {
+ // A review, like tier-drift: the status names the last error and
+ // the backoff, and the repair depends on what the error is.
+ code: "freshen-failing",
+ issueClass: "background index refresh keeps failing",
+ nextCommand: "o2b search status",
+ autoRepairable: false,
+ },
{
code: "entity-label-malformed",
issueClass: "entity label fails the quality gate",
@@ -1173,6 +1181,12 @@ export interface PlanRepairOptions {
* page counts.
*/
readonly readable?: (rel: string) => boolean;
+ /**
+ * The search-index path, when the doctor runs here, for its index-backed
+ * checks (`tier-drift`, `freshen-failing`). Absent, those checks skip,
+ * as they do in `runDoctor`.
+ */
+ readonly dbPath?: string;
}
/**
@@ -1190,7 +1204,7 @@ export interface PlanRepairOptions {
* the same findings, each with its next-command hint.
*/
export function planRepair(vault: string, opts: PlanRepairOptions = {}): RepairPlan {
- const issues = opts.issues ?? collectDoctorIssues(vault, opts.readable);
+ const issues = opts.issues ?? collectDoctorIssues(vault, opts.readable, opts.dbPath);
const fixes: RepairItem[] = [];
for (const fixer of FIXERS) {
@@ -1226,8 +1240,12 @@ export function planRepair(vault: string, opts: PlanRepairOptions = {}): RepairP
function collectDoctorIssues(
vault: string,
readable: ((rel: string) => boolean) | undefined,
+ dbPath: string | undefined,
): ReadonlyArray<DoctorIssue> {
- const doctor = runDoctor(vault, readable !== undefined ? { readable } : {});
+ const doctor = runDoctor(vault, {
+ ...(readable !== undefined ? { readable } : {}),
+ ...(dbPath !== undefined ? { dbPath } : {}),
+ });
return [...doctor.errors, ...doctor.warnings];
}
@@ -1289,6 +1307,12 @@ export interface ApplyRepairOptions {
* that never held them. Absent, every page counts.
*/
readonly readable?: (rel: string) => boolean;
+ /**
+ * The search-index path for the doctor's index-backed checks, so the
+ * plan lists `tier-drift` and `freshen-failing` as the report does.
+ * Absent, those checks skip.
+ */
+ readonly dbPath?: string;
}
/** The fields a doctor finding names artifacts in. */
@@ -1365,9 +1389,12 @@ export function applyRepair(vault: string, opts: ApplyRepairOptions): RepairOutc
// are bounded again by what each one would name, so a fix that reaches
// a withheld page through its detail is not planned either.
const plan = view.filtersNothing
- ? planRepair(vault, opts.readable !== undefined ? { readable: opts.readable } : {})
+ ? planRepair(vault, {
+ ...(opts.readable !== undefined ? { readable: opts.readable } : {}),
+ ...(opts.dbPath !== undefined ? { dbPath: opts.dbPath } : {}),
+ })
: planRepair(vault, {
- issues: view.keep(collectDoctorIssues(vault, opts.readable), (i) =>
+ issues: view.keep(collectDoctorIssues(vault, opts.readable, opts.dbPath), (i) =>
doctorIssueRefs(vault, i),
),
});
diff --git a/src/core/brain/doctor.ts b/src/core/brain/doctor.ts
index d7d81914..0c79c41b 100644
--- a/src/core/brain/doctor.ts
+++ b/src/core/brain/doctor.ts
@@ -91,6 +91,7 @@ import { logChainCheck } from "./doctor/log-chain-check.ts";
import { checkSemanticHealth } from "./doctor/semantic-health-check.ts";
import {
danglingWorkrunCheck,
+ freshenFailureCheck,
symlinkEscapeCheck,
syncConflictLogCheck,
tierDriftCheck,
@@ -175,6 +176,7 @@ const DOCTOR_CHECKS: ReadonlyArray<DoctorCheck> = Object.freeze([
preferenceCheck,
retiredCheck,
tierDriftCheck,
+ freshenFailureCheck,
duplicateIdCheck,
logShardCheck,
brokenBacklinkCheck,
diff --git a/src/core/brain/doctor/store-integrity.ts b/src/core/brain/doctor/store-integrity.ts
index 6d805880..74b87648 100644
--- a/src/core/brain/doctor/store-integrity.ts
+++ b/src/core/brain/doctor/store-integrity.ts
@@ -8,9 +8,10 @@
* all of them are conditions only a walk of the tree can see.
*/
-import { join } from "node:path";
+import { dirname, join } from "node:path";
import { ensureInsideVault, realpathInsideVault, vaultRelative } from "../../path-safety.ts";
+import { readFreshenState } from "../../search/freshen.ts";
import { BRAIN_LOG_AUDIT_DIRS } from "../audit-dirs.ts";
import { continuityLogDir } from "../continuity/store.ts";
import { scanDanglingWorkruns } from "../dream-workrun.ts";
@@ -36,13 +37,14 @@ import {
/**
* Tier guard (write-time-integrity-governance): staged identity
* hand-edits the index post-pass detected. Fail-soft index read - a
- * missing index or pre-v6 schema simply skips the check.
+ * missing index or pre-v6 schema simply skips the check. Counted over
+ * the pages the caller may read, like every check handed `readable`.
*/
export const tierDriftCheck: DoctorCheck = {
failSoft: false,
- run({ dbPath }, { issues }) {
+ run({ dbPath, readable }, { issues }) {
if (dbPath === undefined) return;
- const driftCount = readTierDriftCount(dbPath);
+ const driftCount = readTierDriftCount(dbPath, readable);
if (driftCount > 0) {
issues.push({
severity: "warning",
@@ -53,14 +55,38 @@ export const tierDriftCheck: DoctorCheck = {
},
};
+/**
+ * Freshen on read (index-freshness): background index runs started by
+ * stale reads that keep failing. Three in a row is a broken index, not a
+ * hiccup - every search meanwhile answers from an ageing index.
+ */
+export const freshenFailureCheck: DoctorCheck = {
+ failSoft: false,
+ run({ dbPath }, { issues }) {
+ if (dbPath === undefined) return;
+ const state = readFreshenState(dirname(dbPath));
+ if (state.lastOutcome !== "failed" || state.failures < FRESHEN_FAILURE_WARN_AT) return;
+ issues.push({
+ severity: "warning",
+ code: FRESHEN_FAILING_CODE,
+ message:
+ `background index refresh failed ${state.failures} times in a row ` +
+ `(last: ${state.lastError ?? "unknown error"}); the index is not being kept current`,
+ });
+ },
+};
+
+/** Failures in a row from which {@link freshenFailureCheck} warns. */
+export const FRESHEN_FAILURE_WARN_AT = 3;
+
+/** The code of the {@link freshenFailureCheck} warning. */
+export const FRESHEN_FAILING_CODE = "freshen-failing";
+
/** The code of the {@link tierDriftCheck} warning. */
export const TIER_DRIFT_CODE = "tier-drift";
-/**
- * The {@link tierDriftCheck} warning text for `count` staged drift rows,
- * shared with a surface that recounts the rows a caller may read.
- */
-export function tierDriftMessage(count: number): string {
+/** The {@link tierDriftCheck} warning text for `count` staged drift rows. */
+function tierDriftMessage(count: number): string {
return `${count} identity-field hand-edit(s) staged - review with: o2b brain tiers check`;
}
diff --git a/src/core/brain/frontmatter-tiers.ts b/src/core/brain/frontmatter-tiers.ts
index 94415e95..1089e636 100644
--- a/src/core/brain/frontmatter-tiers.ts
+++ b/src/core/brain/frontmatter-tiers.ts
@@ -220,12 +220,24 @@ function sameScalar(left: unknown, right: unknown): boolean {
* Count the staged tier-drift findings straight from the index
* database (doctor surface). Read-only, fail-soft: a missing index
* file, a pre-v6 schema, or any read error counts as zero.
+ *
+ * With `readable`, only the rows on pages the caller may read count, so
+ * a drifted page it cannot read moves the count no more than an absent
+ * one does.
*/
-export function readTierDriftCount(dbPath: string): number {
+export function readTierDriftCount(dbPath: string, readable?: (rel: string) => boolean): number {
if (!existsSync(dbPath)) return 0;
let db: Database | null = null;
try {
db = new Database(dbPath, { readonly: true });
+ if (readable !== undefined) {
+ return db
+ .query<{ path: string }, []>(
+ "SELECT d.path FROM tier_drift t JOIN documents d ON d.id = t.document_id",
+ )
+ .all()
+ .filter((row) => readable(row.path)).length;
+ }
return db.query<{ n: number }, []>("SELECT COUNT(*) AS n FROM tier_drift").get()?.n ?? 0;
} catch {
return 0;
diff --git a/src/core/maintenance/ensure-current.ts b/src/core/maintenance/ensure-current.ts
index 8d6c9348..34974c7a 100644
--- a/src/core/maintenance/ensure-current.ts
+++ b/src/core/maintenance/ensure-current.ts
@@ -17,17 +17,17 @@
* migrations and downgrades correctly.
*/
import { existsSync } from "node:fs";
-import { dirname, join, resolve } from "node:path";
-import { fileURLToPath } from "node:url";
import { Database } from "bun:sqlite";
import { defaultConfigPath } from "../config.ts";
+import { o2bCommand } from "./o2b-command.ts";
import { brainConfigPath } from "../brain/paths.ts";
import { resolveSearchConfig } from "../search/index.ts";
import { reindexVault } from "../search/indexer.ts";
import { CHUNKER_VERSION } from "../search/chunker.ts";
import { LATEST_SCHEMA_VERSION, readSchemaVersion } from "../search/schema.ts";
-import { CHUNKER_VERSION_STATE_KEY } from "../search/store/state.ts";
+import { CHUNKER_VERSION_STATE_KEY, LAST_INDEXED_AT_STATE_KEY } from "../search/store/state.ts";
+import { maybeFreshenIndex, type FreshenDecision } from "../search/freshen.ts";
import { isWriterLockHeld } from "../search/store/writer-lock.ts";
import type { ResolvedSearchConfig } from "../search/types.ts";
import {
@@ -95,6 +95,12 @@ export interface EnsureCurrentResult {
* foreground and there was no child to decide about.
*/
readonly reindexSpawn: SelfHealSpawnDecision | null;
+ /**
+ * Freshen on read at session start: whether a current index that has
+ * gone stale got a background incremental run, or why not. Null when the
+ * index needed a rebuild (self-heal owns it) or was not looked at.
+ */
+ readonly freshen: FreshenDecision | null;
/** Non-empty when nothing ran, e.g. "not-initialized". */
readonly skipped: string;
/** Best-effort: per-step failures, never thrown. */
@@ -111,6 +117,8 @@ export interface EnsureCurrentOptions {
* (re)built matches the one the caller's server actually uses, e.g. under
* `o2b mcp --config <custom>`. */
readonly configPath?: string;
+ /** Test seam: replaces the detached spawn of a freshen-on-read run. */
+ readonly freshenSpawn?: (argv: string[]) => void;
}
function message(e: unknown): string {
@@ -131,6 +139,27 @@ function chunksAreStale(db: Database): boolean {
return db.query<{ one: number }, []>("SELECT 1 AS one FROM documents LIMIT 1").get() !== null;
}
+/** The index's `last_indexed_at`, or null when it cannot be read. */
+function readLastIndexedAt(dbPath: string): string | null {
+ let db: Database;
+ try {
+ db = new Database(dbPath, { readonly: true });
+ } catch {
+ return null;
+ }
+ try {
+ return (
+ db
+ .query<{ value: string }, [string]>("SELECT value FROM index_state WHERE key = ? LIMIT 1")
+ .get(LAST_INDEXED_AT_STATE_KEY)?.value ?? null
+ );
+ } catch {
+ return null;
+ } finally {
+ closeDatabase(db);
+ }
+}
+
/**
* Cheap, read-only check: is the search index absent, on a stale schema,
* or chunked by older rules?
@@ -152,22 +181,6 @@ function indexNeedsRebuild(config: ResolvedSearchConfig): boolean {
}
}
-/**
- * argv prefix that runs this checkout's CLI (current plugin version).
- *
- * POSIX goes through `scripts/o2b`, which also applies the macOS SQLite
- * setup. Native Windows cannot execute that bash launcher, so it runs the
- * TypeScript entry point with the Bun that is running this process.
- */
-function o2bCommand(): string[] {
- // src/core/maintenance/ensure-current.ts -> repo root
- const repo = resolve(dirname(fileURLToPath(import.meta.url)), "..", "..", "..");
- if (process.platform === "win32") {
- return [process.execPath, "run", join(repo, "src", "cli", "main.ts")];
- }
- return [join(repo, "scripts", "o2b")];
-}
-
/**
* Start the managed-file upgrade in a detached worker, unless a recorded
* failure is cooling down or another worker has the vault.
@@ -349,6 +362,7 @@ export async function ensureVaultCurrent(
let brainUpgradeOutcome: SelfHealUpgradeOutcome | null = null;
let reindexTriggered = false;
let reindexSpawn: SelfHealSpawnDecision | null = null;
+ let freshen: FreshenDecision | null = null;
// Only an already-initialised vault is an upgrade target. A missing
// `_brain.yaml` means "not set up yet" - that is `o2b init`'s job, not ours.
@@ -359,6 +373,7 @@ export async function ensureVaultCurrent(
brainUpgradeOutcome: null,
reindexTriggered: false,
reindexSpawn: null,
+ freshen: null,
skipped: "not-initialized",
errors: [],
};
@@ -398,6 +413,13 @@ export async function ensureVaultCurrent(
await reindexVault(config);
reindexTriggered = true;
}
+ } else {
+ // 3. A current index that has gone stale: catch it up in the
+ // background so the session's first read is not the one that pays.
+ freshen = maybeFreshenIndex(config, {
+ lastIndexedAt: readLastIndexedAt(config.dbPath),
+ ...(opts.freshenSpawn !== undefined ? { spawn: opts.freshenSpawn } : {}),
+ });
}
} catch (e) {
errors.push(`search-reindex: ${message(e)}`);
@@ -409,6 +431,7 @@ export async function ensureVaultCurrent(
brainUpgradeOutcome,
reindexTriggered,
reindexSpawn,
+ freshen,
skipped: "",
errors,
};
diff --git a/src/core/maintenance/o2b-command.ts b/src/core/maintenance/o2b-command.ts
new file mode 100644
index 00000000..59c16866
--- /dev/null
+++ b/src/core/maintenance/o2b-command.ts
@@ -0,0 +1,19 @@
+import { dirname, join, resolve } from "node:path";
+import { fileURLToPath } from "node:url";
+
+/**
+ * argv prefix that runs this checkout's CLI (current plugin version), for
+ * the detached children the self-heal and freshen paths start.
+ *
+ * POSIX goes through `scripts/o2b`, which also applies the macOS SQLite
+ * setup. Native Windows cannot execute that bash launcher, so it runs the
+ * TypeScript entry point with the Bun that is running this process.
+ */
+export function o2bCommand(): string[] {
+ // src/core/maintenance/o2b-command.ts -> repo root
+ const repo = resolve(dirname(fileURLToPath(import.meta.url)), "..", "..", "..");
+ if (process.platform === "win32") {
+ return [process.execPath, "run", join(repo, "src", "cli", "main.ts")];
+ }
+ return [join(repo, "scripts", "o2b")];
+}
diff --git a/src/core/search/freshen.ts b/src/core/search/freshen.ts
new file mode 100644
index 00000000..74d4ef62
--- /dev/null
+++ b/src/core/search/freshen.ts
@@ -0,0 +1,533 @@
+/**
+ * Freshen on read (index-freshness design): a reader that finds the search
+ * index older than `search_freshen_interval_s` answers from the index it has
+ * and starts ONE detached, low-priority incremental index run, so the next
+ * read sees the current vault. Nothing stays resident: the only triggers are
+ * reads (`search()`) and session start (`ensureVaultCurrent`), both of which
+ * only happen while an agent works.
+ *
+ * Everything here is per device and lives next to the index
+ * (`<dirname(dbPath)>/`), never in `Brain/`: the claim that keeps concurrent
+ * readers from starting a run each, and the state file the child writes its
+ * outcome and backoff to.
+ *
+ * `maybeFreshenIndex` never throws and never waits for the run: a reader
+ * of a fresh index pays nothing; a due one pays one state-file read, one
+ * lock probe and one exclusive create.
+ */
+import { randomUUID } from "node:crypto";
+import { readFileSync, renameSync, statSync, unlinkSync, writeFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+
+import { appendMetric } from "../brain/metrics.ts";
+import { atomicWriteFileSync } from "../fs-atomic.ts";
+import { o2bCommand } from "../maintenance/o2b-command.ts";
+import { isWriterLockHeld } from "./store/writer-lock.ts";
+import type { ResolvedSearchConfig } from "./types.ts";
+
+/** The claim a reader takes before it starts a run; the child releases it. */
+export const FRESHEN_CLAIM_FILE = "freshen.claim";
+
+/** The child's outcome, failure streak and backoff. */
+export const FRESHEN_STATE_FILE = "freshen-state.json";
+
+/**
+ * Metrics surface for background runs. A row is written only when a run
+ * changed the index or failed: an idle machine would otherwise append one
+ * row a minute to a synced file for nothing.
+ */
+export const INDEX_FRESHEN_SURFACE = "index_freshen";
+
+/** A claim older than this belongs to a run that died; it is taken over. */
+export const FRESHEN_CLAIM_ABANDONED_MS = 10 * 60 * 1000;
+
+const BACKOFF_BASE_MS = 60_000;
+const BACKOFF_CAP_MS = 3_600_000;
+
+/** Why a reader did not start a run. */
+export const FRESHEN_SKIP = Object.freeze({
+ off: "off",
+ noIndex: "no_index",
+ fresh: "fresh",
+ backoff: "backoff",
+ readOnly: "read_only",
+ writerLock: "writer_lock",
+ claimed: "claimed",
+ /** The index directory refused the claim (read-only mount, no permission). */
+ unwritable: "unwritable",
+ spawnFailed: "spawn_failed",
+} as const);
+
+export type FreshenSkip = (typeof FRESHEN_SKIP)[keyof typeof FRESHEN_SKIP];
+
+/** What a reader did: started a run, or the reason it did not. */
+export type FreshenDecision = "spawned" | FreshenSkip;
+
+export interface FreshenState {
+ readonly failures: number;
+ readonly backoffUntil: string | null;
+ readonly lastOutcome: "completed" | "failed" | null;
+ readonly lastRunAt: string | null;
+ readonly lastDurationMs: number | null;
+ readonly lastError: string | null;
+ /** Documents the last completed run added, updated or deleted. */
+ readonly lastChanged: number | null;
+}
+
+export const EMPTY_FRESHEN_STATE: FreshenState = Object.freeze({
+ failures: 0,
+ backoffUntil: null,
+ lastOutcome: null,
+ lastRunAt: null,
+ lastDurationMs: null,
+ lastError: null,
+ lastChanged: null,
+});
+
+export interface DecideFreshenInput {
+ readonly nowMs: number;
+ readonly lastIndexedAt: string | null;
+ readonly intervalSeconds: number;
+ readonly backoffUntilMs: number | null;
+}
+
+/**
+ * Whether a read should start a run. Pure. An index that was never
+ * stamped belongs to self-heal (a missing or unbuilt index is rebuilt
+ * there); an unparseable stamp is treated as due, since claiming it fresh
+ * would leave the index stale forever. Freshness is decided before the
+ * backoff, so a reader can skip the state file while the index is fresh.
+ */
+export function decideFreshen(
+ input: DecideFreshenInput,
+): { readonly action: "spawn" } | { readonly action: "skip"; readonly reason: FreshenSkip } {
+ if (input.intervalSeconds <= 0) return { action: "skip", reason: FRESHEN_SKIP.off };
+ if (input.lastIndexedAt === null) return { action: "skip", reason: FRESHEN_SKIP.noIndex };
+ const indexedMs = Date.parse(input.lastIndexedAt);
+ if (Number.isFinite(indexedMs) && input.nowMs - indexedMs < input.intervalSeconds * 1000) {
+ return { action: "skip", reason: FRESHEN_SKIP.fresh };
+ }
+ if (input.backoffUntilMs !== null && input.backoffUntilMs > input.nowMs) {
+ return { action: "skip", reason: FRESHEN_SKIP.backoff };
+ }
+ return { action: "spawn" };
+}
+
+/** Seconds since `lastIndexedAt`, or null when it is absent or unparseable. */
+export function indexAgeSeconds(lastIndexedAt: string | null, nowMs: number): number | null {
+ if (lastIndexedAt === null) return null;
+ const ms = Date.parse(lastIndexedAt);
+ if (!Number.isFinite(ms)) return null;
+ return Math.max(0, Math.floor((nowMs - ms) / 1000));
+}
+
+/** Backoff after the `failures`-th failure in a row: 1 min doubling to 1 h. */
+export function nextBackoffMs(failures: number): number {
+ const n = Math.max(1, failures);
+ return Math.min(BACKOFF_BASE_MS * 2 ** (n - 1), BACKOFF_CAP_MS);
+}
+
+/**
+ * A claim file that cannot be parsed is normally one being written: the
+ * exclusive create and the write are two steps. Only past this grace is a
+ * torn claim taken for a dead run's.
+ */
+const CLAIM_TORN_GRACE_MS = 10_000;
+
+/** A takeover lock older than this belongs to a reader that died mid-takeover. */
+const TAKEOVER_STALE_MS = 30_000;
+
+/** A claim taken, or why it was not. */
+export type FreshenClaim =
+ | { readonly token: string }
+ | { readonly skip: typeof FRESHEN_SKIP.claimed | typeof FRESHEN_SKIP.unwritable };
+
+/**
+ * Take the claim, or return null when a live run holds it or the index
+ * directory refuses it. {@link tryClaimFreshen} tells the two apart.
+ */
+export function claimFreshen(dir: string, nowMs: number): string | null {
+ const claim = tryClaimFreshen(dir, nowMs);
+ return "token" in claim ? claim.token : null;
+}
+
+/**
+ * Take the claim. The claim is an exclusive create, so of N readers racing
+ * on a stale index exactly one wins; a create that fails for any reason
+ * other than an existing claim (EROFS, EACCES) is reported as unwritable,
+ * since no reader there can ever start a run. A claim older than
+ * {@link FRESHEN_CLAIM_ABANDONED_MS} (or torn and older than
+ * {@link CLAIM_TORN_GRACE_MS}) is a run that died (a reboot, an OOM kill)
+ * and is taken over - but only by the reader holding the exclusive
+ * takeover lock, which re-reads the claim under it and replaces it with an
+ * atomic rename, so the slot is never empty and a fresh claim is never
+ * touched.
+ */
+export function tryClaimFreshen(dir: string, nowMs: number): FreshenClaim {
+ const path = join(dir, FRESHEN_CLAIM_FILE);
+ const token = `fr-${randomUUID()}`;
+ const body = JSON.stringify({ token, at: nowMs });
+ const claimed = { skip: FRESHEN_SKIP.claimed } as const;
+ for (let attempt = 0; attempt < 2; attempt++) {
+ try {
+ writeFileSync(path, body, { flag: "wx" });
+ return { token };
+ } catch (e) {
+ if ((e as NodeJS.ErrnoException).code !== "EEXIST") {
+ return { skip: FRESHEN_SKIP.unwritable };
+ }
+ }
+ const seen = readClaimBody(path);
+ if (seen === null) continue; // released since: race the create again
+ if (!claimAbandoned(path, seen, nowMs)) return claimed;
+ const taken = takeOver(path, token, body, nowMs);
+ return taken === null ? claimed : { token: taken };
+ }
+ return claimed;
+}
+
+function takeOver(path: string, token: string, body: string, nowMs: number): string | null {
+ const lock = `${path}.takeover`;
+ if (!createExclusive(lock)) {
+ let age = 0;
+ try {
+ age = nowMs - statSync(lock).mtimeMs;
+ } catch {
+ return null;
+ }
+ if (age <= TAKEOVER_STALE_MS) return null;
+ unlinkQuietly(lock);
+ if (!createExclusive(lock)) return null;
+ }
+ try {
+ const current = readClaimBody(path);
+ if (current !== null && !claimAbandoned(path, current, nowMs)) return null;
+ const tmp = `${path}.${token}.tmp`;
+ writeFileSync(tmp, body);
+ renameSync(tmp, path);
+ return token;
+ } catch {
+ return null;
+ } finally {
+ unlinkQuietly(lock);
+ }
+}
+
+function createExclusive(path: string): boolean {
+ try {
+ writeFileSync(path, String(process.pid), { flag: "wx" });
+ return true;
+ } catch {
+ return false;
+ }
+}
+
+function readClaimBody(path: string): string | null {
+ try {
+ return readFileSync(path, "utf8");
+ } catch {
+ return null;
+ }
+}
+
+function claimAbandoned(path: string, body: string, nowMs: number): boolean {
+ try {
+ const parsed = JSON.parse(body) as { at?: unknown };
+ if (typeof parsed.at === "number") return nowMs - parsed.at > FRESHEN_CLAIM_ABANDONED_MS;
+ } catch {
+ // Torn or empty: judged by age below.
+ }
+ try {
+ return nowMs - statSync(path).mtimeMs > CLAIM_TORN_GRACE_MS;
+ } catch {
+ return false;
+ }
+}
+
+function unlinkQuietly(path: string): void {
+ try {
+ unlinkSync(path);
+ } catch {
+ // Already gone.
+ }
+}
+
+/** Release the claim, but only while it is still ours. */
+export function releaseFreshen(dir: string, token: string): void {
+ const path = join(dir, FRESHEN_CLAIM_FILE);
+ try {
+ const parsed = JSON.parse(readFileSync(path, "utf8")) as { token?: unknown };
+ if (parsed.token === token) unlinkSync(path);
+ } catch {
+ // Absent or unreadable: nothing of ours to release.
+ }
+}
+
+function str(v: unknown): string | null {
+ return typeof v === "string" ? v : null;
+}
+
+function num(v: unknown): number | null {
+ return typeof v === "number" && Number.isFinite(v) ? v : null;
+}
+
+/** The state file, or the empty state when it is absent or torn. */
+export function readFreshenState(dir: string): FreshenState {
+ try {
+ const raw = JSON.parse(readFileSync(join(dir, FRESHEN_STATE_FILE), "utf8")) as Record<
+ string,
+ unknown
+ >;
+ const outcome = raw["lastOutcome"];
+ return Object.freeze({
+ failures: num(raw["failures"]) ?? 0,
+ backoffUntil: str(raw["backoffUntil"]),
+ lastOutcome: outcome === "completed" || outcome === "failed" ? outcome : null,
+ lastRunAt: str(raw["lastRunAt"]),
+ lastDurationMs: num(raw["lastDurationMs"]),
+ lastError: str(raw["lastError"]),
+ lastChanged: num(raw["lastChanged"]),
+ });
+ } catch {
+ return EMPTY_FRESHEN_STATE;
+ }
+}
+
+/** Write the state file atomically. */
+export function writeFreshenState(dir: string, state: FreshenState): void {
+ atomicWriteFileSync(join(dir, FRESHEN_STATE_FILE), JSON.stringify(state, null, 2) + "\n");
+}
+
+/**
+ * Prefix the child's argv so it runs at idle I/O priority where the
+ * platform has a tool for it; the child lowers its own CPU priority.
+ */
+export function freshenCommand(
+ base: ReadonlyArray<string>,
+ has: (tool: string) => boolean,
+ platform: NodeJS.Platform,
+): string[] {
+ if (platform === "linux" && has("ionice")) return ["ionice", "-c3", ...base];
+ if (platform === "darwin" && has("taskpolicy")) return ["taskpolicy", "-b", ...base];
+ return [...base];
+}
+
+/**
+ * The options the freshen child is spawned with. Detached on every
+ * platform: a non-detached child dies with its parent on Windows, and on
+ * POSIX it stays in the reader's process group, so a short-lived hook
+ * whose group is signalled on exit would take the run down with it.
+ */
+export function freshenSpawnOptions(env: NodeJS.ProcessEnv): {
+ readonly stdin: "ignore";
+ readonly stdout: "ignore";
+ readonly stderr: "ignore";
+ readonly windowsHide: true;
+ readonly detached: true;
+ readonly env: Record<string, string | undefined>;
+} {
+ return {
+ stdin: "ignore",
+ stdout: "ignore",
+ stderr: "ignore",
+ windowsHide: true,
+ detached: true,
+ // Bun gives a child spawned without `env` the environment this process
+ // STARTED with; a config chosen after start must reach the child.
+ env: { ...env },
+ };
+}
+
+function spawnDetached(argv: string[]): void {
+ const proc = Bun.spawn(argv, freshenSpawnOptions(process.env));
+ proc.unref();
+}
+
+export interface MaybeFreshenOptions {
+ readonly lastIndexedAt: string | null;
+ /** A read-only open (cross-vault, recall sources): never write there. */
+ readonly readOnly?: boolean;
+ readonly nowMs?: number;
+ /** Test seam: replaces the detached spawn. */
+ readonly spawn?: (argv: string[]) => void;
+}
+
+/**
+ * Start a background incremental index run when the index is stale.
+ * Never throws; returns what it did.
+ */
+export function maybeFreshenIndex(
+ config: ResolvedSearchConfig,
+ opts: MaybeFreshenOptions,
+): FreshenDecision {
+ try {
+ const freshen = config.freshen;
+ if (freshen === undefined || freshen.intervalSeconds <= 0) return FRESHEN_SKIP.off;
+ if (opts.readOnly === true) return FRESHEN_SKIP.readOnly;
+ const nowMs = opts.nowMs ?? Date.now();
+ const dir = dirname(config.dbPath);
+ const input = {
+ nowMs,
+ lastIndexedAt: opts.lastIndexedAt,
+ intervalSeconds: freshen.intervalSeconds,
+ };
+ // Every search lands here: decide on the index stamp alone first and
+ // read the state file for the backoff only once the index is due.
+ const due = decideFreshen({ ...input, backoffUntilMs: null });
+ if (due.action === "skip") return due.reason;
+ const backoffUntil = readFreshenState(dir).backoffUntil;
+ const backoffMs = backoffUntil === null ? Number.NaN : Date.parse(backoffUntil);
+ const decision = decideFreshen({
+ ...input,
+ backoffUntilMs: Number.isFinite(backoffMs) ? backoffMs : null,
+ });
+ if (decision.action === "skip") return decision.reason;
+ if (isWriterLockHeld(config.dbPath)) return FRESHEN_SKIP.writerLock;
+ const claim = tryClaimFreshen(dir, nowMs);
+ if (!("token" in claim)) return claim.skip;
+ const token = claim.token;
+ const argv = freshenCommand(
+ [
+ ...o2bCommand(),
+ "search",
+ "index",
+ "--vault",
+ config.vault,
+ ...(freshen.configPath !== null ? ["--config", freshen.configPath] : []),
+ // Where the claim and the state live, so a child that fails before
+ // it can resolve its config can still record it and release.
+ "--freshen-state",
+ dir,
+ "--freshen",
+ token,
+ ],
+ (tool) => Bun.which(tool) !== null,
+ process.platform,
+ );
+ try {
+ (opts.spawn ?? spawnDetached)(argv);
+ } catch {
+ releaseFreshen(dir, token);
+ return FRESHEN_SKIP.spawnFailed;
+ }
+ return "spawned";
+ } catch {
+ // A reader must never fail because freshening could not decide.
+ return FRESHEN_SKIP.spawnFailed;
+ }
+}
+
+export interface FreshenOutcome {
+ readonly outcome: "completed" | "failed";
+ readonly durationMs: number;
+ /** Documents added, updated or deleted (completed runs). */
+ readonly changed?: number;
+ readonly error?: string;
+}
+
+/**
+ * What the background child records when it ends: the state file (outcome,
+ * streak, backoff) and, when it changed something or failed, one metric
+ * row. Never throws: a run must not fail because its bookkeeping did.
+ */
+export function recordFreshenOutcome(
+ vault: string | null,
+ dir: string,
+ result: FreshenOutcome,
+ nowMs: number = Date.now(),
+): void {
+ const previous = readFreshenState(dir);
+ const failed = result.outcome === "failed";
+ const failures = failed ? previous.failures + 1 : 0;
+ const runAt = new Date(nowMs).toISOString();
+ try {
+ writeFreshenState(dir, {
+ failures,
+ backoffUntil: failed ? new Date(nowMs + nextBackoffMs(failures)).toISOString() : null,
+ lastOutcome: result.outcome,
+ lastRunAt: runAt,
+ lastDurationMs: result.durationMs,
+ lastError: failed ? (result.error ?? "failed") : null,
+ lastChanged: failed ? null : (result.changed ?? 0),
+ });
+ } catch {
+ // Best effort, like the self-heal outcome rows.
+ }
+ if (vault === null || (!failed && (result.changed ?? 0) === 0)) return;
+ try {
+ appendMetric(vault, {
+ surface: INDEX_FRESHEN_SURFACE,
+ runAt,
+ payload: {
+ outcome: result.outcome,
+ duration_ms: result.durationMs,
+ ...(failed ? { error: result.error ?? "failed", failures } : {}),
+ ...(!failed ? { changed: result.changed ?? 0 } : {}),
+ },
+ });
+ } catch {
+ // Same: metrics are diagnostics, never a reason to fail.
+ }
+}
+
+/** What `o2b search status` and the doctor show about freshen on read. */
+export interface FreshenStatus {
+ /** 0 when freshening is off. */
+ readonly intervalSeconds: number;
+ readonly indexAgeSeconds: number | null;
+ readonly state: FreshenState;
+ /** The backoff end, only while it is still in the future. */
+ readonly activeBackoffUntil: string | null;
+}
+
+export function freshenStatus(
+ config: ResolvedSearchConfig,
+ lastIndexedAt: string | null,
+ nowMs: number = Date.now(),
+): FreshenStatus {
+ const state = readFreshenState(dirname(config.dbPath));
+ const backoffMs = state.backoffUntil === null ? Number.NaN : Date.parse(state.backoffUntil);
+ return Object.freeze({
+ intervalSeconds: config.freshen?.intervalSeconds ?? 0,
+ indexAgeSeconds: indexAgeSeconds(lastIndexedAt, nowMs),
+ state,
+ activeBackoffUntil: Number.isFinite(backoffMs) && backoffMs > nowMs ? state.backoffUntil : null,
+ });
+}
+
+/** The status lines, aligned with the rest of `o2b search status`. */
+export function renderFreshenStatus(s: FreshenStatus): string[] {
+ const lines = [
+ `freshen: ${s.intervalSeconds > 0 ? `every ${s.intervalSeconds}s` : "off"}`,
+ `index_age: ${s.indexAgeSeconds === null ? "(unknown)" : `${s.indexAgeSeconds}s`}`,
+ ];
+ const st = s.state;
+ if (st.lastOutcome === "completed") {
+ lines.push(`last_freshen: completed ${st.lastRunAt} (${st.lastChanged ?? 0} changed)`);
+ } else if (st.lastOutcome === "failed") {
+ lines.push(
+ `last_freshen: failed ${st.lastRunAt} (${st.failures} in a row): ${st.lastError}`,
+ );
+ } else {
+ lines.push("last_freshen: (none)");
+ }
+ // The label stays within the 19 columns every status label fits in.
+ if (s.activeBackoffUntil !== null) {
+ lines.push(`freshen_backoff: until ${s.activeBackoffUntil}`);
+ }
+ return lines;
+}
+
+/** The `freshen` object of `o2b search status --json`. */
+export function serializeFreshenStatus(s: FreshenStatus): Record<string, unknown> {
+ return {
+ interval_s: s.intervalSeconds,
+ index_age_s: s.indexAgeSeconds,
+ last_outcome: s.state.lastOutcome,
+ last_run_at: s.state.lastRunAt,
+ last_changed: s.state.lastChanged,
+ last_error: s.state.lastError,
+ failures: s.state.failures,
+ backoff_until: s.activeBackoffUntil,
+ };
+}
diff --git a/src/core/search/index.ts b/src/core/search/index.ts
index ef8c320e..4405bdfd 100644
--- a/src/core/search/index.ts
+++ b/src/core/search/index.ts
@@ -41,6 +41,7 @@ import {
import { SearchError } from "./types.ts";
import type {
ResolvedEmbeddingConfig,
+ ResolvedFreshenConfig,
ResolvedRecallConfig,
ResolvedRerankConfig,
ResolvedSearchConfig,
@@ -71,6 +72,7 @@ export type {
IndexStatusSnapshot,
PendingVectorCensus,
ResolvedEmbeddingConfig,
+ ResolvedFreshenConfig,
ResolvedRecallConfig,
ResolvedRerankConfig,
ResolvedSearchConfig,
@@ -1377,6 +1379,31 @@ export function resolveSearchConfig(opts: {
trigramPrefilterMaxSelectivity,
});
+ const freshen: ResolvedFreshenConfig = Object.freeze({
+ intervalSeconds: parseInteger(
+ envOrConfig(
+ env,
+ config,
+ "OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S",
+ "search_freshen_interval_s",
+ ),
+ 60,
+ "search_freshen_interval_s",
+ { min: 0 },
+ ),
+ embeddings: parseBool(
+ envOrConfig(
+ env,
+ config,
+ "OPEN_SECOND_BRAIN_SEARCH_FRESHEN_EMBEDDINGS",
+ "search_freshen_embeddings",
+ ),
+ false,
+ "search_freshen_embeddings",
+ ),
+ configPath: opts.configPath ?? null,
+ });
+
const base: ResolvedSearchConfig = Object.freeze({
vault: opts.vault,
dbPath,
@@ -1395,6 +1422,7 @@ export function resolveSearchConfig(opts: {
resumeReindex,
hybridDeadlineMs,
ftsTokenize,
+ freshen,
});
// The env variable names an extra-body refusal only when its value is
diff --git a/src/core/search/indexer.ts b/src/core/search/indexer.ts
index a60cb3e7..bf4a0c19 100644
--- a/src/core/search/indexer.ts
+++ b/src/core/search/indexer.ts
@@ -104,6 +104,7 @@ import {
EMBEDDING_PREFIX_PASSAGE_STATE_KEY,
LAST_FULL_INDEX_AT_STATE_KEY,
LAST_INDEXED_AT_STATE_KEY,
+ LINK_RESOLUTION_PENDING_STATE_KEY,
} from "./store.ts";
import { LATEST_SCHEMA_VERSION } from "./schema.ts";
import { chunkWindowDiagnosticCode, SearchError } from "./types.ts";
@@ -187,6 +188,7 @@ interface MutableStats {
relationViolations: IndexStats["relationViolations"];
tierDrift: IndexStats["tierDrift"];
aliasResolved: number;
+ linkResolutionSkipped: boolean;
eventAnchorsPending: number;
backend: IndexStats["backend"];
deferredReason: IndexStats["deferredReason"];
@@ -216,6 +218,7 @@ function newStats(): MutableStats {
relationViolations: [],
tierDrift: [],
aliasResolved: 0,
+ linkResolutionSkipped: false,
// Counted at the end of the run, once every changed document has
// been upserted: what is left is what this run did not reach.
eventAnchorsPending: 0,
@@ -257,6 +260,7 @@ function freezeStats(s: MutableStats, durationMs: number): IndexStats {
relationViolations: Object.freeze([...s.relationViolations]),
tierDrift: Object.freeze([...s.tierDrift]),
aliasResolved: s.aliasResolved,
+ linkResolutionSkipped: s.linkResolutionSkipped,
eventAnchorsPending: s.eventAnchorsPending,
backend: s.backend,
deferredReason: s.deferredReason,
@@ -455,6 +459,12 @@ async function indexIntoRun(
const ownsStore = !storeOverride;
const store = storeOverride ?? (await Store.open(config, { mode: "write" }));
const stats = newStats();
+ let linkResolutionMarked = false;
+ const markLinkResolutionPending = (): void => {
+ if (linkResolutionMarked) return;
+ linkResolutionMarked = true;
+ store.setState(LINK_RESOLUTION_PENDING_STATE_KEY, "1");
+ };
try {
const existing = store.listDocuments();
@@ -518,6 +528,7 @@ async function indexIntoRun(
continue;
}
+ markLinkResolutionPending();
const filenameBase = basename(file.relPath, ".md");
const chunkResult = chunkMarkdown(content, filenameBase, {
maxTokens: config.chunkSize,
@@ -610,8 +621,8 @@ async function indexIntoRun(
tokenCount: c.tokenCount,
headingPath: c.headingPath,
}));
- // An unchanged chunk keeps its stored vector (vector carry-over),
- // so the embedding phase below never re-pays for it.
+ // An unchanged chunk is kept in place, and a moved one carries its
+ // stored vector, so the embedding phase below never re-pays for it.
const replaced = store.replaceDocumentChunks(docId, chunkInputs);
const chunkIds = replaced.chunkIds;
stats.embeddingsReused += replaced.embeddingsReused;
@@ -630,8 +641,11 @@ async function indexIntoRun(
});
}
// Entity-boosted retrieval (v0.13.0): persist the chunk's
- // deterministic entity set alongside its links.
- store.replaceEntities(cid, extractEntities(content));
+ // deterministic entity set alongside its links. A chunk kept in
+ // place has the entities of the content it still holds - so a
+ // release that changes `extractEntities` must bump CHUNKER_VERSION,
+ // which rebuilds every chunk, or kept rows keep the old entities.
+ if (!replaced.keptChunkIds.has(cid)) store.replaceEntities(cid, extractEntities(content));
}
// Typed graph semantics (v3): frontmatter relation fields
// (related / extends / contradicts / superseded_by) become typed
@@ -666,16 +680,35 @@ async function indexIntoRun(
for (const [path] of existing) {
if (!seen.has(path)) {
+ markLinkResolutionPending();
store.deleteDocument(path);
stats.deleted++;
opts?.onFile?.({ path, kind: "deleted" });
}
}
- store.resolveLinkTargets();
- // Alias post-pass (v7): exact path matches above always win; this
- // only fills still-unresolved slash-free targets from doc_aliases.
- stats.aliasResolved = store.resolveAliasTargets();
+ // Link and alias resolution are a pure function of the documents and
+ // their links. A run that changed no document (and was not forced)
+ // would only rewrite every link row to the value it already holds -
+ // the alias pass resets and re-sets each alias-resolved link - which
+ // is what made a no-change run write to disk. Skipped then; the
+ // constraint and tier passes below still run.
+ // A run killed between its document writes and this point leaves the
+ // pending marker set, and the next run resolves even if it changes
+ // nothing itself.
+ if (
+ stats.added + stats.updated + stats.deleted === 0 &&
+ opts?.force !== true &&
+ store.getState(LINK_RESOLUTION_PENDING_STATE_KEY) !== "1"
+ ) {
+ stats.linkResolutionSkipped = true;
+ } else {
+ store.resolveLinkTargets();
+ // Alias post-pass (v7): exact path matches above always win; this
+ // only fills still-unresolved slash-free targets from doc_aliases.
+ stats.aliasResolved = store.resolveAliasTargets();
+ store.setState(LINK_RESOLUTION_PENDING_STATE_KEY, "0");
+ }
// Link-constraint materialization post-pass
// (write-time-integrity-governance): recompute every typed edge's
diff --git a/src/core/search/pipeline/outcome.ts b/src/core/search/pipeline/outcome.ts
index ab5a8742..d47c832a 100644
--- a/src/core/search/pipeline/outcome.ts
+++ b/src/core/search/pipeline/outcome.ts
@@ -19,7 +19,12 @@ import {
type TransportReach,
} from "../../graph/transport-reach.ts";
import { buildEvidencePack, downrankTerminalEvidenceResults } from "../evidence-pack.ts";
-import { buildRetrievalTrail, corpusStatementFor } from "../retrieval-trail.ts";
+import {
+ buildRetrievalTrail,
+ corpusStatementFor,
+ INDEX_STALE_SECONDS,
+ RETRIEVAL_DEGRADATION,
+} from "../retrieval-trail.ts";
import { buildEvidenceVerification, coverageOverResults } from "../evidence-verification.ts";
import { fnv1aHex } from "../feedback.ts";
import {
@@ -251,6 +256,40 @@ export function buildSearchOutcome(input: OutcomeInput): SearchOutcome {
});
}
+/**
+ * Name a stale index on an outcome's trail, at read time. The code is not
+ * part of the computed answer: the query cache stores outcomes without it
+ * (an index run that changes nothing keeps the cache generation, so a
+ * stored code would outlive the staleness it reports), and every answer,
+ * hit or miss, gains it here from the index age it is served at. The
+ * result is the trail the pipeline would have built with the code in its
+ * sink: the code leads, and a corpus statement gives way to it.
+ */
+export function withIndexStale(outcome: SearchOutcome, ageSeconds: number | null): SearchOutcome {
+ if (ageSeconds === null || ageSeconds <= INDEX_STALE_SECONDS) return outcome;
+ const stale = Object.freeze({
+ code: RETRIEVAL_DEGRADATION.indexStale,
+ detail: Object.freeze({ ageSeconds }),
+ });
+ const trail = outcome.retrievalTrail;
+ const retrievalTrail = Object.freeze({
+ retrieved: trail?.retrieved ?? outcome.cards?.length ?? outcome.results.length,
+ pool: trail?.pool ?? outcome.total,
+ degraded: Object.freeze([stale, ...(trail?.degraded ?? [])]),
+ });
+ if (trail !== undefined) return Object.freeze({ ...outcome, retrievalTrail });
+ // A healthy answer has no trail; it goes where the outcome builders put
+ // it, right after the match quality, so the serialized shape matches a
+ // freshly built degraded answer.
+ const out: Record<string, unknown> = {};
+ for (const [key, value] of Object.entries(outcome)) {
+ out[key] = value;
+ if (key === "idfWeightedCoverage") out["retrievalTrail"] = retrievalTrail;
+ }
+ if (!("retrievalTrail" in out)) out["retrievalTrail"] = retrievalTrail;
+ return Object.freeze(out) as unknown as SearchOutcome;
+}
+
/**
* Both root sets unresolved. Handed to the classifier when the index facts
* could not be read at all, which it reports as
diff --git a/src/core/search/retrieval-trail.ts b/src/core/search/retrieval-trail.ts
index c41ac501..11ee87fc 100644
--- a/src/core/search/retrieval-trail.ts
+++ b/src/core/search/retrieval-trail.ts
@@ -65,7 +65,20 @@ import type {
* are NOT invented for conditions nothing reports: a code with no
* producer would be a promise the search path does not keep.
*/
+/** How old the index may be before a search names it on the trail. */
+export const INDEX_STALE_SECONDS = 600;
+
export const RETRIEVAL_DEGRADATION = Object.freeze({
+ /**
+ * The index was last updated more than {@link INDEX_STALE_SECONDS} ago, so
+ * notes changed since then may be missing from the answer. Detail:
+ * `ageSeconds`. The code reports the age only, not that a refresh is
+ * running: freshen on read starts a background run when it can and
+ * starts none when it is off, backing off, blocked by another run or
+ * writer, read-only for this reader, or unable to spawn. Added at read time (`withIndexStale`), never
+ * stored in the query cache.
+ */
+ indexStale: "index-stale",
/**
* The query tokenised to an empty FTS match, so the keyword lane never
* ran. The purest silent empty in the tree: `fts.ts` returned no hits
@@ -193,6 +206,7 @@ export type RetrievalDegradationCode =
/** Membership list, ordered lane by lane along the pipeline. */
export const RETRIEVAL_DEGRADATION_CODES: ReadonlyArray<RetrievalDegradationCode> = Object.freeze([
+ RETRIEVAL_DEGRADATION.indexStale,
RETRIEVAL_DEGRADATION.keywordFtsMatchEmpty,
RETRIEVAL_DEGRADATION.keywordTrigramLaneFault,
RETRIEVAL_DEGRADATION.semanticEmbeddingsAbsent,
@@ -354,6 +368,8 @@ export function semanticLaneMissing(degraded: ReadonlyArray<RetrievalDegradation
*/
export function describeRetrievalDegradation(code: RetrievalDegradationCode): string {
switch (code) {
+ case RETRIEVAL_DEGRADATION.indexStale:
+ return "the index was last updated more than ten minutes ago, so notes changed since then may be missing";
case RETRIEVAL_DEGRADATION.keywordFtsMatchEmpty:
return "the query carried no term the keyword index can match, so the keyword lane never ran";
case RETRIEVAL_DEGRADATION.keywordTrigramLaneFault:
diff --git a/src/core/search/search.ts b/src/core/search/search.ts
index b4a9eb1d..844a8fd6 100644
--- a/src/core/search/search.ts
+++ b/src/core/search/search.ts
@@ -22,6 +22,7 @@ import {
buildSearchOutcome,
corpusStatementForEmptyWindow,
emptyOutcome,
+ withIndexStale,
} from "./pipeline/outcome.ts";
import { applyPostRankPhases } from "./pipeline/post-rank.ts";
import { resolveQueryShape } from "./pipeline/query-shape.ts";
@@ -40,6 +41,8 @@ import { isAbortError } from "./embeddings/http-util.ts";
import type { CacheProbe } from "./pipeline/cache-slot.ts";
import type { RetrievalDegradationSink } from "./retrieval-trail.ts";
import { RETRIEVAL_DEGRADATION, noteDegradation } from "./retrieval-trail.ts";
+import { indexAgeSeconds, maybeFreshenIndex } from "./freshen.ts";
+import { LAST_INDEXED_AT_STATE_KEY } from "./store/state.ts";
import type { FrontmatterCache } from "./result-filters.ts";
import { Store } from "./store.ts";
import type { ResolvedSearchConfig, SearchOptions, SearchOutcome } from "./types.ts";
@@ -233,6 +236,20 @@ export async function search(
// caller can tell an exhausted corpus from a broken embedder without
// matching on prose.
const degraded: RetrievalDegradationSink = [];
+ // Freshen on read: an index older than the interval starts one
+ // background incremental run (never into a read-only origin); this
+ // answer comes from the index as it is. A badly stale index is named
+ // on the trail whether or not a run could start, at read time on every
+ // answer, cache hits included, and never in the cached row.
+ const lastIndexedAt = store.getState(LAST_INDEXED_AT_STATE_KEY);
+ maybeFreshenIndex(effectiveConfig, {
+ lastIndexedAt,
+ readOnly: opts.selfHeal === false,
+ nowMs,
+ ...(opts.freshenSpawn !== undefined ? { spawn: opts.freshenSpawn } : {}),
+ });
+ const indexAge = indexAgeSeconds(lastIndexedAt, nowMs);
+ const served = (outcome: SearchOutcome): SearchOutcome => withIndexStale(outcome, indexAge);
// Shared across every frontmatter-reading stage below (Plan 1, 1.3)
// so a candidate path already read by one stage is not re-read
// and re-parsed by the next.
@@ -281,7 +298,7 @@ export async function search(
planHash: shape.basePlan.planHash,
})
: CACHE_BYPASSED;
- if (cache.hit !== null) return cache.hit;
+ if (cache.hit !== null) return served(cache.hit);
// A deadline-degraded answer is keyword-only under a key that promises
// the hybrid one: serve it, but never cache it. A spend-gated one is
// the same, and the key carries no gate or price either, so a cached
@@ -411,17 +428,19 @@ export async function search(
degraded,
opts.transportReach,
);
- return finalize(
- emptyOutcome({
- store,
- opts,
- query,
- pathPrefix,
- warnings,
- routedSurface,
- degraded,
- corpus,
- }),
+ return served(
+ finalize(
+ emptyOutcome({
+ store,
+ opts,
+ query,
+ pathPrefix,
+ warnings,
+ routedSurface,
+ degraded,
+ corpus,
+ }),
+ ),
);
}
@@ -537,26 +556,28 @@ export async function search(
);
const emit =
postRank.decisionFallback === true || rerankUnavailable ? (o: SearchOutcome) => o : finalize;
- return emit(
- buildSearchOutcome({
- store,
- config: effectiveConfig,
- opts,
- query,
- pathPrefix,
- results,
- warnings,
- secondPass: retry.secondPass,
- routedSurface,
- trustReceipts: postRank.trustReceipts,
- frontmatterCache,
- poolSize,
- degraded,
- corpus,
- ...(postRank.decisionModel?.answerable !== undefined
- ? { decisionModel: { answerable: postRank.decisionModel.answerable } }
- : {}),
- }),
+ return served(
+ emit(
+ buildSearchOutcome({
+ store,
+ config: effectiveConfig,
+ opts,
+ query,
+ pathPrefix,
+ results,
+ warnings,
+ secondPass: retry.secondPass,
+ routedSurface,
+ trustReceipts: postRank.trustReceipts,
+ frontmatterCache,
+ poolSize,
+ degraded,
+ corpus,
+ ...(postRank.decisionModel?.answerable !== undefined
+ ? { decisionModel: { answerable: postRank.decisionModel.answerable } }
+ : {}),
+ }),
+ ),
);
} finally {
await store.close();
diff --git a/src/core/search/store.ts b/src/core/search/store.ts
index 11c302e3..d96b2f31 100644
--- a/src/core/search/store.ts
+++ b/src/core/search/store.ts
@@ -65,6 +65,7 @@ export {
INDEX_REVISION_STATE_KEY,
LAST_FULL_INDEX_AT_STATE_KEY,
LAST_INDEXED_AT_STATE_KEY,
+ LINK_RESOLUTION_PENDING_STATE_KEY,
peekCorpusGenerationSync,
readCorpusGenerationSync,
} from "./store/state.ts";
diff --git a/src/core/search/store/chunks.ts b/src/core/search/store/chunks.ts
index c207b9a3..b80491a1 100644
--- a/src/core/search/store/chunks.ts
+++ b/src/core/search/store/chunks.ts
@@ -126,25 +126,48 @@ export function getChunksByDocument(db: Database, documentId: number): ChunkRow[
/** What one document's chunk replacement wrote. */
export interface ChunkReplacement {
- /** The new chunk ids, in `chunkIndex` order. */
+ /** The chunk ids, in `chunkIndex` order: kept rows keep theirs. */
readonly chunkIds: number[];
/**
- * New chunks that kept a stored vector of an old chunk with the same
- * `content_hash` (vector carry-over), so the embedding phase will not
- * pay for them again.
+ * Chunks that kept a stored vector: kept in place with a vector of the
+ * recorded identity, or new rows that carried the vector of an old chunk
+ * with the same `content_hash` (vector carry-over). The embedding phase
+ * will not pay for them again.
*/
readonly embeddingsReused: number;
+ /**
+ * Ids of the chunks kept in place: same position, same content, same
+ * lines and heading. Nothing derived from their content (full-text row,
+ * entities, vector) needs to be written again.
+ */
+ readonly keptChunkIds: ReadonlySet<number>;
+}
+
+interface StoredChunk {
+ id: number;
+ chunk_index: number;
+ content_hash: string;
+ start_line: number;
+ end_line: number;
+ token_count: number;
+ heading_path: string;
+ fts_content: string;
}
/**
- * Atomically replace every chunk for a document, carrying the stored
- * vector of every new chunk whose content an old chunk of this document
- * already embedded under the recorded model and dimension (the rule and
- * its guard live in `vector-carry-over.ts`). Only the vec rows that are
- * not carried are purged; the old `chunk_vec_map` and `embeddings` rows
- * go with their chunks through the FK cascade, and the carried ones are
- * re-attached to the new ids. FTS5 stays in sync via the
- * chunks_ai/ad/au triggers.
+ * Replace a document's chunks, rewriting only the ones that changed.
+ *
+ * A chunk at the same `chunkIndex` with the same content hash, lines,
+ * token count, heading and full-text form is KEPT: its row, id, full-text row, entities
+ * and vector stay as they are (a kept vector of another embedding
+ * identity is dropped, as a replaced one would be). Every other old chunk
+ * is deleted and every other new chunk inserted, carrying the stored
+ * vector of an old chunk with the same content where the identity allows
+ * (the rule and its guard live in `vector-carry-over.ts`). A daily log
+ * that grows all day therefore costs its appended tail per run, not the
+ * whole file. The old `chunk_vec_map` and `embeddings` rows of deleted
+ * chunks go with them through the FK cascade; FTS5 stays in sync through
+ * the chunks_ai/ad/au triggers.
*/
export function replaceDocumentChunks(
db: Database,
@@ -154,18 +177,63 @@ export function replaceDocumentChunks(
): ChunkReplacement {
const ids: number[] = [];
let embeddingsReused = 0;
+ const kept = new Set<number>();
db.exec("BEGIN");
try {
+ const stored = db
+ .query<StoredChunk, [number]>(
+ "SELECT id, chunk_index, content_hash, start_line, end_line, token_count, heading_path, " +
+ "fts_content FROM chunks WHERE document_id = ?",
+ )
+ .all(documentId);
+ const byIndex = new Map(stored.map((r) => [r.chunk_index, r]));
+ const keepAt = chunks.map((c) => {
+ const old = byIndex.get(c.chunkIndex);
+ const same =
+ old !== undefined &&
+ old.content_hash === c.contentHash &&
+ old.start_line === c.startLine &&
+ old.end_line === c.endLine &&
+ old.token_count === c.tokenCount &&
+ old.heading_path === (c.headingPath ?? "") &&
+ // The stored full-text form is derived from the content; a release
+ // that derives it differently must not leave kept rows behind.
+ old.fts_content === (c.ftsContent ?? c.content);
+ if (same) kept.add(old.id);
+ return same ? old.id : null;
+ });
+
+ const candidates = readCarryCandidates(db, vecLoaded, documentId);
+ // A kept chunk whose vector is not of the recorded identity loses it,
+ // exactly as a replaced chunk's vector would not be carried.
+ const validKept = new Set(candidates.filter((c) => kept.has(c.chunkId)).map((c) => c.chunkId));
+ const keptStale = [...kept].filter((id) => !validKept.has(id));
+ if (keptStale.length > 0) {
+ purgeVecRowsByChunkIds(db, vecLoaded, keptStale);
+ const placeholders = sqlPlaceholders(keptStale);
+ db.run(`DELETE FROM chunk_vec_map WHERE chunk_id IN (${placeholders})`, keptStale);
+ db.run(`DELETE FROM embeddings WHERE chunk_id IN (${placeholders})`, keptStale);
+ }
+
+ const newPositions: number[] = [];
+ keepAt.forEach((id, position) => {
+ if (id === null) newPositions.push(position);
+ });
const carried = matchCarriedVectors(
- readCarryCandidates(db, vecLoaded, documentId),
- chunks.map((c) => c.contentHash),
+ candidates.filter((c) => !kept.has(c.chunkId)),
+ newPositions.map((p) => chunks[p]!.contentHash),
+ ).map((m) => ({ position: newPositions[m.position]!, candidate: m.candidate }));
+ const carriedOld = new Set(carried.map((m) => m.candidate.chunkId));
+ const removed = stored.map((r) => r.id).filter((id) => !kept.has(id));
+ purgeVecRowsByChunkIds(
+ db,
+ vecLoaded,
+ removed.filter((id) => !carriedOld.has(id)),
);
- const keptChunkIds = new Set(carried.map((m) => m.candidate.chunkId));
- const purged = chunksForDocument(db, documentId)
- .map((c) => c.id)
- .filter((id) => !keptChunkIds.has(id));
- purgeVecRowsByChunkIds(db, vecLoaded, purged);
- db.run("DELETE FROM chunks WHERE document_id = ?", [documentId]);
+ if (removed.length > 0) {
+ db.run(`DELETE FROM chunks WHERE id IN (${sqlPlaceholders(removed)})`, removed);
+ }
+
const insert = db.prepare<
{ id: number },
[number, number, string, string, string, number, number, number, string, string, string]
@@ -174,7 +242,12 @@ export function replaceDocumentChunks(
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id",
);
const now = nowIso();
- for (const c of chunks) {
+ chunks.forEach((c, position) => {
+ const keptId = keepAt[position];
+ if (keptId !== null && keptId !== undefined) {
+ ids.push(keptId);
+ return;
+ }
const row = insert.get(
documentId,
c.chunkIndex,
@@ -190,15 +263,15 @@ export function replaceDocumentChunks(
);
if (!row) throw new SearchError("INDEX_UNREADABLE", "chunk insert returned no id");
ids.push(row.id);
- }
+ });
restoreCarriedVectors(db, ids, carried);
- embeddingsReused = carried.length;
+ embeddingsReused = validKept.size + carried.length;
db.exec("COMMIT");
} catch (e) {
db.exec("ROLLBACK");
throw e;
}
- return { chunkIds: ids, embeddingsReused };
+ return { chunkIds: ids, embeddingsReused, keptChunkIds: kept };
}
/**
diff --git a/src/core/search/store/state.ts b/src/core/search/store/state.ts
index 7dca0ae7..fedcc9fd 100644
--- a/src/core/search/store/state.ts
+++ b/src/core/search/store/state.ts
@@ -53,6 +53,14 @@ export const EMBEDDING_VEC_VERSION_STATE_KEY = "embedding_vec_version";
* dangling count (context-integrity-gates, unit G).
*/
export const LAST_INDEXED_AT_STATE_KEY = "last_indexed_at";
+
+/**
+ * "1" from the moment an index run writes its first document change until
+ * it has resolved links and aliases; "0" after. A run killed in between
+ * leaves "1", so the next run resolves links even when it changes nothing
+ * itself (index-freshness: a no-change run otherwise skips resolution).
+ */
+export const LINK_RESOLUTION_PENDING_STATE_KEY = "link_resolution_pending";
export const LAST_FULL_INDEX_AT_STATE_KEY = "last_full_index_at";
/**
diff --git a/src/core/search/types.ts b/src/core/search/types.ts
index 5e46e6c9..15666045 100644
--- a/src/core/search/types.ts
+++ b/src/core/search/types.ts
@@ -313,6 +313,12 @@ export interface IndexStats {
* (link-recall-intelligence, v7).
*/
readonly aliasResolved: number;
+ /**
+ * True when the run added, updated and deleted no document and was not
+ * forced, so the link and alias resolution passes - a pure function of
+ * the documents and their links - were skipped (index-freshness).
+ */
+ readonly linkResolutionSkipped: boolean;
/**
* Backend that processed this run, resolved lazily after content
* detection (offline code-only extraction, t_85252236). `"offline"`
@@ -1046,6 +1052,12 @@ export interface SearchOptions {
* index surfaces as a per-origin warning instead.
*/
readonly selfHeal?: boolean;
+ /**
+ * Test seam for freshen-on-read: replaces the detached spawn of the
+ * background index run a stale index starts. Production callers leave
+ * it unset.
+ */
+ readonly freshenSpawn?: (argv: string[]) => void;
/**
* Skip rerank kind `decision-model` for this call. Set by the hook
* surfaces (recall inject), whose time budget is shorter than a decision
@@ -1542,6 +1554,16 @@ export interface ResolvedRecallConfig {
readonly metadataBoostGateEnabled?: boolean;
}
+/** What freshen-on-read does; see `src/core/search/freshen.ts`. */
+export interface ResolvedFreshenConfig {
+ /** Seconds the index may age before a read refreshes it; 0 turns it off. */
+ readonly intervalSeconds: number;
+ /** Whether the background run also computes embeddings (paid). */
+ readonly embeddings: boolean;
+ /** The config file the background run must resolve, or null for the default. */
+ readonly configPath: string | null;
+}
+
export interface ResolvedSearchConfig {
readonly vault: string;
readonly dbPath: string;
@@ -1627,4 +1649,10 @@ export interface ResolvedSearchConfig {
* shadow index is untouched by this clause.
*/
readonly ftsTokenize: string;
+ /**
+ * Freshen-on-read (index-freshness): how old the index may get before a
+ * read starts one background incremental run, and what that run does.
+ * Absent on a hand-built config, which reads as off.
+ */
+ readonly freshen?: ResolvedFreshenConfig;
}
diff --git a/src/core/state/surfaces.ts b/src/core/state/surfaces.ts
index 98f74149..5bb22679 100644
--- a/src/core/state/surfaces.ts
+++ b/src/core/state/surfaces.ts
@@ -123,6 +123,7 @@ export const STATE_SURFACE_ID = Object.freeze({
hookAudit: "hook_audit",
hookSessionState: "hook_session_state",
selfHealUpgradeMarker: "self_heal_upgrade_marker",
+ freshenState: "freshen_state",
watchdogAudit: "watchdog_audit",
injectFailopenCache: "inject_failopen_cache",
aiderContext: "aider_context_artifact",
@@ -285,6 +286,7 @@ const MAINTENANCE_LEASE_FILE = "maintenance.sqlite";
const MAINTENANCE_JOURNAL_FILE = `${MAINTENANCE_JOURNAL_STEM}.${JSONL_LEDGER_EXT}`;
const HOOK_STATE_DIR = "hook-state";
const SELF_HEAL_UPGRADE_MARKER_FILE = "self-heal-upgrade.json";
+const FRESHEN_STATE_FILE = "freshen-state.json";
const INJECT_CACHE_DIR = "inject-cache";
const AIDER_CONTEXT_FILE = "aider-context.md";
const DREAM_RUNS_DIR = "dream-runs";
@@ -593,6 +595,23 @@ export const STATE_SURFACES: ReadonlyArray<StateSurface> = Object.freeze([
"drops the failure from `o2b doctor`; the next start simply tries again.",
sources: ["src/core/maintenance/self-heal-upgrade-state.ts"],
},
+ {
+ id: STATE_SURFACE_ID.freshenState,
+ label: "freshen-on-read state",
+ tier: STATE_TIER.derived,
+ // Beside the index, wherever the override puts it: `freshen.ts` keeps
+ // its state in `dirname(config.dbPath)`.
+ derive: (vault, override) => join(dirname(indexPath(vault, override)), FRESHEN_STATE_FILE),
+ override_env: SEARCH_DB_ENV,
+ override_config_key: SEARCH_DB_CONFIG_KEY,
+ carries_memory: false,
+ reason:
+ "What the last background index run started by a stale read did on this device: its " +
+ "outcome, how many documents it changed, the failure streak and when the next run may " +
+ "start. Deleting it ends any backoff early and forgets the last outcome; the next stale " +
+ "read simply starts a run again.",
+ sources: ["src/core/search/freshen.ts"],
+ },
{
id: STATE_SURFACE_ID.watchdogAudit,
label: "watchdog audit fallback",
diff --git a/src/mcp/brain/health-tools.ts b/src/mcp/brain/health-tools.ts
index 3af76df7..fd8c0b1e 100644
--- a/src/mcp/brain/health-tools.ts
+++ b/src/mcp/brain/health-tools.ts
@@ -30,10 +30,7 @@ import { findingRefs } from "./hygiene-tools.ts";
import { everyArtifactRefView } from "../../core/brain/artifact-ref-view.ts";
import { reachView } from "../../core/brain/reach-view.ts";
import { contextReach } from "../tool-contract.ts";
-import { TRANSPORT_REACH } from "../../core/graph/transport-reach.ts";
-import { Store } from "../../core/search/store.ts";
import { computeTrustVerdict } from "../../core/brain/trust/compute-trust-verdict.ts";
-import { TIER_DRIFT_CODE, tierDriftMessage } from "../../core/brain/doctor/store-integrity.ts";
/**
* One reported issue, as an MCP caller sees it.
@@ -70,38 +67,6 @@ const NO_VERIFICATION_DELTA = Object.freeze({
missing_evidence: 0,
});
-/**
- * The `tier-drift` warning restated over the drift rows the caller may
- * read, the rows `brain_tiers check` lists for it.
- *
- * The warning carries no path - only the count of every staged row - so
- * the reference view cannot judge it, and the count alone moves when a
- * page the caller may not read drifts. Below local reach the count is
- * taken again over the readable rows; none left drops the warning, as
- * for a vault with no drift.
- */
-async function recountTierDrift<T extends { readonly code: string; readonly message: string }>(
- ctx: ServerContext,
- warnings: ReadonlyArray<T>,
-): Promise<ReadonlyArray<T>> {
- if (!warnings.some((w) => w.code === TIER_DRIFT_CODE)) return warnings;
- const readable = readableAtContextReach(ctx);
- const store = await Store.open(
- resolveSearchConfig({ vault: ctx.vault, configPath: ctx.configPath ?? undefined }),
- { mode: "read" },
- );
- let count: number;
- try {
- count = store.listTierDrift().filter((row) => readable(row.path)).length;
- } finally {
- await store.close();
- }
- return warnings.flatMap((w) => {
- if (w.code !== TIER_DRIFT_CODE) return [w];
- return count === 0 ? [] : [{ ...w, message: tierDriftMessage(count) }];
- });
-}
-
async function toolBrainDoctor(
ctx: ServerContext,
args: Record<string, unknown>,
@@ -130,6 +95,11 @@ async function toolBrainDoctor(
const outcome = applyRepair(ctx.vault, {
dryRun: !apply,
...(ctx.configPath !== null ? { configPath: ctx.configPath } : {}),
+ // The index-backed findings (`tier-drift`, `freshen-failing`) join
+ // the plan as they join the report below. `tier-drift` is counted
+ // over `readable`, so a withheld drifted page adds nothing to it.
+ dbPath: resolveSearchConfig({ vault: ctx.vault, configPath: ctx.configPath ?? undefined })
+ .dbPath,
// The repair branch returns HERE, before the diagnostic streams
// below are filtered - so until this argument existed a scoped
// caller both read and REWROTE another owner's preferences
@@ -175,10 +145,10 @@ async function toolBrainDoctor(
const visibleIssues = <T extends DoctorIssueNaming>(issues: ReadonlyArray<T>): ReadonlyArray<T> =>
view.keep(issues, (i) => doctorIssueRefs(ctx.vault, i));
const errors = visibleIssues(result.errors);
- const warnings =
- reach === TRANSPORT_REACH.local
- ? visibleIssues(result.warnings)
- : await recountTierDrift(ctx, visibleIssues(result.warnings));
+ // `tier-drift` names no page, so the view cannot judge it; the doctor
+ // counted it over `readable` instead, the rows `brain_tiers check`
+ // lists for this caller.
+ const warnings = visibleIssues(result.warnings);
const uncertain = result.uncertain === undefined ? undefined : visibleIssues(result.uncertain);
// The verdict summarises the streams, so it is taken again over the
// streams this caller sees: a verdict computed over a dropped error
diff --git a/tests/bench/measure-index-writes.ts b/tests/bench/measure-index-writes.ts
new file mode 100644
index 00000000..d2119f82
--- /dev/null
+++ b/tests/bench/measure-index-writes.ts
@@ -0,0 +1,126 @@
+#!/usr/bin/env bun
+/**
+ * Measure how many bytes an incremental index run writes, per pass.
+ *
+ * bun run tests/bench/measure-index-writes.ts <vault> [changed-notes=200 | largest:N]
+ *
+ * Copies the vault's Markdown (and `Brain/_brain.yaml`) to a temp dir,
+ * builds an index there, then runs three incremental passes and prints the
+ * bytes each one wrote, split by the store methods that do the writing:
+ * 1. no change at all,
+ * 2. `changed-notes` notes edited (the catch-up a long-idle index meets),
+ * or with `largest:N` the N largest notes (an append-only daily log that
+ * grows all day is the note a frequent background run meets most),
+ * 3. no change again.
+ *
+ * Linux only: the numbers are `write_bytes` deltas from `/proc/self/io`, the
+ * bytes this process caused to be sent to storage. The source vault is only
+ * read. Development tool for the index-freshness work; prints, never writes
+ * outside its temp dir.
+ */
+import { appendFileSync, cpSync, mkdtempSync, readFileSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join, relative } from "node:path";
+
+import { resolveSearchConfig } from "../../src/core/search/index.ts";
+import { indexVault } from "../../src/core/search/indexer.ts";
+import { Store } from "../../src/core/search/store.ts";
+
+function writeBytes(): number {
+ const m = /write_bytes:\s+(\d+)/.exec(readFileSync("/proc/self/io", "utf8"));
+ if (m === null) throw new Error("/proc/self/io has no write_bytes (Linux only)");
+ return Number(m[1]);
+}
+
+const PASSES = [
+ "replaceDocumentChunks",
+ "replaceLinks",
+ "deleteDocument",
+ "resolveLinkTargets",
+ "resolveAliasTargets",
+ "recomputeRelationConstraintFlags",
+ "setState",
+ "close",
+] as const;
+
+const tally = new Map<string, number>();
+const proto = Store.prototype as unknown as Record<string, (...args: unknown[]) => unknown>;
+for (const name of PASSES) {
+ const original = proto[name];
+ if (typeof original !== "function") continue;
+ proto[name] = function (this: unknown, ...args: unknown[]) {
+ const before = writeBytes();
+ const result = original.apply(this, args);
+ const settle = (): void => {
+ tally.set(name, (tally.get(name) ?? 0) + (writeBytes() - before));
+ };
+ if (result instanceof Promise) return result.finally(settle);
+ settle();
+ return result;
+ };
+}
+
+function mb(n: number): string {
+ return (n / 1024 / 1024).toFixed(1).padStart(8);
+}
+
+async function measured(label: string, vault: string): Promise<void> {
+ tally.clear();
+ const before = writeBytes();
+ const t0 = performance.now();
+ const stats = await indexVault(resolveSearchConfig({ vault }));
+ const total = writeBytes() - before;
+ const ms = Math.round(performance.now() - t0);
+ console.log(
+ `\n${label}: ${mb(total)} MB in ${ms} ms ` +
+ `(added ${stats.added}, updated ${stats.updated}, deleted ${stats.deleted}, ` +
+ `link resolution ${stats.linkResolutionSkipped ? "skipped" : "ran"})`,
+ );
+ let counted = 0;
+ for (const [name, bytes] of [...tally].toSorted((a, b) => b[1] - a[1])) {
+ counted += bytes;
+ console.log(` ${name.padEnd(34)}${mb(bytes)} MB`);
+ }
+ console.log(` ${"(elsewhere)".padEnd(34)}${mb(total - counted)} MB`);
+}
+
+const [source, changedArg] = process.argv.slice(2);
+if (source === undefined) {
+ console.error(
+ "usage: bun run tests/bench/measure-index-writes.ts <vault> [changed-notes=200 | largest:N]",
+ );
+ process.exit(2);
+}
+const largest = changedArg?.startsWith("largest:") === true;
+const changed = Number(largest ? changedArg!.slice("largest:".length) : (changedArg ?? "200"));
+
+const work = mkdtempSync(join(tmpdir(), "osb-measure-writes-"));
+const vault = join(work, "vault");
+try {
+ cpSync(source, vault, {
+ recursive: true,
+ filter: (src) =>
+ !relative(source, src).startsWith(".") &&
+ (!src.includes(".") || src.endsWith(".md") || src.endsWith("_brain.yaml")),
+ });
+ console.log(`copied ${source} -> ${vault}; building the index...`);
+ await indexVault(resolveSearchConfig({ vault }));
+
+ await measured("1. no change", vault);
+
+ const glob = new Bun.Glob("**/*.md");
+ const all = [...glob.scanSync({ cwd: vault })].toSorted();
+ const notes = largest
+ ? all
+ .map((n) => ({ n, size: Bun.file(join(vault, n)).size }))
+ .toSorted((a, b) => b.size - a.size)
+ .slice(0, changed)
+ .map((x) => x.n)
+ : all.slice(0, changed);
+ for (const note of notes) appendFileSync(join(vault, note), "\n<!-- measure -->\n");
+ await measured(`2. ${notes.length} ${largest ? "largest " : ""}notes edited`, vault);
+
+ await measured("3. no change again", vault);
+} finally {
+ rmSync(work, { recursive: true, force: true });
+}
diff --git a/tests/cli/search-index-freshen.test.ts b/tests/cli/search-index-freshen.test.ts
new file mode 100644
index 00000000..f9bf99b4
--- /dev/null
+++ b/tests/cli/search-index-freshen.test.ts
@@ -0,0 +1,137 @@
+/**
+ * The background child a stale read starts: `o2b search index --freshen
+ * <token>` records its outcome in the per-device state file, backs off
+ * after a failure, writes a metric row only when it changed something or
+ * failed, and releases the claim it was handed in every case.
+ */
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { listMetrics } from "../../src/core/brain/metrics.ts";
+import {
+ claimFreshen,
+ FRESHEN_CLAIM_FILE,
+ INDEX_FRESHEN_SURFACE,
+ readFreshenState,
+} from "../../src/core/search/freshen.ts";
+import { acquireWriterLockSync } from "../../src/core/search/store/writer-lock.ts";
+import { runCli } from "../helpers/run-cli.ts";
+
+let tmp: string;
+let vault: string;
+let derived: string;
+let configPath: string;
+
+const env = () => ({ OPEN_SECOND_BRAIN_CONFIG: configPath });
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "o2b-freshen-child-"));
+ vault = join(tmp, "vault");
+ derived = join(vault, ".open-second-brain");
+ mkdirSync(join(vault, "notes"), { recursive: true });
+ mkdirSync(derived, { recursive: true });
+ configPath = join(tmp, "config.yaml");
+ writeFileSync(configPath, `vault: ${vault}\n`);
+ writeFileSync(join(vault, "notes", "a.md"), "# A\n\nfirst note about herons.\n");
+});
+
+afterEach(() => rmSync(tmp, { recursive: true, force: true }));
+
+function freshenRun(): Promise<{ returncode: number }> {
+ const token = claimFreshen(derived, Date.now());
+ expect(token).toBeString();
+ return runCli(["search", "index", "--freshen", token!], { env: env() });
+}
+
+test("a completed run records its outcome, changes and a metric row, and releases the claim", async () => {
+ const result = await freshenRun();
+ expect(result.returncode).toBe(0);
+ const state = readFreshenState(derived);
+ expect(state.lastOutcome).toBe("completed");
+ expect(state.lastChanged).toBe(1);
+ expect(state.failures).toBe(0);
+ expect(state.backoffUntil).toBeNull();
+ expect(existsSync(join(derived, FRESHEN_CLAIM_FILE))).toBe(false);
+ const rows = listMetrics(vault, { surface: INDEX_FRESHEN_SURFACE });
+ expect(rows).toHaveLength(1);
+ expect(rows[0]!.payload["outcome"]).toBe("completed");
+});
+
+test("a run that changed nothing writes no metric row", async () => {
+ await freshenRun();
+ await freshenRun();
+ expect(readFreshenState(derived).lastChanged).toBe(0);
+ expect(listMetrics(vault, { surface: INDEX_FRESHEN_SURFACE })).toHaveLength(1);
+});
+
+test("a failed run backs off, doubling on the next failure, and still releases the claim", async () => {
+ // A directory squatting on the index path makes every open fail.
+ mkdirSync(join(derived, "brain.sqlite"));
+ const before = Date.now();
+ const first = await freshenRun();
+ expect(first.returncode).not.toBe(0);
+ let state = readFreshenState(derived);
+ expect(state.lastOutcome).toBe("failed");
+ expect(state.failures).toBe(1);
+ expect(state.lastError).toBeString();
+ const firstWait = Date.parse(state.backoffUntil!) - before;
+ expect(firstWait).toBeGreaterThanOrEqual(55_000);
+ expect(existsSync(join(derived, FRESHEN_CLAIM_FILE))).toBe(false);
+
+ await freshenRun();
+ state = readFreshenState(derived);
+ expect(state.failures).toBe(2);
+ expect(Date.parse(state.backoffUntil!) - Date.now()).toBeGreaterThan(110_000);
+ const rows = listMetrics(vault, { surface: INDEX_FRESHEN_SURFACE });
+ expect(rows.map((r) => r.payload["outcome"])).toEqual(["failed", "failed"]);
+});
+
+test("a success after failures clears the streak and the backoff", async () => {
+ mkdirSync(join(derived, "brain.sqlite"));
+ await freshenRun();
+ rmSync(join(derived, "brain.sqlite"), { recursive: true });
+ await freshenRun();
+ const state = readFreshenState(derived);
+ expect(state.lastOutcome).toBe("completed");
+ expect(state.failures).toBe(0);
+ expect(state.backoffUntil).toBeNull();
+});
+
+test("a run that meets another index writer is a skip, not a failure", async () => {
+ const release = acquireWriterLockSync(join(derived, "brain.sqlite"));
+ try {
+ await freshenRun();
+ } finally {
+ release();
+ }
+ const state = readFreshenState(derived);
+ expect(state.failures).toBe(0);
+ expect(state.backoffUntil).toBeNull();
+ expect(state.lastOutcome).toBeNull();
+ expect(existsSync(join(derived, FRESHEN_CLAIM_FILE))).toBe(false);
+ expect(listMetrics(vault, { surface: INDEX_FRESHEN_SURFACE })).toHaveLength(0);
+});
+
+test("a run that fails before it can resolve its config still records the failure and releases the claim", async () => {
+ const token = claimFreshen(derived, Date.now())!;
+ const result = await runCli(
+ [
+ "search",
+ "index",
+ "--config",
+ join(tmp, "missing.yaml"),
+ "--freshen",
+ token,
+ "--freshen-state",
+ derived,
+ ],
+ { env: env() },
+ );
+ expect(result.returncode).not.toBe(0);
+ const state = readFreshenState(derived);
+ expect(state.lastOutcome).toBe("failed");
+ expect(state.failures).toBe(1);
+ expect(existsSync(join(derived, FRESHEN_CLAIM_FILE))).toBe(false);
+});
diff --git a/tests/cli/search-status-freshen.test.ts b/tests/cli/search-status-freshen.test.ts
new file mode 100644
index 00000000..67693756
--- /dev/null
+++ b/tests/cli/search-status-freshen.test.ts
@@ -0,0 +1,114 @@
+/**
+ * `o2b search status` shows freshen on read: the interval (or off), how
+ * old the index is, what the last background run did, and an active
+ * backoff - in the human output and under `freshen` in `--json`.
+ */
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { writeFreshenState } from "../../src/core/search/freshen.ts";
+import { runCli } from "../helpers/run-cli.ts";
+
+let tmp: string;
+let vault: string;
+let configPath: string;
+
+beforeEach(async () => {
+ tmp = mkdtempSync(join(tmpdir(), "o2b-status-freshen-"));
+ vault = join(tmp, "vault");
+ mkdirSync(join(vault, "notes"), { recursive: true });
+ configPath = join(tmp, "config.yaml");
+ writeFileSync(configPath, `vault: ${vault}\n`);
+ writeFileSync(join(vault, "notes", "a.md"), "# A\n\nbody\n");
+ await runCli(["search", "index"], { env: env("60") });
+});
+
+afterEach(() => rmSync(tmp, { recursive: true, force: true }));
+
+function env(interval: string): Record<string, string> {
+ return {
+ OPEN_SECOND_BRAIN_CONFIG: configPath,
+ OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S: interval,
+ };
+}
+
+test("status names the interval, the index age and that no background run happened yet", async () => {
+ const out = await runCli(["search", "status"], { env: env("60") });
+ expect(out.returncode).toBe(0);
+ expect(out.stdout).toContain("freshen: every 60s");
+ expect(out.stdout).toMatch(/index_age:\s+\d+s/);
+ expect(out.stdout).toContain("last_freshen: (none)");
+ expect(out.stdout).not.toContain("freshen_backoff");
+});
+
+test("an interval of 0 reads as off", async () => {
+ const out = await runCli(["search", "status"], { env: env("0") });
+ expect(out.stdout).toContain("freshen: off");
+});
+
+test("a failed run shows its error and the active backoff, in text and JSON", async () => {
+ const backoffUntil = new Date(Date.now() + 120_000).toISOString();
+ writeFreshenState(join(vault, ".open-second-brain"), {
+ failures: 2,
+ backoffUntil,
+ lastOutcome: "failed",
+ lastRunAt: "2026-10-07T12:00:00.000Z",
+ lastDurationMs: 40,
+ lastError: "disk full",
+ lastChanged: null,
+ });
+ const text = await runCli(["search", "status"], { env: env("60") });
+ expect(text.stdout).toContain(
+ "last_freshen: failed 2026-10-07T12:00:00.000Z (2 in a row): disk full",
+ );
+ expect(text.stdout).toContain(`freshen_backoff: until ${backoffUntil}`);
+
+ const json = JSON.parse(
+ (await runCli(["search", "status", "--json"], { env: env("60") })).stdout,
+ );
+ expect(json.freshen).toMatchObject({
+ interval_s: 60,
+ last_outcome: "failed",
+ last_error: "disk full",
+ failures: 2,
+ backoff_until: backoffUntil,
+ });
+ expect(json.freshen.index_age_s).toBeGreaterThanOrEqual(0);
+});
+
+test("every freshen line keeps the 21-column value alignment of the status block", async () => {
+ writeFreshenState(join(vault, ".open-second-brain"), {
+ failures: 1,
+ backoffUntil: new Date(Date.now() + 120_000).toISOString(),
+ lastOutcome: "failed",
+ lastRunAt: "2026-10-07T12:00:00.000Z",
+ lastDurationMs: 40,
+ lastError: "disk full",
+ lastChanged: null,
+ });
+ const out = await runCli(["search", "status"], { env: env("60") });
+ const lines = out.stdout.split("\n").filter((l) => /^(freshen|index_age|last_freshen)/.test(l));
+ expect(lines).toHaveLength(4);
+ for (const line of lines) {
+ expect(line.slice(0, 21)).toMatch(/^[a-z_]+: +$/);
+ expect(line[21]).not.toBe(" ");
+ }
+});
+
+test("a completed run shows how many documents it changed", async () => {
+ writeFreshenState(join(vault, ".open-second-brain"), {
+ failures: 0,
+ backoffUntil: null,
+ lastOutcome: "completed",
+ lastRunAt: "2026-10-07T12:00:00.000Z",
+ lastDurationMs: 300,
+ lastError: null,
+ lastChanged: 3,
+ });
+ const out = await runCli(["search", "status"], { env: env("60") });
+ expect(out.stdout).toContain(
+ "last_freshen: completed 2026-10-07T12:00:00.000Z (3 changed)",
+ );
+});
diff --git a/tests/core/architecture/state-surface-census.test.ts b/tests/core/architecture/state-surface-census.test.ts
index e72fbff9..c4250102 100644
--- a/tests/core/architecture/state-surface-census.test.ts
+++ b/tests/core/architecture/state-surface-census.test.ts
@@ -80,7 +80,7 @@ const STORE_DIR_IDENTIFIER = "DERIVED_STORE_DIR";
const SWEPT_POPULATION_SIZE = 23;
/** Declared surfaces today. Pinned for the same reason. */
-const DECLARED_SURFACE_COUNT = 46;
+const DECLARED_SURFACE_COUNT = 47;
/** An exclusion reason has to be an argument, not a label. */
const MIN_REASON_LENGTH = 80;
diff --git a/tests/core/architecture/write-site-census.test.ts b/tests/core/architecture/write-site-census.test.ts
index 24353441..22876fbe 100644
--- a/tests/core/architecture/write-site-census.test.ts
+++ b/tests/core/architecture/write-site-census.test.ts
@@ -912,6 +912,18 @@ const DIRECT_WRITE_EXCLUSIONS: Readonly<Record<string, WriteExclusion>> = Object
"this tool's own per-device state, never a note.",
},
+ "src/core/search/freshen.ts": {
+ categories: [C.lockPrimitive, C.retentionDelete],
+ calls: ["renameSync", "unlinkSync", "writeFileSync"],
+ reason:
+ "the freshen-on-read claim is an exclusive create (`wx`) of a token file in " +
+ "`.open-second-brain/`, released by unlinking it; a dead run's claim is replaced by " +
+ "an atomic rename under an exclusive takeover lock; routing it through " +
+ "the atomic writer would lose the exclusivity that keeps concurrent readers from " +
+ "starting a background index run each. The run's state file goes through " +
+ "`atomicWriteFileSync`. Both are this tool's own per-device state, never a note.",
+ },
+
"src/core/state/migrate.ts": {
categories: [C.archiveTransfer, C.lifecycleMove, C.retentionDelete],
calls: ["copyFileSync", "rmSync", "rmdirSync"],
@@ -1221,8 +1233,10 @@ const DIRECT_ROWS = ROWS.filter((row) => row.directCalls.length > 0);
* removes the failed-upgrade marker once nothing is pending.
* 75 -> 76: the doctor self-test seeds and journals a throwaway store in
* temp storage and removes the tree when the run ends.
+ * 76 -> 77: `src/core/search/freshen.ts` claims and releases the
+ * freshen-on-read claim (an exclusive create) in `.open-second-brain/`.
*/
-const DIRECT_WRITE_ROWS = 76;
+const DIRECT_WRITE_ROWS = 77;
/**
* Measured modules reaching a write through a shared helper. An equality.
@@ -1285,8 +1299,11 @@ const DIRECT_WRITE_ROWS = 76;
* candidates through the shared atomic JSONL writer.
* 108 -> 109: `src/core/search/session-focus.ts` writes the focus through
* `atomicWriteText` (mode 0600, a leaf link is replaced, not followed).
+ *
+ * 109 -> 110: `src/core/search/freshen.ts` writes the freshen-on-read state
+ * file through `atomicWriteFileSync`.
*/
-const SHARED_HELPER_ROWS = 109;
+const SHARED_HELPER_ROWS = 110;
// ----- Origin-channel coverage boundary (Unit C) ----------------------------
@@ -1357,8 +1374,9 @@ const STAMPED_PATHS: ReadonlySet<string> = new Set(
* 72 -> 73: the payload registry's touch (metadata only).
* 73 -> 74: the automatic-upgrade lock and marker removal (state, not notes).
* 74 -> 75: the doctor self-test's throwaway-store writes (temp storage).
+ * 75 -> 76: the freshen-on-read claim and its release (state, not notes).
*/
-const UNSTAMPED_DIRECT_ROWS = 75;
+const UNSTAMPED_DIRECT_ROWS = 76;
/**
* Shared-helper write sites the stamp does not reach, measured the same
@@ -1371,9 +1389,10 @@ const UNSTAMPED_DIRECT_ROWS = 75;
* and the dedup index cache, a derived file outside the vault (102 -> 103),
* and the orphan-repair detach (103 -> 104). 104 -> 105: the repair lane
* stages hub candidates through the shared atomic JSONL writer. 105 -> 106:
- * the session focus writes through the shared atomic text writer.
+ * the session focus writes through the shared atomic text writer. 106 -> 107:
+ * the freshen-on-read state file writes through the shared atomic writer.
*/
-const UNSTAMPED_SHARED_ROWS = 106;
+const UNSTAMPED_SHARED_ROWS = 107;
describe("in-vault write-site census", () => {
test("every direct-fs write site carries a written exclusion", () => {
diff --git a/tests/core/brain/doctor-exit-census.test.ts b/tests/core/brain/doctor-exit-census.test.ts
index 242c5ff1..b9e2dd5b 100644
--- a/tests/core/brain/doctor-exit-census.test.ts
+++ b/tests/core/brain/doctor-exit-census.test.ts
@@ -188,6 +188,7 @@ const DOCTOR_REGISTERED_CODES: ReadonlyArray<string> = [
"embeddings-backlog",
"entity-label-malformed",
"entity-quote-variant-collision",
+ "freshen-failing",
"inbox-archivable",
"log-chain-broken",
"low-evidence-confirmed",
diff --git a/tests/core/brain/doctor-freshen.test.ts b/tests/core/brain/doctor-freshen.test.ts
new file mode 100644
index 00000000..8faf7574
--- /dev/null
+++ b/tests/core/brain/doctor-freshen.test.ts
@@ -0,0 +1,104 @@
+/**
+ * The doctor warns when background index runs started by stale reads keep
+ * failing: three in a row is a broken index, not a transient hiccup, and
+ * until it is fixed every search answers from an ageing index.
+ */
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { dirname, join } from "node:path";
+
+import { applyRepair } from "../../../src/core/brain/diagnostics.ts";
+import { runDoctor } from "../../../src/core/brain/doctor.ts";
+import { bootstrapBrain } from "../../../src/core/brain/init.ts";
+import { writeFreshenState } from "../../../src/core/search/freshen.ts";
+import { resolveSearchConfig } from "../../../src/core/search/index.ts";
+import { MCPServer } from "../../../src/mcp/server.ts";
+import { runCli } from "../../helpers/run-cli.ts";
+
+let tmp: string;
+let vault: string;
+let configPath: string;
+let derived: string;
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "o2b-doctor-freshen-"));
+ vault = join(tmp, "vault");
+ configPath = join(tmp, "config.yaml");
+ writeFileSync(configPath, `vault: ${vault}\n`);
+ bootstrapBrain(vault, { configPath });
+ derived = join(vault, ".open-second-brain");
+ // The surfaces below resolve the index path themselves; the state file
+ // has to sit where they look.
+ expect(dirname(resolveSearchConfig({ vault, configPath }).dbPath)).toBe(derived);
+ mkdirSync(derived, { recursive: true });
+});
+
+afterEach(() => rmSync(tmp, { recursive: true, force: true }));
+
+function failures(n: number): void {
+ writeFreshenState(derived, {
+ failures: n,
+ backoffUntil: new Date(Date.now() + 60_000).toISOString(),
+ lastOutcome: "failed",
+ lastRunAt: new Date().toISOString(),
+ lastDurationMs: 10,
+ lastError: "database disk image is malformed",
+ lastChanged: null,
+ });
+}
+
+function freshenWarnings() {
+ return runDoctor(vault, { dbPath: join(derived, "brain.sqlite") }).warnings.filter(
+ (w) => w.code === "freshen-failing",
+ );
+}
+
+test("three failed background runs in a row are a warning naming the last error", () => {
+ failures(3);
+ const hits = freshenWarnings();
+ expect(hits).toHaveLength(1);
+ expect(hits[0]!.message).toContain("3");
+ expect(hits[0]!.message).toContain("database disk image is malformed");
+});
+
+test("fewer than three failures stay quiet", () => {
+ failures(2);
+ expect(freshenWarnings()).toEqual([]);
+});
+
+test("no state file is no warning", () => {
+ expect(freshenWarnings()).toEqual([]);
+});
+
+test("the repair plan lists a failing freshen as not auto-repairable", () => {
+ failures(3);
+ const outcome = applyRepair(vault, { dryRun: true, dbPath: join(derived, "brain.sqlite") });
+ expect(outcome.unfixable.map((u) => u.code)).toContain("freshen-failing");
+});
+
+test("o2b brain doctor --repair reaches the failing freshen", async () => {
+ failures(3);
+ const r = await runCli(["brain", "doctor", "--vault", vault, "--repair", "--json"], {
+ env: { OPEN_SECOND_BRAIN_CONFIG: configPath },
+ });
+ expect(r.returncode).toBe(0);
+ const outcome = JSON.parse(r.stdout) as { unfixable: ReadonlyArray<{ code: string }> };
+ expect(outcome.unfixable.map((u) => u.code)).toContain("freshen-failing");
+});
+
+test("brain_doctor reaches the failing freshen in its report and its repair preview", async () => {
+ failures(3);
+ const saved = process.env["OPEN_SECOND_BRAIN_CONFIG"];
+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = configPath;
+ try {
+ const server = new MCPServer({ vault, configPath });
+ const report = JSON.stringify(await server.callTool("brain_doctor", {}));
+ expect(report).toContain("freshen-failing");
+ const repair = JSON.stringify(await server.callTool("brain_doctor", { repair: true }));
+ expect(repair).toContain("freshen-failing");
+ } finally {
+ if (saved === undefined) delete process.env["OPEN_SECOND_BRAIN_CONFIG"];
+ else process.env["OPEN_SECOND_BRAIN_CONFIG"] = saved;
+ }
+});
diff --git a/tests/core/brain/tier-drift.test.ts b/tests/core/brain/tier-drift.test.ts
index 32e908de..e7bab29e 100644
--- a/tests/core/brain/tier-drift.test.ts
+++ b/tests/core/brain/tier-drift.test.ts
@@ -12,6 +12,8 @@ import { afterEach, beforeEach, expect, test } from "bun:test";
import { mkdirSync, writeFileSync } from "node:fs";
import { join } from "node:path";
+import { runDoctor } from "../../../src/core/brain/doctor.ts";
+import { TIER_DRIFT_CODE } from "../../../src/core/brain/doctor/store-integrity.ts";
import { indexVault } from "../../../src/core/search/indexer.ts";
import { Store } from "../../../src/core/search/store.ts";
import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";
@@ -152,3 +154,19 @@ test("deleting the document drops its drift rows", async () => {
await indexVault(config);
expect(await drift()).toHaveLength(0);
});
+
+test("the doctor counts only the drift rows on pages the caller may read", async () => {
+ const config = makeConfig({ vault, dbPath });
+ writePref("pref-spaces");
+ await indexVault(config);
+ writePref("pref-tabs");
+ await indexVault(config);
+ const driftWarnings = (readable?: (rel: string) => boolean) =>
+ runDoctor(vault, { dbPath, ...(readable !== undefined ? { readable } : {}) }).warnings.filter(
+ (w) => w.code === TIER_DRIFT_CODE,
+ );
+ expect(driftWarnings()).toHaveLength(1);
+ expect(driftWarnings((rel) => rel === "Brain/preferences/pref-spaces.md")).toHaveLength(1);
+ // A caller that may not read the drifted page is told nothing about it.
+ expect(driftWarnings(() => false)).toEqual([]);
+});
diff --git a/tests/core/maintenance/ensure-current.test.ts b/tests/core/maintenance/ensure-current.test.ts
index e46efd56..9ff686f4 100644
--- a/tests/core/maintenance/ensure-current.test.ts
+++ b/tests/core/maintenance/ensure-current.test.ts
@@ -208,3 +208,60 @@ describe("ensureVaultCurrent: chunks cut by older chunking rules (#186)", () =>
expect(r.reindexTriggered).toBe(false);
});
});
+
+describe("ensureVaultCurrent: freshen on session start", () => {
+ const INTERVAL_ENV = "OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S";
+ let prevInterval: string | undefined;
+ let calls: string[][];
+ const freshenSpawn = (argv: string[]): void => {
+ calls.push(argv);
+ };
+
+ beforeEach(() => {
+ prevInterval = process.env[INTERVAL_ENV];
+ process.env[INTERVAL_ENV] = "60";
+ calls = [];
+ });
+
+ afterEach(() => {
+ if (prevInterval === undefined) delete process.env[INTERVAL_ENV];
+ else process.env[INTERVAL_ENV] = prevInterval;
+ });
+
+ function ageIndex(seconds: number): void {
+ const db = new Database(dbPath());
+ db.query("UPDATE index_state SET value = ? WHERE key = 'last_indexed_at'").run(
+ new Date(Date.now() - seconds * 1000).toISOString(),
+ );
+ db.close();
+ }
+
+ test("a current index that has gone stale is caught up in the background", async () => {
+ bootstrapBrain(vault, { configPath });
+ await ensureVaultCurrent(vault, { background: false });
+ ageIndex(300);
+ const r = await ensureVaultCurrent(vault, { background: false, freshenSpawn });
+ expect(r.freshen).toBe("spawned");
+ expect(calls).toHaveLength(1);
+ });
+
+ test("a fresh index starts nothing", async () => {
+ bootstrapBrain(vault, { configPath });
+ await ensureVaultCurrent(vault, { background: false });
+ const r = await ensureVaultCurrent(vault, { background: false, freshenSpawn });
+ expect(r.freshen).toBe("fresh");
+ expect(calls).toHaveLength(0);
+ });
+
+ test("an index that needs a rebuild is left to self-heal, not freshened", async () => {
+ bootstrapBrain(vault, { configPath });
+ await ensureVaultCurrent(vault, { background: false });
+ const db = new Database(dbPath());
+ db.run("UPDATE index_state SET value = '1' WHERE key = 'schema_version'");
+ db.close();
+ const r = await ensureVaultCurrent(vault, { background: false, freshenSpawn });
+ expect(r.reindexTriggered).toBe(true);
+ expect(r.freshen).toBeNull();
+ expect(calls).toHaveLength(0);
+ });
+});
diff --git a/tests/core/search/chunk-preserving-replace.test.ts b/tests/core/search/chunk-preserving-replace.test.ts
new file mode 100644
index 00000000..b02daa7d
--- /dev/null
+++ b/tests/core/search/chunk-preserving-replace.test.ts
@@ -0,0 +1,165 @@
+/**
+ * Re-indexing a changed document keeps every chunk whose position and
+ * content did not change in place (same row, same id) and rewrites only
+ * the chunks that did. A daily log that grows all day then costs the
+ * appended tail on each run, not the whole file: before, every chunk was
+ * deleted and re-inserted, full-text rows included.
+ */
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import { Database } from "bun:sqlite";
+
+import { indexVault } from "../../../src/core/search/indexer.ts";
+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";
+
+let vault: string;
+let dbPath: string;
+let cleanup: () => void;
+
+/** A section of about 900 words, more than one 800-word chunk can hold alone. */
+function section(name: string): string {
+ const words = Array.from({ length: 900 }, (_, i) => `${name}${i % 50}`).join(" ");
+ return `## ${name}\n\n${words}\n`;
+}
+
+const LOG = "notes/log.md";
+
+beforeEach(() => {
+ ({ vault, dbPath, cleanup } = createTempVault("chunk-preserving"));
+ writeMd(vault, LOG, `# Log\n\n${section("alpha")}\n${section("bravo")}\n${section("charlie")}`);
+ // Indexed after the log, so the log's chunks are not the highest rowids:
+ // SQLite would otherwise hand a delete-and-reinsert the same ids back.
+ writeMd(vault, "notes/zz-other.md", `# Other\n\n${section("zulu")}`);
+});
+
+afterEach(() => cleanup());
+
+interface ChunkRow {
+ id: number;
+ chunk_index: number;
+ content_hash: string;
+}
+
+function chunks(): ChunkRow[] {
+ const db = new Database(dbPath, { readonly: true });
+ try {
+ return db
+ .query<ChunkRow, [string]>(
+ "SELECT c.id, c.chunk_index, c.content_hash FROM chunks c " +
+ "JOIN documents d ON d.id = c.document_id WHERE d.path = ? ORDER BY c.chunk_index",
+ )
+ .all(LOG);
+ } finally {
+ db.close();
+ }
+}
+
+function ftsHits(term: string): number {
+ const db = new Database(dbPath, { readonly: true });
+ try {
+ return db
+ .query<{ n: number }, [string]>("SELECT count(*) AS n FROM chunk_fts WHERE chunk_fts MATCH ?")
+ .get(term)!.n;
+ } finally {
+ db.close();
+ }
+}
+
+function entityRows(): number {
+ const db = new Database(dbPath, { readonly: true });
+ try {
+ return db.query<{ n: number }, []>("SELECT count(*) AS n FROM chunk_entities").get()!.n;
+ } finally {
+ db.close();
+ }
+}
+
+test("appending to a document keeps every earlier chunk in place", async () => {
+ const config = makeConfig({ vault, dbPath });
+ await indexVault(config);
+ const before = chunks();
+ expect(before.length).toBeGreaterThanOrEqual(3);
+
+ writeMd(
+ vault,
+ LOG,
+ `# Log\n\n${section("alpha")}\n${section("bravo")}\n${section("charlie")}\n${section("delta")}`,
+ );
+ await indexVault(config);
+ const after = chunks();
+
+ expect(after.length).toBeGreaterThan(before.length);
+ // Every chunk but the last of the old document survives with its id.
+ const kept = before.slice(0, -1);
+ expect(after.slice(0, kept.length)).toEqual(kept);
+ expect(ftsHits("delta7")).toBeGreaterThan(0);
+ expect(ftsHits("alpha7")).toBeGreaterThan(0);
+});
+
+test("editing the middle rewrites only the chunks that changed", async () => {
+ const config = makeConfig({ vault, dbPath });
+ await indexVault(config);
+ const before = chunks();
+
+ writeMd(vault, LOG, `# Log\n\n${section("alpha")}\n${section("bravo2")}\n${section("charlie")}`);
+ await indexVault(config);
+ const after = chunks();
+
+ const sameHash = (a: ChunkRow, b: ChunkRow): boolean =>
+ a.chunk_index === b.chunk_index && a.content_hash === b.content_hash;
+ for (const row of after) {
+ const old = before.find((b) => sameHash(b, row));
+ // A chunk at the same position with the same content keeps its id;
+ // a changed one is a new row.
+ if (old !== undefined) expect(row.id).toBe(old.id);
+ else expect(before.map((b) => b.id)).not.toContain(row.id);
+ }
+ expect(after.filter((r) => before.some((b) => b.id === r.id)).length).toBeGreaterThan(0);
+ expect(ftsHits("bravo27")).toBeGreaterThan(0);
+ expect(ftsHits("bravo7")).toBe(0);
+});
+
+test("shrinking a document removes the chunks past its new end, full-text rows included", async () => {
+ const config = makeConfig({ vault, dbPath });
+ await indexVault(config);
+ writeMd(vault, LOG, `# Log\n\n${section("alpha")}`);
+ await indexVault(config);
+ expect(ftsHits("charlie7")).toBe(0);
+ expect(ftsHits("alpha7")).toBeGreaterThan(0);
+ const after = chunks();
+ expect(after.map((c) => c.chunk_index)).toEqual(after.map((_, i) => i));
+});
+
+test("a kept chunk keeps its entities and a rewritten one gets fresh ones", async () => {
+ writeMd(vault, LOG, `# Log\n\nMeeting with Ada Lovelace.\n\n${section("alpha")}`);
+ const config = makeConfig({ vault, dbPath });
+ await indexVault(config);
+ const entitiesBefore = entityRows();
+ writeMd(
+ vault,
+ LOG,
+ `# Log\n\nMeeting with Ada Lovelace.\n\n${section("alpha")}\n${section("bravo")}`,
+ );
+ await indexVault(config);
+ expect(entityRows()).toBeGreaterThanOrEqual(entitiesBefore);
+});
+
+test("a chunk whose stored full-text form is out of date is rewritten, not kept", async () => {
+ const config = makeConfig({ vault, dbPath });
+ await indexVault(config);
+ // What a release that changes how full-text content is derived would
+ // leave behind: same content and hash, a different stored FTS form.
+ const db = new Database(dbPath);
+ db.run(
+ "UPDATE chunks SET fts_content = 'stale-form' WHERE chunk_index = 0 AND document_id = " +
+ "(SELECT id FROM documents WHERE path = ?)",
+ [LOG],
+ );
+ db.close();
+ writeMd(
+ vault,
+ LOG,
+ `# Log\n\n${section("alpha")}\n${section("bravo")}\n${section("charlie")}\n${section("delta")}`,
+ );
+ await indexVault(config);
+ expect(ftsHits("stale")).toBe(0);
+});
diff --git a/tests/core/search/freshen-config.test.ts b/tests/core/search/freshen-config.test.ts
new file mode 100644
index 00000000..c999eb0f
--- /dev/null
+++ b/tests/core/search/freshen-config.test.ts
@@ -0,0 +1,71 @@
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { resolveSearchConfig } from "../../../src/core/search/index.ts";
+import { SearchError } from "../../../src/core/search/types.ts";
+
+// The test preload pins the interval to 0 so the suite never spawns a
+// background indexer; these tests own the variable while they run.
+const ENV_KEYS = [
+ "OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S",
+ "OPEN_SECOND_BRAIN_SEARCH_FRESHEN_EMBEDDINGS",
+];
+
+let tmp: string;
+let configPath: string;
+const saved: Record<string, string | undefined> = {};
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "osb-freshen-cfg-"));
+ configPath = join(tmp, "config.yaml");
+ for (const k of ENV_KEYS) {
+ saved[k] = process.env[k];
+ delete process.env[k];
+ }
+});
+
+afterEach(() => {
+ for (const k of ENV_KEYS) {
+ if (saved[k] === undefined) delete process.env[k];
+ else process.env[k] = saved[k];
+ }
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+function resolve(extra = ""): ReturnType<typeof resolveSearchConfig> {
+ writeFileSync(configPath, `vault: "${tmp}"\n${extra}`);
+ return resolveSearchConfig({ vault: tmp, configPath });
+}
+
+test("freshening is on every 60 s by default, keyword-only, with the config path", () => {
+ expect(resolve().freshen).toEqual({ intervalSeconds: 60, embeddings: false, configPath });
+});
+
+test("the config key sets the interval and 0 turns freshening off", () => {
+ expect(resolve('search_freshen_interval_s: "300"\n').freshen?.intervalSeconds).toBe(300);
+ expect(resolve('search_freshen_interval_s: "0"\n').freshen?.intervalSeconds).toBe(0);
+});
+
+test("the env variable wins over the config key", () => {
+ process.env["OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S"] = "0";
+ expect(resolve('search_freshen_interval_s: "300"\n').freshen?.intervalSeconds).toBe(0);
+});
+
+test("a negative or non-numeric interval is refused by name", () => {
+ for (const bad of ['"-1"', '"abc"']) {
+ expect(() => resolve(`search_freshen_interval_s: ${bad}\n`)).toThrow(SearchError);
+ expect(() => resolve(`search_freshen_interval_s: ${bad}\n`)).toThrow(
+ /search_freshen_interval_s/,
+ );
+ }
+});
+
+test("background embeddings are opt-in", () => {
+ expect(resolve('search_freshen_embeddings: "true"\n').freshen?.embeddings).toBe(true);
+});
+
+test("a config resolved without a file carries no config path", () => {
+ expect(resolveSearchConfig({ vault: tmp }).freshen?.configPath).toBeNull();
+});
diff --git a/tests/core/search/freshen-search.test.ts b/tests/core/search/freshen-search.test.ts
new file mode 100644
index 00000000..432459ca
--- /dev/null
+++ b/tests/core/search/freshen-search.test.ts
@@ -0,0 +1,153 @@
+/**
+ * Freshen on read at the `search()` seam: a stale index starts one
+ * background run and the answer still comes from the index as it is; a
+ * read-only open never starts one; an index more than ten minutes old is
+ * named on the trail as `index-stale` whatever the freshen setting. The
+ * query cache stores answers without that code and the trail gains it at
+ * read time, so a cache hit names the index age it is served at.
+ */
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import { Database } from "bun:sqlite";
+
+import { indexVault } from "../../../src/core/search/indexer.ts";
+import { RETRIEVAL_DEGRADATION } from "../../../src/core/search/retrieval-trail.ts";
+import { search } from "../../../src/core/search/search.ts";
+import { LAST_INDEXED_AT_STATE_KEY } from "../../../src/core/search/store/state.ts";
+import type { ResolvedSearchConfig } from "../../../src/core/search/types.ts";
+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";
+
+let vault: string;
+let dbPath: string;
+let cleanup: () => void;
+let calls: string[][];
+const spawn = (argv: string[]): void => {
+ calls.push(argv);
+};
+
+beforeEach(async () => {
+ const v = createTempVault("freshen-search");
+ vault = v.vault;
+ dbPath = v.dbPath;
+ cleanup = v.cleanup;
+ calls = [];
+ writeMd(vault, "notes/heron.md", "# Heron\n\nThe grey heron waits by the river.\n");
+ await indexVault(config(60));
+});
+
+afterEach(() => cleanup());
+
+function config(intervalSeconds: number, cacheEnabled = false): ResolvedSearchConfig {
+ return Object.freeze({
+ ...makeConfig({ vault, dbPath, cacheEnabled }),
+ freshen: { intervalSeconds, embeddings: false, configPath: null },
+ });
+}
+
+function setIndexAge(seconds: number): void {
+ const db = new Database(dbPath);
+ try {
+ db.query("UPDATE index_state SET value = ? WHERE key = ?").run(
+ new Date(Date.now() - seconds * 1000).toISOString(),
+ LAST_INDEXED_AT_STATE_KEY,
+ );
+ } finally {
+ db.close();
+ }
+}
+
+function staleCodes(outcome: Awaited<ReturnType<typeof search>>) {
+ return (outcome.retrievalTrail?.degraded ?? []).filter(
+ (d) => d.code === RETRIEVAL_DEGRADATION.indexStale,
+ );
+}
+
+test("a stale index starts one background run and still answers", async () => {
+ setIndexAge(300);
+ const outcome = await search(config(60), { query: "heron", freshenSpawn: spawn });
+ expect(outcome.results.length).toBeGreaterThan(0);
+ expect(calls).toHaveLength(1);
+ expect(calls[0]!.join(" ")).toContain("search index --vault");
+});
+
+test("a fresh index starts nothing", async () => {
+ await search(config(60), { query: "heron", freshenSpawn: spawn });
+ expect(calls).toHaveLength(0);
+});
+
+test("a read-only open never starts a run", async () => {
+ setIndexAge(3600);
+ await search(config(60), { query: "heron", freshenSpawn: spawn, selfHeal: false });
+ expect(calls).toHaveLength(0);
+});
+
+test("an index older than ten minutes is named on the trail with its age", async () => {
+ setIndexAge(11 * 60);
+ const outcome = await search(config(60), { query: "heron", freshenSpawn: spawn });
+ const stale = staleCodes(outcome);
+ expect(stale).toHaveLength(1);
+ expect(stale[0]!.detail?.["ageSeconds"]).toBeGreaterThanOrEqual(660);
+});
+
+test("a lag of a couple of minutes is not reported", async () => {
+ setIndexAge(120);
+ expect(staleCodes(await search(config(60), { query: "heron", freshenSpawn: spawn }))).toEqual([]);
+});
+
+test("with freshening off a stale index is still reported but nothing starts", async () => {
+ setIndexAge(11 * 60);
+ const outcome = await search(config(0), { query: "heron", freshenSpawn: spawn });
+ expect(staleCodes(outcome)).toHaveLength(1);
+ expect(calls).toHaveLength(0);
+});
+
+function cachedPayloads(): string[] {
+ const db = new Database(dbPath, { readonly: true });
+ try {
+ return db
+ .query<{ payload: string }, []>("SELECT payload FROM query_cache")
+ .all()
+ .map((r) => r.payload);
+ } finally {
+ db.close();
+ }
+}
+
+/** Mark every cached row, so an answer carrying the mark was a cache hit. */
+function markCachedRows(total: number): void {
+ const db = new Database(dbPath);
+ try {
+ db.query("UPDATE query_cache SET payload = json_set(payload, '$.total', ?)").run(total);
+ } finally {
+ db.close();
+ }
+}
+
+test("with freshening off a stale-index answer is cached without its stale code", async () => {
+ setIndexAge(11 * 60);
+ const outcome = await search(config(0, true), { query: "heron", freshenSpawn: spawn });
+ expect(staleCodes(outcome)).toHaveLength(1);
+ const rows = cachedPayloads();
+ expect(rows).toHaveLength(1);
+ expect(rows[0]).not.toContain(RETRIEVAL_DEGRADATION.indexStale);
+});
+
+test("a cache hit over an index that has since gone stale names it", async () => {
+ await search(config(0, true), { query: "heron", freshenSpawn: spawn });
+ markCachedRows(4242);
+ setIndexAge(11 * 60);
+ const outcome = await search(config(0, true), { query: "heron", freshenSpawn: spawn });
+ expect(outcome.total).toBe(4242);
+ expect(staleCodes(outcome)).toHaveLength(1);
+ expect(outcome.retrievalTrail?.retrieved).toBe(outcome.results.length);
+});
+
+test("a cache hit over a fresh index carries no stale code", async () => {
+ setIndexAge(11 * 60);
+ await search(config(0, true), { query: "heron", freshenSpawn: spawn });
+ markCachedRows(4242);
+ setIndexAge(0);
+ const outcome = await search(config(0, true), { query: "heron", freshenSpawn: spawn });
+ expect(outcome.total).toBe(4242);
+ expect(staleCodes(outcome)).toEqual([]);
+ expect(outcome.retrievalTrail).toBeUndefined();
+});
diff --git a/tests/core/search/freshen.test.ts b/tests/core/search/freshen.test.ts
new file mode 100644
index 00000000..d825e80e
--- /dev/null
+++ b/tests/core/search/freshen.test.ts
@@ -0,0 +1,384 @@
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import {
+ chmodSync,
+ existsSync,
+ mkdirSync,
+ mkdtempSync,
+ readFileSync,
+ rmSync,
+ utimesSync,
+ writeFileSync,
+} from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import {
+ claimFreshen,
+ decideFreshen,
+ FRESHEN_CLAIM_FILE,
+ FRESHEN_SKIP,
+ freshenCommand,
+ freshenSpawnOptions,
+ maybeFreshenIndex,
+ nextBackoffMs,
+ readFreshenState,
+ releaseFreshen,
+ writeFreshenState,
+} from "../../../src/core/search/freshen.ts";
+import { acquireWriterLockSync } from "../../../src/core/search/store/writer-lock.ts";
+import type { ResolvedSearchConfig } from "../../../src/core/search/types.ts";
+import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";
+
+const NOW = Date.parse("2026-10-07T12:00:00.000Z");
+const ago = (s: number): string => new Date(NOW - s * 1000).toISOString();
+
+describe("decideFreshen", () => {
+ const base = { nowMs: NOW, intervalSeconds: 60, backoffUntilMs: null };
+
+ test("an interval of 0 is off", () => {
+ expect(decideFreshen({ ...base, intervalSeconds: 0, lastIndexedAt: ago(3600) })).toEqual({
+ action: "skip",
+ reason: FRESHEN_SKIP.off,
+ });
+ });
+
+ test("an index never stamped is left to self-heal", () => {
+ expect(decideFreshen({ ...base, lastIndexedAt: null })).toEqual({
+ action: "skip",
+ reason: FRESHEN_SKIP.noIndex,
+ });
+ });
+
+ test("an index younger than the interval is fresh", () => {
+ expect(decideFreshen({ ...base, lastIndexedAt: ago(30) })).toEqual({
+ action: "skip",
+ reason: FRESHEN_SKIP.fresh,
+ });
+ });
+
+ test("an index older than the interval is due", () => {
+ expect(decideFreshen({ ...base, lastIndexedAt: ago(61) })).toEqual({ action: "spawn" });
+ });
+
+ test("an active backoff wins over a stale index", () => {
+ expect(
+ decideFreshen({ ...base, lastIndexedAt: ago(3600), backoffUntilMs: NOW + 1000 }),
+ ).toEqual({ action: "skip", reason: FRESHEN_SKIP.backoff });
+ });
+
+ test("a fresh index is fresh even while a backoff is active", () => {
+ expect(decideFreshen({ ...base, lastIndexedAt: ago(30), backoffUntilMs: NOW + 1000 })).toEqual({
+ action: "skip",
+ reason: FRESHEN_SKIP.fresh,
+ });
+ });
+
+ test("an unparseable stamp counts as due rather than fresh", () => {
+ expect(decideFreshen({ ...base, lastIndexedAt: "not a date" })).toEqual({ action: "spawn" });
+ });
+});
+
+describe("claim, state and backoff", () => {
+ let dir: string;
+ beforeEach(() => {
+ dir = mkdtempSync(join(tmpdir(), "osb-freshen-"));
+ });
+ afterEach(() => rmSync(dir, { recursive: true, force: true }));
+
+ test("only the first of two claims wins", () => {
+ expect(claimFreshen(dir, NOW)).toBeString();
+ expect(claimFreshen(dir, NOW)).toBeNull();
+ });
+
+ test("a claim abandoned for more than ten minutes is taken over", () => {
+ writeFileSync(
+ join(dir, FRESHEN_CLAIM_FILE),
+ JSON.stringify({ token: "dead", at: NOW - 601_000 }),
+ );
+ const token = claimFreshen(dir, NOW);
+ expect(token).toBeString();
+ expect(token).not.toBe("dead");
+ });
+
+ test("a torn claim file older than the grace is taken over too", () => {
+ const path = join(dir, FRESHEN_CLAIM_FILE);
+ writeFileSync(path, "{");
+ const old = new Date(Date.now() - 60_000);
+ utimesSync(path, old, old);
+ expect(claimFreshen(dir, Date.now())).toBeString();
+ });
+
+ test("releasing with a foreign token leaves the claim in place", () => {
+ const token = claimFreshen(dir, NOW)!;
+ releaseFreshen(dir, "someone-else");
+ expect(existsSync(join(dir, FRESHEN_CLAIM_FILE))).toBe(true);
+ releaseFreshen(dir, token);
+ expect(existsSync(join(dir, FRESHEN_CLAIM_FILE))).toBe(false);
+ });
+
+ test("state round-trips and a missing or torn file reads as empty", () => {
+ expect(readFreshenState(dir).failures).toBe(0);
+ writeFileSync(join(dir, "freshen-state.json"), "{");
+ expect(readFreshenState(dir).lastOutcome).toBeNull();
+ const state = {
+ failures: 2,
+ backoffUntil: ago(-120),
+ lastOutcome: "failed" as const,
+ lastRunAt: ago(5),
+ lastDurationMs: 300,
+ lastError: "disk full",
+ lastChanged: null,
+ };
+ writeFreshenState(dir, state);
+ expect(readFreshenState(dir)).toEqual(state);
+ });
+
+ test("backoff doubles from a minute and stops at an hour", () => {
+ expect(nextBackoffMs(1)).toBe(60_000);
+ expect(nextBackoffMs(2)).toBe(120_000);
+ expect(nextBackoffMs(10)).toBe(3_600_000);
+ });
+});
+
+describe("freshenCommand", () => {
+ const base = ["o2b", "search", "index"];
+ test("linux runs the child at idle I/O priority when ionice exists", () => {
+ expect(freshenCommand(base, (t) => t === "ionice", "linux")).toEqual([
+ "ionice",
+ "-c3",
+ ...base,
+ ]);
+ expect(freshenCommand(base, () => false, "linux")).toEqual(base);
+ });
+ test("macOS uses taskpolicy and Windows needs no prefix", () => {
+ expect(freshenCommand(base, (t) => t === "taskpolicy", "darwin")).toEqual([
+ "taskpolicy",
+ "-b",
+ ...base,
+ ]);
+ expect(freshenCommand(base, () => true, "win32")).toEqual(base);
+ });
+});
+
+describe("freshenSpawnOptions", () => {
+ test("the child is detached on every platform so it outlives a short-lived hook", () => {
+ const opts = freshenSpawnOptions({ O2B_TEST: "1" });
+ expect(opts.detached).toBe(true);
+ expect(opts.windowsHide).toBe(true);
+ expect([opts.stdin, opts.stdout, opts.stderr]).toEqual(["ignore", "ignore", "ignore"]);
+ });
+
+ test("the child gets a copy of the current environment, not the start-up one", () => {
+ const env = { O2B_CONFIG: "/later/config.yaml" };
+ const opts = freshenSpawnOptions(env);
+ expect(opts.env).toEqual(env);
+ expect(opts.env).not.toBe(env);
+ });
+});
+
+describe("maybeFreshenIndex", () => {
+ let vault: string;
+ let config: ResolvedSearchConfig;
+ let calls: string[][];
+ const spawn = (argv: string[]): void => {
+ calls.push(argv);
+ };
+
+ beforeEach(() => {
+ vault = mkdtempSync(join(tmpdir(), "osb-freshen-vault-"));
+ mkdirSync(join(vault, ".open-second-brain"), { recursive: true });
+ calls = [];
+ config = {
+ vault,
+ dbPath: join(vault, ".open-second-brain", "brain.sqlite"),
+ freshen: { intervalSeconds: 60, embeddings: false, configPath: "/cfg/config.yaml" },
+ } as unknown as ResolvedSearchConfig;
+ });
+ afterEach(() => rmSync(vault, { recursive: true, force: true }));
+
+ test("a stale index spawns one index run carrying the vault, config and claim", () => {
+ expect(maybeFreshenIndex(config, { lastIndexedAt: ago(300), nowMs: NOW, spawn })).toBe(
+ "spawned",
+ );
+ expect(calls).toHaveLength(1);
+ const argv = calls[0]!;
+ expect(argv.join(" ")).toContain(`search index --vault ${vault} --config /cfg/config.yaml`);
+ const claim = JSON.parse(
+ readFileSync(join(vault, ".open-second-brain", FRESHEN_CLAIM_FILE), "utf8"),
+ ) as { token: string };
+ expect(argv.slice(-2)).toEqual(["--freshen", claim.token]);
+ });
+
+ test("a second reader while the first run holds the claim spawns nothing", () => {
+ maybeFreshenIndex(config, { lastIndexedAt: ago(300), nowMs: NOW, spawn });
+ expect(maybeFreshenIndex(config, { lastIndexedAt: ago(300), nowMs: NOW, spawn })).toBe(
+ FRESHEN_SKIP.claimed,
+ );
+ expect(calls).toHaveLength(1);
+ });
+
+ test("a fresh index spawns nothing", () => {
+ expect(maybeFreshenIndex(config, { lastIndexedAt: ago(5), nowMs: NOW, spawn })).toBe(
+ FRESHEN_SKIP.fresh,
+ );
+ expect(calls).toHaveLength(0);
+ });
+
+ test("a read-only open never freshens a vault it does not own", () => {
+ expect(
+ maybeFreshenIndex(config, { lastIndexedAt: ago(3600), nowMs: NOW, spawn, readOnly: true }),
+ ).toBe(FRESHEN_SKIP.readOnly);
+ expect(calls).toHaveLength(0);
+ });
+
+ test("a config without freshen settings is off", () => {
+ const { freshen: _drop, ...rest } = config as unknown as Record<string, unknown>;
+ expect(
+ maybeFreshenIndex(rest as unknown as ResolvedSearchConfig, {
+ lastIndexedAt: ago(3600),
+ nowMs: NOW,
+ spawn,
+ }),
+ ).toBe(FRESHEN_SKIP.off);
+ });
+
+ test("an indexer already holding the writer lock is left alone", () => {
+ const release = acquireWriterLockSync(config.dbPath);
+ try {
+ expect(maybeFreshenIndex(config, { lastIndexedAt: ago(300), nowMs: NOW, spawn })).toBe(
+ FRESHEN_SKIP.writerLock,
+ );
+ expect(calls).toHaveLength(0);
+ } finally {
+ release();
+ }
+ });
+
+ test("an active backoff from a failed run spawns nothing", () => {
+ writeFreshenState(join(vault, ".open-second-brain"), {
+ failures: 1,
+ backoffUntil: new Date(NOW + 30_000).toISOString(),
+ lastOutcome: "failed",
+ lastRunAt: ago(10),
+ lastDurationMs: 100,
+ lastError: "boom",
+ lastChanged: null,
+ });
+ expect(maybeFreshenIndex(config, { lastIndexedAt: ago(300), nowMs: NOW, spawn })).toBe(
+ FRESHEN_SKIP.backoff,
+ );
+ });
+
+ test.skipIf(CHMOD_CANNOT_DENY)(
+ "an index directory that refuses the claim is reported as unwritable, not claimed",
+ () => {
+ const dir = join(vault, ".open-second-brain");
+ chmodSync(dir, 0o555);
+ try {
+ expect(maybeFreshenIndex(config, { lastIndexedAt: ago(300), nowMs: NOW, spawn })).toBe(
+ FRESHEN_SKIP.unwritable,
+ );
+ expect(calls).toHaveLength(0);
+ } finally {
+ chmodSync(dir, 0o755);
+ }
+ },
+ );
+
+ test("a fresh index is decided without the state file, so a backoff there is not consulted", () => {
+ writeFreshenState(join(vault, ".open-second-brain"), {
+ failures: 1,
+ backoffUntil: new Date(NOW + 30_000).toISOString(),
+ lastOutcome: "failed",
+ lastRunAt: ago(10),
+ lastDurationMs: 100,
+ lastError: "boom",
+ lastChanged: null,
+ });
+ expect(maybeFreshenIndex(config, { lastIndexedAt: ago(5), nowMs: NOW, spawn })).toBe(
+ FRESHEN_SKIP.fresh,
+ );
+ });
+
+ test("a spawner that throws is reported as a skip and releases the claim", () => {
+ const decision = maybeFreshenIndex(config, {
+ lastIndexedAt: ago(300),
+ nowMs: NOW,
+ spawn: () => {
+ throw new Error("no fork");
+ },
+ });
+ expect(decision).toBe(FRESHEN_SKIP.spawnFailed);
+ expect(existsSync(join(vault, ".open-second-brain", FRESHEN_CLAIM_FILE))).toBe(false);
+ });
+});
+
+describe("claim races", () => {
+ let dir: string;
+ beforeEach(() => {
+ dir = mkdtempSync(join(tmpdir(), "osb-freshen-race-"));
+ });
+ afterEach(() => rmSync(dir, { recursive: true, force: true }));
+
+ test("a zero-byte claim younger than the grace is a claim being written, not a dead one", () => {
+ writeFileSync(join(dir, FRESHEN_CLAIM_FILE), "");
+ expect(claimFreshen(dir, Date.now())).toBeNull();
+ });
+
+ test("a zero-byte claim older than the grace is taken over", () => {
+ const path = join(dir, FRESHEN_CLAIM_FILE);
+ writeFileSync(path, "");
+ const old = new Date(Date.now() - 60_000);
+ utimesSync(path, old, old);
+ expect(claimFreshen(dir, Date.now())).toBeString();
+ });
+
+ test("a stale claim taken over by one reader is not taken over again by the next", () => {
+ writeFileSync(
+ join(dir, FRESHEN_CLAIM_FILE),
+ JSON.stringify({ token: "dead", at: Date.now() - 601_000 }),
+ );
+ expect(claimFreshen(dir, Date.now())).toBeString();
+ expect(claimFreshen(dir, Date.now())).toBeNull();
+ });
+
+ test("of eight processes racing on a stale claim exactly one wins", async () => {
+ writeFileSync(
+ join(dir, FRESHEN_CLAIM_FILE),
+ JSON.stringify({ token: "dead", at: Date.now() - 601_000 }),
+ );
+ const module = join(import.meta.dir, "..", "..", "..", "src", "core", "search", "freshen.ts");
+ const script = `const { claimFreshen } = await import(${JSON.stringify(module)});
+ const go = Number(process.argv[1]);
+ while (Date.now() < go) {}
+ console.log(claimFreshen(process.argv[2], Date.now()) === null ? "lost" : "won");`;
+ const go = String(Date.now() + 1500);
+ const procs = Array.from({ length: 8 }, () =>
+ Bun.spawn(["bun", "-e", script, go, dir], { stdout: "pipe", stderr: "pipe" }),
+ );
+ const outs = await Promise.all(procs.map((p) => new Response(p.stdout).text()));
+ expect(outs.map((o) => o.trim()).filter((o) => o === "won")).toHaveLength(1);
+ });
+});
+
+test("the spawned run is told where its state lives", () => {
+ const vault = mkdtempSync(join(tmpdir(), "osb-freshen-statedir-"));
+ try {
+ mkdirSync(join(vault, ".open-second-brain"), { recursive: true });
+ const calls: string[][] = [];
+ const config = {
+ vault,
+ dbPath: join(vault, ".open-second-brain", "brain.sqlite"),
+ freshen: { intervalSeconds: 60, embeddings: false, configPath: null },
+ } as unknown as ResolvedSearchConfig;
+ maybeFreshenIndex(config, {
+ lastIndexedAt: new Date(Date.now() - 300_000).toISOString(),
+ spawn: (argv) => calls.push(argv),
+ });
+ const argv = calls[0]!;
+ const at = argv.indexOf("--freshen-state");
+ expect(argv[at + 1]).toBe(join(vault, ".open-second-brain"));
+ } finally {
+ rmSync(vault, { recursive: true, force: true });
+ }
+});
diff --git a/tests/core/search/indexer-noop.test.ts b/tests/core/search/indexer-noop.test.ts
new file mode 100644
index 00000000..dfb2183e
--- /dev/null
+++ b/tests/core/search/indexer-noop.test.ts
@@ -0,0 +1,128 @@
+/**
+ * A run that adds, updates and deletes no document skips the link and
+ * alias resolution passes: they are a pure function of the documents and
+ * their links, which did not change, and rewriting every link row on every
+ * run is what made frequent background runs write to disk for nothing.
+ * The relation-constraint pass still runs (it reads, and writes only
+ * changed flags), so a schema-pack edit alone still takes effect.
+ */
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import { Database } from "bun:sqlite";
+
+import { indexVault } from "../../../src/core/search/indexer.ts";
+import {
+ LAST_INDEXED_AT_STATE_KEY,
+ LINK_RESOLUTION_PENDING_STATE_KEY,
+} from "../../../src/core/search/store/state.ts";
+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";
+
+let vault: string;
+let dbPath: string;
+let cleanup: () => void;
+
+beforeEach(() => {
+ ({ vault, dbPath, cleanup } = createTempVault("indexer-noop"));
+ // `[[grey-heron]]` resolves only through the frontmatter alias, in the
+ // alias pass that resets and re-sets it each run - the link a skip must
+ // not lose. `[[notes/pond.md]]` resolves by exact path.
+ writeMd(vault, "notes/birds/heron.md", "---\naliases: [grey-heron]\n---\n# Heron\n\nA wader.\n");
+ writeMd(
+ vault,
+ "notes/river.md",
+ "# River\n\nHome of the [[grey-heron]] and [[notes/pond.md]].\n",
+ );
+ writeMd(vault, "notes/pond.md", "# Pond\n\nStill water.\n");
+});
+
+afterEach(() => cleanup());
+
+function linkTargets(): Array<{ target_path: string; resolved: number }> {
+ const db = new Database(dbPath, { readonly: true });
+ try {
+ return db
+ .query<{ target_path: string; resolved: number }, []>(
+ "SELECT target_path, target_document_id IS NOT NULL AS resolved FROM links " +
+ "WHERE target_path IS NOT NULL ORDER BY target_path",
+ )
+ .all();
+ } finally {
+ db.close();
+ }
+}
+
+function lastIndexedAt(): string | null {
+ const db = new Database(dbPath, { readonly: true });
+ try {
+ return (
+ db
+ .query<{ value: string }, [string]>("SELECT value FROM index_state WHERE key = ?")
+ .get(LAST_INDEXED_AT_STATE_KEY)?.value ?? null
+ );
+ } finally {
+ db.close();
+ }
+}
+
+test("a run with no document change skips link resolution and keeps every link resolved", async () => {
+ const config = makeConfig({ vault, dbPath });
+ const first = await indexVault(config);
+ expect(first.linkResolutionSkipped).toBe(false);
+ const before = linkTargets();
+ expect(before.every((l) => l.resolved === 1)).toBe(true);
+ const stampBefore = lastIndexedAt();
+
+ await Bun.sleep(5);
+ const second = await indexVault(config);
+ expect(second.linkResolutionSkipped).toBe(true);
+ expect(linkTargets()).toEqual(before);
+ expect(lastIndexedAt()).not.toBe(stampBefore);
+});
+
+test("a changed note runs link resolution again and resolves a new link", async () => {
+ const config = makeConfig({ vault, dbPath });
+ await indexVault(config);
+ writeMd(vault, "notes/lake.md", "# Lake\n\nWhere the [[grey-heron]] winters.\n");
+ const stats = await indexVault(config);
+ expect(stats.linkResolutionSkipped).toBe(false);
+ expect(
+ linkTargets()
+ .filter((l) => l.target_path === "grey-heron")
+ .every((l) => l.resolved === 1),
+ ).toBe(true);
+});
+
+test("a deleted target is noticed: its links become unresolved", async () => {
+ const config = makeConfig({ vault, dbPath });
+ await indexVault(config);
+ const { rmSync } = await import("node:fs");
+ rmSync(`${vault}/notes/pond.md`);
+ const stats = await indexVault(config);
+ expect(stats.linkResolutionSkipped).toBe(false);
+ expect(linkTargets().find((l) => l.target_path === "notes/pond.md")?.resolved).toBe(0);
+});
+
+test("a forced run never skips", async () => {
+ const config = makeConfig({ vault, dbPath });
+ await indexVault(config);
+ expect((await indexVault(config, { force: true })).linkResolutionSkipped).toBe(false);
+});
+
+test("a run killed after writing documents but before resolving links is finished by the next run", async () => {
+ const config = makeConfig({ vault, dbPath });
+ await indexVault(config);
+ // What a run killed mid-way leaves: documents committed, the alias
+ // link not resolved yet, and the pending marker it set before writing.
+ const db = new Database(dbPath);
+ db.run("UPDATE links SET target_document_id = NULL WHERE target_path = 'grey-heron'");
+ db.query("INSERT OR REPLACE INTO index_state(key, value, updated_at) VALUES (?, '1', ?)").run(
+ LINK_RESOLUTION_PENDING_STATE_KEY,
+ new Date().toISOString(),
+ );
+ db.close();
+
+ const stats = await indexVault(config);
+ expect(stats.linkResolutionSkipped).toBe(false);
+ expect(linkTargets().find((l) => l.target_path === "grey-heron")?.resolved).toBe(1);
+ // Resolved now, so the run after that may skip again.
+ expect((await indexVault(config)).linkResolutionSkipped).toBe(true);
+});
diff --git a/tests/core/search/vector-carry-over.test.ts b/tests/core/search/vector-carry-over.test.ts
index 1fd1074f..b8a6c198 100644
--- a/tests/core/search/vector-carry-over.test.ts
+++ b/tests/core/search/vector-carry-over.test.ts
@@ -213,18 +213,39 @@ describe("replaceDocumentChunks carry-over", () => {
const { store, docId, oldIds } = await seededStore();
const before = embeddingTimes(dbPath, oldIds[0]!);
await Bun.sleep(5);
+ // "h0" moves from position 0 to 1, so it is a new row that carries
+ // the old vector rather than a chunk kept in place.
const replaced = store.replaceDocumentChunks(docId, [
- chunkInput(0, "h0"),
- chunkInput(1, "h1-edited"),
+ chunkInput(0, "h1-edited"),
+ chunkInput(1, "h0"),
chunkInput(2, "h2"),
]);
await store.close();
- const after = embeddingTimes(dbPath, replaced.chunkIds[0]!);
+ const after = embeddingTimes(dbPath, replaced.chunkIds[1]!);
expect(after.createdAt).toBe(before.createdAt);
expect(after.updatedAt).not.toBe(before.updatedAt);
},
);
+ test.skipIf(!VEC_LOADABLE)(
+ "a chunk kept in place keeps its row, its vector and both timestamps",
+ async () => {
+ const { store, docId, oldIds } = await seededStore();
+ const before = embeddingTimes(dbPath, oldIds[2]!);
+ await Bun.sleep(5);
+ const replaced = store.replaceDocumentChunks(docId, [
+ chunkInput(0, "h1-edited"),
+ chunkInput(1, "h0"),
+ chunkInput(2, "h2"),
+ ]);
+ expect(replaced.chunkIds[2]).toBe(oldIds[2]!);
+ expect(replaced.keptChunkIds.has(oldIds[2]!)).toBe(true);
+ expect(store.getEmbeddingHash(oldIds[2]!)).toBe("emb-2");
+ await store.close();
+ expect(embeddingTimes(dbPath, oldIds[2]!)).toEqual(before);
+ },
+ );
+
test.skipIf(!VEC_LOADABLE)(
"the plain replaceChunks form carries too and still returns the ids",
async () => {
@@ -267,6 +288,35 @@ describe("replaceDocumentChunks carry-over", () => {
},
);
+ for (const [what, key, value] of [
+ ["model", EMBEDDING_MODEL_STATE_KEY, "another-model"],
+ ["dimension", EMBEDDING_DIMENSION_STATE_KEY, String(STORE_DIMENSION * 2)],
+ ] as const) {
+ test.skipIf(!VEC_LOADABLE)(
+ `a chunk kept in place whose vector has another ${what} loses it and is pending again`,
+ async () => {
+ const { store, docId, oldIds } = await seededStore();
+ store.setState(key, value);
+ const replaced = store.replaceDocumentChunks(docId, [
+ chunkInput(0, "h0"),
+ chunkInput(1, "h1-edited"),
+ chunkInput(2, "h2"),
+ ]);
+ const kept = [oldIds[0]!, oldIds[2]!];
+ expect([replaced.chunkIds[0], replaced.chunkIds[2]]).toEqual(kept);
+ for (const id of kept) {
+ expect(replaced.keptChunkIds.has(id)).toBe(true);
+ expect(store.getEmbeddingHash(id)).toBeNull();
+ }
+ expect(replaced.embeddingsReused).toBe(0);
+ const pending = store.findChunksWithoutEmbeddings().map((p) => p.chunkId);
+ for (const id of kept) expect(pending).toContain(id);
+ await store.close();
+ expect(vecRowCount(dbPath)).toBe(0);
+ },
+ );
+ }
+
test.skipIf(!VEC_LOADABLE)(
"nothing is carried when the embedding identity is unrecorded",
async () => {
diff --git a/tests/core/state/surfaces.test.ts b/tests/core/state/surfaces.test.ts
index f20f1b83..c52b93fa 100644
--- a/tests/core/state/surfaces.test.ts
+++ b/tests/core/state/surfaces.test.ts
@@ -30,6 +30,11 @@
import { afterEach, describe, expect, test } from "bun:test";
import { selfHealUpgradeMarkerPath } from "../../../src/core/maintenance/self-heal-upgrade-state.ts";
+import {
+ EMPTY_FRESHEN_STATE,
+ FRESHEN_STATE_FILE,
+ writeFreshenState,
+} from "../../../src/core/search/freshen.ts";
import { chmodSync, existsSync, mkdirSync, statSync, symlinkSync, writeFileSync } from "node:fs";
import { dirname, join, resolve, sep } from "node:path";
@@ -262,6 +267,9 @@ const RESOLVER_BINDINGS: ReadonlyArray<readonly [StateSurfaceId, (vault: string)
["hook_audit", (v) => hookAuditDir(v)],
["hook_session_state", (v) => dirname(hookStateFilePath(v, null))],
["self_heal_upgrade_marker", (v) => selfHealUpgradeMarkerPath(v)],
+ // `freshen.ts` takes the directory its caller hands it, always
+ // `dirname(config.dbPath)`, so the row is bound to the index resolver.
+ ["freshen_state", (v) => join(dirname(resolveIndexPath(v, null)), FRESHEN_STATE_FILE)],
["aider_context_artifact", (v) => resolveAiderSidecarPath({ vault: v } as InstallEnv, {})],
// --- Under `<vault>/Brain/` --------------------------------------------
["brain_log", (v) => brainDirs(v).log],
@@ -454,6 +462,24 @@ describe("each row is bound to the resolver that owns it", () => {
expect(row("search_session_focus").derive(vault, elsewhere)).toBe(
dirname(sessionFocusPath({ vault, dbPath: elsewhere } as ResolvedSearchConfig, "scope")),
);
+ expect(row("freshen_state").derive(vault, elsewhere)).toBe(
+ join(dirname(elsewhere), FRESHEN_STATE_FILE),
+ );
+ });
+
+ test("a relocated index moves the freshen state to where the run writes it", () => {
+ const home = tempVault();
+ const elsewhere = join(tempVault(), "brain.sqlite");
+ writeFreshenState(dirname(elsewhere), EMPTY_FRESHEN_STATE);
+ const inventory = inventoryStateSurfaces({
+ vault: home,
+ env: { OPEN_SECOND_BRAIN_SEARCH_DB: elsewhere },
+ config: EMPTY_CONFIG,
+ });
+ const state = inventory.surfaces.find((s) => s.id === "freshen_state")!;
+ expect(state.path).toBe(join(dirname(elsewhere), FRESHEN_STATE_FILE));
+ expect(state.origin).toBe(CONFIG_ORIGIN.env);
+ expect(existsSync(state.path)).toBe(true);
});
for (const binding of DRIVEN_BINDINGS) {
diff --git a/tests/e2e/index-freshen.test.ts b/tests/e2e/index-freshen.test.ts
new file mode 100644
index 00000000..b6045776
--- /dev/null
+++ b/tests/e2e/index-freshen.test.ts
@@ -0,0 +1,61 @@
+/**
+ * Freshen on read end to end, with a real detached child: a note written
+ * after the last index run is not found by the first search, which starts
+ * a background `o2b search index --freshen`; once that run has recorded
+ * its outcome, the next search finds the note. No test seam is used.
+ */
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { FRESHEN_CLAIM_FILE, readFreshenState } from "../../src/core/search/freshen.ts";
+import { runCli } from "../helpers/run-cli.ts";
+
+let tmp: string;
+let vault: string;
+let configPath: string;
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "o2b-freshen-e2e-"));
+ vault = join(tmp, "vault");
+ mkdirSync(join(vault, "notes"), { recursive: true });
+ configPath = join(tmp, "config.yaml");
+ writeFileSync(configPath, `vault: ${vault}\n`);
+ writeFileSync(join(vault, "notes", "old.md"), "# Old\n\nan older note about kestrels.\n");
+});
+
+afterEach(() => rmSync(tmp, { recursive: true, force: true }));
+
+const env = (interval: string) => ({
+ OPEN_SECOND_BRAIN_CONFIG: configPath,
+ OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S: interval,
+});
+
+test(
+ "a stale read starts a background run and the next read finds the new note",
+ async () => {
+ expect((await runCli(["search", "index"], { env: env("0") })).returncode).toBe(0);
+ await Bun.sleep(1_100);
+ writeFileSync(join(vault, "notes", "new.md"), "# New\n\na fresh note about wagtails.\n");
+
+ const first = await runCli(["search", "query", "wagtails", "--json"], { env: env("1") });
+ expect(first.returncode).toBe(0);
+ expect(JSON.parse(first.stdout).results ?? []).toHaveLength(0);
+
+ const derived = join(vault, ".open-second-brain");
+ const deadline = Date.now() + 30_000;
+ while (readFreshenState(derived).lastOutcome === null && Date.now() < deadline) {
+ await Bun.sleep(200);
+ }
+ const state = readFreshenState(derived);
+ expect(state.lastOutcome).toBe("completed");
+ expect(state.lastChanged).toBe(1);
+ expect(existsSync(join(derived, FRESHEN_CLAIM_FILE))).toBe(false);
+
+ const second = await runCli(["search", "query", "wagtails", "--json"], { env: env("0") });
+ const paths = (JSON.parse(second.stdout).results as Array<{ path: string }>).map((r) => r.path);
+ expect(paths).toContain("notes/new.md");
+ },
+ { timeout: 60_000 },
+);
diff --git a/tests/helpers/platform-freshen.test.ts b/tests/helpers/platform-freshen.test.ts
new file mode 100644
index 00000000..e9a282dd
--- /dev/null
+++ b/tests/helpers/platform-freshen.test.ts
@@ -0,0 +1,10 @@
+import { expect, test } from "bun:test";
+
+import { homeEnv } from "./platform.ts";
+
+// A hook or MCP server spawned with the minimal environment would
+// otherwise fall back to freshening every 60 s and start real background
+// indexers into temp vaults the test is about to delete.
+test("the minimal child environment keeps freshen on read off", () => {
+ expect(homeEnv("/tmp/h")["OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S"]).toBe("0");
+});
diff --git a/tests/helpers/platform.ts b/tests/helpers/platform.ts
index a0328247..7efd6b4e 100644
--- a/tests/helpers/platform.ts
+++ b/tests/helpers/platform.ts
@@ -43,8 +43,13 @@ export const CHMOD_CANNOT_DENY = RUNNING_AS_ROOT || IS_WINDOWS;
* three Windows variables under `home`, the layout Windows itself uses.
*/
export function homeEnv(home: string): Record<string, string> {
- if (!IS_WINDOWS) return { HOME: home };
+ // The preload pins freshen on read off for this process; a child spawned
+ // with a minimal environment needs it pinned too, or a search it runs
+ // starts real background indexers into a vault the test then deletes.
+ const freshenOff = { OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S: "0" };
+ if (!IS_WINDOWS) return { HOME: home, ...freshenOff };
return {
+ ...freshenOff,
HOME: home,
USERPROFILE: home,
LOCALAPPDATA: join(home, "AppData", "Local"),
diff --git a/tests/mcp/doctor-schema-reach.test.ts b/tests/mcp/doctor-schema-reach.test.ts
index 67de99ba..a9ca1de4 100644
--- a/tests/mcp/doctor-schema-reach.test.ts
+++ b/tests/mcp/doctor-schema-reach.test.ts
@@ -85,6 +85,7 @@ async function answer(
describe("a malformed private preference answers as an absent one at remote reach", () => {
for (const [tool, args] of [
["brain_doctor", {}],
+ ["brain_doctor", { repair: true }],
["schema_inspect", { view: "lint" }],
["schema_inspect", { view: "orphans" }],
["schema_inspect", { view: "stats" }],
@@ -102,6 +103,8 @@ describe("a malformed private preference answers as an absent one at remote reac
const doctor = await answer(a, "brain_doctor", {}, TRANSPORT_REACH.local);
expect(doctor).toContain(PREF);
expect(doctor).toContain("tier-drift");
+ const repair = await answer(a, "brain_doctor", { repair: true }, TRANSPORT_REACH.local);
+ expect(repair).toContain("tier-drift");
expect(await answer(a, "schema_inspect", { view: "orphans" }, TRANSPORT_REACH.local)).toContain(
PREF,
);
diff --git a/tests/setup.ts b/tests/setup.ts
index 737e83da..13cf3d3f 100644
--- a/tests/setup.ts
+++ b/tests/setup.ts
@@ -56,6 +56,14 @@ if (process.env["O2B_DEVICE_ID"] === undefined) {
process.env["O2B_DEVICE_ID"] = "";
}
+// Freshen-on-read starts a detached `o2b search index` whenever a search
+// meets an index older than the interval. A test suite must never leave
+// background indexers behind, so the default is off here; the tests of the
+// feature own the variable while they run.
+if (process.env["OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S"] === undefined) {
+ process.env["OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S"] = "0";
+}
+
if (!process.env["OPEN_SECOND_BRAIN_CONFIG"]) {
const root = mkdtempSync(join(tmpdir(), "osb-test-default-"));
// A preload's afterAll runs once, after the last file: the throwaway
diff --git a/uv.lock b/uv.lock
index 219a29e9..f6ef08d7 100644
--- a/uv.lock
+++ b/uv.lock
@@ -4,5 +4,5 @@ requires-python = ">=3.11"
[[package]]
name = "open-second-brain"
-version = "1.75.0"
+version = "1.76.0"
source = { editable = "." }