Local-first π§ memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly π΄ dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
feat: freshen the search index on read, rewriting only changed chunks (v1.76.0) (#239)
Serg Eroshe646a49df5e61
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.jsonindex be2dad1a..5d7ef4ce 100644--- a/.claude-plugin/plugin.json+++ b/.claude-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.75.0",+ "version": "1.76.0","description": "Plugin-first second brain package for AI agents and humans.","author": {"name": "Open Second Brain contributors"diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.jsonindex 95b8fb6c..062e7ba9 100644--- a/.codex-plugin/plugin.json+++ b/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.75.0",+ "version": "1.76.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 83f4e7f2..97220ec5 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,6 +5,23 @@ All notable changes to this project will be documented in this file.The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).+## [1.76.0] - 2026-10-08++Open Second Brain now keeps its search index current without a scheduler: a search or a session start that finds the index more than a minute old answers from it as it is and starts one low-priority background incremental run, so the next read sees the current vault. Nothing stays resident and nothing runs while no agent works. Two indexer changes make those frequent runs cheap: a run that changed no document writes almost nothing, and a changed document rewrites only the chunks that changed, so an edit to a 1 MB daily log costs about 0.2 MB of writes instead of about 9 MB.++### Added++- **Freshen on read.** `search()` (every reading MCP tool, the recall-inject hook, `o2b search query`) and `ensureVaultCurrent` (SessionStart and MCP start) start a detached `o2b search index --freshen <token>` when `last_indexed_at` is older than `search_freshen_interval_s` (default `60`, `0` turns it off, env `OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S`). The reader never waits for the run. An exclusive claim next to the index (`freshen.claim`) keeps concurrent readers from starting a run each; a dead run's claim is taken over after ten minutes by exactly one reader, under an exclusive takeover lock. A held writer lock or an index directory that refuses the claim (read-only mount, no permission) skips the spawn, a run that still meets another writer ends as a skip rather than a failure, and the child is told where its state lives (`--freshen-state`), so even a failure before it resolves its config is recorded and releases the claim; and a read-only open (cross-vault, recall sources) never starts one. The child lowers its CPU priority and runs under `ionice -c3` on Linux or `taskpolicy -b` on macOS when the tool exists. It is keyword-only unless `search_freshen_embeddings` is on.+- **Backoff and visibility.** The child records its outcome in the per-device `freshen-state.json` beside the index (a new state surface that follows `OPEN_SECOND_BRAIN_SEARCH_DB` / `search_db_path`, so `o2b state status` and `o2b state migrate` report and leave it where a relocated index keeps it); a failed run backs off one minute, doubling to an hour. `o2b search status` prints `freshen`, `index_age`, `last_freshen` and an active `freshen_backoff: until <time>` (`freshen` object under `--json`); `o2b doctor` warns with the new code `freshen-failing` after three failures in a row; runs that changed documents or failed append one row to the new `index_freshen` metrics surface.+- **Trail code `index-stale`.** A search over an index more than ten minutes old names it on the retrieval trail with `detail.ageSeconds`. The code is added at read time, also to a query-cache hit, and never stored in the cache, so a cached answer reports the index age it is served at and an operator who reindexes on a schedule keeps the cache.++### Changed++- **A no-change index run skips link and alias resolution.** Both are a pure function of the documents and their links; rewriting every link row on every run was the only write a no-change run made. The relation-constraint and tier passes still run, so a schema-pack edit alone still takes effect. `IndexStats.linkResolutionSkipped` reports it. A run killed between its document writes and link resolution leaves a pending marker (`link_resolution_pending`), so the next run resolves even if it changes nothing itself.+- **Re-indexing a changed document keeps its unchanged chunks in place.** A chunk at the same position with the same content hash, lines, token count, heading and stored full-text form keeps its row, id, full-text entry, entities and vector; only changed chunks are deleted and inserted, and moved ones still carry their vectors. Measured on a 5,812-note vault: one edit to a 1 MB daily log went from 9.3 MB written and 2.1 s to 0.2 MB and 0.17 s; 200 edited notes from 57 MB and 1.8 s to 28 MB and 0.58 s.+- **The doctor's repair plan sees the index.** `o2b brain doctor --repair` and `brain_doctor` with `repair` now hand the doctor the resolved index path, so `freshen-failing` and `tier-drift` are listed as not auto-repairable, as the report already listed them. `tier-drift` is counted over the pages the caller may read, so below local reach a drifted page the caller cannot read moves the count no more than an absent one, in the report and the repair plan alike.+- **Docs:** `docs/cli-reference.md` gains "Freshen on read", the two keys, the `--freshen` flag, the status fields and the `index-stale` row; `docs/how-it-works.md` "Keeping the index current"; `docs/architecture.md` states that background work starts only from agent activity; `docs/hermes-cron.md` and `docs/updating.md` say a scheduled reindex is now only needed for embeddings; `docs/metrics.md` lists `index_freshen`. The README names this release.+## [1.75.0] - 2026-10-07Open Second Brain now reaches the two moments the session-start injection never could: the delegated subagent that performs the writes, and the end of the turn. A carrier hands the operator's context to every delegated subagent once, on the SubagentStart event, before the subagent's first prompt, with the write-shaped PostToolUse registration kept as the fallback for runtimes without that event; an opt-in Stop-hook digest surfaces pending Brain hygiene findings as one line, once per change, on the runtimes whose Stop channel is non-blocking and stays silent elsewhere; and an injection body over its budget degrades its sections to headline tiers - the top three ranked bullets each - least important section first, until the body fits, before it drops a whole section.@@ -8292,6 +8309,7 @@ plugin config (vault field)`, and exits with a clear- Sandbox vault and plugin manifest fixtures for tests.- GitHub release workflow for tag-based and manually dispatched releases.+[1.76.0]: https://github.com/itechmeat/open-second-brain/compare/v1.75.0...v1.76.0[1.75.0]: https://github.com/itechmeat/open-second-brain/compare/v1.74.1...v1.75.0[1.74.1]: https://github.com/itechmeat/open-second-brain/compare/v1.74.0...v1.74.1[1.74.0]: https://github.com/itechmeat/open-second-brain/compare/v1.73.1...v1.74.0diff --git a/README.md b/README.mdindex 19deaa56..b57a6ca8 100644--- a/README.md+++ b/README.md@@ -113,7 +113,7 @@ The full router with readiness criteria is [`install.md`](install.md); native Wi## What is new-1.73.0 makes the query embed honest and upgrades safe. Every query embed, from the search lane behind the MCP tools and hooks to the `brain_context_pack` semantic belief order, passes one gate: under a positive `embedding_cost_gate_usd` an unpriced model is refused for any caller that is not local (a hybrid search falls back to keyword-only and says so by trail code), and a query longer than the model's input window is cut to it and disclosed. `embedding_input_window_tokens` declares the window of a model the curated table does not list, and `embedding_extra_body` sends extra request fields to an OpenAI-compatible endpoint. `brain_recall_feedback` and the benchmark, eval and tune reports name the degradations their searches met, and `brain_tune` no longer saves a winner measured without the semantic lane. `o2b brain upgrade` and the automatic upgrade worker apply exactly the plan they computed and refuse a managed file edited in between instead of overwriting it. Every release is described in the [CHANGELOG](CHANGELOG.md).+1.76.0 keeps the search index current without a scheduler. A search or a session start that finds the index more than a minute old answers from it as it is and starts one low-priority background run, so the next read sees the current vault; nothing stays resident, and nothing runs while no agent works. A changed note now rewrites only the parts of the index that changed, so appending to a large daily log costs a fraction of what it did, and `o2b search status` shows how fresh the index is. Every release is described in the [CHANGELOG](CHANGELOG.md).## Documentationdiff --git a/docs/architecture.md b/docs/architecture.mdindex 9de5bfa5..a31befef 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -265,7 +265,7 @@ writes it, and they act only on a path the caller named.Three architectural invariants:-- **Filesystem-first.** No database, no daemon. Every artifact is plain Markdown with YAML frontmatter; backup is `cp -r` or `tar`.+- **Filesystem-first.** No database, no daemon. Every artifact is plain Markdown with YAML frontmatter; backup is `cp -r` or `tar`. Background work is started only by agent activity and exits when done: the search index is refreshed by a short detached run a stale read starts (freshen on read), never by a resident process.- **Deterministic core.** The `dream` algorithm is a pure function of inputs (signals, preferences, log, configuration, current time). No LLM calls inside the core. Semantic merging, if needed, is delegated to external agents via the same CLI / MCP surface. External judgment (the bench judge, the hygiene conflict resolver) goes through one sanctioned fail-open boundary: the operator-configured command bridge (`src/core/reliability/command-bridge.ts`).- **Pre-run snapshot + atomic per-file writes.** Each `dream` run takes a `.snapshots/<run_id>.tar.zst` before any state change (streamed through a staging file, and published under its final name only once complete); per-file writes go through `fs-atomic` (temp + rename). Combined with retention of the most-recent N snapshots, this gives reversible, audit-friendly mutation.diff --git a/docs/brainstorm/index-freshness/design.md b/docs/brainstorm/index-freshness/design.mdnew file mode 100644index 00000000..77c6a5bd--- /dev/null+++ b/docs/brainstorm/index-freshness/design.md@@ -0,0 +1,101 @@+# Index freshness on read - the search index keeps itself current while agents work++**Status:** approved design, not implemented+**Audience:** implementation++## Problem statement++Nothing keeps the search index current. No write path updates it: the MCP writers, ingest, dream and the hooks all write Markdown and leave `brain.sqlite` as it was. The only automatic rebuild (`ensureVaultCurrent`) fires on a missing index or an old schema or chunker version, never on changed content. Freshness therefore depends on the operator running `o2b search index`, keeping `o2b search watch` in the foreground, or installing one of the printed cron recipes. On the reference machine the index went six days without a run: 223 new or changed notes were invisible to every search, and nothing reported that the index was behind.++## Measurements (reference vault, 5,811 Markdown files, 16 cores, NVMe)++| Run | Wall | CPU | Peak RSS | Disk write |+|---|---|---|---|---|+| no change | 0.30 s | 0.35 s | ~110 MB | ~0.6 MB |+| one note added or deleted | 0.30 s | 0.35 s | ~110 MB | ~1 MB |+| six days of drift (215 added, 8 updated) | 8.4 s | 5.1 s (60% of one core) | ~160 MB | ~1 GB |++The wall times include a bash start. An incremental run is cheap enough to run often. The catch-up run is cheap in CPU but writes about 1 GB, from two sources: the indexer works in WAL and consolidates to `journal_mode=DELETE` on close (a full checkpoint), and every run, an empty one included, repeats the whole-index post-passes (link and alias resolution, constraint flags, tier guard).++## Goals++1. A note written by an agent, or synced in while an agent works, becomes findable without anyone running a command.+2. Indexing never blocks the caller and does not get in the way of other work on the machine.+3. Only agent activity triggers indexing. No resident process, no OS scheduler, no Hermes cron.+4. How far the index is behind is visible to the operator and, when it matters, to the agent.++## Out of scope++- Any daemon, file watcher or timer that runs while no agent works. `o2b search watch` stays as the opt-in foreground watcher.+- Installing scheduler entries. The print-never-install rule for cron recipes stands.+- Embedding computation in the background by default (it costs money; see Configuration).+- Freshness of other derived state (clusters, bridges, dream). The maintenance lane keeps owning those.++## Chosen approach: refresh on read, plus catch-up at session start++A reader that finds the index older than a threshold answers from the index it has and starts one detached incremental index run in the background; the next read sees the result. `ensureVaultCurrent` (SessionStart hook and MCP server start) applies the same check, so a session starts from a caught-up index. Freshness is paid for where it is needed, at read time, and one seam covers agent writes and external changes (Syncthing, Obsidian) alike.++Rejected alternatives:++- **Catch-up at session start only.** Smallest change, but a session can live for days, and notes written during it stay invisible until the next one.+- **Index after each write.** The agent finds what it wrote, but changes made outside the agent stay invisible, and the trigger has to be threaded through every write path.++## Design++### 1. Trigger and entry points++- New module `src/core/search/freshen.ts` exporting `maybeFreshenIndex(...)`. It decides and, when due, spawns; it never throws and never awaits the run.+- Call sites:+ 1. `search()` in `src/core/search/search.ts`, right after the store is opened. Every reading surface goes through it (`brain_search`, the recall tools, `brain_context_pack`, the recall-inject hook, the CLI).+ 2. `ensureVaultCurrent` in `src/core/maintenance/ensure-current.ts`, so session start and MCP start catch up before the first read.+- A read-only open (`opts.selfHeal === false`: cross-vault and recall-source search) never freshens: it would write an index into a vault this device does not own. Same rule as self-heal today.+- Due when `last_indexed_at` in `index_state` is older than `search_freshen_interval_s` (default 60). One read of `index_state`; the vault is not walked by the caller.+- The run: a detached `o2b search index --vault <v> --config <c> --freshen <runId>`, spawned like the self-heal reindex (streams ignored, `unref`, `detached` on Windows, `env: { ...process.env }`). Keyword-only unless `search_freshen_embeddings` is on.++### 2. Single flight, throttling and priority++- **Claim.** Before spawning, the caller makes an exclusive create of `<vault>/.open-second-brain/freshen.claim` (`wx`), as the self-heal upgrade lock does. The winner spawns and hands the claim to the child, which removes it when it ends. A claim older than 10 minutes is treated as abandoned and taken over. A held writer lock (`isWriterLockHeld`) skips the spawn: some indexer (manual, self-heal, maintenance) is already running.+- **Throttle.** A successful run stamps `last_indexed_at`, so for the next interval every session's reads find the index fresh and spawn nothing. Under active use that is at most one run per interval.+- **Backoff.** A failed run writes a failure marker next to the claim. Spawns then wait 1 min, doubling up to 1 h, so a broken vault does not spawn a failing process per query. `doctor` reports the streak.+- **Priority.** The child lowers its CPU priority at start with `os.setPriority` (all platforms). On Linux the command is prefixed with `ionice -c3` and on macOS with `taskpolicy -b` when the tool exists; without it the run proceeds at normal I/O priority.+- **Caller cost.** One `index_state` read, one lock probe and one exclusive create: single-digit milliseconds. The run is always in the background.++### 3. Write cost and visibility++- **A no-change run writes nothing heavy.** When a run added, updated and deleted no document, the whole-index post-passes are skipped and only `last_indexed_at` is stamped, in one small transaction. Frequent runs must not wear the disk.+- **Catch-up write volume, measured then fixed.** Per-phase written bytes are recorded during a run (from `/proc/self/io` on Linux; the measurement is skipped elsewhere). The dominant phase is fixed: candidates are the WAL-plus-checkpoint double write, whole-index link and alias resolution where only changed documents need it, and FTS segment merges. Target: a 200-file catch-up writes at most ~100 MB instead of ~1 GB. If the target needs a larger rework than this feature warrants, the work stops and the numbers go back to the owner before going further.+- **Visibility:**+ - `o2b search status` shows the index age, the outcome of the last background run and any active backoff.+ - `o2b doctor --readiness` gains a freshness line and warns on a failure streak.+ - A search whose index is more than 10 minutes old carries the new trail code `index-stale` with `detail.ageSeconds`; a normal lag of under a minute adds nothing.+ - Every spawn decision and outcome is recorded like `self_heal_reindex` (skipped with reason, spawned, succeeded, failed with reason).++### 4. Configuration++Machine config, each key with an `OPEN_SECOND_BRAIN_*` environment override:++| Key | Meaning | Default |+|---|---|---|+| `search_freshen_interval_s` | how old the index may get before a read refreshes it; `0` turns the feature off | `60` |+| `search_freshen_embeddings` | also compute embeddings in the background run | `false` |++On by default: a feature nobody turns on fixes nothing. The CHANGELOG says that a search may now start a short background process and how to turn it off.++## Compatibility++- `o2b search watch`, `o2b search query --auto-refresh` and the cron recipes are unchanged.+- The docs of `o2b search reindex --cron-template` say it is no longer needed for keyword freshness, only for a periodic embedding pass.+- `docs/architecture.md` "no daemon": a short background process started in response to agent work is not a daemon; nothing stays resident.+- `docs/hermes-cron.md`: a Hermes reindex job becomes optional.++## Testing++1. `maybeFreshenIndex` in isolation, with the spawner injected: index age below and above the interval, `0` disables, own and foreign claim, abandoned claim takeover, writer lock held, backoff active and expired.+2. `search()`: a stale index triggers one spawn, a fresh one none, a read-only open never.+3. Indexer: a no-change run skips the post-passes and stamps only `last_indexed_at`.+4. One end-to-end test with a real child: a new note becomes findable after the background run. Runs on Linux and in the Windows shards.+5. Write volume: a measurement script, not a unit test; before and after numbers go into the PR.++## Release++One PR, minor version (next minor after the current `main`), with the bump, the CHANGELOG entry and its compare link, `docs/cli-reference.md`, `docs/how-it-works.md`, `docs/architecture.md`, `docs/hermes-cron.md`, the README release line, and regenerated Codex mirrors if a skill changes.diff --git a/docs/brainstorm/index-freshness/plan.md b/docs/brainstorm/index-freshness/plan.mdnew file mode 100644index 00000000..81720ae3--- /dev/null+++ b/docs/brainstorm/index-freshness/plan.md@@ -0,0 +1,172 @@+# Index Freshness on Read Implementation Plan++> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.++**Goal:** A read that finds the search index older than a threshold starts one low-priority background incremental index run, so the index stays current while agents work, with no daemon and no scheduler.++**Architecture:** A new `src/core/search/freshen.ts` owns the decision (pure), the per-device state file and claim, and the detached spawn. `search()` and `ensureVaultCurrent` call it. The child is `o2b search index --freshen <runId>`. The indexer skips its whole-index post-passes on a no-change run, and the catch-up write volume is measured per phase and its dominant cause fixed.++**Tech Stack:** Bun 1.4 + TypeScript, `bun:sqlite`, `bun:test`.++**Spec:** `docs/brainstorm/index-freshness/design.md`++## Global Constraints++- No resident process, timer, watcher or scheduler entry; indexing starts only from an agent-driven read or session start.+- A read never waits for indexing; `maybeFreshenIndex` never throws.+- A read-only open (`opts.selfHeal === false`) never freshens.+- Config keys: `search_freshen_interval_s` (default `60`, `0` = off, env `OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S`), `search_freshen_embeddings` (default `false`, env `OPEN_SECOND_BRAIN_SEARCH_FRESHEN_EMBEDDINGS`).+- Claim file `<dirname(dbPath)>/freshen.claim`, abandoned after 10 minutes; state file `<dirname(dbPath)>/freshen-state.json` (per device, never in `Brain/`).+- Backoff after a failed run: 60 s doubling, capped at 3600 s.+- Trail code `index-stale` with `detail.ageSeconds` when the index is older than 600 s.+- Background child: CPU priority lowered with `os.setPriority`; `ionice -c3` (Linux) or `taskpolicy -b` (macOS) prefix when on PATH.+- Catch-up write target: a ~200-file catch-up writes at most ~100 MB; if that needs a larger rework, stop and report numbers.+- Tests never spawn a real child except the one end-to-end test; `tests/setup.ts` pins the interval to `0`.+- Version: next minor after `main`, bump inside the PR; English only; no AI co-author trailers.++## Review Focus++1. Many sessions searching at once on a stale index: exactly one child runs; the others skip without error.+2. A child killed mid-run (reboot, OOM): the claim is taken over after 10 minutes and no failure streak is invented.+3. A vault whose index cannot be written (read-only disk, corrupt file): the child fails, backoff grows, searches keep working on the old index, `doctor` names the streak.+4. Cross-vault (`global`) and recall-source searches: never spawn into a foreign vault.+5. A schema-pack edit with no document change: the no-change fast path must still recompute relation-constraint flags.++Each item has a test in the owning task (Tasks 2, 2, 3, 4, 5 respectively).++---++### Task 1: Freshen configuration++**Files:**+- Modify: `src/core/search/types.ts` (add `ResolvedFreshenConfig`, optional `freshen` on `ResolvedSearchConfig`)+- Modify: `src/core/search/index.ts` (`resolveSearchConfig`)+- Modify: `tests/setup.ts` (pin `OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S=0` unless set)+- Test: `tests/core/search/freshen-config.test.ts`++**Interfaces:**+- Produces: `interface ResolvedFreshenConfig { readonly intervalSeconds: number; readonly embeddings: boolean; readonly configPath: string | null }`; `ResolvedSearchConfig.freshen?: ResolvedFreshenConfig` (absent = off, so hand-built configs in tests stay off).++- [ ] **Step 1: Failing tests** β default resolves `{ intervalSeconds: 60, embeddings: false, configPath }`; env `OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S=0` gives `0`; config key `search_freshen_interval_s: "300"` gives `300`; `"-1"` and `"abc"` throw naming the key; `search_freshen_embeddings: "true"` gives `true`. Tests save and restore the env var because the preload pins it.+- [ ] **Step 2: Run** `bun test tests/core/search/freshen-config.test.ts` β FAIL (`freshen` undefined).+- [ ] **Step 3: Implement** with the existing `parseInteger(envOrConfig(...), 60, "search_freshen_interval_s", { min: 0 })` and `parseBool` helpers; `configPath: opts.configPath ?? null`.+- [ ] **Step 4: Preload** β in `tests/setup.ts`, next to `O2B_DEVICE_ID`: `if (process.env["OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S"] === undefined) process.env["OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S"] = "0";` with a comment that the suite must never spawn background indexers.+- [ ] **Step 5: Run** the new test and `bun run typecheck` β PASS.+- [ ] **Step 6: Commit** `feat(search): freshen interval and embeddings config`.++### Task 2: Decision, claim, state and spawn (`freshen.ts`)++**Files:**+- Create: `src/core/search/freshen.ts`+- Test: `tests/core/search/freshen.test.ts`++**Interfaces:**+- Consumes: `ResolvedFreshenConfig` (Task 1), `isWriterLockHeld(dbPath)` from `store/writer-lock.ts`.+- Produces:+ - `FRESHEN_SKIP = { off, fresh, backoff, writerLock, claimed, readOnly, noIndex }` (string values `off`, `fresh`, `backoff`, `writer_lock`, `claimed`, `read_only`, `no_index`).+ - `decideFreshen(input: { nowMs: number; lastIndexedAt: string | null; intervalSeconds: number; backoffUntilMs: number | null }): { action: "spawn" } | { action: "skip"; reason: FreshenSkip }` (pure; lock and claim are checked after it).+ - `claimFreshen(dir: string, nowMs: number): string | null` β exclusive create of `freshen.claim` holding `{ token, at }`; takes over a claim older than 600 s; returns the token or `null`.+ - `releaseFreshen(dir: string, token: string): void` β removes the claim only if it still holds `token`.+ - `readFreshenState(dir) / writeFreshenState(dir, state)` with `FreshenState { failures: number; backoffUntil: string | null; lastOutcome: "completed" | "failed" | null; lastRunAt: string | null; lastDurationMs: number | null; lastError: string | null; lastChanged: number | null }`; a missing or torn file reads as the empty state.+ - `nextBackoffMs(failures: number): number` β `min(60_000 * 2 ** (failures - 1), 3_600_000)`.+ - `freshenCommand(base: string[], has: (tool: string) => boolean, platform: NodeJS.Platform): string[]` β prefixes `ionice -c3` on linux / `taskpolicy -b` on darwin when `has(tool)`.+ - `maybeFreshenIndex(config: ResolvedSearchConfig, opts: { lastIndexedAt: string | null; readOnly?: boolean; nowMs?: number; spawn?: (argv: string[]) => void }): FreshenDecision` β never throws; returns `"spawned"` or the skip reason.++- [ ] **Step 1: Failing tests** for `decideFreshen`: `intervalSeconds 0` β `off`; `lastIndexedAt null` β `no_index` (self-heal owns a missing index); age 30 s / interval 60 β `fresh`; age 61 s β `spawn`; `backoffUntilMs > now` β `backoff`.+- [ ] **Step 2: Failing tests** for the claim: first claim returns a token; a second returns `null` (`claimed`); a claim file with `at` 601 s ago is taken over; `releaseFreshen` with a foreign token leaves the file (Review Focus 1, 2).+- [ ] **Step 3: Failing tests** for state and backoff: round-trip; torn JSON reads as empty; `nextBackoffMs(1) === 60_000`, `(2) === 120_000`, `(10) === 3_600_000`.+- [ ] **Step 4: Failing tests** for `freshenCommand`: linux with `ionice` β `["ionice", "-c3", ...base]`; linux without β `base`; darwin with `taskpolicy` β `["taskpolicy", "-b", ...base]`; win32 β `base`.+- [ ] **Step 5: Failing tests** for `maybeFreshenIndex` with an injected `spawn` and a temp dir: stale index β spawn called once with `search index --vault <v> --freshen <id>` (plus `--config <p>` when set); fresh β not called; `readOnly: true` β `read_only`, not called (Review Focus 4); held writer lock (create the lock dir the way `writer-lock.ts` does) β `writer_lock`; a `spawn` that throws β returns a skip, does not throw.+- [ ] **Step 6: Run** `bun test tests/core/search/freshen.test.ts` β FAIL.+- [ ] **Step 7: Implement** `freshen.ts`. The real spawner reuses the self-heal pattern from `src/core/maintenance/ensure-current.ts` (`Bun.spawn`, streams `ignore`, `windowsHide`, `detached` on win32, `env: { ...process.env }`, `unref`) and the `o2bCommand()` resolution (move it to a small shared helper `src/core/maintenance/o2b-command.ts` and import it from both). The claim token is passed as `--freshen <token>`.+- [ ] **Step 8: Run** tests and typecheck β PASS.+- [ ] **Step 9: Commit** `feat(search): freshen decision, claim, state and spawn`.++### Task 3: The background child (`o2b search index --freshen`)++**Files:**+- Modify: `src/cli/search/verbs/indexing.ts` (`cmdSearchIndex`)+- Modify: `src/core/brain/metrics.ts` only if surfaces are registered there (add `index_freshen`)+- Test: `tests/cli/search-index-freshen.test.ts`++**Interfaces:**+- Consumes: `releaseFreshen`, `readFreshenState`, `writeFreshenState`, `nextBackoffMs` (Task 2).+- Produces: flag `--freshen <token>`; on success the state gets `failures: 0, lastOutcome: "completed", lastChanged: added+updated+deleted`; on failure `failures+1`, `backoffUntil = now + nextBackoffMs(failures)`, `lastError`; a metric row on surface `index_freshen` only on failure or when `lastChanged > 0`; the claim is released in `finally`.++- [ ] **Step 1: Failing tests** (in-process `cmdSearchIndex` with a temp vault): `--freshen <token>` after `claimFreshen` β exit 0, claim gone, state `completed`; a vault whose db path is unwritable (a directory squatting `brain.sqlite`) β state `failed`, `failures 1`, `backoffUntil` set, claim gone (Review Focus 3); a second failure β `failures 2` and a doubled backoff.+- [ ] **Step 2: Run** β FAIL.+- [ ] **Step 3: Implement**: parse `freshen`; when present, `os.setPriority(process.pid, 10)` in a try; `embeddings: cfg.freshen?.embeddings === true` unless `--embeddings` was given; wrap the run in try/catch/finally writing state, the metric row and releasing the claim; the exit code stays as today.+- [ ] **Step 4: Run** β PASS. **Step 5: Commit** `feat(search): background freshen child records its outcome`.++### Task 4: Call sites and the `index-stale` trail code++**Files:**+- Modify: `src/core/search/search.ts` (after the store opens)+- Modify: `src/core/maintenance/ensure-current.ts` (after the rebuild check)+- Modify: `src/core/search/retrieval-trail.ts` (code, membership, description)+- Modify: `docs/cli-reference.md` (trail table row; required by `tests/docs/retrieval-trail-doc.test.ts`)+- Test: `tests/core/search/freshen-search.test.ts`, extend `tests/core/maintenance/ensure-current*.test.ts`++**Interfaces:**+- Consumes: `maybeFreshenIndex` (Task 2). A test seam: `SearchOptions.freshenSpawn?: (argv: string[]) => void` passed through to `maybeFreshenIndex({ spawn })`.+- Produces: `RETRIEVAL_DEGRADATION.indexStale = "index-stale"` with `detail: { ageSeconds }`; `EnsureCurrentResult.freshen: FreshenDecision | null`.++- [ ] **Step 1: Failing tests**: a search over an index whose `last_indexed_at` is 5 min old with `freshen.intervalSeconds 60` calls the injected spawn once and returns results; `selfHeal: false` never calls it; age 11 min adds `{ code: "index-stale", detail: { ageSeconds } }` to `degraded`; age 2 min does not; interval `0` with age 11 min still reports `index-stale` but never spawns. `ensureVaultCurrent` on a current-schema stale index returns `freshen: "spawned"` (spawn injected through `EnsureCurrentOptions.freshenSpawn`), and on a rebuild-needed index leaves `freshen: null`.+- [ ] **Step 2: Run** β FAIL.+- [ ] **Step 3: Implement**: in `search()` read `store.getState(LAST_INDEXED_AT_STATE_KEY)` once; call `maybeFreshenIndex(effectiveConfig, { lastIndexedAt, readOnly: opts.selfHeal === false, spawn: opts.freshenSpawn })`; push `index-stale` when the age exceeds 600 s. In `ensureVaultCurrent`, when no rebuild is needed, open the index read-only, read the stamp, call `maybeFreshenIndex`.+- [ ] **Step 4: Add the trail code** to `RETRIEVAL_DEGRADATION`, `RETRIEVAL_DEGRADATION_CODES` (after the keyword codes) and `describeRetrievalDegradation`; add the `docs/cli-reference.md` row: "the index was last updated more than ten minutes ago, so notes changed since then may be missing; a background run has been started unless freshening is off or backing off".+- [ ] **Step 5: Run** the new tests, `tests/docs/retrieval-trail-doc.test.ts`, `tests/core/search/` β PASS. **Step 6: Commit** `feat(search): reads refresh a stale index in the background`.++### Task 5: No-change run skips the whole-index post-passes++**Files:**+- Modify: `src/core/search/indexer.ts` (around the post-passes after the delete loop)+- Modify: `src/core/search/store/state.ts` (new key `schema_pack_digest`)+- Test: `tests/core/search/indexer-noop.test.ts`++**Interfaces:**+- Produces: state key `SCHEMA_PACK_DIGEST_STATE_KEY = "schema_pack_digest"`; `IndexStats.postPassesSkipped: boolean`.++- [ ] **Step 1: Failing tests**: index a small vault, index again with no change β `postPassesSkipped: true` and `last_indexed_at` advanced; change a note β `false`; change only `Brain/_brain.yaml` `link_constraints` β `false` and the blocked flag recomputed (Review Focus 5); a forced run β `false`.+- [ ] **Step 2: Run** β FAIL.+- [ ] **Step 3: Implement**: compute a SHA-256 of the raw `_brain.yaml` bytes (empty string when absent); skip `resolveLinkTargets`, `resolveAliasTargets`, `recomputeRelationConstraintFlags` and the tier pass when `added + updated + deleted === 0`, not forced, and the digest equals the stored one; store the digest after the post-passes run. Keep the stamp, the census and the deferred-reason reads.+- [ ] **Step 4: Run** the new test and `tests/core/search/indexer*.test.ts` β PASS. **Step 5: Commit** `perf(search): a no-change index run skips the whole-index post-passes`.++### Task 6: Catch-up write volume, measured then fixed++**Files:**+- Create: `scripts/measure-index-writes.ts` (Linux; reads `/proc/self/io` `write_bytes` around phases; prints a table)+- Modify: `src/core/search/indexer.ts` (optional `onPhase` hook in `IndexOptions`, no behaviour change)+- Modify: the module of the dominant phase (decided by the measurement)++- [ ] **Step 1:** Add `onPhase?: (phase: "walk" | "post" | "embed" | "stamp" | "close") => void` to the index options, called at phase boundaries. Commit.+- [ ] **Step 2:** The script copies the reference vault to a temp dir, indexes it, rewinds the copy's `last_indexed_at` and touches 200 notes with new content, then runs a catch-up index and prints bytes per phase plus the close (WAL checkpoint). Run it; record the table in the PR description.+- [ ] **Step 3:** Fix the dominant phase. Expected candidates and fixes: per-file transactions β one transaction per batch of files; full-index link/alias resolution β restrict to links touching changed documents; WAL-to-DELETE checkpoint on every close β keep WAL between runs where the backing allows it. Write a regression test for the behaviour changed (not for byte counts).+- [ ] **Step 4:** Re-run the script; target β€ ~100 MB. If not reachable without a broader rework, stop, keep what helped, and report the numbers. Commit `perf(search): <what was fixed>`.++### Task 7: Freshness in `search status` and `doctor`++**Files:**+- Modify: `src/cli/search/verbs/status.ts`+- Modify: the readiness check module under `src/core/brain/doctor/` that prints index lines+- Test: extend the status and readiness tests++- [ ] **Step 1: Failing tests**: `o2b search status` prints `freshen: every 60s` (or `off`), `index_age: <n>s`, `last_freshen: completed <ts> (<n> changed)` / `failed <ts>: <error>`, and `freshen_backoff_until: <ts>` when active; `--json` carries the same fields under `freshen`. Readiness warns when `failures >= 3`.+- [ ] **Step 2: Run** β FAIL. **Step 3: Implement** from `readFreshenState(dirname(dbPath))`. **Step 4: Run** β PASS. **Step 5: Commit** `feat(search): show index freshness in status and doctor`.++### Task 8: End-to-end with a real child++**Files:**+- Test: `tests/e2e/index-freshen.test.ts`++- [ ] **Step 1:** Temp vault + config, index it, write a new note, set `OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S=1` for a spawned `o2b search query` (via `runCli` helpers), wait up to 20 s polling the state file for `lastOutcome: "completed"`, then search finds the new note. Timeout generous for Windows.+- [ ] **Step 2: Run** on Linux β PASS. **Step 3: Commit** `test(search): background freshen end to end`.++### Task 9: Docs, version, gates, PR++**Files:** `docs/cli-reference.md` (keys, `--freshen`, status fields), `docs/how-it-works.md`, `docs/architecture.md` ("no daemon" clarification), `docs/hermes-cron.md`, the reindex `--cron-template` help text, `README.md` release line, `CHANGELOG.md` (+ compare link), `package.json` + `bun run scripts/sync-version.ts`, `bun run sync-plugin-mirrors` if a skill changed.++- [ ] **Step 1:** Write the docs and CHANGELOG (Added: freshen on read, trail code, status fields; Changed: no-change runs, catch-up writes with the measured numbers; note that a search may start a short background process and `search_freshen_interval_s: 0` turns it off).+- [ ] **Step 2:** Bump the minor version, sync, `--check`.+- [ ] **Step 3:** Gates: `bun run typecheck`, `bun run lint`, `bun run fmt:check`, `bun run link-ratchet:check`, `bun run sync-plugin-mirrors:check`, the OpenClaw bundle diff with Bun 1.4.0, `bun test --parallel`.+- [ ] **Step 4:** Commit, push the branch, open the PR with the measurement tables; wait for CI and review bots; fix findings.diff --git a/docs/cli-reference.md b/docs/cli-reference.mdindex 0188422d..fd52a8da 100644--- a/docs/cli-reference.md+++ b/docs/cli-reference.md@@ -1827,6 +1827,9 @@ o2b search reindex Rebuild the SQLite + FTS5 index from scratchmachine-local pido2b search index Incrementally update the index; --embeddings computes vectors, --progress watches it--force-cost bypasses the embedding cost gate (since v0.36.0)+ --freshen <token> [--freshen-state <dir>] is the background run freshen on+ read starts (since v1.76.0): lowered priority, no output, outcome in+ <dir>/freshen-state.json; meeting another index writer is a skipo2b search vector-backfill Run the vector phase ALONE for indexed chunks that have no vector -no vault walk, no re-chunking, no frontmatter pass (since v1.43.0).Dry-run by DEFAULT: it counts the pending chunks, reports the@@ -1875,6 +1878,11 @@ o2b search status Index status; since v0.36.0 also reports the activ<n> in the last 30 days); --json carries it as the event_time object(documents, with_event_time, earliest, latest, recent_window_days,in_recent_window)+ Since v1.76.0 it prints freshen (every <n>s or off), index_age, last_freshen+ (completed <ts> (<n> changed) / failed <ts> (<n> in a row): <error> /+ (none)) and freshen_backoff (until <ts>) while a backoff is active; --json+ carries them as the freshen object (interval_s, index_age_s, last_outcome,+ last_run_at, last_changed, last_error, failures, backoff_until)o2b search check Pre-flight diagnostics: vault, index directory, SQLite/FTS5, thevector extension, the embedding key, the provider, the vector ABI stamp--integrity additionally runs a full PRAGMA quick_check over the index@@ -2027,6 +2035,7 @@ so there is no separate lane field that could drift from it:| Code | The narrowing it reports || ---- | ------------------------ |+| `index-stale` | the index was last updated more than ten minutes ago (`detail.ageSeconds`), so notes changed since then may be missing; the code reports the index age only and does not say a refresh is running: freshen on read starts a background run when it can, and starts none when it is off, a failed run is backing off, another run or writer holds the index, the index is read-only for this reader, or the run cannot be spawned || `keyword-fts-match-empty` | the query tokenised to an empty FTS match, so the keyword lane never ran || `keyword-trigram-lane-fault` | the trigram candidate lane could not be read; `detail.fault` carries that lane's own classification || `semantic-embeddings-absent` | the index holds no compatible embedding |@@ -2363,11 +2372,57 @@ Two opt-in ranking guards and one deadline join the suite:| `search_relational_rerank_pin` | `OPEN_SECOND_BRAIN_SEARCH_RELATIONAL_RERANK_PIN` | `false` | Rerank may promote a relational-origin candidate but never sink it below its pre-rerank order (the relevance floor is untouched) || `search_metadata_boost_gate` | `OPEN_SECOND_BRAIN_SEARCH_METADATA_BOOST_GATE` | `false` | A query whose keyword lane returned no hits contributes zero from every additive metadata/structural boost layer || `search_hybrid_deadline_ms` | `OPEN_SECOND_BRAIN_SEARCH_HYBRID_DEADLINE` | `15000` | Wall-clock budget over the whole composite hybrid path (embed, semantic top-k, rerank, second pass); on expiry the abandoned embed and rerank requests are aborted, the search completes with what it has (keyword-only, or the pre-rerank order with exclusions, trust gate and the other post-rank phases still applied), reports `hybridDeadlineExceeded` and is not cached; `0` disables |+| `search_freshen_interval_s` | `OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S` | `60` | Freshen on read (since v1.76.0): a search or a session start that finds the index older than this many seconds starts one background incremental `o2b search index` at low CPU and I/O priority, and answers from the index it has; `0` turns it off. See "Freshen on read" below |+| `search_freshen_embeddings` | `OPEN_SECOND_BRAIN_SEARCH_FRESHEN_EMBEDDINGS` | `false` | Let that background run compute embeddings too (it costs money); off, it indexes keyword-only and leaves vectors to an explicit `--embeddings` run |Both guards are off by default and leave every score byte-identical;the deadline is on by default because its lane budgets already summedto more, and it bounds exactly the phases with no budget of their own.+### Freshen on read (since v1.76.0)++Nothing schedules index runs: no daemon, no OS timer, no Hermes job. A+search (every reading MCP tool, the recall-inject hook, `o2b search+query`) and a session start (`ensureVaultCurrent`) read the index's+`last_indexed_at`; when it is older than `search_freshen_interval_s`+(default 60 s, `0` turns it off), the reader answers from the index it+has and starts one detached `o2b search index --freshen <token>`, so the+next read sees the current vault. Only agent activity triggers it.++- **One run at a time.** The reader takes an exclusive claim,+ `<index dir>/freshen.claim`, and skips when another run holds it, an+ indexer holds the writer lock, or the index directory refuses the claim+ (a read-only mount, no permission). A claim older than ten minutes belongs+ to a run that died and is taken over by exactly one reader, under an+ exclusive takeover lock. A run that still meets another writer ends as+ a skip, not a failure.+- **Out of the way.** The child lowers its CPU priority and runs under+ `ionice -c3` on Linux or `taskpolicy -b` on macOS when the tool exists.+ It indexes keyword-only unless `search_freshen_embeddings` is on.+- **Cheap when nothing changed.** A run that added, updated and deleted+ no document skips link and alias resolution and writes almost nothing.+ A changed document rewrites only the chunks that changed, so a daily+ log that grows all day costs its appended tail per run.+- **Failures back off.** A failed run records its error in+ `<index dir>/freshen-state.json` and the next run waits 1 minute,+ doubling to an hour. `o2b search status` shows the last outcome and+ the backoff; `o2b doctor` warns with `freshen-failing` after three+ failures in a row. Runs that changed something or failed append one+ row to the `index_freshen` metrics surface.+- **Staleness is named.** A search over an index more than ten minutes+ old carries the trail code `index-stale` with `detail.ageSeconds`.+ The code names the age only; whether a background run started is not+ part of it (see the cases above where freshen on read starts none).+ The code is added when the answer is served, also on a query-cache+ hit, and is never stored in the cache.+- **Never a foreign vault.** Cross-vault and recall-source reads open+ other indexes read-only and never start a run there.++`o2b search watch` stays available for a foreground watcher, and the+cron recipes of `o2b search reindex --cron-template` and the maintenance+lane are still printed, but keyword freshness no longer needs them; a+scheduled run remains the way to compute embeddings periodically.+### The reserved visibility token (since v1.54.0)```textdiff --git a/docs/hermes-cron.md b/docs/hermes-cron.mdindex da2841ad..b31c8f07 100644--- a/docs/hermes-cron.md+++ b/docs/hermes-cron.md@@ -78,7 +78,7 @@ When `--silent-if-empty` is set and the window has no changes, the command write`dream` is itself idempotent β running it on top of an unchanged Brain is a no-op (no log entry written, no snapshot taken). The chained form is safe at any cadence.-The alternative to a hand-chained dream job is the maintenance lane, which runs dream, reindex, bridges and clusters under one lease and only inside its window: `o2b brain maintenance run --vault /path/to/vault --cron-template` prints a script and the `hermes cron create` line that schedules it (hourly by default; `--interval`, `--window H-H --tz ZONE` narrow it). The script is silent when the lane succeeds or a gate skips, and prints the lane's JSON report when a task failed, timed out or was refused, so Hermes delivers only the runs that need attention. Every recipe this CLI prints - the maintenance lane, `o2b search reindex --cron-template` and `o2b partner codegraph resync --cron-template` - also prints a systemd user timer with `--format systemd`, for a host where Hermes does not own the schedule.+The alternative to a hand-chained dream job is the maintenance lane, which runs dream, reindex, bridges and clusters under one lease and only inside its window: `o2b brain maintenance run --vault /path/to/vault --cron-template` prints a script and the `hermes cron create` line that schedules it (hourly by default; `--interval`, `--window H-H --tz ZONE` narrow it). The script is silent when the lane succeeds or a gate skips, and prints the lane's JSON report when a task failed, timed out or was refused, so Hermes delivers only the runs that need attention. Since v1.76.0 the keyword index needs no scheduled reindex: freshen on read keeps it current while agents work (see "Freshen on read" in `docs/cli-reference.md`), so a scheduled reindex is only worth keeping to compute embeddings periodically. Every recipe this CLI prints - the maintenance lane, `o2b search reindex --cron-template` and `o2b partner codegraph resync --cron-template` - also prints a systemd user timer with `--format systemd`, for a host where Hermes does not own the schedule.## Discipline report (daily logging-discipline sanity-check)diff --git a/docs/how-it-works.md b/docs/how-it-works.mdindex 7e8962bb..5f8062bc 100644--- a/docs/how-it-works.md+++ b/docs/how-it-works.md@@ -1342,6 +1342,22 @@ Full design and migration notes:and the matching implementation plan[`docs/plans/2026-05-16-brain-search-impl.md`](plans/2026-05-16-brain-search-impl.md).+### Keeping the index current (since v1.76.0)++The index is kept current by the reads that need it. When a search or a+session start finds `last_indexed_at` older than+`search_freshen_interval_s` (60 s by default), it answers from the+index as it is and starts one detached, low-priority incremental run+(`o2b search index --freshen`); an exclusive claim next to the index+keeps concurrent readers from starting one each, and a failed run backs+off from one minute to an hour. Nothing stays resident: with no agent+working, nothing indexes. Two indexer changes make frequent runs cheap:+a run that changed no document skips link and alias resolution, and a+changed document keeps every chunk whose position and content did not+change in place (row, full-text entry, entities, vector), so appending+to a 1 MB daily log costs about 0.2 MB of writes instead of about 9 MB.+Details: "Freshen on read" in `docs/cli-reference.md`.+## Workspace reach and proactive insight (since v0.38.0)Two suites extend where the Brain can be reached and what itdiff --git a/docs/metrics.md b/docs/metrics.mdindex 7051f663..e17254d0 100644--- a/docs/metrics.md+++ b/docs/metrics.md@@ -82,6 +82,7 @@ writes, and a metric record is a summary, not a report.| `vault_vitals` | `o2b brain vitals` | `preferences_scanned`, `domain_diversity`, `connectivity_index`, `orphan_count`, `gap_pressure` || `self_heal_upgrade` | `o2b brain upgrade --self-heal` (the detached worker `ensureVaultCurrent` starts) | `outcome` (`applied` / `failed`), `duration_ms`, `files` (rewritten, or pending on `failed`), `error` (on `failed`). An attempt that finds nothing pending writes no row || `self_heal_reindex` | `ensureVaultCurrent` (parent) and `o2b search reindex --self-heal <run-id>` (child) | parent row: `decision` (`spawned` / `skipped_writer_lock`), `run_id` (on `spawned`). Child row: `outcome` (`completed` / `failed`), `run_id`, `duration_ms`, `error` (on `failed`) |+| `index_freshen` (since v1.76.0) | `o2b search index --freshen <token>` (the background run freshen on read starts) | `outcome` (`completed` / `failed`), `duration_ms`, `changed` (on `completed`), `error` and `failures` (on `failed`). A row is written only when the run changed documents or failed || `maintenance_spend` (since v1.64.0) | `o2b brain maintenance run`, MCP `brain_maintenance` (`run`) | `task` (`reindex`), `model` (null when the config names none), `tokens`, `estimated_usd`, `forced` (`force-cost` overrode a positive gate), `lane` (always true: both front doors are the lane) |Payload fields marked "(lane runs)" appear only on maintenance-lanediff --git a/docs/updating.md b/docs/updating.mdindex 06e47d90..312d6791 100644--- a/docs/updating.md+++ b/docs/updating.md@@ -47,6 +47,15 @@ instruction files such as `CLAUDE.md`/`AGENTS.md`, installed`.claude/skills/`) and warns with the exact replacement for any stalereference it finds (`removed-tool-reference`).+## Upgrading to 1.76.0++No step is required. Searches and session starts now keep the search+index current on their own: an index older than a minute gets one+background incremental run. If you scheduled `o2b search index` or+`o2b search reindex` only to keep keyword search fresh, that job can go;+keep it if it computes embeddings. To turn the background runs off, set+`search_freshen_interval_s: "0"`.+## Upgrading to 1.73.0No step is required unless you set a positive `embedding_cost_gate_usd`diff --git a/openclaw.plugin.json b/openclaw.plugin.jsonindex 8bd04bc2..fcb8a5fc 100644--- a/openclaw.plugin.json+++ b/openclaw.plugin.json@@ -2,7 +2,7 @@"id": "open-second-brain","name": "Open Second Brain","description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",- "version": "1.75.0",+ "version": "1.76.0","activation": { "onStartup": true },"skills": ["./skills"],"contracts": {diff --git a/package.json b/package.jsonindex 0cdd443e..3be6572f 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.75.0",+ "version": "1.76.0","private": false,"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.","keywords": [diff --git a/plugin.yaml b/plugin.yamlindex 75c93f72..cd8f1d6a 100644--- a/plugin.yaml+++ b/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.75.0"+version: "1.76.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.jsonindex 47ed2f3e..42e7b65c 100644--- a/plugins/codex/.codex-plugin/plugin.json+++ b/plugins/codex/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.75.0",+ "version": "1.76.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/plugins/codex/README.md b/plugins/codex/README.mdindex d92cbfc8..4e3b51fd 100644--- a/plugins/codex/README.md+++ b/plugins/codex/README.md@@ -113,7 +113,7 @@ The full router with readiness criteria is [`install.md`](https://github.com/ite## What is new-1.73.0 makes the query embed honest and upgrades safe. Every query embed, from the search lane behind the MCP tools and hooks to the `brain_context_pack` semantic belief order, passes one gate: under a positive `embedding_cost_gate_usd` an unpriced model is refused for any caller that is not local (a hybrid search falls back to keyword-only and says so by trail code), and a query longer than the model's input window is cut to it and disclosed. `embedding_input_window_tokens` declares the window of a model the curated table does not list, and `embedding_extra_body` sends extra request fields to an OpenAI-compatible endpoint. `brain_recall_feedback` and the benchmark, eval and tune reports name the degradations their searches met, and `brain_tune` no longer saves a winner measured without the semantic lane. `o2b brain upgrade` and the automatic upgrade worker apply exactly the plan they computed and refuse a managed file edited in between instead of overwriting it. Every release is described in the [CHANGELOG](https://github.com/itechmeat/open-second-brain/blob/main/CHANGELOG.md).+1.76.0 keeps the search index current without a scheduler. A search or a session start that finds the index more than a minute old answers from it as it is and starts one low-priority background run, so the next read sees the current vault; nothing stays resident, and nothing runs while no agent works. A changed note now rewrites only the parts of the index that changed, so appending to a large daily log costs a fraction of what it did, and `o2b search status` shows how fresh the index is. Every release is described in the [CHANGELOG](https://github.com/itechmeat/open-second-brain/blob/main/CHANGELOG.md).## Documentationdiff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yamlindex 75c93f72..cd8f1d6a 100644--- a/plugins/hermes/plugin.yaml+++ b/plugins/hermes/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.75.0"+version: "1.76.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/pyproject.toml b/pyproject.tomlindex 1707c93c..a4818511 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"# CLI entry points (those moved to `package.json` `bin`).[project]name = "open-second-brain"-version = "1.75.0"+version = "1.76.0"description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."readme = "README.md"requires-python = ">=3.11"diff --git a/src/cli/brain/verbs/doctor.ts b/src/cli/brain/verbs/doctor.tsindex e55014ee..492636b4 100644--- a/src/cli/brain/verbs/doctor.ts+++ b/src/cli/brain/verbs/doctor.ts@@ -105,6 +105,7 @@ export async function cmdBrainDoctor(argv: string[]): Promise<number> {dryRun,agent: resolveBrainAgent(flags, config),configPath: config,+ dbPath: resolveSearchConfig({ vault, configPath: config ?? undefined }).dbPath,});return renderRepair(outcome, Boolean(flags["json"]));} catch (exc) {diff --git a/src/cli/command-manifest.ts b/src/cli/command-manifest.tsindex 7b8deedf..3c5368d9 100644--- a/src/cli/command-manifest.ts+++ b/src/cli/command-manifest.ts@@ -813,6 +813,8 @@ export const CLI_COMMAND_MANIFEST: CliRootManifest = Object.freeze({flag("concurrency", "string"),flag("verbose", "boolean"),flag("progress", "boolean"),+ flag("freshen", "string"),+ flag("freshen-state", "string"),]),command("reindex", "Rebuild the search index", [flag("vault", "string"),diff --git a/src/cli/search/verbs/indexing.ts b/src/cli/search/verbs/indexing.tsindex 6598fbc6..123c3682 100644--- a/src/cli/search/verbs/indexing.ts+++ b/src/cli/search/verbs/indexing.ts@@ -6,7 +6,11 @@* name once the index covers every document.*/+import { setPriority } from "node:os";+import { dirname } from "node:path";+import { formatDegradationNotice } from "../../../core/integrity/degradation.ts";+import { recordFreshenOutcome, releaseFreshen } from "../../../core/search/freshen.ts";import {createSafeguard,resolveSafeguardTimeoutMs,@@ -176,7 +180,13 @@ export async function cmdSearchIndex(argv: ReadonlyArray<string>): Promise<numbeverbose: { type: "boolean" },progress: { type: "boolean" },json: { type: "boolean" },+ freshen: { type: "string" },+ "freshen-state": { type: "string" },});+ const freshenToken = flagString(flags, "freshen");+ if (freshenToken !== undefined) {+ return runFreshenChild(flags, freshenToken, flagString(flags, "freshen-state"));+ }const cfg = resolveConfig(flags);const verbose = flagBoolean(flags, "verbose");@@ -214,6 +224,56 @@ export async function cmdSearchIndex(argv: ReadonlyArray<string>): Promise<numbereturn 0;}+/**+ * `--freshen <token>`: the background run a stale read started (see+ * `src/core/search/freshen.ts`). It runs at lowered CPU priority, indexes+ * incrementally (embeddings only when `search_freshen_embeddings` is on),+ * records its outcome and backoff in the per-device state file, and+ * releases the claim it was handed whatever happens. Nothing is printed:+ * every stream of a detached child is ignored.+ */+async function runFreshenChild(+ flags: SearchVerbFlags,+ token: string,+ stateDirFlag: string | undefined,+): Promise<number> {+ try {+ setPriority(process.pid, 10);+ } catch {+ // Not permitted here (or not supported): run at normal priority.+ }+ const startedAt = Date.now();+ let cfg: ResolvedSearchConfig | undefined;+ let stateDir = stateDirFlag;+ try {+ cfg = resolveConfig(flags);+ stateDir ??= dirname(cfg.dbPath);+ const stats = await indexVault(cfg, {+ embeddings: cfg.freshen?.embeddings === true,+ });+ recordFreshenOutcome(cfg.vault, stateDir, {+ outcome: "completed",+ durationMs: Date.now() - startedAt,+ changed: stats.added + stats.updated + stats.deleted,+ });+ return 0;+ } catch (e) {+ // Another writer holds the index (a manual run, the watcher, the+ // maintenance lane): nothing is wrong, the next stale read retries.+ if (e instanceof SearchError && e.code === "INDEX_LOCKED") return 0;+ if (stateDir !== undefined) {+ recordFreshenOutcome(cfg?.vault ?? null, stateDir, {+ outcome: "failed",+ durationMs: Date.now() - startedAt,+ error: e instanceof Error ? e.message : String(e),+ });+ }+ return 1;+ } finally {+ if (stateDir !== undefined) releaseFreshen(stateDir, token);+ }+}+/*** Rebuild the index from scratch.*diff --git a/src/cli/search/verbs/status.ts b/src/cli/search/verbs/status.tsindex b977662f..59e7a529 100644--- a/src/cli/search/verbs/status.ts+++ b/src/cli/search/verbs/status.ts@@ -4,6 +4,12 @@*/import { formatEstimatedUsd } from "../../../core/search/embedding-spend.ts";+import {+ freshenStatus,+ renderFreshenStatus,+ serializeFreshenStatus,+ type FreshenStatus,+} from "../../../core/search/freshen.ts";import {eventTimeStatus,indexStatus,@@ -31,11 +37,13 @@ export async function cmdSearchStatus(argv: ReadonlyArray<string>): Promise<numbconst status = await indexStatus(cfg);// The event-time summary reads the index, so it exists only with one.const eventTime = status.exists ? await eventTimeStatus(cfg) : null;+ const freshen = status.exists ? freshenStatus(cfg, status.lastIndexedAt) : null;if (flagBoolean(flags, "json")) {process.stdout.write(JSON.stringify({...(serializeIndexStatus(status) as Record<string, unknown>),...(eventTime !== null ? { event_time: serializeEventTimeStatus(eventTime) } : {}),+ ...(freshen !== null ? { freshen: serializeFreshenStatus(freshen) } : {}),// Same polarity as the human twin below, so the two conditions// read as one rule rather than as each other's negation....(!status.exists ? nextCommandField("search-index-missing") : {}),@@ -43,7 +51,7 @@ export async function cmdSearchStatus(argv: ReadonlyArray<string>): Promise<numb);return 0;}- process.stdout.write(renderStatusHuman(status, eventTime));+ process.stdout.write(renderStatusHuman(status, eventTime, freshen));// no-dead-ends, phase 3: the pointer used to be spliced into the// renderer's first line, which is a second emission mechanism AND// beyond the reach of a scan over writer call sites. The rail decides@@ -56,7 +64,11 @@ export async function cmdSearchStatus(argv: ReadonlyArray<string>): Promise<numbreturn 0;}-function renderStatusHuman(s: IndexStatusSnapshot, eventTime: EventTimeStatus | null): string {+function renderStatusHuman(+ s: IndexStatusSnapshot,+ eventTime: EventTimeStatus | null,+ freshen: FreshenStatus | null,+): string {if (!s.exists) {return `index: not initialised\n path: ${s.indexPath}\n`;}@@ -77,6 +89,7 @@ function renderStatusHuman(s: IndexStatusSnapshot, eventTime: EventTimeStatus |lines.push(`embedding_key: ${s.embeddingKeyPresent ? "present" : "missing"}`);lines.push(`last_indexed_at: ${s.lastIndexedAt ?? "(never)"}`);lines.push(`last_full_index_at: ${s.lastFullIndexAt ?? "(never)"}`);+ if (freshen !== null) lines.push(...renderFreshenStatus(freshen));if (eventTime !== null) lines.push(`event_time: ${renderEventTimeStatus(eventTime)}`);// The oversize-chunk census when it could NOT run. Above the warnings// and not among them, deliberately: this line reports a check that diddiff --git a/src/core/brain/diagnostics.ts b/src/core/brain/diagnostics.tsindex 52ec8222..c4d11248 100644--- a/src/core/brain/diagnostics.ts+++ b/src/core/brain/diagnostics.ts@@ -715,6 +715,14 @@ export const DIAGNOSTIC_SIGNALS: ReadonlyMap<string, DiagnosticSignal> = new MapnextCommand: "o2b brain tiers check",autoRepairable: false,},+ {+ // A review, like tier-drift: the status names the last error and+ // the backoff, and the repair depends on what the error is.+ code: "freshen-failing",+ issueClass: "background index refresh keeps failing",+ nextCommand: "o2b search status",+ autoRepairable: false,+ },{code: "entity-label-malformed",issueClass: "entity label fails the quality gate",@@ -1173,6 +1181,12 @@ export interface PlanRepairOptions {* page counts.*/readonly readable?: (rel: string) => boolean;+ /**+ * The search-index path, when the doctor runs here, for its index-backed+ * checks (`tier-drift`, `freshen-failing`). Absent, those checks skip,+ * as they do in `runDoctor`.+ */+ readonly dbPath?: string;}/**@@ -1190,7 +1204,7 @@ export interface PlanRepairOptions {* the same findings, each with its next-command hint.*/export function planRepair(vault: string, opts: PlanRepairOptions = {}): RepairPlan {- const issues = opts.issues ?? collectDoctorIssues(vault, opts.readable);+ const issues = opts.issues ?? collectDoctorIssues(vault, opts.readable, opts.dbPath);const fixes: RepairItem[] = [];for (const fixer of FIXERS) {@@ -1226,8 +1240,12 @@ export function planRepair(vault: string, opts: PlanRepairOptions = {}): RepairPfunction collectDoctorIssues(vault: string,readable: ((rel: string) => boolean) | undefined,+ dbPath: string | undefined,): ReadonlyArray<DoctorIssue> {- const doctor = runDoctor(vault, readable !== undefined ? { readable } : {});+ const doctor = runDoctor(vault, {+ ...(readable !== undefined ? { readable } : {}),+ ...(dbPath !== undefined ? { dbPath } : {}),+ });return [...doctor.errors, ...doctor.warnings];}@@ -1289,6 +1307,12 @@ export interface ApplyRepairOptions {* that never held them. Absent, every page counts.*/readonly readable?: (rel: string) => boolean;+ /**+ * The search-index path for the doctor's index-backed checks, so the+ * plan lists `tier-drift` and `freshen-failing` as the report does.+ * Absent, those checks skip.+ */+ readonly dbPath?: string;}/** The fields a doctor finding names artifacts in. */@@ -1365,9 +1389,12 @@ export function applyRepair(vault: string, opts: ApplyRepairOptions): RepairOutc// are bounded again by what each one would name, so a fix that reaches// a withheld page through its detail is not planned either.const plan = view.filtersNothing- ? planRepair(vault, opts.readable !== undefined ? { readable: opts.readable } : {})+ ? planRepair(vault, {+ ...(opts.readable !== undefined ? { readable: opts.readable } : {}),+ ...(opts.dbPath !== undefined ? { dbPath: opts.dbPath } : {}),+ }): planRepair(vault, {- issues: view.keep(collectDoctorIssues(vault, opts.readable), (i) =>+ issues: view.keep(collectDoctorIssues(vault, opts.readable, opts.dbPath), (i) =>doctorIssueRefs(vault, i),),});diff --git a/src/core/brain/doctor.ts b/src/core/brain/doctor.tsindex d7d81914..0c79c41b 100644--- a/src/core/brain/doctor.ts+++ b/src/core/brain/doctor.ts@@ -91,6 +91,7 @@ import { logChainCheck } from "./doctor/log-chain-check.ts";import { checkSemanticHealth } from "./doctor/semantic-health-check.ts";import {danglingWorkrunCheck,+ freshenFailureCheck,symlinkEscapeCheck,syncConflictLogCheck,tierDriftCheck,@@ -175,6 +176,7 @@ const DOCTOR_CHECKS: ReadonlyArray<DoctorCheck> = Object.freeze([preferenceCheck,retiredCheck,tierDriftCheck,+ freshenFailureCheck,duplicateIdCheck,logShardCheck,brokenBacklinkCheck,diff --git a/src/core/brain/doctor/store-integrity.ts b/src/core/brain/doctor/store-integrity.tsindex 6d805880..74b87648 100644--- a/src/core/brain/doctor/store-integrity.ts+++ b/src/core/brain/doctor/store-integrity.ts@@ -8,9 +8,10 @@* all of them are conditions only a walk of the tree can see.*/-import { join } from "node:path";+import { dirname, join } from "node:path";import { ensureInsideVault, realpathInsideVault, vaultRelative } from "../../path-safety.ts";+import { readFreshenState } from "../../search/freshen.ts";import { BRAIN_LOG_AUDIT_DIRS } from "../audit-dirs.ts";import { continuityLogDir } from "../continuity/store.ts";import { scanDanglingWorkruns } from "../dream-workrun.ts";@@ -36,13 +37,14 @@ import {/*** Tier guard (write-time-integrity-governance): staged identity* hand-edits the index post-pass detected. Fail-soft index read - a- * missing index or pre-v6 schema simply skips the check.+ * missing index or pre-v6 schema simply skips the check. Counted over+ * the pages the caller may read, like every check handed `readable`.*/export const tierDriftCheck: DoctorCheck = {failSoft: false,- run({ dbPath }, { issues }) {+ run({ dbPath, readable }, { issues }) {if (dbPath === undefined) return;- const driftCount = readTierDriftCount(dbPath);+ const driftCount = readTierDriftCount(dbPath, readable);if (driftCount > 0) {issues.push({severity: "warning",@@ -53,14 +55,38 @@ export const tierDriftCheck: DoctorCheck = {},};+/**+ * Freshen on read (index-freshness): background index runs started by+ * stale reads that keep failing. Three in a row is a broken index, not a+ * hiccup - every search meanwhile answers from an ageing index.+ */+export const freshenFailureCheck: DoctorCheck = {+ failSoft: false,+ run({ dbPath }, { issues }) {+ if (dbPath === undefined) return;+ const state = readFreshenState(dirname(dbPath));+ if (state.lastOutcome !== "failed" || state.failures < FRESHEN_FAILURE_WARN_AT) return;+ issues.push({+ severity: "warning",+ code: FRESHEN_FAILING_CODE,+ message:+ `background index refresh failed ${state.failures} times in a row ` ++ `(last: ${state.lastError ?? "unknown error"}); the index is not being kept current`,+ });+ },+};++/** Failures in a row from which {@link freshenFailureCheck} warns. */+export const FRESHEN_FAILURE_WARN_AT = 3;++/** The code of the {@link freshenFailureCheck} warning. */+export const FRESHEN_FAILING_CODE = "freshen-failing";+/** The code of the {@link tierDriftCheck} warning. */export const TIER_DRIFT_CODE = "tier-drift";-/**- * The {@link tierDriftCheck} warning text for `count` staged drift rows,- * shared with a surface that recounts the rows a caller may read.- */-export function tierDriftMessage(count: number): string {+/** The {@link tierDriftCheck} warning text for `count` staged drift rows. */+function tierDriftMessage(count: number): string {return `${count} identity-field hand-edit(s) staged - review with: o2b brain tiers check`;}diff --git a/src/core/brain/frontmatter-tiers.ts b/src/core/brain/frontmatter-tiers.tsindex 94415e95..1089e636 100644--- a/src/core/brain/frontmatter-tiers.ts+++ b/src/core/brain/frontmatter-tiers.ts@@ -220,12 +220,24 @@ function sameScalar(left: unknown, right: unknown): boolean {* Count the staged tier-drift findings straight from the index* database (doctor surface). Read-only, fail-soft: a missing index* file, a pre-v6 schema, or any read error counts as zero.+ *+ * With `readable`, only the rows on pages the caller may read count, so+ * a drifted page it cannot read moves the count no more than an absent+ * one does.*/-export function readTierDriftCount(dbPath: string): number {+export function readTierDriftCount(dbPath: string, readable?: (rel: string) => boolean): number {if (!existsSync(dbPath)) return 0;let db: Database | null = null;try {db = new Database(dbPath, { readonly: true });+ if (readable !== undefined) {+ return db+ .query<{ path: string }, []>(+ "SELECT d.path FROM tier_drift t JOIN documents d ON d.id = t.document_id",+ )+ .all()+ .filter((row) => readable(row.path)).length;+ }return db.query<{ n: number }, []>("SELECT COUNT(*) AS n FROM tier_drift").get()?.n ?? 0;} catch {return 0;diff --git a/src/core/maintenance/ensure-current.ts b/src/core/maintenance/ensure-current.tsindex 8d6c9348..34974c7a 100644--- a/src/core/maintenance/ensure-current.ts+++ b/src/core/maintenance/ensure-current.ts@@ -17,17 +17,17 @@* migrations and downgrades correctly.*/import { existsSync } from "node:fs";-import { dirname, join, resolve } from "node:path";-import { fileURLToPath } from "node:url";import { Database } from "bun:sqlite";import { defaultConfigPath } from "../config.ts";+import { o2bCommand } from "./o2b-command.ts";import { brainConfigPath } from "../brain/paths.ts";import { resolveSearchConfig } from "../search/index.ts";import { reindexVault } from "../search/indexer.ts";import { CHUNKER_VERSION } from "../search/chunker.ts";import { LATEST_SCHEMA_VERSION, readSchemaVersion } from "../search/schema.ts";-import { CHUNKER_VERSION_STATE_KEY } from "../search/store/state.ts";+import { CHUNKER_VERSION_STATE_KEY, LAST_INDEXED_AT_STATE_KEY } from "../search/store/state.ts";+import { maybeFreshenIndex, type FreshenDecision } from "../search/freshen.ts";import { isWriterLockHeld } from "../search/store/writer-lock.ts";import type { ResolvedSearchConfig } from "../search/types.ts";import {@@ -95,6 +95,12 @@ export interface EnsureCurrentResult {* foreground and there was no child to decide about.*/readonly reindexSpawn: SelfHealSpawnDecision | null;+ /**+ * Freshen on read at session start: whether a current index that has+ * gone stale got a background incremental run, or why not. Null when the+ * index needed a rebuild (self-heal owns it) or was not looked at.+ */+ readonly freshen: FreshenDecision | null;/** Non-empty when nothing ran, e.g. "not-initialized". */readonly skipped: string;/** Best-effort: per-step failures, never thrown. */@@ -111,6 +117,8 @@ export interface EnsureCurrentOptions {* (re)built matches the one the caller's server actually uses, e.g. under* `o2b mcp --config <custom>`. */readonly configPath?: string;+ /** Test seam: replaces the detached spawn of a freshen-on-read run. */+ readonly freshenSpawn?: (argv: string[]) => void;}function message(e: unknown): string {@@ -131,6 +139,27 @@ function chunksAreStale(db: Database): boolean {return db.query<{ one: number }, []>("SELECT 1 AS one FROM documents LIMIT 1").get() !== null;}+/** The index's `last_indexed_at`, or null when it cannot be read. */+function readLastIndexedAt(dbPath: string): string | null {+ let db: Database;+ try {+ db = new Database(dbPath, { readonly: true });+ } catch {+ return null;+ }+ try {+ return (+ db+ .query<{ value: string }, [string]>("SELECT value FROM index_state WHERE key = ? LIMIT 1")+ .get(LAST_INDEXED_AT_STATE_KEY)?.value ?? null+ );+ } catch {+ return null;+ } finally {+ closeDatabase(db);+ }+}+/*** Cheap, read-only check: is the search index absent, on a stale schema,* or chunked by older rules?@@ -152,22 +181,6 @@ function indexNeedsRebuild(config: ResolvedSearchConfig): boolean {}}-/**- * argv prefix that runs this checkout's CLI (current plugin version).- *- * POSIX goes through `scripts/o2b`, which also applies the macOS SQLite- * setup. Native Windows cannot execute that bash launcher, so it runs the- * TypeScript entry point with the Bun that is running this process.- */-function o2bCommand(): string[] {- // src/core/maintenance/ensure-current.ts -> repo root- const repo = resolve(dirname(fileURLToPath(import.meta.url)), "..", "..", "..");- if (process.platform === "win32") {- return [process.execPath, "run", join(repo, "src", "cli", "main.ts")];- }- return [join(repo, "scripts", "o2b")];-}-/*** Start the managed-file upgrade in a detached worker, unless a recorded* failure is cooling down or another worker has the vault.@@ -349,6 +362,7 @@ export async function ensureVaultCurrent(let brainUpgradeOutcome: SelfHealUpgradeOutcome | null = null;let reindexTriggered = false;let reindexSpawn: SelfHealSpawnDecision | null = null;+ let freshen: FreshenDecision | null = null;// Only an already-initialised vault is an upgrade target. A missing// `_brain.yaml` means "not set up yet" - that is `o2b init`'s job, not ours.@@ -359,6 +373,7 @@ export async function ensureVaultCurrent(brainUpgradeOutcome: null,reindexTriggered: false,reindexSpawn: null,+ freshen: null,skipped: "not-initialized",errors: [],};@@ -398,6 +413,13 @@ export async function ensureVaultCurrent(await reindexVault(config);reindexTriggered = true;}+ } else {+ // 3. A current index that has gone stale: catch it up in the+ // background so the session's first read is not the one that pays.+ freshen = maybeFreshenIndex(config, {+ lastIndexedAt: readLastIndexedAt(config.dbPath),+ ...(opts.freshenSpawn !== undefined ? { spawn: opts.freshenSpawn } : {}),+ });}} catch (e) {errors.push(`search-reindex: ${message(e)}`);@@ -409,6 +431,7 @@ export async function ensureVaultCurrent(brainUpgradeOutcome,reindexTriggered,reindexSpawn,+ freshen,skipped: "",errors,};diff --git a/src/core/maintenance/o2b-command.ts b/src/core/maintenance/o2b-command.tsnew file mode 100644index 00000000..59c16866--- /dev/null+++ b/src/core/maintenance/o2b-command.ts@@ -0,0 +1,19 @@+import { dirname, join, resolve } from "node:path";+import { fileURLToPath } from "node:url";++/**+ * argv prefix that runs this checkout's CLI (current plugin version), for+ * the detached children the self-heal and freshen paths start.+ *+ * POSIX goes through `scripts/o2b`, which also applies the macOS SQLite+ * setup. Native Windows cannot execute that bash launcher, so it runs the+ * TypeScript entry point with the Bun that is running this process.+ */+export function o2bCommand(): string[] {+ // src/core/maintenance/o2b-command.ts -> repo root+ const repo = resolve(dirname(fileURLToPath(import.meta.url)), "..", "..", "..");+ if (process.platform === "win32") {+ return [process.execPath, "run", join(repo, "src", "cli", "main.ts")];+ }+ return [join(repo, "scripts", "o2b")];+}diff --git a/src/core/search/freshen.ts b/src/core/search/freshen.tsnew file mode 100644index 00000000..74d4ef62--- /dev/null+++ b/src/core/search/freshen.ts@@ -0,0 +1,533 @@+/**+ * Freshen on read (index-freshness design): a reader that finds the search+ * index older than `search_freshen_interval_s` answers from the index it has+ * and starts ONE detached, low-priority incremental index run, so the next+ * read sees the current vault. Nothing stays resident: the only triggers are+ * reads (`search()`) and session start (`ensureVaultCurrent`), both of which+ * only happen while an agent works.+ *+ * Everything here is per device and lives next to the index+ * (`<dirname(dbPath)>/`), never in `Brain/`: the claim that keeps concurrent+ * readers from starting a run each, and the state file the child writes its+ * outcome and backoff to.+ *+ * `maybeFreshenIndex` never throws and never waits for the run: a reader+ * of a fresh index pays nothing; a due one pays one state-file read, one+ * lock probe and one exclusive create.+ */+import { randomUUID } from "node:crypto";+import { readFileSync, renameSync, statSync, unlinkSync, writeFileSync } from "node:fs";+import { dirname, join } from "node:path";++import { appendMetric } from "../brain/metrics.ts";+import { atomicWriteFileSync } from "../fs-atomic.ts";+import { o2bCommand } from "../maintenance/o2b-command.ts";+import { isWriterLockHeld } from "./store/writer-lock.ts";+import type { ResolvedSearchConfig } from "./types.ts";++/** The claim a reader takes before it starts a run; the child releases it. */+export const FRESHEN_CLAIM_FILE = "freshen.claim";++/** The child's outcome, failure streak and backoff. */+export const FRESHEN_STATE_FILE = "freshen-state.json";++/**+ * Metrics surface for background runs. A row is written only when a run+ * changed the index or failed: an idle machine would otherwise append one+ * row a minute to a synced file for nothing.+ */+export const INDEX_FRESHEN_SURFACE = "index_freshen";++/** A claim older than this belongs to a run that died; it is taken over. */+export const FRESHEN_CLAIM_ABANDONED_MS = 10 * 60 * 1000;++const BACKOFF_BASE_MS = 60_000;+const BACKOFF_CAP_MS = 3_600_000;++/** Why a reader did not start a run. */+export const FRESHEN_SKIP = Object.freeze({+ off: "off",+ noIndex: "no_index",+ fresh: "fresh",+ backoff: "backoff",+ readOnly: "read_only",+ writerLock: "writer_lock",+ claimed: "claimed",+ /** The index directory refused the claim (read-only mount, no permission). */+ unwritable: "unwritable",+ spawnFailed: "spawn_failed",+} as const);++export type FreshenSkip = (typeof FRESHEN_SKIP)[keyof typeof FRESHEN_SKIP];++/** What a reader did: started a run, or the reason it did not. */+export type FreshenDecision = "spawned" | FreshenSkip;++export interface FreshenState {+ readonly failures: number;+ readonly backoffUntil: string | null;+ readonly lastOutcome: "completed" | "failed" | null;+ readonly lastRunAt: string | null;+ readonly lastDurationMs: number | null;+ readonly lastError: string | null;+ /** Documents the last completed run added, updated or deleted. */+ readonly lastChanged: number | null;+}++export const EMPTY_FRESHEN_STATE: FreshenState = Object.freeze({+ failures: 0,+ backoffUntil: null,+ lastOutcome: null,+ lastRunAt: null,+ lastDurationMs: null,+ lastError: null,+ lastChanged: null,+});++export interface DecideFreshenInput {+ readonly nowMs: number;+ readonly lastIndexedAt: string | null;+ readonly intervalSeconds: number;+ readonly backoffUntilMs: number | null;+}++/**+ * Whether a read should start a run. Pure. An index that was never+ * stamped belongs to self-heal (a missing or unbuilt index is rebuilt+ * there); an unparseable stamp is treated as due, since claiming it fresh+ * would leave the index stale forever. Freshness is decided before the+ * backoff, so a reader can skip the state file while the index is fresh.+ */+export function decideFreshen(+ input: DecideFreshenInput,+): { readonly action: "spawn" } | { readonly action: "skip"; readonly reason: FreshenSkip } {+ if (input.intervalSeconds <= 0) return { action: "skip", reason: FRESHEN_SKIP.off };+ if (input.lastIndexedAt === null) return { action: "skip", reason: FRESHEN_SKIP.noIndex };+ const indexedMs = Date.parse(input.lastIndexedAt);+ if (Number.isFinite(indexedMs) && input.nowMs - indexedMs < input.intervalSeconds * 1000) {+ return { action: "skip", reason: FRESHEN_SKIP.fresh };+ }+ if (input.backoffUntilMs !== null && input.backoffUntilMs > input.nowMs) {+ return { action: "skip", reason: FRESHEN_SKIP.backoff };+ }+ return { action: "spawn" };+}++/** Seconds since `lastIndexedAt`, or null when it is absent or unparseable. */+export function indexAgeSeconds(lastIndexedAt: string | null, nowMs: number): number | null {+ if (lastIndexedAt === null) return null;+ const ms = Date.parse(lastIndexedAt);+ if (!Number.isFinite(ms)) return null;+ return Math.max(0, Math.floor((nowMs - ms) / 1000));+}++/** Backoff after the `failures`-th failure in a row: 1 min doubling to 1 h. */+export function nextBackoffMs(failures: number): number {+ const n = Math.max(1, failures);+ return Math.min(BACKOFF_BASE_MS * 2 ** (n - 1), BACKOFF_CAP_MS);+}++/**+ * A claim file that cannot be parsed is normally one being written: the+ * exclusive create and the write are two steps. Only past this grace is a+ * torn claim taken for a dead run's.+ */+const CLAIM_TORN_GRACE_MS = 10_000;++/** A takeover lock older than this belongs to a reader that died mid-takeover. */+const TAKEOVER_STALE_MS = 30_000;++/** A claim taken, or why it was not. */+export type FreshenClaim =+ | { readonly token: string }+ | { readonly skip: typeof FRESHEN_SKIP.claimed | typeof FRESHEN_SKIP.unwritable };++/**+ * Take the claim, or return null when a live run holds it or the index+ * directory refuses it. {@link tryClaimFreshen} tells the two apart.+ */+export function claimFreshen(dir: string, nowMs: number): string | null {+ const claim = tryClaimFreshen(dir, nowMs);+ return "token" in claim ? claim.token : null;+}++/**+ * Take the claim. The claim is an exclusive create, so of N readers racing+ * on a stale index exactly one wins; a create that fails for any reason+ * other than an existing claim (EROFS, EACCES) is reported as unwritable,+ * since no reader there can ever start a run. A claim older than+ * {@link FRESHEN_CLAIM_ABANDONED_MS} (or torn and older than+ * {@link CLAIM_TORN_GRACE_MS}) is a run that died (a reboot, an OOM kill)+ * and is taken over - but only by the reader holding the exclusive+ * takeover lock, which re-reads the claim under it and replaces it with an+ * atomic rename, so the slot is never empty and a fresh claim is never+ * touched.+ */+export function tryClaimFreshen(dir: string, nowMs: number): FreshenClaim {+ const path = join(dir, FRESHEN_CLAIM_FILE);+ const token = `fr-${randomUUID()}`;+ const body = JSON.stringify({ token, at: nowMs });+ const claimed = { skip: FRESHEN_SKIP.claimed } as const;+ for (let attempt = 0; attempt < 2; attempt++) {+ try {+ writeFileSync(path, body, { flag: "wx" });+ return { token };+ } catch (e) {+ if ((e as NodeJS.ErrnoException).code !== "EEXIST") {+ return { skip: FRESHEN_SKIP.unwritable };+ }+ }+ const seen = readClaimBody(path);+ if (seen === null) continue; // released since: race the create again+ if (!claimAbandoned(path, seen, nowMs)) return claimed;+ const taken = takeOver(path, token, body, nowMs);+ return taken === null ? claimed : { token: taken };+ }+ return claimed;+}++function takeOver(path: string, token: string, body: string, nowMs: number): string | null {+ const lock = `${path}.takeover`;+ if (!createExclusive(lock)) {+ let age = 0;+ try {+ age = nowMs - statSync(lock).mtimeMs;+ } catch {+ return null;+ }+ if (age <= TAKEOVER_STALE_MS) return null;+ unlinkQuietly(lock);+ if (!createExclusive(lock)) return null;+ }+ try {+ const current = readClaimBody(path);+ if (current !== null && !claimAbandoned(path, current, nowMs)) return null;+ const tmp = `${path}.${token}.tmp`;+ writeFileSync(tmp, body);+ renameSync(tmp, path);+ return token;+ } catch {+ return null;+ } finally {+ unlinkQuietly(lock);+ }+}++function createExclusive(path: string): boolean {+ try {+ writeFileSync(path, String(process.pid), { flag: "wx" });+ return true;+ } catch {+ return false;+ }+}++function readClaimBody(path: string): string | null {+ try {+ return readFileSync(path, "utf8");+ } catch {+ return null;+ }+}++function claimAbandoned(path: string, body: string, nowMs: number): boolean {+ try {+ const parsed = JSON.parse(body) as { at?: unknown };+ if (typeof parsed.at === "number") return nowMs - parsed.at > FRESHEN_CLAIM_ABANDONED_MS;+ } catch {+ // Torn or empty: judged by age below.+ }+ try {+ return nowMs - statSync(path).mtimeMs > CLAIM_TORN_GRACE_MS;+ } catch {+ return false;+ }+}++function unlinkQuietly(path: string): void {+ try {+ unlinkSync(path);+ } catch {+ // Already gone.+ }+}++/** Release the claim, but only while it is still ours. */+export function releaseFreshen(dir: string, token: string): void {+ const path = join(dir, FRESHEN_CLAIM_FILE);+ try {+ const parsed = JSON.parse(readFileSync(path, "utf8")) as { token?: unknown };+ if (parsed.token === token) unlinkSync(path);+ } catch {+ // Absent or unreadable: nothing of ours to release.+ }+}++function str(v: unknown): string | null {+ return typeof v === "string" ? v : null;+}++function num(v: unknown): number | null {+ return typeof v === "number" && Number.isFinite(v) ? v : null;+}++/** The state file, or the empty state when it is absent or torn. */+export function readFreshenState(dir: string): FreshenState {+ try {+ const raw = JSON.parse(readFileSync(join(dir, FRESHEN_STATE_FILE), "utf8")) as Record<+ string,+ unknown+ >;+ const outcome = raw["lastOutcome"];+ return Object.freeze({+ failures: num(raw["failures"]) ?? 0,+ backoffUntil: str(raw["backoffUntil"]),+ lastOutcome: outcome === "completed" || outcome === "failed" ? outcome : null,+ lastRunAt: str(raw["lastRunAt"]),+ lastDurationMs: num(raw["lastDurationMs"]),+ lastError: str(raw["lastError"]),+ lastChanged: num(raw["lastChanged"]),+ });+ } catch {+ return EMPTY_FRESHEN_STATE;+ }+}++/** Write the state file atomically. */+export function writeFreshenState(dir: string, state: FreshenState): void {+ atomicWriteFileSync(join(dir, FRESHEN_STATE_FILE), JSON.stringify(state, null, 2) + "\n");+}++/**+ * Prefix the child's argv so it runs at idle I/O priority where the+ * platform has a tool for it; the child lowers its own CPU priority.+ */+export function freshenCommand(+ base: ReadonlyArray<string>,+ has: (tool: string) => boolean,+ platform: NodeJS.Platform,+): string[] {+ if (platform === "linux" && has("ionice")) return ["ionice", "-c3", ...base];+ if (platform === "darwin" && has("taskpolicy")) return ["taskpolicy", "-b", ...base];+ return [...base];+}++/**+ * The options the freshen child is spawned with. Detached on every+ * platform: a non-detached child dies with its parent on Windows, and on+ * POSIX it stays in the reader's process group, so a short-lived hook+ * whose group is signalled on exit would take the run down with it.+ */+export function freshenSpawnOptions(env: NodeJS.ProcessEnv): {+ readonly stdin: "ignore";+ readonly stdout: "ignore";+ readonly stderr: "ignore";+ readonly windowsHide: true;+ readonly detached: true;+ readonly env: Record<string, string | undefined>;+} {+ return {+ stdin: "ignore",+ stdout: "ignore",+ stderr: "ignore",+ windowsHide: true,+ detached: true,+ // Bun gives a child spawned without `env` the environment this process+ // STARTED with; a config chosen after start must reach the child.+ env: { ...env },+ };+}++function spawnDetached(argv: string[]): void {+ const proc = Bun.spawn(argv, freshenSpawnOptions(process.env));+ proc.unref();+}++export interface MaybeFreshenOptions {+ readonly lastIndexedAt: string | null;+ /** A read-only open (cross-vault, recall sources): never write there. */+ readonly readOnly?: boolean;+ readonly nowMs?: number;+ /** Test seam: replaces the detached spawn. */+ readonly spawn?: (argv: string[]) => void;+}++/**+ * Start a background incremental index run when the index is stale.+ * Never throws; returns what it did.+ */+export function maybeFreshenIndex(+ config: ResolvedSearchConfig,+ opts: MaybeFreshenOptions,+): FreshenDecision {+ try {+ const freshen = config.freshen;+ if (freshen === undefined || freshen.intervalSeconds <= 0) return FRESHEN_SKIP.off;+ if (opts.readOnly === true) return FRESHEN_SKIP.readOnly;+ const nowMs = opts.nowMs ?? Date.now();+ const dir = dirname(config.dbPath);+ const input = {+ nowMs,+ lastIndexedAt: opts.lastIndexedAt,+ intervalSeconds: freshen.intervalSeconds,+ };+ // Every search lands here: decide on the index stamp alone first and+ // read the state file for the backoff only once the index is due.+ const due = decideFreshen({ ...input, backoffUntilMs: null });+ if (due.action === "skip") return due.reason;+ const backoffUntil = readFreshenState(dir).backoffUntil;+ const backoffMs = backoffUntil === null ? Number.NaN : Date.parse(backoffUntil);+ const decision = decideFreshen({+ ...input,+ backoffUntilMs: Number.isFinite(backoffMs) ? backoffMs : null,+ });+ if (decision.action === "skip") return decision.reason;+ if (isWriterLockHeld(config.dbPath)) return FRESHEN_SKIP.writerLock;+ const claim = tryClaimFreshen(dir, nowMs);+ if (!("token" in claim)) return claim.skip;+ const token = claim.token;+ const argv = freshenCommand(+ [+ ...o2bCommand(),+ "search",+ "index",+ "--vault",+ config.vault,+ ...(freshen.configPath !== null ? ["--config", freshen.configPath] : []),+ // Where the claim and the state live, so a child that fails before+ // it can resolve its config can still record it and release.+ "--freshen-state",+ dir,+ "--freshen",+ token,+ ],+ (tool) => Bun.which(tool) !== null,+ process.platform,+ );+ try {+ (opts.spawn ?? spawnDetached)(argv);+ } catch {+ releaseFreshen(dir, token);+ return FRESHEN_SKIP.spawnFailed;+ }+ return "spawned";+ } catch {+ // A reader must never fail because freshening could not decide.+ return FRESHEN_SKIP.spawnFailed;+ }+}++export interface FreshenOutcome {+ readonly outcome: "completed" | "failed";+ readonly durationMs: number;+ /** Documents added, updated or deleted (completed runs). */+ readonly changed?: number;+ readonly error?: string;+}++/**+ * What the background child records when it ends: the state file (outcome,+ * streak, backoff) and, when it changed something or failed, one metric+ * row. Never throws: a run must not fail because its bookkeeping did.+ */+export function recordFreshenOutcome(+ vault: string | null,+ dir: string,+ result: FreshenOutcome,+ nowMs: number = Date.now(),+): void {+ const previous = readFreshenState(dir);+ const failed = result.outcome === "failed";+ const failures = failed ? previous.failures + 1 : 0;+ const runAt = new Date(nowMs).toISOString();+ try {+ writeFreshenState(dir, {+ failures,+ backoffUntil: failed ? new Date(nowMs + nextBackoffMs(failures)).toISOString() : null,+ lastOutcome: result.outcome,+ lastRunAt: runAt,+ lastDurationMs: result.durationMs,+ lastError: failed ? (result.error ?? "failed") : null,+ lastChanged: failed ? null : (result.changed ?? 0),+ });+ } catch {+ // Best effort, like the self-heal outcome rows.+ }+ if (vault === null || (!failed && (result.changed ?? 0) === 0)) return;+ try {+ appendMetric(vault, {+ surface: INDEX_FRESHEN_SURFACE,+ runAt,+ payload: {+ outcome: result.outcome,+ duration_ms: result.durationMs,+ ...(failed ? { error: result.error ?? "failed", failures } : {}),+ ...(!failed ? { changed: result.changed ?? 0 } : {}),+ },+ });+ } catch {+ // Same: metrics are diagnostics, never a reason to fail.+ }+}++/** What `o2b search status` and the doctor show about freshen on read. */+export interface FreshenStatus {+ /** 0 when freshening is off. */+ readonly intervalSeconds: number;+ readonly indexAgeSeconds: number | null;+ readonly state: FreshenState;+ /** The backoff end, only while it is still in the future. */+ readonly activeBackoffUntil: string | null;+}++export function freshenStatus(+ config: ResolvedSearchConfig,+ lastIndexedAt: string | null,+ nowMs: number = Date.now(),+): FreshenStatus {+ const state = readFreshenState(dirname(config.dbPath));+ const backoffMs = state.backoffUntil === null ? Number.NaN : Date.parse(state.backoffUntil);+ return Object.freeze({+ intervalSeconds: config.freshen?.intervalSeconds ?? 0,+ indexAgeSeconds: indexAgeSeconds(lastIndexedAt, nowMs),+ state,+ activeBackoffUntil: Number.isFinite(backoffMs) && backoffMs > nowMs ? state.backoffUntil : null,+ });+}++/** The status lines, aligned with the rest of `o2b search status`. */+export function renderFreshenStatus(s: FreshenStatus): string[] {+ const lines = [+ `freshen: ${s.intervalSeconds > 0 ? `every ${s.intervalSeconds}s` : "off"}`,+ `index_age: ${s.indexAgeSeconds === null ? "(unknown)" : `${s.indexAgeSeconds}s`}`,+ ];+ const st = s.state;+ if (st.lastOutcome === "completed") {+ lines.push(`last_freshen: completed ${st.lastRunAt} (${st.lastChanged ?? 0} changed)`);+ } else if (st.lastOutcome === "failed") {+ lines.push(+ `last_freshen: failed ${st.lastRunAt} (${st.failures} in a row): ${st.lastError}`,+ );+ } else {+ lines.push("last_freshen: (none)");+ }+ // The label stays within the 19 columns every status label fits in.+ if (s.activeBackoffUntil !== null) {+ lines.push(`freshen_backoff: until ${s.activeBackoffUntil}`);+ }+ return lines;+}++/** The `freshen` object of `o2b search status --json`. */+export function serializeFreshenStatus(s: FreshenStatus): Record<string, unknown> {+ return {+ interval_s: s.intervalSeconds,+ index_age_s: s.indexAgeSeconds,+ last_outcome: s.state.lastOutcome,+ last_run_at: s.state.lastRunAt,+ last_changed: s.state.lastChanged,+ last_error: s.state.lastError,+ failures: s.state.failures,+ backoff_until: s.activeBackoffUntil,+ };+}diff --git a/src/core/search/index.ts b/src/core/search/index.tsindex ef8c320e..4405bdfd 100644--- a/src/core/search/index.ts+++ b/src/core/search/index.ts@@ -41,6 +41,7 @@ import {import { SearchError } from "./types.ts";import type {ResolvedEmbeddingConfig,+ ResolvedFreshenConfig,ResolvedRecallConfig,ResolvedRerankConfig,ResolvedSearchConfig,@@ -71,6 +72,7 @@ export type {IndexStatusSnapshot,PendingVectorCensus,ResolvedEmbeddingConfig,+ ResolvedFreshenConfig,ResolvedRecallConfig,ResolvedRerankConfig,ResolvedSearchConfig,@@ -1377,6 +1379,31 @@ export function resolveSearchConfig(opts: {trigramPrefilterMaxSelectivity,});+ const freshen: ResolvedFreshenConfig = Object.freeze({+ intervalSeconds: parseInteger(+ envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S",+ "search_freshen_interval_s",+ ),+ 60,+ "search_freshen_interval_s",+ { min: 0 },+ ),+ embeddings: parseBool(+ envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_SEARCH_FRESHEN_EMBEDDINGS",+ "search_freshen_embeddings",+ ),+ false,+ "search_freshen_embeddings",+ ),+ configPath: opts.configPath ?? null,+ });+const base: ResolvedSearchConfig = Object.freeze({vault: opts.vault,dbPath,@@ -1395,6 +1422,7 @@ export function resolveSearchConfig(opts: {resumeReindex,hybridDeadlineMs,ftsTokenize,+ freshen,});// The env variable names an extra-body refusal only when its value isdiff --git a/src/core/search/indexer.ts b/src/core/search/indexer.tsindex a60cb3e7..bf4a0c19 100644--- a/src/core/search/indexer.ts+++ b/src/core/search/indexer.ts@@ -104,6 +104,7 @@ import {EMBEDDING_PREFIX_PASSAGE_STATE_KEY,LAST_FULL_INDEX_AT_STATE_KEY,LAST_INDEXED_AT_STATE_KEY,+ LINK_RESOLUTION_PENDING_STATE_KEY,} from "./store.ts";import { LATEST_SCHEMA_VERSION } from "./schema.ts";import { chunkWindowDiagnosticCode, SearchError } from "./types.ts";@@ -187,6 +188,7 @@ interface MutableStats {relationViolations: IndexStats["relationViolations"];tierDrift: IndexStats["tierDrift"];aliasResolved: number;+ linkResolutionSkipped: boolean;eventAnchorsPending: number;backend: IndexStats["backend"];deferredReason: IndexStats["deferredReason"];@@ -216,6 +218,7 @@ function newStats(): MutableStats {relationViolations: [],tierDrift: [],aliasResolved: 0,+ linkResolutionSkipped: false,// Counted at the end of the run, once every changed document has// been upserted: what is left is what this run did not reach.eventAnchorsPending: 0,@@ -257,6 +260,7 @@ function freezeStats(s: MutableStats, durationMs: number): IndexStats {relationViolations: Object.freeze([...s.relationViolations]),tierDrift: Object.freeze([...s.tierDrift]),aliasResolved: s.aliasResolved,+ linkResolutionSkipped: s.linkResolutionSkipped,eventAnchorsPending: s.eventAnchorsPending,backend: s.backend,deferredReason: s.deferredReason,@@ -455,6 +459,12 @@ async function indexIntoRun(const ownsStore = !storeOverride;const store = storeOverride ?? (await Store.open(config, { mode: "write" }));const stats = newStats();+ let linkResolutionMarked = false;+ const markLinkResolutionPending = (): void => {+ if (linkResolutionMarked) return;+ linkResolutionMarked = true;+ store.setState(LINK_RESOLUTION_PENDING_STATE_KEY, "1");+ };try {const existing = store.listDocuments();@@ -518,6 +528,7 @@ async function indexIntoRun(continue;}+ markLinkResolutionPending();const filenameBase = basename(file.relPath, ".md");const chunkResult = chunkMarkdown(content, filenameBase, {maxTokens: config.chunkSize,@@ -610,8 +621,8 @@ async function indexIntoRun(tokenCount: c.tokenCount,headingPath: c.headingPath,}));- // An unchanged chunk keeps its stored vector (vector carry-over),- // so the embedding phase below never re-pays for it.+ // An unchanged chunk is kept in place, and a moved one carries its+ // stored vector, so the embedding phase below never re-pays for it.const replaced = store.replaceDocumentChunks(docId, chunkInputs);const chunkIds = replaced.chunkIds;stats.embeddingsReused += replaced.embeddingsReused;@@ -630,8 +641,11 @@ async function indexIntoRun(});}// Entity-boosted retrieval (v0.13.0): persist the chunk's- // deterministic entity set alongside its links.- store.replaceEntities(cid, extractEntities(content));+ // deterministic entity set alongside its links. A chunk kept in+ // place has the entities of the content it still holds - so a+ // release that changes `extractEntities` must bump CHUNKER_VERSION,+ // which rebuilds every chunk, or kept rows keep the old entities.+ if (!replaced.keptChunkIds.has(cid)) store.replaceEntities(cid, extractEntities(content));}// Typed graph semantics (v3): frontmatter relation fields// (related / extends / contradicts / superseded_by) become typed@@ -666,16 +680,35 @@ async function indexIntoRun(for (const [path] of existing) {if (!seen.has(path)) {+ markLinkResolutionPending();store.deleteDocument(path);stats.deleted++;opts?.onFile?.({ path, kind: "deleted" });}}- store.resolveLinkTargets();- // Alias post-pass (v7): exact path matches above always win; this- // only fills still-unresolved slash-free targets from doc_aliases.- stats.aliasResolved = store.resolveAliasTargets();+ // Link and alias resolution are a pure function of the documents and+ // their links. A run that changed no document (and was not forced)+ // would only rewrite every link row to the value it already holds -+ // the alias pass resets and re-sets each alias-resolved link - which+ // is what made a no-change run write to disk. Skipped then; the+ // constraint and tier passes below still run.+ // A run killed between its document writes and this point leaves the+ // pending marker set, and the next run resolves even if it changes+ // nothing itself.+ if (+ stats.added + stats.updated + stats.deleted === 0 &&+ opts?.force !== true &&+ store.getState(LINK_RESOLUTION_PENDING_STATE_KEY) !== "1"+ ) {+ stats.linkResolutionSkipped = true;+ } else {+ store.resolveLinkTargets();+ // Alias post-pass (v7): exact path matches above always win; this+ // only fills still-unresolved slash-free targets from doc_aliases.+ stats.aliasResolved = store.resolveAliasTargets();+ store.setState(LINK_RESOLUTION_PENDING_STATE_KEY, "0");+ }// Link-constraint materialization post-pass// (write-time-integrity-governance): recompute every typed edge'sdiff --git a/src/core/search/pipeline/outcome.ts b/src/core/search/pipeline/outcome.tsindex ab5a8742..d47c832a 100644--- a/src/core/search/pipeline/outcome.ts+++ b/src/core/search/pipeline/outcome.ts@@ -19,7 +19,12 @@ import {type TransportReach,} from "../../graph/transport-reach.ts";import { buildEvidencePack, downrankTerminalEvidenceResults } from "../evidence-pack.ts";-import { buildRetrievalTrail, corpusStatementFor } from "../retrieval-trail.ts";+import {+ buildRetrievalTrail,+ corpusStatementFor,+ INDEX_STALE_SECONDS,+ RETRIEVAL_DEGRADATION,+} from "../retrieval-trail.ts";import { buildEvidenceVerification, coverageOverResults } from "../evidence-verification.ts";import { fnv1aHex } from "../feedback.ts";import {@@ -251,6 +256,40 @@ export function buildSearchOutcome(input: OutcomeInput): SearchOutcome {});}+/**+ * Name a stale index on an outcome's trail, at read time. The code is not+ * part of the computed answer: the query cache stores outcomes without it+ * (an index run that changes nothing keeps the cache generation, so a+ * stored code would outlive the staleness it reports), and every answer,+ * hit or miss, gains it here from the index age it is served at. The+ * result is the trail the pipeline would have built with the code in its+ * sink: the code leads, and a corpus statement gives way to it.+ */+export function withIndexStale(outcome: SearchOutcome, ageSeconds: number | null): SearchOutcome {+ if (ageSeconds === null || ageSeconds <= INDEX_STALE_SECONDS) return outcome;+ const stale = Object.freeze({+ code: RETRIEVAL_DEGRADATION.indexStale,+ detail: Object.freeze({ ageSeconds }),+ });+ const trail = outcome.retrievalTrail;+ const retrievalTrail = Object.freeze({+ retrieved: trail?.retrieved ?? outcome.cards?.length ?? outcome.results.length,+ pool: trail?.pool ?? outcome.total,+ degraded: Object.freeze([stale, ...(trail?.degraded ?? [])]),+ });+ if (trail !== undefined) return Object.freeze({ ...outcome, retrievalTrail });+ // A healthy answer has no trail; it goes where the outcome builders put+ // it, right after the match quality, so the serialized shape matches a+ // freshly built degraded answer.+ const out: Record<string, unknown> = {};+ for (const [key, value] of Object.entries(outcome)) {+ out[key] = value;+ if (key === "idfWeightedCoverage") out["retrievalTrail"] = retrievalTrail;+ }+ if (!("retrievalTrail" in out)) out["retrievalTrail"] = retrievalTrail;+ return Object.freeze(out) as unknown as SearchOutcome;+}+/*** Both root sets unresolved. Handed to the classifier when the index facts* could not be read at all, which it reports asdiff --git a/src/core/search/retrieval-trail.ts b/src/core/search/retrieval-trail.tsindex c41ac501..11ee87fc 100644--- a/src/core/search/retrieval-trail.ts+++ b/src/core/search/retrieval-trail.ts@@ -65,7 +65,20 @@ import type {* are NOT invented for conditions nothing reports: a code with no* producer would be a promise the search path does not keep.*/+/** How old the index may be before a search names it on the trail. */+export const INDEX_STALE_SECONDS = 600;+export const RETRIEVAL_DEGRADATION = Object.freeze({+ /**+ * The index was last updated more than {@link INDEX_STALE_SECONDS} ago, so+ * notes changed since then may be missing from the answer. Detail:+ * `ageSeconds`. The code reports the age only, not that a refresh is+ * running: freshen on read starts a background run when it can and+ * starts none when it is off, backing off, blocked by another run or+ * writer, read-only for this reader, or unable to spawn. Added at read time (`withIndexStale`), never+ * stored in the query cache.+ */+ indexStale: "index-stale",/*** The query tokenised to an empty FTS match, so the keyword lane never* ran. The purest silent empty in the tree: `fts.ts` returned no hits@@ -193,6 +206,7 @@ export type RetrievalDegradationCode =/** Membership list, ordered lane by lane along the pipeline. */export const RETRIEVAL_DEGRADATION_CODES: ReadonlyArray<RetrievalDegradationCode> = Object.freeze([+ RETRIEVAL_DEGRADATION.indexStale,RETRIEVAL_DEGRADATION.keywordFtsMatchEmpty,RETRIEVAL_DEGRADATION.keywordTrigramLaneFault,RETRIEVAL_DEGRADATION.semanticEmbeddingsAbsent,@@ -354,6 +368,8 @@ export function semanticLaneMissing(degraded: ReadonlyArray<RetrievalDegradation*/export function describeRetrievalDegradation(code: RetrievalDegradationCode): string {switch (code) {+ case RETRIEVAL_DEGRADATION.indexStale:+ return "the index was last updated more than ten minutes ago, so notes changed since then may be missing";case RETRIEVAL_DEGRADATION.keywordFtsMatchEmpty:return "the query carried no term the keyword index can match, so the keyword lane never ran";case RETRIEVAL_DEGRADATION.keywordTrigramLaneFault:diff --git a/src/core/search/search.ts b/src/core/search/search.tsindex b4a9eb1d..844a8fd6 100644--- a/src/core/search/search.ts+++ b/src/core/search/search.ts@@ -22,6 +22,7 @@ import {buildSearchOutcome,corpusStatementForEmptyWindow,emptyOutcome,+ withIndexStale,} from "./pipeline/outcome.ts";import { applyPostRankPhases } from "./pipeline/post-rank.ts";import { resolveQueryShape } from "./pipeline/query-shape.ts";@@ -40,6 +41,8 @@ import { isAbortError } from "./embeddings/http-util.ts";import type { CacheProbe } from "./pipeline/cache-slot.ts";import type { RetrievalDegradationSink } from "./retrieval-trail.ts";import { RETRIEVAL_DEGRADATION, noteDegradation } from "./retrieval-trail.ts";+import { indexAgeSeconds, maybeFreshenIndex } from "./freshen.ts";+import { LAST_INDEXED_AT_STATE_KEY } from "./store/state.ts";import type { FrontmatterCache } from "./result-filters.ts";import { Store } from "./store.ts";import type { ResolvedSearchConfig, SearchOptions, SearchOutcome } from "./types.ts";@@ -233,6 +236,20 @@ export async function search(// caller can tell an exhausted corpus from a broken embedder without// matching on prose.const degraded: RetrievalDegradationSink = [];+ // Freshen on read: an index older than the interval starts one+ // background incremental run (never into a read-only origin); this+ // answer comes from the index as it is. A badly stale index is named+ // on the trail whether or not a run could start, at read time on every+ // answer, cache hits included, and never in the cached row.+ const lastIndexedAt = store.getState(LAST_INDEXED_AT_STATE_KEY);+ maybeFreshenIndex(effectiveConfig, {+ lastIndexedAt,+ readOnly: opts.selfHeal === false,+ nowMs,+ ...(opts.freshenSpawn !== undefined ? { spawn: opts.freshenSpawn } : {}),+ });+ const indexAge = indexAgeSeconds(lastIndexedAt, nowMs);+ const served = (outcome: SearchOutcome): SearchOutcome => withIndexStale(outcome, indexAge);// Shared across every frontmatter-reading stage below (Plan 1, 1.3)// so a candidate path already read by one stage is not re-read// and re-parsed by the next.@@ -281,7 +298,7 @@ export async function search(planHash: shape.basePlan.planHash,}): CACHE_BYPASSED;- if (cache.hit !== null) return cache.hit;+ if (cache.hit !== null) return served(cache.hit);// A deadline-degraded answer is keyword-only under a key that promises// the hybrid one: serve it, but never cache it. A spend-gated one is// the same, and the key carries no gate or price either, so a cached@@ -411,17 +428,19 @@ export async function search(degraded,opts.transportReach,);- return finalize(- emptyOutcome({- store,- opts,- query,- pathPrefix,- warnings,- routedSurface,- degraded,- corpus,- }),+ return served(+ finalize(+ emptyOutcome({+ store,+ opts,+ query,+ pathPrefix,+ warnings,+ routedSurface,+ degraded,+ corpus,+ }),+ ),);}@@ -537,26 +556,28 @@ export async function search();const emit =postRank.decisionFallback === true || rerankUnavailable ? (o: SearchOutcome) => o : finalize;- return emit(- buildSearchOutcome({- store,- config: effectiveConfig,- opts,- query,- pathPrefix,- results,- warnings,- secondPass: retry.secondPass,- routedSurface,- trustReceipts: postRank.trustReceipts,- frontmatterCache,- poolSize,- degraded,- corpus,- ...(postRank.decisionModel?.answerable !== undefined- ? { decisionModel: { answerable: postRank.decisionModel.answerable } }- : {}),- }),+ return served(+ emit(+ buildSearchOutcome({+ store,+ config: effectiveConfig,+ opts,+ query,+ pathPrefix,+ results,+ warnings,+ secondPass: retry.secondPass,+ routedSurface,+ trustReceipts: postRank.trustReceipts,+ frontmatterCache,+ poolSize,+ degraded,+ corpus,+ ...(postRank.decisionModel?.answerable !== undefined+ ? { decisionModel: { answerable: postRank.decisionModel.answerable } }+ : {}),+ }),+ ),);} finally {await store.close();diff --git a/src/core/search/store.ts b/src/core/search/store.tsindex 11c302e3..d96b2f31 100644--- a/src/core/search/store.ts+++ b/src/core/search/store.ts@@ -65,6 +65,7 @@ export {INDEX_REVISION_STATE_KEY,LAST_FULL_INDEX_AT_STATE_KEY,LAST_INDEXED_AT_STATE_KEY,+ LINK_RESOLUTION_PENDING_STATE_KEY,peekCorpusGenerationSync,readCorpusGenerationSync,} from "./store/state.ts";diff --git a/src/core/search/store/chunks.ts b/src/core/search/store/chunks.tsindex c207b9a3..b80491a1 100644--- a/src/core/search/store/chunks.ts+++ b/src/core/search/store/chunks.ts@@ -126,25 +126,48 @@ export function getChunksByDocument(db: Database, documentId: number): ChunkRow[/** What one document's chunk replacement wrote. */export interface ChunkReplacement {- /** The new chunk ids, in `chunkIndex` order. */+ /** The chunk ids, in `chunkIndex` order: kept rows keep theirs. */readonly chunkIds: number[];/**- * New chunks that kept a stored vector of an old chunk with the same- * `content_hash` (vector carry-over), so the embedding phase will not- * pay for them again.+ * Chunks that kept a stored vector: kept in place with a vector of the+ * recorded identity, or new rows that carried the vector of an old chunk+ * with the same `content_hash` (vector carry-over). The embedding phase+ * will not pay for them again.*/readonly embeddingsReused: number;+ /**+ * Ids of the chunks kept in place: same position, same content, same+ * lines and heading. Nothing derived from their content (full-text row,+ * entities, vector) needs to be written again.+ */+ readonly keptChunkIds: ReadonlySet<number>;+}++interface StoredChunk {+ id: number;+ chunk_index: number;+ content_hash: string;+ start_line: number;+ end_line: number;+ token_count: number;+ heading_path: string;+ fts_content: string;}/**- * Atomically replace every chunk for a document, carrying the stored- * vector of every new chunk whose content an old chunk of this document- * already embedded under the recorded model and dimension (the rule and- * its guard live in `vector-carry-over.ts`). Only the vec rows that are- * not carried are purged; the old `chunk_vec_map` and `embeddings` rows- * go with their chunks through the FK cascade, and the carried ones are- * re-attached to the new ids. FTS5 stays in sync via the- * chunks_ai/ad/au triggers.+ * Replace a document's chunks, rewriting only the ones that changed.+ *+ * A chunk at the same `chunkIndex` with the same content hash, lines,+ * token count, heading and full-text form is KEPT: its row, id, full-text row, entities+ * and vector stay as they are (a kept vector of another embedding+ * identity is dropped, as a replaced one would be). Every other old chunk+ * is deleted and every other new chunk inserted, carrying the stored+ * vector of an old chunk with the same content where the identity allows+ * (the rule and its guard live in `vector-carry-over.ts`). A daily log+ * that grows all day therefore costs its appended tail per run, not the+ * whole file. The old `chunk_vec_map` and `embeddings` rows of deleted+ * chunks go with them through the FK cascade; FTS5 stays in sync through+ * the chunks_ai/ad/au triggers.*/export function replaceDocumentChunks(db: Database,@@ -154,18 +177,63 @@ export function replaceDocumentChunks(): ChunkReplacement {const ids: number[] = [];let embeddingsReused = 0;+ const kept = new Set<number>();db.exec("BEGIN");try {+ const stored = db+ .query<StoredChunk, [number]>(+ "SELECT id, chunk_index, content_hash, start_line, end_line, token_count, heading_path, " ++ "fts_content FROM chunks WHERE document_id = ?",+ )+ .all(documentId);+ const byIndex = new Map(stored.map((r) => [r.chunk_index, r]));+ const keepAt = chunks.map((c) => {+ const old = byIndex.get(c.chunkIndex);+ const same =+ old !== undefined &&+ old.content_hash === c.contentHash &&+ old.start_line === c.startLine &&+ old.end_line === c.endLine &&+ old.token_count === c.tokenCount &&+ old.heading_path === (c.headingPath ?? "") &&+ // The stored full-text form is derived from the content; a release+ // that derives it differently must not leave kept rows behind.+ old.fts_content === (c.ftsContent ?? c.content);+ if (same) kept.add(old.id);+ return same ? old.id : null;+ });++ const candidates = readCarryCandidates(db, vecLoaded, documentId);+ // A kept chunk whose vector is not of the recorded identity loses it,+ // exactly as a replaced chunk's vector would not be carried.+ const validKept = new Set(candidates.filter((c) => kept.has(c.chunkId)).map((c) => c.chunkId));+ const keptStale = [...kept].filter((id) => !validKept.has(id));+ if (keptStale.length > 0) {+ purgeVecRowsByChunkIds(db, vecLoaded, keptStale);+ const placeholders = sqlPlaceholders(keptStale);+ db.run(`DELETE FROM chunk_vec_map WHERE chunk_id IN (${placeholders})`, keptStale);+ db.run(`DELETE FROM embeddings WHERE chunk_id IN (${placeholders})`, keptStale);+ }++ const newPositions: number[] = [];+ keepAt.forEach((id, position) => {+ if (id === null) newPositions.push(position);+ });const carried = matchCarriedVectors(- readCarryCandidates(db, vecLoaded, documentId),- chunks.map((c) => c.contentHash),+ candidates.filter((c) => !kept.has(c.chunkId)),+ newPositions.map((p) => chunks[p]!.contentHash),+ ).map((m) => ({ position: newPositions[m.position]!, candidate: m.candidate }));+ const carriedOld = new Set(carried.map((m) => m.candidate.chunkId));+ const removed = stored.map((r) => r.id).filter((id) => !kept.has(id));+ purgeVecRowsByChunkIds(+ db,+ vecLoaded,+ removed.filter((id) => !carriedOld.has(id)),);- const keptChunkIds = new Set(carried.map((m) => m.candidate.chunkId));- const purged = chunksForDocument(db, documentId)- .map((c) => c.id)- .filter((id) => !keptChunkIds.has(id));- purgeVecRowsByChunkIds(db, vecLoaded, purged);- db.run("DELETE FROM chunks WHERE document_id = ?", [documentId]);+ if (removed.length > 0) {+ db.run(`DELETE FROM chunks WHERE id IN (${sqlPlaceholders(removed)})`, removed);+ }+const insert = db.prepare<{ id: number },[number, number, string, string, string, number, number, number, string, string, string]@@ -174,7 +242,12 @@ export function replaceDocumentChunks("VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id",);const now = nowIso();- for (const c of chunks) {+ chunks.forEach((c, position) => {+ const keptId = keepAt[position];+ if (keptId !== null && keptId !== undefined) {+ ids.push(keptId);+ return;+ }const row = insert.get(documentId,c.chunkIndex,@@ -190,15 +263,15 @@ export function replaceDocumentChunks();if (!row) throw new SearchError("INDEX_UNREADABLE", "chunk insert returned no id");ids.push(row.id);- }+ });restoreCarriedVectors(db, ids, carried);- embeddingsReused = carried.length;+ embeddingsReused = validKept.size + carried.length;db.exec("COMMIT");} catch (e) {db.exec("ROLLBACK");throw e;}- return { chunkIds: ids, embeddingsReused };+ return { chunkIds: ids, embeddingsReused, keptChunkIds: kept };}/**diff --git a/src/core/search/store/state.ts b/src/core/search/store/state.tsindex 7dca0ae7..fedcc9fd 100644--- a/src/core/search/store/state.ts+++ b/src/core/search/store/state.ts@@ -53,6 +53,14 @@ export const EMBEDDING_VEC_VERSION_STATE_KEY = "embedding_vec_version";* dangling count (context-integrity-gates, unit G).*/export const LAST_INDEXED_AT_STATE_KEY = "last_indexed_at";++/**+ * "1" from the moment an index run writes its first document change until+ * it has resolved links and aliases; "0" after. A run killed in between+ * leaves "1", so the next run resolves links even when it changes nothing+ * itself (index-freshness: a no-change run otherwise skips resolution).+ */+export const LINK_RESOLUTION_PENDING_STATE_KEY = "link_resolution_pending";export const LAST_FULL_INDEX_AT_STATE_KEY = "last_full_index_at";/**diff --git a/src/core/search/types.ts b/src/core/search/types.tsindex 5e46e6c9..15666045 100644--- a/src/core/search/types.ts+++ b/src/core/search/types.ts@@ -313,6 +313,12 @@ export interface IndexStats {* (link-recall-intelligence, v7).*/readonly aliasResolved: number;+ /**+ * True when the run added, updated and deleted no document and was not+ * forced, so the link and alias resolution passes - a pure function of+ * the documents and their links - were skipped (index-freshness).+ */+ readonly linkResolutionSkipped: boolean;/*** Backend that processed this run, resolved lazily after content* detection (offline code-only extraction, t_85252236). `"offline"`@@ -1046,6 +1052,12 @@ export interface SearchOptions {* index surfaces as a per-origin warning instead.*/readonly selfHeal?: boolean;+ /**+ * Test seam for freshen-on-read: replaces the detached spawn of the+ * background index run a stale index starts. Production callers leave+ * it unset.+ */+ readonly freshenSpawn?: (argv: string[]) => void;/*** Skip rerank kind `decision-model` for this call. Set by the hook* surfaces (recall inject), whose time budget is shorter than a decision@@ -1542,6 +1554,16 @@ export interface ResolvedRecallConfig {readonly metadataBoostGateEnabled?: boolean;}+/** What freshen-on-read does; see `src/core/search/freshen.ts`. */+export interface ResolvedFreshenConfig {+ /** Seconds the index may age before a read refreshes it; 0 turns it off. */+ readonly intervalSeconds: number;+ /** Whether the background run also computes embeddings (paid). */+ readonly embeddings: boolean;+ /** The config file the background run must resolve, or null for the default. */+ readonly configPath: string | null;+}+export interface ResolvedSearchConfig {readonly vault: string;readonly dbPath: string;@@ -1627,4 +1649,10 @@ export interface ResolvedSearchConfig {* shadow index is untouched by this clause.*/readonly ftsTokenize: string;+ /**+ * Freshen-on-read (index-freshness): how old the index may get before a+ * read starts one background incremental run, and what that run does.+ * Absent on a hand-built config, which reads as off.+ */+ readonly freshen?: ResolvedFreshenConfig;}diff --git a/src/core/state/surfaces.ts b/src/core/state/surfaces.tsindex 98f74149..5bb22679 100644--- a/src/core/state/surfaces.ts+++ b/src/core/state/surfaces.ts@@ -123,6 +123,7 @@ export const STATE_SURFACE_ID = Object.freeze({hookAudit: "hook_audit",hookSessionState: "hook_session_state",selfHealUpgradeMarker: "self_heal_upgrade_marker",+ freshenState: "freshen_state",watchdogAudit: "watchdog_audit",injectFailopenCache: "inject_failopen_cache",aiderContext: "aider_context_artifact",@@ -285,6 +286,7 @@ const MAINTENANCE_LEASE_FILE = "maintenance.sqlite";const MAINTENANCE_JOURNAL_FILE = `${MAINTENANCE_JOURNAL_STEM}.${JSONL_LEDGER_EXT}`;const HOOK_STATE_DIR = "hook-state";const SELF_HEAL_UPGRADE_MARKER_FILE = "self-heal-upgrade.json";+const FRESHEN_STATE_FILE = "freshen-state.json";const INJECT_CACHE_DIR = "inject-cache";const AIDER_CONTEXT_FILE = "aider-context.md";const DREAM_RUNS_DIR = "dream-runs";@@ -593,6 +595,23 @@ export const STATE_SURFACES: ReadonlyArray<StateSurface> = Object.freeze(["drops the failure from `o2b doctor`; the next start simply tries again.",sources: ["src/core/maintenance/self-heal-upgrade-state.ts"],},+ {+ id: STATE_SURFACE_ID.freshenState,+ label: "freshen-on-read state",+ tier: STATE_TIER.derived,+ // Beside the index, wherever the override puts it: `freshen.ts` keeps+ // its state in `dirname(config.dbPath)`.+ derive: (vault, override) => join(dirname(indexPath(vault, override)), FRESHEN_STATE_FILE),+ override_env: SEARCH_DB_ENV,+ override_config_key: SEARCH_DB_CONFIG_KEY,+ carries_memory: false,+ reason:+ "What the last background index run started by a stale read did on this device: its " ++ "outcome, how many documents it changed, the failure streak and when the next run may " ++ "start. Deleting it ends any backoff early and forgets the last outcome; the next stale " ++ "read simply starts a run again.",+ sources: ["src/core/search/freshen.ts"],+ },{id: STATE_SURFACE_ID.watchdogAudit,label: "watchdog audit fallback",diff --git a/src/mcp/brain/health-tools.ts b/src/mcp/brain/health-tools.tsindex 3af76df7..fd8c0b1e 100644--- a/src/mcp/brain/health-tools.ts+++ b/src/mcp/brain/health-tools.ts@@ -30,10 +30,7 @@ import { findingRefs } from "./hygiene-tools.ts";import { everyArtifactRefView } from "../../core/brain/artifact-ref-view.ts";import { reachView } from "../../core/brain/reach-view.ts";import { contextReach } from "../tool-contract.ts";-import { TRANSPORT_REACH } from "../../core/graph/transport-reach.ts";-import { Store } from "../../core/search/store.ts";import { computeTrustVerdict } from "../../core/brain/trust/compute-trust-verdict.ts";-import { TIER_DRIFT_CODE, tierDriftMessage } from "../../core/brain/doctor/store-integrity.ts";/*** One reported issue, as an MCP caller sees it.@@ -70,38 +67,6 @@ const NO_VERIFICATION_DELTA = Object.freeze({missing_evidence: 0,});-/**- * The `tier-drift` warning restated over the drift rows the caller may- * read, the rows `brain_tiers check` lists for it.- *- * The warning carries no path - only the count of every staged row - so- * the reference view cannot judge it, and the count alone moves when a- * page the caller may not read drifts. Below local reach the count is- * taken again over the readable rows; none left drops the warning, as- * for a vault with no drift.- */-async function recountTierDrift<T extends { readonly code: string; readonly message: string }>(- ctx: ServerContext,- warnings: ReadonlyArray<T>,-): Promise<ReadonlyArray<T>> {- if (!warnings.some((w) => w.code === TIER_DRIFT_CODE)) return warnings;- const readable = readableAtContextReach(ctx);- const store = await Store.open(- resolveSearchConfig({ vault: ctx.vault, configPath: ctx.configPath ?? undefined }),- { mode: "read" },- );- let count: number;- try {- count = store.listTierDrift().filter((row) => readable(row.path)).length;- } finally {- await store.close();- }- return warnings.flatMap((w) => {- if (w.code !== TIER_DRIFT_CODE) return [w];- return count === 0 ? [] : [{ ...w, message: tierDriftMessage(count) }];- });-}-async function toolBrainDoctor(ctx: ServerContext,args: Record<string, unknown>,@@ -130,6 +95,11 @@ async function toolBrainDoctor(const outcome = applyRepair(ctx.vault, {dryRun: !apply,...(ctx.configPath !== null ? { configPath: ctx.configPath } : {}),+ // The index-backed findings (`tier-drift`, `freshen-failing`) join+ // the plan as they join the report below. `tier-drift` is counted+ // over `readable`, so a withheld drifted page adds nothing to it.+ dbPath: resolveSearchConfig({ vault: ctx.vault, configPath: ctx.configPath ?? undefined })+ .dbPath,// The repair branch returns HERE, before the diagnostic streams// below are filtered - so until this argument existed a scoped// caller both read and REWROTE another owner's preferences@@ -175,10 +145,10 @@ async function toolBrainDoctor(const visibleIssues = <T extends DoctorIssueNaming>(issues: ReadonlyArray<T>): ReadonlyArray<T> =>view.keep(issues, (i) => doctorIssueRefs(ctx.vault, i));const errors = visibleIssues(result.errors);- const warnings =- reach === TRANSPORT_REACH.local- ? visibleIssues(result.warnings)- : await recountTierDrift(ctx, visibleIssues(result.warnings));+ // `tier-drift` names no page, so the view cannot judge it; the doctor+ // counted it over `readable` instead, the rows `brain_tiers check`+ // lists for this caller.+ const warnings = visibleIssues(result.warnings);const uncertain = result.uncertain === undefined ? undefined : visibleIssues(result.uncertain);// The verdict summarises the streams, so it is taken again over the// streams this caller sees: a verdict computed over a dropped errordiff --git a/tests/bench/measure-index-writes.ts b/tests/bench/measure-index-writes.tsnew file mode 100644index 00000000..d2119f82--- /dev/null+++ b/tests/bench/measure-index-writes.ts@@ -0,0 +1,126 @@+#!/usr/bin/env bun+/**+ * Measure how many bytes an incremental index run writes, per pass.+ *+ * bun run tests/bench/measure-index-writes.ts <vault> [changed-notes=200 | largest:N]+ *+ * Copies the vault's Markdown (and `Brain/_brain.yaml`) to a temp dir,+ * builds an index there, then runs three incremental passes and prints the+ * bytes each one wrote, split by the store methods that do the writing:+ * 1. no change at all,+ * 2. `changed-notes` notes edited (the catch-up a long-idle index meets),+ * or with `largest:N` the N largest notes (an append-only daily log that+ * grows all day is the note a frequent background run meets most),+ * 3. no change again.+ *+ * Linux only: the numbers are `write_bytes` deltas from `/proc/self/io`, the+ * bytes this process caused to be sent to storage. The source vault is only+ * read. Development tool for the index-freshness work; prints, never writes+ * outside its temp dir.+ */+import { appendFileSync, cpSync, mkdtempSync, readFileSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join, relative } from "node:path";++import { resolveSearchConfig } from "../../src/core/search/index.ts";+import { indexVault } from "../../src/core/search/indexer.ts";+import { Store } from "../../src/core/search/store.ts";++function writeBytes(): number {+ const m = /write_bytes:\s+(\d+)/.exec(readFileSync("/proc/self/io", "utf8"));+ if (m === null) throw new Error("/proc/self/io has no write_bytes (Linux only)");+ return Number(m[1]);+}++const PASSES = [+ "replaceDocumentChunks",+ "replaceLinks",+ "deleteDocument",+ "resolveLinkTargets",+ "resolveAliasTargets",+ "recomputeRelationConstraintFlags",+ "setState",+ "close",+] as const;++const tally = new Map<string, number>();+const proto = Store.prototype as unknown as Record<string, (...args: unknown[]) => unknown>;+for (const name of PASSES) {+ const original = proto[name];+ if (typeof original !== "function") continue;+ proto[name] = function (this: unknown, ...args: unknown[]) {+ const before = writeBytes();+ const result = original.apply(this, args);+ const settle = (): void => {+ tally.set(name, (tally.get(name) ?? 0) + (writeBytes() - before));+ };+ if (result instanceof Promise) return result.finally(settle);+ settle();+ return result;+ };+}++function mb(n: number): string {+ return (n / 1024 / 1024).toFixed(1).padStart(8);+}++async function measured(label: string, vault: string): Promise<void> {+ tally.clear();+ const before = writeBytes();+ const t0 = performance.now();+ const stats = await indexVault(resolveSearchConfig({ vault }));+ const total = writeBytes() - before;+ const ms = Math.round(performance.now() - t0);+ console.log(+ `\n${label}: ${mb(total)} MB in ${ms} ms ` ++ `(added ${stats.added}, updated ${stats.updated}, deleted ${stats.deleted}, ` ++ `link resolution ${stats.linkResolutionSkipped ? "skipped" : "ran"})`,+ );+ let counted = 0;+ for (const [name, bytes] of [...tally].toSorted((a, b) => b[1] - a[1])) {+ counted += bytes;+ console.log(` ${name.padEnd(34)}${mb(bytes)} MB`);+ }+ console.log(` ${"(elsewhere)".padEnd(34)}${mb(total - counted)} MB`);+}++const [source, changedArg] = process.argv.slice(2);+if (source === undefined) {+ console.error(+ "usage: bun run tests/bench/measure-index-writes.ts <vault> [changed-notes=200 | largest:N]",+ );+ process.exit(2);+}+const largest = changedArg?.startsWith("largest:") === true;+const changed = Number(largest ? changedArg!.slice("largest:".length) : (changedArg ?? "200"));++const work = mkdtempSync(join(tmpdir(), "osb-measure-writes-"));+const vault = join(work, "vault");+try {+ cpSync(source, vault, {+ recursive: true,+ filter: (src) =>+ !relative(source, src).startsWith(".") &&+ (!src.includes(".") || src.endsWith(".md") || src.endsWith("_brain.yaml")),+ });+ console.log(`copied ${source} -> ${vault}; building the index...`);+ await indexVault(resolveSearchConfig({ vault }));++ await measured("1. no change", vault);++ const glob = new Bun.Glob("**/*.md");+ const all = [...glob.scanSync({ cwd: vault })].toSorted();+ const notes = largest+ ? all+ .map((n) => ({ n, size: Bun.file(join(vault, n)).size }))+ .toSorted((a, b) => b.size - a.size)+ .slice(0, changed)+ .map((x) => x.n)+ : all.slice(0, changed);+ for (const note of notes) appendFileSync(join(vault, note), "\n<!-- measure -->\n");+ await measured(`2. ${notes.length} ${largest ? "largest " : ""}notes edited`, vault);++ await measured("3. no change again", vault);+} finally {+ rmSync(work, { recursive: true, force: true });+}diff --git a/tests/cli/search-index-freshen.test.ts b/tests/cli/search-index-freshen.test.tsnew file mode 100644index 00000000..f9bf99b4--- /dev/null+++ b/tests/cli/search-index-freshen.test.ts@@ -0,0 +1,137 @@+/**+ * The background child a stale read starts: `o2b search index --freshen+ * <token>` records its outcome in the per-device state file, backs off+ * after a failure, writes a metric row only when it changed something or+ * failed, and releases the claim it was handed in every case.+ */+import { afterEach, beforeEach, expect, test } from "bun:test";+import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { listMetrics } from "../../src/core/brain/metrics.ts";+import {+ claimFreshen,+ FRESHEN_CLAIM_FILE,+ INDEX_FRESHEN_SURFACE,+ readFreshenState,+} from "../../src/core/search/freshen.ts";+import { acquireWriterLockSync } from "../../src/core/search/store/writer-lock.ts";+import { runCli } from "../helpers/run-cli.ts";++let tmp: string;+let vault: string;+let derived: string;+let configPath: string;++const env = () => ({ OPEN_SECOND_BRAIN_CONFIG: configPath });++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-freshen-child-"));+ vault = join(tmp, "vault");+ derived = join(vault, ".open-second-brain");+ mkdirSync(join(vault, "notes"), { recursive: true });+ mkdirSync(derived, { recursive: true });+ configPath = join(tmp, "config.yaml");+ writeFileSync(configPath, `vault: ${vault}\n`);+ writeFileSync(join(vault, "notes", "a.md"), "# A\n\nfirst note about herons.\n");+});++afterEach(() => rmSync(tmp, { recursive: true, force: true }));++function freshenRun(): Promise<{ returncode: number }> {+ const token = claimFreshen(derived, Date.now());+ expect(token).toBeString();+ return runCli(["search", "index", "--freshen", token!], { env: env() });+}++test("a completed run records its outcome, changes and a metric row, and releases the claim", async () => {+ const result = await freshenRun();+ expect(result.returncode).toBe(0);+ const state = readFreshenState(derived);+ expect(state.lastOutcome).toBe("completed");+ expect(state.lastChanged).toBe(1);+ expect(state.failures).toBe(0);+ expect(state.backoffUntil).toBeNull();+ expect(existsSync(join(derived, FRESHEN_CLAIM_FILE))).toBe(false);+ const rows = listMetrics(vault, { surface: INDEX_FRESHEN_SURFACE });+ expect(rows).toHaveLength(1);+ expect(rows[0]!.payload["outcome"]).toBe("completed");+});++test("a run that changed nothing writes no metric row", async () => {+ await freshenRun();+ await freshenRun();+ expect(readFreshenState(derived).lastChanged).toBe(0);+ expect(listMetrics(vault, { surface: INDEX_FRESHEN_SURFACE })).toHaveLength(1);+});++test("a failed run backs off, doubling on the next failure, and still releases the claim", async () => {+ // A directory squatting on the index path makes every open fail.+ mkdirSync(join(derived, "brain.sqlite"));+ const before = Date.now();+ const first = await freshenRun();+ expect(first.returncode).not.toBe(0);+ let state = readFreshenState(derived);+ expect(state.lastOutcome).toBe("failed");+ expect(state.failures).toBe(1);+ expect(state.lastError).toBeString();+ const firstWait = Date.parse(state.backoffUntil!) - before;+ expect(firstWait).toBeGreaterThanOrEqual(55_000);+ expect(existsSync(join(derived, FRESHEN_CLAIM_FILE))).toBe(false);++ await freshenRun();+ state = readFreshenState(derived);+ expect(state.failures).toBe(2);+ expect(Date.parse(state.backoffUntil!) - Date.now()).toBeGreaterThan(110_000);+ const rows = listMetrics(vault, { surface: INDEX_FRESHEN_SURFACE });+ expect(rows.map((r) => r.payload["outcome"])).toEqual(["failed", "failed"]);+});++test("a success after failures clears the streak and the backoff", async () => {+ mkdirSync(join(derived, "brain.sqlite"));+ await freshenRun();+ rmSync(join(derived, "brain.sqlite"), { recursive: true });+ await freshenRun();+ const state = readFreshenState(derived);+ expect(state.lastOutcome).toBe("completed");+ expect(state.failures).toBe(0);+ expect(state.backoffUntil).toBeNull();+});++test("a run that meets another index writer is a skip, not a failure", async () => {+ const release = acquireWriterLockSync(join(derived, "brain.sqlite"));+ try {+ await freshenRun();+ } finally {+ release();+ }+ const state = readFreshenState(derived);+ expect(state.failures).toBe(0);+ expect(state.backoffUntil).toBeNull();+ expect(state.lastOutcome).toBeNull();+ expect(existsSync(join(derived, FRESHEN_CLAIM_FILE))).toBe(false);+ expect(listMetrics(vault, { surface: INDEX_FRESHEN_SURFACE })).toHaveLength(0);+});++test("a run that fails before it can resolve its config still records the failure and releases the claim", async () => {+ const token = claimFreshen(derived, Date.now())!;+ const result = await runCli(+ [+ "search",+ "index",+ "--config",+ join(tmp, "missing.yaml"),+ "--freshen",+ token,+ "--freshen-state",+ derived,+ ],+ { env: env() },+ );+ expect(result.returncode).not.toBe(0);+ const state = readFreshenState(derived);+ expect(state.lastOutcome).toBe("failed");+ expect(state.failures).toBe(1);+ expect(existsSync(join(derived, FRESHEN_CLAIM_FILE))).toBe(false);+});diff --git a/tests/cli/search-status-freshen.test.ts b/tests/cli/search-status-freshen.test.tsnew file mode 100644index 00000000..67693756--- /dev/null+++ b/tests/cli/search-status-freshen.test.ts@@ -0,0 +1,114 @@+/**+ * `o2b search status` shows freshen on read: the interval (or off), how+ * old the index is, what the last background run did, and an active+ * backoff - in the human output and under `freshen` in `--json`.+ */+import { afterEach, beforeEach, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { writeFreshenState } from "../../src/core/search/freshen.ts";+import { runCli } from "../helpers/run-cli.ts";++let tmp: string;+let vault: string;+let configPath: string;++beforeEach(async () => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-status-freshen-"));+ vault = join(tmp, "vault");+ mkdirSync(join(vault, "notes"), { recursive: true });+ configPath = join(tmp, "config.yaml");+ writeFileSync(configPath, `vault: ${vault}\n`);+ writeFileSync(join(vault, "notes", "a.md"), "# A\n\nbody\n");+ await runCli(["search", "index"], { env: env("60") });+});++afterEach(() => rmSync(tmp, { recursive: true, force: true }));++function env(interval: string): Record<string, string> {+ return {+ OPEN_SECOND_BRAIN_CONFIG: configPath,+ OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S: interval,+ };+}++test("status names the interval, the index age and that no background run happened yet", async () => {+ const out = await runCli(["search", "status"], { env: env("60") });+ expect(out.returncode).toBe(0);+ expect(out.stdout).toContain("freshen: every 60s");+ expect(out.stdout).toMatch(/index_age:\s+\d+s/);+ expect(out.stdout).toContain("last_freshen: (none)");+ expect(out.stdout).not.toContain("freshen_backoff");+});++test("an interval of 0 reads as off", async () => {+ const out = await runCli(["search", "status"], { env: env("0") });+ expect(out.stdout).toContain("freshen: off");+});++test("a failed run shows its error and the active backoff, in text and JSON", async () => {+ const backoffUntil = new Date(Date.now() + 120_000).toISOString();+ writeFreshenState(join(vault, ".open-second-brain"), {+ failures: 2,+ backoffUntil,+ lastOutcome: "failed",+ lastRunAt: "2026-10-07T12:00:00.000Z",+ lastDurationMs: 40,+ lastError: "disk full",+ lastChanged: null,+ });+ const text = await runCli(["search", "status"], { env: env("60") });+ expect(text.stdout).toContain(+ "last_freshen: failed 2026-10-07T12:00:00.000Z (2 in a row): disk full",+ );+ expect(text.stdout).toContain(`freshen_backoff: until ${backoffUntil}`);++ const json = JSON.parse(+ (await runCli(["search", "status", "--json"], { env: env("60") })).stdout,+ );+ expect(json.freshen).toMatchObject({+ interval_s: 60,+ last_outcome: "failed",+ last_error: "disk full",+ failures: 2,+ backoff_until: backoffUntil,+ });+ expect(json.freshen.index_age_s).toBeGreaterThanOrEqual(0);+});++test("every freshen line keeps the 21-column value alignment of the status block", async () => {+ writeFreshenState(join(vault, ".open-second-brain"), {+ failures: 1,+ backoffUntil: new Date(Date.now() + 120_000).toISOString(),+ lastOutcome: "failed",+ lastRunAt: "2026-10-07T12:00:00.000Z",+ lastDurationMs: 40,+ lastError: "disk full",+ lastChanged: null,+ });+ const out = await runCli(["search", "status"], { env: env("60") });+ const lines = out.stdout.split("\n").filter((l) => /^(freshen|index_age|last_freshen)/.test(l));+ expect(lines).toHaveLength(4);+ for (const line of lines) {+ expect(line.slice(0, 21)).toMatch(/^[a-z_]+: +$/);+ expect(line[21]).not.toBe(" ");+ }+});++test("a completed run shows how many documents it changed", async () => {+ writeFreshenState(join(vault, ".open-second-brain"), {+ failures: 0,+ backoffUntil: null,+ lastOutcome: "completed",+ lastRunAt: "2026-10-07T12:00:00.000Z",+ lastDurationMs: 300,+ lastError: null,+ lastChanged: 3,+ });+ const out = await runCli(["search", "status"], { env: env("60") });+ expect(out.stdout).toContain(+ "last_freshen: completed 2026-10-07T12:00:00.000Z (3 changed)",+ );+});diff --git a/tests/core/architecture/state-surface-census.test.ts b/tests/core/architecture/state-surface-census.test.tsindex e72fbff9..c4250102 100644--- a/tests/core/architecture/state-surface-census.test.ts+++ b/tests/core/architecture/state-surface-census.test.ts@@ -80,7 +80,7 @@ const STORE_DIR_IDENTIFIER = "DERIVED_STORE_DIR";const SWEPT_POPULATION_SIZE = 23;/** Declared surfaces today. Pinned for the same reason. */-const DECLARED_SURFACE_COUNT = 46;+const DECLARED_SURFACE_COUNT = 47;/** An exclusion reason has to be an argument, not a label. */const MIN_REASON_LENGTH = 80;diff --git a/tests/core/architecture/write-site-census.test.ts b/tests/core/architecture/write-site-census.test.tsindex 24353441..22876fbe 100644--- a/tests/core/architecture/write-site-census.test.ts+++ b/tests/core/architecture/write-site-census.test.ts@@ -912,6 +912,18 @@ const DIRECT_WRITE_EXCLUSIONS: Readonly<Record<string, WriteExclusion>> = Object"this tool's own per-device state, never a note.",},+ "src/core/search/freshen.ts": {+ categories: [C.lockPrimitive, C.retentionDelete],+ calls: ["renameSync", "unlinkSync", "writeFileSync"],+ reason:+ "the freshen-on-read claim is an exclusive create (`wx`) of a token file in " ++ "`.open-second-brain/`, released by unlinking it; a dead run's claim is replaced by " ++ "an atomic rename under an exclusive takeover lock; routing it through " ++ "the atomic writer would lose the exclusivity that keeps concurrent readers from " ++ "starting a background index run each. The run's state file goes through " ++ "`atomicWriteFileSync`. Both are this tool's own per-device state, never a note.",+ },+"src/core/state/migrate.ts": {categories: [C.archiveTransfer, C.lifecycleMove, C.retentionDelete],calls: ["copyFileSync", "rmSync", "rmdirSync"],@@ -1221,8 +1233,10 @@ const DIRECT_ROWS = ROWS.filter((row) => row.directCalls.length > 0);* removes the failed-upgrade marker once nothing is pending.* 75 -> 76: the doctor self-test seeds and journals a throwaway store in* temp storage and removes the tree when the run ends.+ * 76 -> 77: `src/core/search/freshen.ts` claims and releases the+ * freshen-on-read claim (an exclusive create) in `.open-second-brain/`.*/-const DIRECT_WRITE_ROWS = 76;+const DIRECT_WRITE_ROWS = 77;/*** Measured modules reaching a write through a shared helper. An equality.@@ -1285,8 +1299,11 @@ const DIRECT_WRITE_ROWS = 76;* candidates through the shared atomic JSONL writer.* 108 -> 109: `src/core/search/session-focus.ts` writes the focus through* `atomicWriteText` (mode 0600, a leaf link is replaced, not followed).+ *+ * 109 -> 110: `src/core/search/freshen.ts` writes the freshen-on-read state+ * file through `atomicWriteFileSync`.*/-const SHARED_HELPER_ROWS = 109;+const SHARED_HELPER_ROWS = 110;// ----- Origin-channel coverage boundary (Unit C) ----------------------------@@ -1357,8 +1374,9 @@ const STAMPED_PATHS: ReadonlySet<string> = new Set(* 72 -> 73: the payload registry's touch (metadata only).* 73 -> 74: the automatic-upgrade lock and marker removal (state, not notes).* 74 -> 75: the doctor self-test's throwaway-store writes (temp storage).+ * 75 -> 76: the freshen-on-read claim and its release (state, not notes).*/-const UNSTAMPED_DIRECT_ROWS = 75;+const UNSTAMPED_DIRECT_ROWS = 76;/*** Shared-helper write sites the stamp does not reach, measured the same@@ -1371,9 +1389,10 @@ const UNSTAMPED_DIRECT_ROWS = 75;* and the dedup index cache, a derived file outside the vault (102 -> 103),* and the orphan-repair detach (103 -> 104). 104 -> 105: the repair lane* stages hub candidates through the shared atomic JSONL writer. 105 -> 106:- * the session focus writes through the shared atomic text writer.+ * the session focus writes through the shared atomic text writer. 106 -> 107:+ * the freshen-on-read state file writes through the shared atomic writer.*/-const UNSTAMPED_SHARED_ROWS = 106;+const UNSTAMPED_SHARED_ROWS = 107;describe("in-vault write-site census", () => {test("every direct-fs write site carries a written exclusion", () => {diff --git a/tests/core/brain/doctor-exit-census.test.ts b/tests/core/brain/doctor-exit-census.test.tsindex 242c5ff1..b9e2dd5b 100644--- a/tests/core/brain/doctor-exit-census.test.ts+++ b/tests/core/brain/doctor-exit-census.test.ts@@ -188,6 +188,7 @@ const DOCTOR_REGISTERED_CODES: ReadonlyArray<string> = ["embeddings-backlog","entity-label-malformed","entity-quote-variant-collision",+ "freshen-failing","inbox-archivable","log-chain-broken","low-evidence-confirmed",diff --git a/tests/core/brain/doctor-freshen.test.ts b/tests/core/brain/doctor-freshen.test.tsnew file mode 100644index 00000000..8faf7574--- /dev/null+++ b/tests/core/brain/doctor-freshen.test.ts@@ -0,0 +1,104 @@+/**+ * The doctor warns when background index runs started by stale reads keep+ * failing: three in a row is a broken index, not a transient hiccup, and+ * until it is fixed every search answers from an ageing index.+ */+import { afterEach, beforeEach, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { dirname, join } from "node:path";++import { applyRepair } from "../../../src/core/brain/diagnostics.ts";+import { runDoctor } from "../../../src/core/brain/doctor.ts";+import { bootstrapBrain } from "../../../src/core/brain/init.ts";+import { writeFreshenState } from "../../../src/core/search/freshen.ts";+import { resolveSearchConfig } from "../../../src/core/search/index.ts";+import { MCPServer } from "../../../src/mcp/server.ts";+import { runCli } from "../../helpers/run-cli.ts";++let tmp: string;+let vault: string;+let configPath: string;+let derived: string;++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-doctor-freshen-"));+ vault = join(tmp, "vault");+ configPath = join(tmp, "config.yaml");+ writeFileSync(configPath, `vault: ${vault}\n`);+ bootstrapBrain(vault, { configPath });+ derived = join(vault, ".open-second-brain");+ // The surfaces below resolve the index path themselves; the state file+ // has to sit where they look.+ expect(dirname(resolveSearchConfig({ vault, configPath }).dbPath)).toBe(derived);+ mkdirSync(derived, { recursive: true });+});++afterEach(() => rmSync(tmp, { recursive: true, force: true }));++function failures(n: number): void {+ writeFreshenState(derived, {+ failures: n,+ backoffUntil: new Date(Date.now() + 60_000).toISOString(),+ lastOutcome: "failed",+ lastRunAt: new Date().toISOString(),+ lastDurationMs: 10,+ lastError: "database disk image is malformed",+ lastChanged: null,+ });+}++function freshenWarnings() {+ return runDoctor(vault, { dbPath: join(derived, "brain.sqlite") }).warnings.filter(+ (w) => w.code === "freshen-failing",+ );+}++test("three failed background runs in a row are a warning naming the last error", () => {+ failures(3);+ const hits = freshenWarnings();+ expect(hits).toHaveLength(1);+ expect(hits[0]!.message).toContain("3");+ expect(hits[0]!.message).toContain("database disk image is malformed");+});++test("fewer than three failures stay quiet", () => {+ failures(2);+ expect(freshenWarnings()).toEqual([]);+});++test("no state file is no warning", () => {+ expect(freshenWarnings()).toEqual([]);+});++test("the repair plan lists a failing freshen as not auto-repairable", () => {+ failures(3);+ const outcome = applyRepair(vault, { dryRun: true, dbPath: join(derived, "brain.sqlite") });+ expect(outcome.unfixable.map((u) => u.code)).toContain("freshen-failing");+});++test("o2b brain doctor --repair reaches the failing freshen", async () => {+ failures(3);+ const r = await runCli(["brain", "doctor", "--vault", vault, "--repair", "--json"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: configPath },+ });+ expect(r.returncode).toBe(0);+ const outcome = JSON.parse(r.stdout) as { unfixable: ReadonlyArray<{ code: string }> };+ expect(outcome.unfixable.map((u) => u.code)).toContain("freshen-failing");+});++test("brain_doctor reaches the failing freshen in its report and its repair preview", async () => {+ failures(3);+ const saved = process.env["OPEN_SECOND_BRAIN_CONFIG"];+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = configPath;+ try {+ const server = new MCPServer({ vault, configPath });+ const report = JSON.stringify(await server.callTool("brain_doctor", {}));+ expect(report).toContain("freshen-failing");+ const repair = JSON.stringify(await server.callTool("brain_doctor", { repair: true }));+ expect(repair).toContain("freshen-failing");+ } finally {+ if (saved === undefined) delete process.env["OPEN_SECOND_BRAIN_CONFIG"];+ else process.env["OPEN_SECOND_BRAIN_CONFIG"] = saved;+ }+});diff --git a/tests/core/brain/tier-drift.test.ts b/tests/core/brain/tier-drift.test.tsindex 32e908de..e7bab29e 100644--- a/tests/core/brain/tier-drift.test.ts+++ b/tests/core/brain/tier-drift.test.ts@@ -12,6 +12,8 @@ import { afterEach, beforeEach, expect, test } from "bun:test";import { mkdirSync, writeFileSync } from "node:fs";import { join } from "node:path";+import { runDoctor } from "../../../src/core/brain/doctor.ts";+import { TIER_DRIFT_CODE } from "../../../src/core/brain/doctor/store-integrity.ts";import { indexVault } from "../../../src/core/search/indexer.ts";import { Store } from "../../../src/core/search/store.ts";import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";@@ -152,3 +154,19 @@ test("deleting the document drops its drift rows", async () => {await indexVault(config);expect(await drift()).toHaveLength(0);});++test("the doctor counts only the drift rows on pages the caller may read", async () => {+ const config = makeConfig({ vault, dbPath });+ writePref("pref-spaces");+ await indexVault(config);+ writePref("pref-tabs");+ await indexVault(config);+ const driftWarnings = (readable?: (rel: string) => boolean) =>+ runDoctor(vault, { dbPath, ...(readable !== undefined ? { readable } : {}) }).warnings.filter(+ (w) => w.code === TIER_DRIFT_CODE,+ );+ expect(driftWarnings()).toHaveLength(1);+ expect(driftWarnings((rel) => rel === "Brain/preferences/pref-spaces.md")).toHaveLength(1);+ // A caller that may not read the drifted page is told nothing about it.+ expect(driftWarnings(() => false)).toEqual([]);+});diff --git a/tests/core/maintenance/ensure-current.test.ts b/tests/core/maintenance/ensure-current.test.tsindex e46efd56..9ff686f4 100644--- a/tests/core/maintenance/ensure-current.test.ts+++ b/tests/core/maintenance/ensure-current.test.ts@@ -208,3 +208,60 @@ describe("ensureVaultCurrent: chunks cut by older chunking rules (#186)", () =>expect(r.reindexTriggered).toBe(false);});});++describe("ensureVaultCurrent: freshen on session start", () => {+ const INTERVAL_ENV = "OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S";+ let prevInterval: string | undefined;+ let calls: string[][];+ const freshenSpawn = (argv: string[]): void => {+ calls.push(argv);+ };++ beforeEach(() => {+ prevInterval = process.env[INTERVAL_ENV];+ process.env[INTERVAL_ENV] = "60";+ calls = [];+ });++ afterEach(() => {+ if (prevInterval === undefined) delete process.env[INTERVAL_ENV];+ else process.env[INTERVAL_ENV] = prevInterval;+ });++ function ageIndex(seconds: number): void {+ const db = new Database(dbPath());+ db.query("UPDATE index_state SET value = ? WHERE key = 'last_indexed_at'").run(+ new Date(Date.now() - seconds * 1000).toISOString(),+ );+ db.close();+ }++ test("a current index that has gone stale is caught up in the background", async () => {+ bootstrapBrain(vault, { configPath });+ await ensureVaultCurrent(vault, { background: false });+ ageIndex(300);+ const r = await ensureVaultCurrent(vault, { background: false, freshenSpawn });+ expect(r.freshen).toBe("spawned");+ expect(calls).toHaveLength(1);+ });++ test("a fresh index starts nothing", async () => {+ bootstrapBrain(vault, { configPath });+ await ensureVaultCurrent(vault, { background: false });+ const r = await ensureVaultCurrent(vault, { background: false, freshenSpawn });+ expect(r.freshen).toBe("fresh");+ expect(calls).toHaveLength(0);+ });++ test("an index that needs a rebuild is left to self-heal, not freshened", async () => {+ bootstrapBrain(vault, { configPath });+ await ensureVaultCurrent(vault, { background: false });+ const db = new Database(dbPath());+ db.run("UPDATE index_state SET value = '1' WHERE key = 'schema_version'");+ db.close();+ const r = await ensureVaultCurrent(vault, { background: false, freshenSpawn });+ expect(r.reindexTriggered).toBe(true);+ expect(r.freshen).toBeNull();+ expect(calls).toHaveLength(0);+ });+});diff --git a/tests/core/search/chunk-preserving-replace.test.ts b/tests/core/search/chunk-preserving-replace.test.tsnew file mode 100644index 00000000..b02daa7d--- /dev/null+++ b/tests/core/search/chunk-preserving-replace.test.ts@@ -0,0 +1,165 @@+/**+ * Re-indexing a changed document keeps every chunk whose position and+ * content did not change in place (same row, same id) and rewrites only+ * the chunks that did. A daily log that grows all day then costs the+ * appended tail on each run, not the whole file: before, every chunk was+ * deleted and re-inserted, full-text rows included.+ */+import { afterEach, beforeEach, expect, test } from "bun:test";+import { Database } from "bun:sqlite";++import { indexVault } from "../../../src/core/search/indexer.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";++let vault: string;+let dbPath: string;+let cleanup: () => void;++/** A section of about 900 words, more than one 800-word chunk can hold alone. */+function section(name: string): string {+ const words = Array.from({ length: 900 }, (_, i) => `${name}${i % 50}`).join(" ");+ return `## ${name}\n\n${words}\n`;+}++const LOG = "notes/log.md";++beforeEach(() => {+ ({ vault, dbPath, cleanup } = createTempVault("chunk-preserving"));+ writeMd(vault, LOG, `# Log\n\n${section("alpha")}\n${section("bravo")}\n${section("charlie")}`);+ // Indexed after the log, so the log's chunks are not the highest rowids:+ // SQLite would otherwise hand a delete-and-reinsert the same ids back.+ writeMd(vault, "notes/zz-other.md", `# Other\n\n${section("zulu")}`);+});++afterEach(() => cleanup());++interface ChunkRow {+ id: number;+ chunk_index: number;+ content_hash: string;+}++function chunks(): ChunkRow[] {+ const db = new Database(dbPath, { readonly: true });+ try {+ return db+ .query<ChunkRow, [string]>(+ "SELECT c.id, c.chunk_index, c.content_hash FROM chunks c " ++ "JOIN documents d ON d.id = c.document_id WHERE d.path = ? ORDER BY c.chunk_index",+ )+ .all(LOG);+ } finally {+ db.close();+ }+}++function ftsHits(term: string): number {+ const db = new Database(dbPath, { readonly: true });+ try {+ return db+ .query<{ n: number }, [string]>("SELECT count(*) AS n FROM chunk_fts WHERE chunk_fts MATCH ?")+ .get(term)!.n;+ } finally {+ db.close();+ }+}++function entityRows(): number {+ const db = new Database(dbPath, { readonly: true });+ try {+ return db.query<{ n: number }, []>("SELECT count(*) AS n FROM chunk_entities").get()!.n;+ } finally {+ db.close();+ }+}++test("appending to a document keeps every earlier chunk in place", async () => {+ const config = makeConfig({ vault, dbPath });+ await indexVault(config);+ const before = chunks();+ expect(before.length).toBeGreaterThanOrEqual(3);++ writeMd(+ vault,+ LOG,+ `# Log\n\n${section("alpha")}\n${section("bravo")}\n${section("charlie")}\n${section("delta")}`,+ );+ await indexVault(config);+ const after = chunks();++ expect(after.length).toBeGreaterThan(before.length);+ // Every chunk but the last of the old document survives with its id.+ const kept = before.slice(0, -1);+ expect(after.slice(0, kept.length)).toEqual(kept);+ expect(ftsHits("delta7")).toBeGreaterThan(0);+ expect(ftsHits("alpha7")).toBeGreaterThan(0);+});++test("editing the middle rewrites only the chunks that changed", async () => {+ const config = makeConfig({ vault, dbPath });+ await indexVault(config);+ const before = chunks();++ writeMd(vault, LOG, `# Log\n\n${section("alpha")}\n${section("bravo2")}\n${section("charlie")}`);+ await indexVault(config);+ const after = chunks();++ const sameHash = (a: ChunkRow, b: ChunkRow): boolean =>+ a.chunk_index === b.chunk_index && a.content_hash === b.content_hash;+ for (const row of after) {+ const old = before.find((b) => sameHash(b, row));+ // A chunk at the same position with the same content keeps its id;+ // a changed one is a new row.+ if (old !== undefined) expect(row.id).toBe(old.id);+ else expect(before.map((b) => b.id)).not.toContain(row.id);+ }+ expect(after.filter((r) => before.some((b) => b.id === r.id)).length).toBeGreaterThan(0);+ expect(ftsHits("bravo27")).toBeGreaterThan(0);+ expect(ftsHits("bravo7")).toBe(0);+});++test("shrinking a document removes the chunks past its new end, full-text rows included", async () => {+ const config = makeConfig({ vault, dbPath });+ await indexVault(config);+ writeMd(vault, LOG, `# Log\n\n${section("alpha")}`);+ await indexVault(config);+ expect(ftsHits("charlie7")).toBe(0);+ expect(ftsHits("alpha7")).toBeGreaterThan(0);+ const after = chunks();+ expect(after.map((c) => c.chunk_index)).toEqual(after.map((_, i) => i));+});++test("a kept chunk keeps its entities and a rewritten one gets fresh ones", async () => {+ writeMd(vault, LOG, `# Log\n\nMeeting with Ada Lovelace.\n\n${section("alpha")}`);+ const config = makeConfig({ vault, dbPath });+ await indexVault(config);+ const entitiesBefore = entityRows();+ writeMd(+ vault,+ LOG,+ `# Log\n\nMeeting with Ada Lovelace.\n\n${section("alpha")}\n${section("bravo")}`,+ );+ await indexVault(config);+ expect(entityRows()).toBeGreaterThanOrEqual(entitiesBefore);+});++test("a chunk whose stored full-text form is out of date is rewritten, not kept", async () => {+ const config = makeConfig({ vault, dbPath });+ await indexVault(config);+ // What a release that changes how full-text content is derived would+ // leave behind: same content and hash, a different stored FTS form.+ const db = new Database(dbPath);+ db.run(+ "UPDATE chunks SET fts_content = 'stale-form' WHERE chunk_index = 0 AND document_id = " ++ "(SELECT id FROM documents WHERE path = ?)",+ [LOG],+ );+ db.close();+ writeMd(+ vault,+ LOG,+ `# Log\n\n${section("alpha")}\n${section("bravo")}\n${section("charlie")}\n${section("delta")}`,+ );+ await indexVault(config);+ expect(ftsHits("stale")).toBe(0);+});diff --git a/tests/core/search/freshen-config.test.ts b/tests/core/search/freshen-config.test.tsnew file mode 100644index 00000000..c999eb0f--- /dev/null+++ b/tests/core/search/freshen-config.test.ts@@ -0,0 +1,71 @@+import { afterEach, beforeEach, expect, test } from "bun:test";+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { resolveSearchConfig } from "../../../src/core/search/index.ts";+import { SearchError } from "../../../src/core/search/types.ts";++// The test preload pins the interval to 0 so the suite never spawns a+// background indexer; these tests own the variable while they run.+const ENV_KEYS = [+ "OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S",+ "OPEN_SECOND_BRAIN_SEARCH_FRESHEN_EMBEDDINGS",+];++let tmp: string;+let configPath: string;+const saved: Record<string, string | undefined> = {};++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "osb-freshen-cfg-"));+ configPath = join(tmp, "config.yaml");+ for (const k of ENV_KEYS) {+ saved[k] = process.env[k];+ delete process.env[k];+ }+});++afterEach(() => {+ for (const k of ENV_KEYS) {+ if (saved[k] === undefined) delete process.env[k];+ else process.env[k] = saved[k];+ }+ rmSync(tmp, { recursive: true, force: true });+});++function resolve(extra = ""): ReturnType<typeof resolveSearchConfig> {+ writeFileSync(configPath, `vault: "${tmp}"\n${extra}`);+ return resolveSearchConfig({ vault: tmp, configPath });+}++test("freshening is on every 60 s by default, keyword-only, with the config path", () => {+ expect(resolve().freshen).toEqual({ intervalSeconds: 60, embeddings: false, configPath });+});++test("the config key sets the interval and 0 turns freshening off", () => {+ expect(resolve('search_freshen_interval_s: "300"\n').freshen?.intervalSeconds).toBe(300);+ expect(resolve('search_freshen_interval_s: "0"\n').freshen?.intervalSeconds).toBe(0);+});++test("the env variable wins over the config key", () => {+ process.env["OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S"] = "0";+ expect(resolve('search_freshen_interval_s: "300"\n').freshen?.intervalSeconds).toBe(0);+});++test("a negative or non-numeric interval is refused by name", () => {+ for (const bad of ['"-1"', '"abc"']) {+ expect(() => resolve(`search_freshen_interval_s: ${bad}\n`)).toThrow(SearchError);+ expect(() => resolve(`search_freshen_interval_s: ${bad}\n`)).toThrow(+ /search_freshen_interval_s/,+ );+ }+});++test("background embeddings are opt-in", () => {+ expect(resolve('search_freshen_embeddings: "true"\n').freshen?.embeddings).toBe(true);+});++test("a config resolved without a file carries no config path", () => {+ expect(resolveSearchConfig({ vault: tmp }).freshen?.configPath).toBeNull();+});diff --git a/tests/core/search/freshen-search.test.ts b/tests/core/search/freshen-search.test.tsnew file mode 100644index 00000000..432459ca--- /dev/null+++ b/tests/core/search/freshen-search.test.ts@@ -0,0 +1,153 @@+/**+ * Freshen on read at the `search()` seam: a stale index starts one+ * background run and the answer still comes from the index as it is; a+ * read-only open never starts one; an index more than ten minutes old is+ * named on the trail as `index-stale` whatever the freshen setting. The+ * query cache stores answers without that code and the trail gains it at+ * read time, so a cache hit names the index age it is served at.+ */+import { afterEach, beforeEach, expect, test } from "bun:test";+import { Database } from "bun:sqlite";++import { indexVault } from "../../../src/core/search/indexer.ts";+import { RETRIEVAL_DEGRADATION } from "../../../src/core/search/retrieval-trail.ts";+import { search } from "../../../src/core/search/search.ts";+import { LAST_INDEXED_AT_STATE_KEY } from "../../../src/core/search/store/state.ts";+import type { ResolvedSearchConfig } from "../../../src/core/search/types.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";++let vault: string;+let dbPath: string;+let cleanup: () => void;+let calls: string[][];+const spawn = (argv: string[]): void => {+ calls.push(argv);+};++beforeEach(async () => {+ const v = createTempVault("freshen-search");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+ calls = [];+ writeMd(vault, "notes/heron.md", "# Heron\n\nThe grey heron waits by the river.\n");+ await indexVault(config(60));+});++afterEach(() => cleanup());++function config(intervalSeconds: number, cacheEnabled = false): ResolvedSearchConfig {+ return Object.freeze({+ ...makeConfig({ vault, dbPath, cacheEnabled }),+ freshen: { intervalSeconds, embeddings: false, configPath: null },+ });+}++function setIndexAge(seconds: number): void {+ const db = new Database(dbPath);+ try {+ db.query("UPDATE index_state SET value = ? WHERE key = ?").run(+ new Date(Date.now() - seconds * 1000).toISOString(),+ LAST_INDEXED_AT_STATE_KEY,+ );+ } finally {+ db.close();+ }+}++function staleCodes(outcome: Awaited<ReturnType<typeof search>>) {+ return (outcome.retrievalTrail?.degraded ?? []).filter(+ (d) => d.code === RETRIEVAL_DEGRADATION.indexStale,+ );+}++test("a stale index starts one background run and still answers", async () => {+ setIndexAge(300);+ const outcome = await search(config(60), { query: "heron", freshenSpawn: spawn });+ expect(outcome.results.length).toBeGreaterThan(0);+ expect(calls).toHaveLength(1);+ expect(calls[0]!.join(" ")).toContain("search index --vault");+});++test("a fresh index starts nothing", async () => {+ await search(config(60), { query: "heron", freshenSpawn: spawn });+ expect(calls).toHaveLength(0);+});++test("a read-only open never starts a run", async () => {+ setIndexAge(3600);+ await search(config(60), { query: "heron", freshenSpawn: spawn, selfHeal: false });+ expect(calls).toHaveLength(0);+});++test("an index older than ten minutes is named on the trail with its age", async () => {+ setIndexAge(11 * 60);+ const outcome = await search(config(60), { query: "heron", freshenSpawn: spawn });+ const stale = staleCodes(outcome);+ expect(stale).toHaveLength(1);+ expect(stale[0]!.detail?.["ageSeconds"]).toBeGreaterThanOrEqual(660);+});++test("a lag of a couple of minutes is not reported", async () => {+ setIndexAge(120);+ expect(staleCodes(await search(config(60), { query: "heron", freshenSpawn: spawn }))).toEqual([]);+});++test("with freshening off a stale index is still reported but nothing starts", async () => {+ setIndexAge(11 * 60);+ const outcome = await search(config(0), { query: "heron", freshenSpawn: spawn });+ expect(staleCodes(outcome)).toHaveLength(1);+ expect(calls).toHaveLength(0);+});++function cachedPayloads(): string[] {+ const db = new Database(dbPath, { readonly: true });+ try {+ return db+ .query<{ payload: string }, []>("SELECT payload FROM query_cache")+ .all()+ .map((r) => r.payload);+ } finally {+ db.close();+ }+}++/** Mark every cached row, so an answer carrying the mark was a cache hit. */+function markCachedRows(total: number): void {+ const db = new Database(dbPath);+ try {+ db.query("UPDATE query_cache SET payload = json_set(payload, '$.total', ?)").run(total);+ } finally {+ db.close();+ }+}++test("with freshening off a stale-index answer is cached without its stale code", async () => {+ setIndexAge(11 * 60);+ const outcome = await search(config(0, true), { query: "heron", freshenSpawn: spawn });+ expect(staleCodes(outcome)).toHaveLength(1);+ const rows = cachedPayloads();+ expect(rows).toHaveLength(1);+ expect(rows[0]).not.toContain(RETRIEVAL_DEGRADATION.indexStale);+});++test("a cache hit over an index that has since gone stale names it", async () => {+ await search(config(0, true), { query: "heron", freshenSpawn: spawn });+ markCachedRows(4242);+ setIndexAge(11 * 60);+ const outcome = await search(config(0, true), { query: "heron", freshenSpawn: spawn });+ expect(outcome.total).toBe(4242);+ expect(staleCodes(outcome)).toHaveLength(1);+ expect(outcome.retrievalTrail?.retrieved).toBe(outcome.results.length);+});++test("a cache hit over a fresh index carries no stale code", async () => {+ setIndexAge(11 * 60);+ await search(config(0, true), { query: "heron", freshenSpawn: spawn });+ markCachedRows(4242);+ setIndexAge(0);+ const outcome = await search(config(0, true), { query: "heron", freshenSpawn: spawn });+ expect(outcome.total).toBe(4242);+ expect(staleCodes(outcome)).toEqual([]);+ expect(outcome.retrievalTrail).toBeUndefined();+});diff --git a/tests/core/search/freshen.test.ts b/tests/core/search/freshen.test.tsnew file mode 100644index 00000000..d825e80e--- /dev/null+++ b/tests/core/search/freshen.test.ts@@ -0,0 +1,384 @@+import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import {+ chmodSync,+ existsSync,+ mkdirSync,+ mkdtempSync,+ readFileSync,+ rmSync,+ utimesSync,+ writeFileSync,+} from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ claimFreshen,+ decideFreshen,+ FRESHEN_CLAIM_FILE,+ FRESHEN_SKIP,+ freshenCommand,+ freshenSpawnOptions,+ maybeFreshenIndex,+ nextBackoffMs,+ readFreshenState,+ releaseFreshen,+ writeFreshenState,+} from "../../../src/core/search/freshen.ts";+import { acquireWriterLockSync } from "../../../src/core/search/store/writer-lock.ts";+import type { ResolvedSearchConfig } from "../../../src/core/search/types.ts";+import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";++const NOW = Date.parse("2026-10-07T12:00:00.000Z");+const ago = (s: number): string => new Date(NOW - s * 1000).toISOString();++describe("decideFreshen", () => {+ const base = { nowMs: NOW, intervalSeconds: 60, backoffUntilMs: null };++ test("an interval of 0 is off", () => {+ expect(decideFreshen({ ...base, intervalSeconds: 0, lastIndexedAt: ago(3600) })).toEqual({+ action: "skip",+ reason: FRESHEN_SKIP.off,+ });+ });++ test("an index never stamped is left to self-heal", () => {+ expect(decideFreshen({ ...base, lastIndexedAt: null })).toEqual({+ action: "skip",+ reason: FRESHEN_SKIP.noIndex,+ });+ });++ test("an index younger than the interval is fresh", () => {+ expect(decideFreshen({ ...base, lastIndexedAt: ago(30) })).toEqual({+ action: "skip",+ reason: FRESHEN_SKIP.fresh,+ });+ });++ test("an index older than the interval is due", () => {+ expect(decideFreshen({ ...base, lastIndexedAt: ago(61) })).toEqual({ action: "spawn" });+ });++ test("an active backoff wins over a stale index", () => {+ expect(+ decideFreshen({ ...base, lastIndexedAt: ago(3600), backoffUntilMs: NOW + 1000 }),+ ).toEqual({ action: "skip", reason: FRESHEN_SKIP.backoff });+ });++ test("a fresh index is fresh even while a backoff is active", () => {+ expect(decideFreshen({ ...base, lastIndexedAt: ago(30), backoffUntilMs: NOW + 1000 })).toEqual({+ action: "skip",+ reason: FRESHEN_SKIP.fresh,+ });+ });++ test("an unparseable stamp counts as due rather than fresh", () => {+ expect(decideFreshen({ ...base, lastIndexedAt: "not a date" })).toEqual({ action: "spawn" });+ });+});++describe("claim, state and backoff", () => {+ let dir: string;+ beforeEach(() => {+ dir = mkdtempSync(join(tmpdir(), "osb-freshen-"));+ });+ afterEach(() => rmSync(dir, { recursive: true, force: true }));++ test("only the first of two claims wins", () => {+ expect(claimFreshen(dir, NOW)).toBeString();+ expect(claimFreshen(dir, NOW)).toBeNull();+ });++ test("a claim abandoned for more than ten minutes is taken over", () => {+ writeFileSync(+ join(dir, FRESHEN_CLAIM_FILE),+ JSON.stringify({ token: "dead", at: NOW - 601_000 }),+ );+ const token = claimFreshen(dir, NOW);+ expect(token).toBeString();+ expect(token).not.toBe("dead");+ });++ test("a torn claim file older than the grace is taken over too", () => {+ const path = join(dir, FRESHEN_CLAIM_FILE);+ writeFileSync(path, "{");+ const old = new Date(Date.now() - 60_000);+ utimesSync(path, old, old);+ expect(claimFreshen(dir, Date.now())).toBeString();+ });++ test("releasing with a foreign token leaves the claim in place", () => {+ const token = claimFreshen(dir, NOW)!;+ releaseFreshen(dir, "someone-else");+ expect(existsSync(join(dir, FRESHEN_CLAIM_FILE))).toBe(true);+ releaseFreshen(dir, token);+ expect(existsSync(join(dir, FRESHEN_CLAIM_FILE))).toBe(false);+ });++ test("state round-trips and a missing or torn file reads as empty", () => {+ expect(readFreshenState(dir).failures).toBe(0);+ writeFileSync(join(dir, "freshen-state.json"), "{");+ expect(readFreshenState(dir).lastOutcome).toBeNull();+ const state = {+ failures: 2,+ backoffUntil: ago(-120),+ lastOutcome: "failed" as const,+ lastRunAt: ago(5),+ lastDurationMs: 300,+ lastError: "disk full",+ lastChanged: null,+ };+ writeFreshenState(dir, state);+ expect(readFreshenState(dir)).toEqual(state);+ });++ test("backoff doubles from a minute and stops at an hour", () => {+ expect(nextBackoffMs(1)).toBe(60_000);+ expect(nextBackoffMs(2)).toBe(120_000);+ expect(nextBackoffMs(10)).toBe(3_600_000);+ });+});++describe("freshenCommand", () => {+ const base = ["o2b", "search", "index"];+ test("linux runs the child at idle I/O priority when ionice exists", () => {+ expect(freshenCommand(base, (t) => t === "ionice", "linux")).toEqual([+ "ionice",+ "-c3",+ ...base,+ ]);+ expect(freshenCommand(base, () => false, "linux")).toEqual(base);+ });+ test("macOS uses taskpolicy and Windows needs no prefix", () => {+ expect(freshenCommand(base, (t) => t === "taskpolicy", "darwin")).toEqual([+ "taskpolicy",+ "-b",+ ...base,+ ]);+ expect(freshenCommand(base, () => true, "win32")).toEqual(base);+ });+});++describe("freshenSpawnOptions", () => {+ test("the child is detached on every platform so it outlives a short-lived hook", () => {+ const opts = freshenSpawnOptions({ O2B_TEST: "1" });+ expect(opts.detached).toBe(true);+ expect(opts.windowsHide).toBe(true);+ expect([opts.stdin, opts.stdout, opts.stderr]).toEqual(["ignore", "ignore", "ignore"]);+ });++ test("the child gets a copy of the current environment, not the start-up one", () => {+ const env = { O2B_CONFIG: "/later/config.yaml" };+ const opts = freshenSpawnOptions(env);+ expect(opts.env).toEqual(env);+ expect(opts.env).not.toBe(env);+ });+});++describe("maybeFreshenIndex", () => {+ let vault: string;+ let config: ResolvedSearchConfig;+ let calls: string[][];+ const spawn = (argv: string[]): void => {+ calls.push(argv);+ };++ beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "osb-freshen-vault-"));+ mkdirSync(join(vault, ".open-second-brain"), { recursive: true });+ calls = [];+ config = {+ vault,+ dbPath: join(vault, ".open-second-brain", "brain.sqlite"),+ freshen: { intervalSeconds: 60, embeddings: false, configPath: "/cfg/config.yaml" },+ } as unknown as ResolvedSearchConfig;+ });+ afterEach(() => rmSync(vault, { recursive: true, force: true }));++ test("a stale index spawns one index run carrying the vault, config and claim", () => {+ expect(maybeFreshenIndex(config, { lastIndexedAt: ago(300), nowMs: NOW, spawn })).toBe(+ "spawned",+ );+ expect(calls).toHaveLength(1);+ const argv = calls[0]!;+ expect(argv.join(" ")).toContain(`search index --vault ${vault} --config /cfg/config.yaml`);+ const claim = JSON.parse(+ readFileSync(join(vault, ".open-second-brain", FRESHEN_CLAIM_FILE), "utf8"),+ ) as { token: string };+ expect(argv.slice(-2)).toEqual(["--freshen", claim.token]);+ });++ test("a second reader while the first run holds the claim spawns nothing", () => {+ maybeFreshenIndex(config, { lastIndexedAt: ago(300), nowMs: NOW, spawn });+ expect(maybeFreshenIndex(config, { lastIndexedAt: ago(300), nowMs: NOW, spawn })).toBe(+ FRESHEN_SKIP.claimed,+ );+ expect(calls).toHaveLength(1);+ });++ test("a fresh index spawns nothing", () => {+ expect(maybeFreshenIndex(config, { lastIndexedAt: ago(5), nowMs: NOW, spawn })).toBe(+ FRESHEN_SKIP.fresh,+ );+ expect(calls).toHaveLength(0);+ });++ test("a read-only open never freshens a vault it does not own", () => {+ expect(+ maybeFreshenIndex(config, { lastIndexedAt: ago(3600), nowMs: NOW, spawn, readOnly: true }),+ ).toBe(FRESHEN_SKIP.readOnly);+ expect(calls).toHaveLength(0);+ });++ test("a config without freshen settings is off", () => {+ const { freshen: _drop, ...rest } = config as unknown as Record<string, unknown>;+ expect(+ maybeFreshenIndex(rest as unknown as ResolvedSearchConfig, {+ lastIndexedAt: ago(3600),+ nowMs: NOW,+ spawn,+ }),+ ).toBe(FRESHEN_SKIP.off);+ });++ test("an indexer already holding the writer lock is left alone", () => {+ const release = acquireWriterLockSync(config.dbPath);+ try {+ expect(maybeFreshenIndex(config, { lastIndexedAt: ago(300), nowMs: NOW, spawn })).toBe(+ FRESHEN_SKIP.writerLock,+ );+ expect(calls).toHaveLength(0);+ } finally {+ release();+ }+ });++ test("an active backoff from a failed run spawns nothing", () => {+ writeFreshenState(join(vault, ".open-second-brain"), {+ failures: 1,+ backoffUntil: new Date(NOW + 30_000).toISOString(),+ lastOutcome: "failed",+ lastRunAt: ago(10),+ lastDurationMs: 100,+ lastError: "boom",+ lastChanged: null,+ });+ expect(maybeFreshenIndex(config, { lastIndexedAt: ago(300), nowMs: NOW, spawn })).toBe(+ FRESHEN_SKIP.backoff,+ );+ });++ test.skipIf(CHMOD_CANNOT_DENY)(+ "an index directory that refuses the claim is reported as unwritable, not claimed",+ () => {+ const dir = join(vault, ".open-second-brain");+ chmodSync(dir, 0o555);+ try {+ expect(maybeFreshenIndex(config, { lastIndexedAt: ago(300), nowMs: NOW, spawn })).toBe(+ FRESHEN_SKIP.unwritable,+ );+ expect(calls).toHaveLength(0);+ } finally {+ chmodSync(dir, 0o755);+ }+ },+ );++ test("a fresh index is decided without the state file, so a backoff there is not consulted", () => {+ writeFreshenState(join(vault, ".open-second-brain"), {+ failures: 1,+ backoffUntil: new Date(NOW + 30_000).toISOString(),+ lastOutcome: "failed",+ lastRunAt: ago(10),+ lastDurationMs: 100,+ lastError: "boom",+ lastChanged: null,+ });+ expect(maybeFreshenIndex(config, { lastIndexedAt: ago(5), nowMs: NOW, spawn })).toBe(+ FRESHEN_SKIP.fresh,+ );+ });++ test("a spawner that throws is reported as a skip and releases the claim", () => {+ const decision = maybeFreshenIndex(config, {+ lastIndexedAt: ago(300),+ nowMs: NOW,+ spawn: () => {+ throw new Error("no fork");+ },+ });+ expect(decision).toBe(FRESHEN_SKIP.spawnFailed);+ expect(existsSync(join(vault, ".open-second-brain", FRESHEN_CLAIM_FILE))).toBe(false);+ });+});++describe("claim races", () => {+ let dir: string;+ beforeEach(() => {+ dir = mkdtempSync(join(tmpdir(), "osb-freshen-race-"));+ });+ afterEach(() => rmSync(dir, { recursive: true, force: true }));++ test("a zero-byte claim younger than the grace is a claim being written, not a dead one", () => {+ writeFileSync(join(dir, FRESHEN_CLAIM_FILE), "");+ expect(claimFreshen(dir, Date.now())).toBeNull();+ });++ test("a zero-byte claim older than the grace is taken over", () => {+ const path = join(dir, FRESHEN_CLAIM_FILE);+ writeFileSync(path, "");+ const old = new Date(Date.now() - 60_000);+ utimesSync(path, old, old);+ expect(claimFreshen(dir, Date.now())).toBeString();+ });++ test("a stale claim taken over by one reader is not taken over again by the next", () => {+ writeFileSync(+ join(dir, FRESHEN_CLAIM_FILE),+ JSON.stringify({ token: "dead", at: Date.now() - 601_000 }),+ );+ expect(claimFreshen(dir, Date.now())).toBeString();+ expect(claimFreshen(dir, Date.now())).toBeNull();+ });++ test("of eight processes racing on a stale claim exactly one wins", async () => {+ writeFileSync(+ join(dir, FRESHEN_CLAIM_FILE),+ JSON.stringify({ token: "dead", at: Date.now() - 601_000 }),+ );+ const module = join(import.meta.dir, "..", "..", "..", "src", "core", "search", "freshen.ts");+ const script = `const { claimFreshen } = await import(${JSON.stringify(module)});+ const go = Number(process.argv[1]);+ while (Date.now() < go) {}+ console.log(claimFreshen(process.argv[2], Date.now()) === null ? "lost" : "won");`;+ const go = String(Date.now() + 1500);+ const procs = Array.from({ length: 8 }, () =>+ Bun.spawn(["bun", "-e", script, go, dir], { stdout: "pipe", stderr: "pipe" }),+ );+ const outs = await Promise.all(procs.map((p) => new Response(p.stdout).text()));+ expect(outs.map((o) => o.trim()).filter((o) => o === "won")).toHaveLength(1);+ });+});++test("the spawned run is told where its state lives", () => {+ const vault = mkdtempSync(join(tmpdir(), "osb-freshen-statedir-"));+ try {+ mkdirSync(join(vault, ".open-second-brain"), { recursive: true });+ const calls: string[][] = [];+ const config = {+ vault,+ dbPath: join(vault, ".open-second-brain", "brain.sqlite"),+ freshen: { intervalSeconds: 60, embeddings: false, configPath: null },+ } as unknown as ResolvedSearchConfig;+ maybeFreshenIndex(config, {+ lastIndexedAt: new Date(Date.now() - 300_000).toISOString(),+ spawn: (argv) => calls.push(argv),+ });+ const argv = calls[0]!;+ const at = argv.indexOf("--freshen-state");+ expect(argv[at + 1]).toBe(join(vault, ".open-second-brain"));+ } finally {+ rmSync(vault, { recursive: true, force: true });+ }+});diff --git a/tests/core/search/indexer-noop.test.ts b/tests/core/search/indexer-noop.test.tsnew file mode 100644index 00000000..dfb2183e--- /dev/null+++ b/tests/core/search/indexer-noop.test.ts@@ -0,0 +1,128 @@+/**+ * A run that adds, updates and deletes no document skips the link and+ * alias resolution passes: they are a pure function of the documents and+ * their links, which did not change, and rewriting every link row on every+ * run is what made frequent background runs write to disk for nothing.+ * The relation-constraint pass still runs (it reads, and writes only+ * changed flags), so a schema-pack edit alone still takes effect.+ */+import { afterEach, beforeEach, expect, test } from "bun:test";+import { Database } from "bun:sqlite";++import { indexVault } from "../../../src/core/search/indexer.ts";+import {+ LAST_INDEXED_AT_STATE_KEY,+ LINK_RESOLUTION_PENDING_STATE_KEY,+} from "../../../src/core/search/store/state.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";++let vault: string;+let dbPath: string;+let cleanup: () => void;++beforeEach(() => {+ ({ vault, dbPath, cleanup } = createTempVault("indexer-noop"));+ // `[[grey-heron]]` resolves only through the frontmatter alias, in the+ // alias pass that resets and re-sets it each run - the link a skip must+ // not lose. `[[notes/pond.md]]` resolves by exact path.+ writeMd(vault, "notes/birds/heron.md", "---\naliases: [grey-heron]\n---\n# Heron\n\nA wader.\n");+ writeMd(+ vault,+ "notes/river.md",+ "# River\n\nHome of the [[grey-heron]] and [[notes/pond.md]].\n",+ );+ writeMd(vault, "notes/pond.md", "# Pond\n\nStill water.\n");+});++afterEach(() => cleanup());++function linkTargets(): Array<{ target_path: string; resolved: number }> {+ const db = new Database(dbPath, { readonly: true });+ try {+ return db+ .query<{ target_path: string; resolved: number }, []>(+ "SELECT target_path, target_document_id IS NOT NULL AS resolved FROM links " ++ "WHERE target_path IS NOT NULL ORDER BY target_path",+ )+ .all();+ } finally {+ db.close();+ }+}++function lastIndexedAt(): string | null {+ const db = new Database(dbPath, { readonly: true });+ try {+ return (+ db+ .query<{ value: string }, [string]>("SELECT value FROM index_state WHERE key = ?")+ .get(LAST_INDEXED_AT_STATE_KEY)?.value ?? null+ );+ } finally {+ db.close();+ }+}++test("a run with no document change skips link resolution and keeps every link resolved", async () => {+ const config = makeConfig({ vault, dbPath });+ const first = await indexVault(config);+ expect(first.linkResolutionSkipped).toBe(false);+ const before = linkTargets();+ expect(before.every((l) => l.resolved === 1)).toBe(true);+ const stampBefore = lastIndexedAt();++ await Bun.sleep(5);+ const second = await indexVault(config);+ expect(second.linkResolutionSkipped).toBe(true);+ expect(linkTargets()).toEqual(before);+ expect(lastIndexedAt()).not.toBe(stampBefore);+});++test("a changed note runs link resolution again and resolves a new link", async () => {+ const config = makeConfig({ vault, dbPath });+ await indexVault(config);+ writeMd(vault, "notes/lake.md", "# Lake\n\nWhere the [[grey-heron]] winters.\n");+ const stats = await indexVault(config);+ expect(stats.linkResolutionSkipped).toBe(false);+ expect(+ linkTargets()+ .filter((l) => l.target_path === "grey-heron")+ .every((l) => l.resolved === 1),+ ).toBe(true);+});++test("a deleted target is noticed: its links become unresolved", async () => {+ const config = makeConfig({ vault, dbPath });+ await indexVault(config);+ const { rmSync } = await import("node:fs");+ rmSync(`${vault}/notes/pond.md`);+ const stats = await indexVault(config);+ expect(stats.linkResolutionSkipped).toBe(false);+ expect(linkTargets().find((l) => l.target_path === "notes/pond.md")?.resolved).toBe(0);+});++test("a forced run never skips", async () => {+ const config = makeConfig({ vault, dbPath });+ await indexVault(config);+ expect((await indexVault(config, { force: true })).linkResolutionSkipped).toBe(false);+});++test("a run killed after writing documents but before resolving links is finished by the next run", async () => {+ const config = makeConfig({ vault, dbPath });+ await indexVault(config);+ // What a run killed mid-way leaves: documents committed, the alias+ // link not resolved yet, and the pending marker it set before writing.+ const db = new Database(dbPath);+ db.run("UPDATE links SET target_document_id = NULL WHERE target_path = 'grey-heron'");+ db.query("INSERT OR REPLACE INTO index_state(key, value, updated_at) VALUES (?, '1', ?)").run(+ LINK_RESOLUTION_PENDING_STATE_KEY,+ new Date().toISOString(),+ );+ db.close();++ const stats = await indexVault(config);+ expect(stats.linkResolutionSkipped).toBe(false);+ expect(linkTargets().find((l) => l.target_path === "grey-heron")?.resolved).toBe(1);+ // Resolved now, so the run after that may skip again.+ expect((await indexVault(config)).linkResolutionSkipped).toBe(true);+});diff --git a/tests/core/search/vector-carry-over.test.ts b/tests/core/search/vector-carry-over.test.tsindex 1fd1074f..b8a6c198 100644--- a/tests/core/search/vector-carry-over.test.ts+++ b/tests/core/search/vector-carry-over.test.ts@@ -213,18 +213,39 @@ describe("replaceDocumentChunks carry-over", () => {const { store, docId, oldIds } = await seededStore();const before = embeddingTimes(dbPath, oldIds[0]!);await Bun.sleep(5);+ // "h0" moves from position 0 to 1, so it is a new row that carries+ // the old vector rather than a chunk kept in place.const replaced = store.replaceDocumentChunks(docId, [- chunkInput(0, "h0"),- chunkInput(1, "h1-edited"),+ chunkInput(0, "h1-edited"),+ chunkInput(1, "h0"),chunkInput(2, "h2"),]);await store.close();- const after = embeddingTimes(dbPath, replaced.chunkIds[0]!);+ const after = embeddingTimes(dbPath, replaced.chunkIds[1]!);expect(after.createdAt).toBe(before.createdAt);expect(after.updatedAt).not.toBe(before.updatedAt);},);+ test.skipIf(!VEC_LOADABLE)(+ "a chunk kept in place keeps its row, its vector and both timestamps",+ async () => {+ const { store, docId, oldIds } = await seededStore();+ const before = embeddingTimes(dbPath, oldIds[2]!);+ await Bun.sleep(5);+ const replaced = store.replaceDocumentChunks(docId, [+ chunkInput(0, "h1-edited"),+ chunkInput(1, "h0"),+ chunkInput(2, "h2"),+ ]);+ expect(replaced.chunkIds[2]).toBe(oldIds[2]!);+ expect(replaced.keptChunkIds.has(oldIds[2]!)).toBe(true);+ expect(store.getEmbeddingHash(oldIds[2]!)).toBe("emb-2");+ await store.close();+ expect(embeddingTimes(dbPath, oldIds[2]!)).toEqual(before);+ },+ );+test.skipIf(!VEC_LOADABLE)("the plain replaceChunks form carries too and still returns the ids",async () => {@@ -267,6 +288,35 @@ describe("replaceDocumentChunks carry-over", () => {},);+ for (const [what, key, value] of [+ ["model", EMBEDDING_MODEL_STATE_KEY, "another-model"],+ ["dimension", EMBEDDING_DIMENSION_STATE_KEY, String(STORE_DIMENSION * 2)],+ ] as const) {+ test.skipIf(!VEC_LOADABLE)(+ `a chunk kept in place whose vector has another ${what} loses it and is pending again`,+ async () => {+ const { store, docId, oldIds } = await seededStore();+ store.setState(key, value);+ const replaced = store.replaceDocumentChunks(docId, [+ chunkInput(0, "h0"),+ chunkInput(1, "h1-edited"),+ chunkInput(2, "h2"),+ ]);+ const kept = [oldIds[0]!, oldIds[2]!];+ expect([replaced.chunkIds[0], replaced.chunkIds[2]]).toEqual(kept);+ for (const id of kept) {+ expect(replaced.keptChunkIds.has(id)).toBe(true);+ expect(store.getEmbeddingHash(id)).toBeNull();+ }+ expect(replaced.embeddingsReused).toBe(0);+ const pending = store.findChunksWithoutEmbeddings().map((p) => p.chunkId);+ for (const id of kept) expect(pending).toContain(id);+ await store.close();+ expect(vecRowCount(dbPath)).toBe(0);+ },+ );+ }+test.skipIf(!VEC_LOADABLE)("nothing is carried when the embedding identity is unrecorded",async () => {diff --git a/tests/core/state/surfaces.test.ts b/tests/core/state/surfaces.test.tsindex f20f1b83..c52b93fa 100644--- a/tests/core/state/surfaces.test.ts+++ b/tests/core/state/surfaces.test.ts@@ -30,6 +30,11 @@import { afterEach, describe, expect, test } from "bun:test";import { selfHealUpgradeMarkerPath } from "../../../src/core/maintenance/self-heal-upgrade-state.ts";+import {+ EMPTY_FRESHEN_STATE,+ FRESHEN_STATE_FILE,+ writeFreshenState,+} from "../../../src/core/search/freshen.ts";import { chmodSync, existsSync, mkdirSync, statSync, symlinkSync, writeFileSync } from "node:fs";import { dirname, join, resolve, sep } from "node:path";@@ -262,6 +267,9 @@ const RESOLVER_BINDINGS: ReadonlyArray<readonly [StateSurfaceId, (vault: string)["hook_audit", (v) => hookAuditDir(v)],["hook_session_state", (v) => dirname(hookStateFilePath(v, null))],["self_heal_upgrade_marker", (v) => selfHealUpgradeMarkerPath(v)],+ // `freshen.ts` takes the directory its caller hands it, always+ // `dirname(config.dbPath)`, so the row is bound to the index resolver.+ ["freshen_state", (v) => join(dirname(resolveIndexPath(v, null)), FRESHEN_STATE_FILE)],["aider_context_artifact", (v) => resolveAiderSidecarPath({ vault: v } as InstallEnv, {})],// --- Under `<vault>/Brain/` --------------------------------------------["brain_log", (v) => brainDirs(v).log],@@ -454,6 +462,24 @@ describe("each row is bound to the resolver that owns it", () => {expect(row("search_session_focus").derive(vault, elsewhere)).toBe(dirname(sessionFocusPath({ vault, dbPath: elsewhere } as ResolvedSearchConfig, "scope")),);+ expect(row("freshen_state").derive(vault, elsewhere)).toBe(+ join(dirname(elsewhere), FRESHEN_STATE_FILE),+ );+ });++ test("a relocated index moves the freshen state to where the run writes it", () => {+ const home = tempVault();+ const elsewhere = join(tempVault(), "brain.sqlite");+ writeFreshenState(dirname(elsewhere), EMPTY_FRESHEN_STATE);+ const inventory = inventoryStateSurfaces({+ vault: home,+ env: { OPEN_SECOND_BRAIN_SEARCH_DB: elsewhere },+ config: EMPTY_CONFIG,+ });+ const state = inventory.surfaces.find((s) => s.id === "freshen_state")!;+ expect(state.path).toBe(join(dirname(elsewhere), FRESHEN_STATE_FILE));+ expect(state.origin).toBe(CONFIG_ORIGIN.env);+ expect(existsSync(state.path)).toBe(true);});for (const binding of DRIVEN_BINDINGS) {diff --git a/tests/e2e/index-freshen.test.ts b/tests/e2e/index-freshen.test.tsnew file mode 100644index 00000000..b6045776--- /dev/null+++ b/tests/e2e/index-freshen.test.ts@@ -0,0 +1,61 @@+/**+ * Freshen on read end to end, with a real detached child: a note written+ * after the last index run is not found by the first search, which starts+ * a background `o2b search index --freshen`; once that run has recorded+ * its outcome, the next search finds the note. No test seam is used.+ */+import { afterEach, beforeEach, expect, test } from "bun:test";+import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { FRESHEN_CLAIM_FILE, readFreshenState } from "../../src/core/search/freshen.ts";+import { runCli } from "../helpers/run-cli.ts";++let tmp: string;+let vault: string;+let configPath: string;++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-freshen-e2e-"));+ vault = join(tmp, "vault");+ mkdirSync(join(vault, "notes"), { recursive: true });+ configPath = join(tmp, "config.yaml");+ writeFileSync(configPath, `vault: ${vault}\n`);+ writeFileSync(join(vault, "notes", "old.md"), "# Old\n\nan older note about kestrels.\n");+});++afterEach(() => rmSync(tmp, { recursive: true, force: true }));++const env = (interval: string) => ({+ OPEN_SECOND_BRAIN_CONFIG: configPath,+ OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S: interval,+});++test(+ "a stale read starts a background run and the next read finds the new note",+ async () => {+ expect((await runCli(["search", "index"], { env: env("0") })).returncode).toBe(0);+ await Bun.sleep(1_100);+ writeFileSync(join(vault, "notes", "new.md"), "# New\n\na fresh note about wagtails.\n");++ const first = await runCli(["search", "query", "wagtails", "--json"], { env: env("1") });+ expect(first.returncode).toBe(0);+ expect(JSON.parse(first.stdout).results ?? []).toHaveLength(0);++ const derived = join(vault, ".open-second-brain");+ const deadline = Date.now() + 30_000;+ while (readFreshenState(derived).lastOutcome === null && Date.now() < deadline) {+ await Bun.sleep(200);+ }+ const state = readFreshenState(derived);+ expect(state.lastOutcome).toBe("completed");+ expect(state.lastChanged).toBe(1);+ expect(existsSync(join(derived, FRESHEN_CLAIM_FILE))).toBe(false);++ const second = await runCli(["search", "query", "wagtails", "--json"], { env: env("0") });+ const paths = (JSON.parse(second.stdout).results as Array<{ path: string }>).map((r) => r.path);+ expect(paths).toContain("notes/new.md");+ },+ { timeout: 60_000 },+);diff --git a/tests/helpers/platform-freshen.test.ts b/tests/helpers/platform-freshen.test.tsnew file mode 100644index 00000000..e9a282dd--- /dev/null+++ b/tests/helpers/platform-freshen.test.ts@@ -0,0 +1,10 @@+import { expect, test } from "bun:test";++import { homeEnv } from "./platform.ts";++// A hook or MCP server spawned with the minimal environment would+// otherwise fall back to freshening every 60 s and start real background+// indexers into temp vaults the test is about to delete.+test("the minimal child environment keeps freshen on read off", () => {+ expect(homeEnv("/tmp/h")["OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S"]).toBe("0");+});diff --git a/tests/helpers/platform.ts b/tests/helpers/platform.tsindex a0328247..7efd6b4e 100644--- a/tests/helpers/platform.ts+++ b/tests/helpers/platform.ts@@ -43,8 +43,13 @@ export const CHMOD_CANNOT_DENY = RUNNING_AS_ROOT || IS_WINDOWS;* three Windows variables under `home`, the layout Windows itself uses.*/export function homeEnv(home: string): Record<string, string> {- if (!IS_WINDOWS) return { HOME: home };+ // The preload pins freshen on read off for this process; a child spawned+ // with a minimal environment needs it pinned too, or a search it runs+ // starts real background indexers into a vault the test then deletes.+ const freshenOff = { OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S: "0" };+ if (!IS_WINDOWS) return { HOME: home, ...freshenOff };return {+ ...freshenOff,HOME: home,USERPROFILE: home,LOCALAPPDATA: join(home, "AppData", "Local"),diff --git a/tests/mcp/doctor-schema-reach.test.ts b/tests/mcp/doctor-schema-reach.test.tsindex 67de99ba..a9ca1de4 100644--- a/tests/mcp/doctor-schema-reach.test.ts+++ b/tests/mcp/doctor-schema-reach.test.ts@@ -85,6 +85,7 @@ async function answer(describe("a malformed private preference answers as an absent one at remote reach", () => {for (const [tool, args] of [["brain_doctor", {}],+ ["brain_doctor", { repair: true }],["schema_inspect", { view: "lint" }],["schema_inspect", { view: "orphans" }],["schema_inspect", { view: "stats" }],@@ -102,6 +103,8 @@ describe("a malformed private preference answers as an absent one at remote reacconst doctor = await answer(a, "brain_doctor", {}, TRANSPORT_REACH.local);expect(doctor).toContain(PREF);expect(doctor).toContain("tier-drift");+ const repair = await answer(a, "brain_doctor", { repair: true }, TRANSPORT_REACH.local);+ expect(repair).toContain("tier-drift");expect(await answer(a, "schema_inspect", { view: "orphans" }, TRANSPORT_REACH.local)).toContain(PREF,);diff --git a/tests/setup.ts b/tests/setup.tsindex 737e83da..13cf3d3f 100644--- a/tests/setup.ts+++ b/tests/setup.ts@@ -56,6 +56,14 @@ if (process.env["O2B_DEVICE_ID"] === undefined) {process.env["O2B_DEVICE_ID"] = "";}+// Freshen-on-read starts a detached `o2b search index` whenever a search+// meets an index older than the interval. A test suite must never leave+// background indexers behind, so the default is off here; the tests of the+// feature own the variable while they run.+if (process.env["OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S"] === undefined) {+ process.env["OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S"] = "0";+}+if (!process.env["OPEN_SECOND_BRAIN_CONFIG"]) {const root = mkdtempSync(join(tmpdir(), "osb-test-default-"));// A preload's afterAll runs once, after the last file: the throwawaydiff --git a/uv.lock b/uv.lockindex 219a29e9..f6ef08d7 100644--- a/uv.lock+++ b/uv.lock@@ -4,5 +4,5 @@ requires-python = ">=3.11"[[package]]name = "open-second-brain"-version = "1.75.0"+version = "1.76.0"source = { editable = "." }