Local-first π§ memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly π΄ dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
Index Freshness on Read Implementation Plan
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: A read that finds the search index older than a threshold starts one low-priority background incremental index run, so the index stays current while agents work, with no daemon and no scheduler.
Architecture: A new src/core/search/freshen.ts owns the decision (pure), the per-device state file and claim, and the detached spawn. search() and ensureVaultCurrent call it. The child is o2b search index --freshen <runId>. The indexer skips its whole-index post-passes on a no-change run, and the catch-up write volume is measured per phase and its dominant cause fixed.
Tech Stack: Bun 1.4 + TypeScript, bun:sqlite, bun:test.
Spec: docs/brainstorm/index-freshness/design.md
Global Constraints
- No resident process, timer, watcher or scheduler entry; indexing starts only from an agent-driven read or session start.
- A read never waits for indexing;
maybeFreshenIndexnever throws. - A read-only open (
opts.selfHeal === false) never freshens. - Config keys:
search_freshen_interval_s(default60,0= off, envOPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S),search_freshen_embeddings(defaultfalse, envOPEN_SECOND_BRAIN_SEARCH_FRESHEN_EMBEDDINGS). - Claim file
<dirname(dbPath)>/freshen.claim, abandoned after 10 minutes; state file<dirname(dbPath)>/freshen-state.json(per device, never inBrain/). - Backoff after a failed run: 60 s doubling, capped at 3600 s.
- Trail code
index-stalewithdetail.ageSecondswhen the index is older than 600 s. - Background child: CPU priority lowered with
os.setPriority;ionice -c3(Linux) ortaskpolicy -b(macOS) prefix when on PATH. - Catch-up write target: a ~200-file catch-up writes at most ~100 MB; if that needs a larger rework, stop and report numbers.
- Tests never spawn a real child except the one end-to-end test;
tests/setup.tspins the interval to0. - Version: next minor after
main, bump inside the PR; English only; no AI co-author trailers.
Review Focus
- Many sessions searching at once on a stale index: exactly one child runs; the others skip without error.
- A child killed mid-run (reboot, OOM): the claim is taken over after 10 minutes and no failure streak is invented.
- A vault whose index cannot be written (read-only disk, corrupt file): the child fails, backoff grows, searches keep working on the old index,
doctornames the streak. - Cross-vault (
global) and recall-source searches: never spawn into a foreign vault. - A schema-pack edit with no document change: the no-change fast path must still recompute relation-constraint flags.
Each item has a test in the owning task (Tasks 2, 2, 3, 4, 5 respectively).
Task 1: Freshen configuration
Files:
- Modify:
src/core/search/types.ts(addResolvedFreshenConfig, optionalfreshenonResolvedSearchConfig) - Modify:
src/core/search/index.ts(resolveSearchConfig) - Modify:
tests/setup.ts(pinOPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S=0unless set) - Test:
tests/core/search/freshen-config.test.ts
Interfaces:
Produces:
interface ResolvedFreshenConfig { readonly intervalSeconds: number; readonly embeddings: boolean; readonly configPath: string | null };ResolvedSearchConfig.freshen?: ResolvedFreshenConfig(absent = off, so hand-built configs in tests stay off).Step 1: Failing tests β default resolves
{ intervalSeconds: 60, embeddings: false, configPath }; envOPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S=0gives0; config keysearch_freshen_interval_s: "300"gives300;"-1"and"abc"throw naming the key;search_freshen_embeddings: "true"givestrue. Tests save and restore the env var because the preload pins it.Step 2: Run
bun test tests/core/search/freshen-config.test.tsβ FAIL (freshenundefined).Step 3: Implement with the existing
parseInteger(envOrConfig(...), 60, "search_freshen_interval_s", { min: 0 })andparseBoolhelpers;configPath: opts.configPath ?? null.Step 4: Preload β in
tests/setup.ts, next toO2B_DEVICE_ID:if (process.env["OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S"] === undefined) process.env["OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S"] = "0";with a comment that the suite must never spawn background indexers.Step 5: Run the new test and
bun run typecheckβ PASS.Step 6: Commit
feat(search): freshen interval and embeddings config.
Task 2: Decision, claim, state and spawn (freshen.ts)
Files:
- Create:
src/core/search/freshen.ts - Test:
tests/core/search/freshen.test.ts
Interfaces:
Consumes:
ResolvedFreshenConfig(Task 1),isWriterLockHeld(dbPath)fromstore/writer-lock.ts.Produces:
FRESHEN_SKIP = { off, fresh, backoff, writerLock, claimed, readOnly, noIndex }(string valuesoff,fresh,backoff,writer_lock,claimed,read_only,no_index).decideFreshen(input: { nowMs: number; lastIndexedAt: string | null; intervalSeconds: number; backoffUntilMs: number | null }): { action: "spawn" } | { action: "skip"; reason: FreshenSkip }(pure; lock and claim are checked after it).claimFreshen(dir: string, nowMs: number): string | nullβ exclusive create offreshen.claimholding{ token, at }; takes over a claim older than 600 s; returns the token ornull.releaseFreshen(dir: string, token: string): voidβ removes the claim only if it still holdstoken.readFreshenState(dir) / writeFreshenState(dir, state)withFreshenState { failures: number; backoffUntil: string | null; lastOutcome: "completed" | "failed" | null; lastRunAt: string | null; lastDurationMs: number | null; lastError: string | null; lastChanged: number | null }; a missing or torn file reads as the empty state.nextBackoffMs(failures: number): numberβmin(60_000 * 2 ** (failures - 1), 3_600_000).freshenCommand(base: string[], has: (tool: string) => boolean, platform: NodeJS.Platform): string[]β prefixesionice -c3on linux /taskpolicy -bon darwin whenhas(tool).maybeFreshenIndex(config: ResolvedSearchConfig, opts: { lastIndexedAt: string | null; readOnly?: boolean; nowMs?: number; spawn?: (argv: string[]) => void }): FreshenDecisionβ never throws; returns"spawned"or the skip reason.
Step 1: Failing tests for
decideFreshen:intervalSeconds 0βoff;lastIndexedAt nullβno_index(self-heal owns a missing index); age 30 s / interval 60 βfresh; age 61 s βspawn;backoffUntilMs > nowβbackoff.Step 2: Failing tests for the claim: first claim returns a token; a second returns
null(claimed); a claim file withat601 s ago is taken over;releaseFreshenwith a foreign token leaves the file (Review Focus 1, 2).Step 3: Failing tests for state and backoff: round-trip; torn JSON reads as empty;
nextBackoffMs(1) === 60_000,(2) === 120_000,(10) === 3_600_000.Step 4: Failing tests for
freshenCommand: linux withioniceβ["ionice", "-c3", ...base]; linux without βbase; darwin withtaskpolicyβ["taskpolicy", "-b", ...base]; win32 βbase.Step 5: Failing tests for
maybeFreshenIndexwith an injectedspawnand a temp dir: stale index β spawn called once withsearch index --vault <v> --freshen <id>(plus--config <p>when set); fresh β not called;readOnly: trueβread_only, not called (Review Focus 4); held writer lock (create the lock dir the waywriter-lock.tsdoes) βwriter_lock; aspawnthat throws β returns a skip, does not throw.Step 6: Run
bun test tests/core/search/freshen.test.tsβ FAIL.Step 7: Implement
freshen.ts. The real spawner reuses the self-heal pattern fromsrc/core/maintenance/ensure-current.ts(Bun.spawn, streamsignore,windowsHide,detachedon win32,env: { ...process.env },unref) and theo2bCommand()resolution (move it to a small shared helpersrc/core/maintenance/o2b-command.tsand import it from both). The claim token is passed as--freshen <token>.Step 8: Run tests and typecheck β PASS.
Step 9: Commit
feat(search): freshen decision, claim, state and spawn.
Task 3: The background child (o2b search index --freshen)
Files:
- Modify:
src/cli/search/verbs/indexing.ts(cmdSearchIndex) - Modify:
src/core/brain/metrics.tsonly if surfaces are registered there (addindex_freshen) - Test:
tests/cli/search-index-freshen.test.ts
Interfaces:
Consumes:
releaseFreshen,readFreshenState,writeFreshenState,nextBackoffMs(Task 2).Produces: flag
--freshen <token>; on success the state getsfailures: 0, lastOutcome: "completed", lastChanged: added+updated+deleted; on failurefailures+1,backoffUntil = now + nextBackoffMs(failures),lastError; a metric row on surfaceindex_freshenonly on failure or whenlastChanged > 0; the claim is released infinally.Step 1: Failing tests (in-process
cmdSearchIndexwith a temp vault):--freshen <token>afterclaimFreshenβ exit 0, claim gone, statecompleted; a vault whose db path is unwritable (a directory squattingbrain.sqlite) β statefailed,failures 1,backoffUntilset, claim gone (Review Focus 3); a second failure βfailures 2and a doubled backoff.Step 2: Run β FAIL.
Step 3: Implement: parse
freshen; when present,os.setPriority(process.pid, 10)in a try;embeddings: cfg.freshen?.embeddings === trueunless--embeddingswas given; wrap the run in try/catch/finally writing state, the metric row and releasing the claim; the exit code stays as today.Step 4: Run β PASS. Step 5: Commit
feat(search): background freshen child records its outcome.
Task 4: Call sites and the index-stale trail code
Files:
- Modify:
src/core/search/search.ts(after the store opens) - Modify:
src/core/maintenance/ensure-current.ts(after the rebuild check) - Modify:
src/core/search/retrieval-trail.ts(code, membership, description) - Modify:
docs/cli-reference.md(trail table row; required bytests/docs/retrieval-trail-doc.test.ts) - Test:
tests/core/search/freshen-search.test.ts, extendtests/core/maintenance/ensure-current*.test.ts
Interfaces:
Consumes:
maybeFreshenIndex(Task 2). A test seam:SearchOptions.freshenSpawn?: (argv: string[]) => voidpassed through tomaybeFreshenIndex({ spawn }).Produces:
RETRIEVAL_DEGRADATION.indexStale = "index-stale"withdetail: { ageSeconds };EnsureCurrentResult.freshen: FreshenDecision | null.Step 1: Failing tests: a search over an index whose
last_indexed_atis 5 min old withfreshen.intervalSeconds 60calls the injected spawn once and returns results;selfHeal: falsenever calls it; age 11 min adds{ code: "index-stale", detail: { ageSeconds } }todegraded; age 2 min does not; interval0with age 11 min still reportsindex-stalebut never spawns.ensureVaultCurrenton a current-schema stale index returnsfreshen: "spawned"(spawn injected throughEnsureCurrentOptions.freshenSpawn), and on a rebuild-needed index leavesfreshen: null.Step 2: Run β FAIL.
Step 3: Implement: in
search()readstore.getState(LAST_INDEXED_AT_STATE_KEY)once; callmaybeFreshenIndex(effectiveConfig, { lastIndexedAt, readOnly: opts.selfHeal === false, spawn: opts.freshenSpawn }); pushindex-stalewhen the age exceeds 600 s. InensureVaultCurrent, when no rebuild is needed, open the index read-only, read the stamp, callmaybeFreshenIndex.Step 4: Add the trail code to
RETRIEVAL_DEGRADATION,RETRIEVAL_DEGRADATION_CODES(after the keyword codes) anddescribeRetrievalDegradation; add thedocs/cli-reference.mdrow: "the index was last updated more than ten minutes ago, so notes changed since then may be missing; a background run has been started unless freshening is off or backing off".Step 5: Run the new tests,
tests/docs/retrieval-trail-doc.test.ts,tests/core/search/β PASS. Step 6: Commitfeat(search): reads refresh a stale index in the background.
Task 5: No-change run skips the whole-index post-passes
Files:
- Modify:
src/core/search/indexer.ts(around the post-passes after the delete loop) - Modify:
src/core/search/store/state.ts(new keyschema_pack_digest) - Test:
tests/core/search/indexer-noop.test.ts
Interfaces:
Produces: state key
SCHEMA_PACK_DIGEST_STATE_KEY = "schema_pack_digest";IndexStats.postPassesSkipped: boolean.Step 1: Failing tests: index a small vault, index again with no change β
postPassesSkipped: trueandlast_indexed_atadvanced; change a note βfalse; change onlyBrain/_brain.yamllink_constraintsβfalseand the blocked flag recomputed (Review Focus 5); a forced run βfalse.Step 2: Run β FAIL.
Step 3: Implement: compute a SHA-256 of the raw
_brain.yamlbytes (empty string when absent); skipresolveLinkTargets,resolveAliasTargets,recomputeRelationConstraintFlagsand the tier pass whenadded + updated + deleted === 0, not forced, and the digest equals the stored one; store the digest after the post-passes run. Keep the stamp, the census and the deferred-reason reads.Step 4: Run the new test and
tests/core/search/indexer*.test.tsβ PASS. Step 5: Commitperf(search): a no-change index run skips the whole-index post-passes.
Task 6: Catch-up write volume, measured then fixed
Files:
Create:
scripts/measure-index-writes.ts(Linux; reads/proc/self/iowrite_bytesaround phases; prints a table)Modify:
src/core/search/indexer.ts(optionalonPhasehook inIndexOptions, no behaviour change)Modify: the module of the dominant phase (decided by the measurement)
Step 1: Add
onPhase?: (phase: "walk" | "post" | "embed" | "stamp" | "close") => voidto the index options, called at phase boundaries. Commit.Step 2: The script copies the reference vault to a temp dir, indexes it, rewinds the copy's
last_indexed_atand touches 200 notes with new content, then runs a catch-up index and prints bytes per phase plus the close (WAL checkpoint). Run it; record the table in the PR description.Step 3: Fix the dominant phase. Expected candidates and fixes: per-file transactions β one transaction per batch of files; full-index link/alias resolution β restrict to links touching changed documents; WAL-to-DELETE checkpoint on every close β keep WAL between runs where the backing allows it. Write a regression test for the behaviour changed (not for byte counts).
Step 4: Re-run the script; target β€ ~100 MB. If not reachable without a broader rework, stop, keep what helped, and report the numbers. Commit
perf(search): <what was fixed>.
Task 7: Freshness in search status and doctor
Files:
Modify:
src/cli/search/verbs/status.tsModify: the readiness check module under
src/core/brain/doctor/that prints index linesTest: extend the status and readiness tests
Step 1: Failing tests:
o2b search statusprintsfreshen: every 60s(oroff),index_age: <n>s,last_freshen: completed <ts> (<n> changed)/failed <ts>: <error>, andfreshen_backoff_until: <ts>when active;--jsoncarries the same fields underfreshen. Readiness warns whenfailures >= 3.Step 2: Run β FAIL. Step 3: Implement from
readFreshenState(dirname(dbPath)). Step 4: Run β PASS. Step 5: Commitfeat(search): show index freshness in status and doctor.
Task 8: End-to-end with a real child
Files:
Test:
tests/e2e/index-freshen.test.tsStep 1: Temp vault + config, index it, write a new note, set
OPEN_SECOND_BRAIN_SEARCH_FRESHEN_INTERVAL_S=1for a spawnedo2b search query(viarunClihelpers), wait up to 20 s polling the state file forlastOutcome: "completed", then search finds the new note. Timeout generous for Windows.Step 2: Run on Linux β PASS. Step 3: Commit
test(search): background freshen end to end.
Task 9: Docs, version, gates, PR
Files: docs/cli-reference.md (keys, --freshen, status fields), docs/how-it-works.md, docs/architecture.md ("no daemon" clarification), docs/hermes-cron.md, the reindex --cron-template help text, README.md release line, CHANGELOG.md (+ compare link), package.json + bun run scripts/sync-version.ts, bun run sync-plugin-mirrors if a skill changed.
- Step 1: Write the docs and CHANGELOG (Added: freshen on read, trail code, status fields; Changed: no-change runs, catch-up writes with the measured numbers; note that a search may start a short background process and
search_freshen_interval_s: 0turns it off). - Step 2: Bump the minor version, sync,
--check. - Step 3: Gates:
bun run typecheck,bun run lint,bun run fmt:check,bun run link-ratchet:check,bun run sync-plugin-mirrors:check, the OpenClaw bundle diff with Bun 1.4.0,bun test --parallel. - Step 4: Commit, push the branch, open the PR with the measurement tables; wait for CI and review bots; fix findings.