Local-first ๐ง memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly ๐ด dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
fix(brain): harden reindex swap, self-heal, hot paths, and continuity contracts (1.24.0) (#125)
Sol Aitken1cde572ff4e1
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.jsonindex a8a85ed0..0dc97149 100644--- a/.claude-plugin/plugin.json+++ b/.claude-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.23.1",+ "version": "1.24.0","description": "Plugin-first second brain package for AI agents and humans.","author": {"name": "Open Second Brain contributors"diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.jsonindex 7f32d806..c54848fc 100644--- a/.codex-plugin/plugin.json+++ b/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.23.1",+ "version": "1.24.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","skills": "./skills","hooks": "./hooks/hooks.json",diff --git a/.gitignore b/.gitignoreindex c1cf3069..1a296f78 100644--- a/.gitignore+++ b/.gitignore@@ -25,6 +25,7 @@ sandbox-vault/.codegraph/.orphaned_at.claude/scheduled_tasks.lock+.claude/settings.local.json.open-second-brain/bench-runs/# Local Kanban/git worktrees (never committed; exclude from lint/fmt)diff --git a/CHANGELOG.md b/CHANGELOG.mdindex de2bca27..a3044e73 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,27 +5,90 @@ All notable changes to this project will be documented in this file.The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).+## [1.24.0] - 2026-07-06++A hardening and correctness pass over the PR #121 refactor. Thirteen fixes+across four classes land as one minor because the `basename` column addition+forces a `LATEST_SCHEMA_VERSION` bump and reindex on upgrade, and because the+`created_at` validation tightening turns previously-accepted garbage into+`INVALID_PARAMS`. The kernel still calls no LLM, and every new field is off or+absent by default with byte-identical output when unconfigured.++### Added++- **Self-heal on an unreadable search index.** `openReadOrSelfHeal` now rebuilds+ on `INDEX_UNREADABLE` alongside the existing `INDEX_MISSING` and+ `SCHEMA_MISMATCH` cases, so a corrupted but present index self-heals instead of+ surfacing a raw read error.++### Changed++- **`basename` column and index in the search store.** Resolved-wikilink+ discovery is now an equality join over a populated `documents.basename` column+ instead of a `documents` SUBSTR scan per link, removing a scan that grew with+ vault age. The column requires a `LATEST_SCHEMA_VERSION` bump, so every+ existing install reindexes once on next open. The reindex is absorbed together+ with the Han-bigram tokenizer change below, so users pay one reindex, not two.+- **Hot-path memoization.** The entity index, backlink index, and heal regex are+ now memoized with mtime and generation keys so they are not rebuilt on every+ query. The memos are opt-in by construction: they introduce a bounded staleness+ window (a write inside mtime granularity can serve one stale read), which is+ documented at the call site and pinned by write-then-read invalidation tests.+- **`links normalize --mode short` rewritten over a basename-to-paths Map.**+ Output is byte-identical to before, captured by a characterization fixture.+- **Continuity latest-record queries exit early in reverse chronological order.**+ The working-memory recompute no longer scans the full history when only the+ most recent record is needed.++### Fixed++- **Search index rebuild race and crash-restore clobber.** A separate reindex+ lock (`acquireReindexLock` / `isReindexInProgress`, distinct from the per-Store+ writer lock) now guards the staging build so a concurrent reindex cannot swap+ an empty staging database over the live index, and the crash-restore preamble+ no longer clobbers a freshly built index with a stale `.bak`. Three regression+ tests pin the path.+- **`created_at` validation tightened.** Continuity and query-demand records now+ require a full canonical UTC timestamp; a `YYYY-MM`-prefix timestamp that was+ previously accepted now fails with a structured `INVALID_PARAMS`. This is the+ contract-tightening gate that places these fixes behind a minor version.+- **Surrogate-safe card snippet truncation.** Snippet truncation now cuts on code+ points instead of UTF-16 code units, so an emoji or other astral character at+ the boundary no longer splits into a lone U+FFFD on the `cards` disclosure+ surface.+- **Second-precision `since` / `until` filtering.** Filter bounds run through+ timestamp normalization before the lexical compare, so millisecond and second+ precision are handled consistently and up to a second of records is no longer+ dropped at each boundary.+- **Secret-exec audit redaction.** The `command` field logged by the secret-exec+ path is now redacted before it reaches the audit trail, so a foreign bare+ credential is no longer persisted in cleartext.+- **Han-bigram tokenizer restricted to contiguous Han spans.** The CJK bigram+ pass no longer emits cross-script bigrams (`nๅฎ`) for mixed tokens, matching+ the docstring's Han-only scope. Pure-Han and pure-ASCII tokenization are+ unchanged.+## [1.23.1] - 2026-07-06### Fixed-- **`o2b init` fails on native Windows** โ `resolve()` produces backslash+- **`o2b init` fails on native Windows** -- `resolve()` produces backslashpaths that `setConfigValue()` rejects via `CONFIG_VALUE_REJECTED_CHARS`.Vault path is now normalized to forward slashes before persisting; forwardslashes work identically in Node/Bun fs APIs on Windows. (#119)-- **Hermes memory provider bridge fails on Windows** โ `scripts/o2b` is a+- **Hermes memory provider bridge fails on Windows** -- `scripts/o2b` is abash script that `subprocess.Popen` cannot execute directly (Windowsdelegates to `cmd.exe`, not a POSIX shell). Added `_resolve_command()`which detects the platform and falls back to `bun run <entry> mcp` when`o2b` is not available as a native executable.-- **`detectTooling()` fails to find tools on Windows** โ PATH was split on+- **`detectTooling()` fails to find tools on Windows** -- PATH was split on`:` only (POSIX); now uses `;` on Windows. Executable probing now honours`PATHEXT` (`.exe`, `.cmd`, `.bat`, `.com`) instead of hardcoding `.exe`.-- **MCP `inputSchema` not recognized by Hermes adapters (all platforms)** โ+- **MCP `inputSchema` not recognized by Hermes adapters (all platforms)** --Hermes adapters (Anthropic, OpenAI, Bedrock) expect tool schemas under`parameters`, but MCP serves them as `inputSchema`. Without this remapbrain tools had no visible arguments on any OS. Both live and static- fallback schemas now remap `inputSchema` โ `parameters` at the provider+ fallback schemas now remap `inputSchema` -> `parameters` at the providerboundary.## [1.23.0] - 2026-07-05@@ -6228,6 +6291,8 @@ plugin config (vault field)`, and exits with a clear- Sandbox vault and plugin manifest fixtures for tests.- GitHub release workflow for tag-based and manually dispatched releases.+[1.24.0]: https://github.com/itechmeat/open-second-brain/compare/v1.23.1...v1.24.0+[1.23.1]: https://github.com/itechmeat/open-second-brain/compare/v1.23.0...v1.23.1[1.23.0]: https://github.com/itechmeat/open-second-brain/compare/v1.22.0...v1.23.0[1.22.0]: https://github.com/itechmeat/open-second-brain/compare/v1.21.0...v1.22.0[1.21.0]: https://github.com/itechmeat/open-second-brain/compare/v1.20.0...v1.21.0diff --git a/README.md b/README.mdindex 0f94adaa..00caa69d 100644--- a/README.md+++ b/README.md@@ -8,7 +8,7 @@ Open Second Brain plugs into [Hermes Agent](https://github.com/NousResearch/herm## What is new-Open Second Brain 1.23.0 adds two optional precision layers that complete the retrieval-precision loop started in 1.22.0, each off by default with byte-identical output when unconfigured. A semantic entity dedup pass surfaces lexical entity-name variants as proposal-only alias-merge candidates when entity_semantic_dedup_enabled is set, via a deterministic jaccard layer with an embedding-cosine reader that reuses the configured embedding provider, and never rewrites the deterministic identity key. An optional cross-encoder rerank stage re-scores the top-K fused search candidates jointly against the query through an OpenAI-compatible endpoint (search_rerank_enabled, off by default), most valuable on the thorough profile, with zero HTTP cost and unchanged ordering when disabled and a graceful degrade to heuristic ordering on endpoint error. The kernel still calls no LLM.+Open Second Brain 1.24.0 is a hardening and correctness pass. The search index rebuild is now guarded by a separate reindex lock so a concurrent or interrupted reindex cannot swap an empty or stale database over the live index, and an unreadable index self-heals instead of surfacing a raw read error. Wikilink resolution runs over a new `documents.basename` column and index instead of a per-link SUBSTR scan, and the entity, backlink, and heal hot paths are memoized so they are not rebuilt on every query. Continuity records now require a full canonical timestamp, card snippet truncation no longer splits an astral character into a lone replacement character, and `since` / `until` filtering handles second precision consistently. The secret-exec audit trail redacts the command field before it is persisted, and the CJK bigram tokenizer stays inside contiguous Han spans. The `basename` column requires a schema bump, so every existing install reindexes once on next open. The kernel still calls no LLM.## Whydiff --git a/docs/brainstorm/pr121-hardening-correctness/cli-output/claude.md b/docs/brainstorm/pr121-hardening-correctness/cli-output/claude.mdnew file mode 100644index 00000000..c06f6c46--- /dev/null+++ b/docs/brainstorm/pr121-hardening-correctness/cli-output/claude.md@@ -0,0 +1,42 @@+## Variant 1 โ Card-sequential single minor (1.24.0), risk-ordered++**Approach:** Ship all 13 fixes as one minor on the shared branch, committed one card at a time in risk order: land t_f27d80fe first (rebase `ec3bad27` in as-is, then extend it with the `INDEX_UNREADABLE`โself-heal change as a follow-on commit), then the low-risk correctness cluster (B.1โB.4 + the Han-bigram tokenizer), then the contract-neutral perf items (C.1, C.3โC.6), and finally C.2 as the release-defining commit carrying the `LATEST_SCHEMA_VERSION` bump, the `basename` column, and the version/CHANGELOG bump. One reindex at release absorbs both C.2 and the tokenizer's re-tokenization.++**Trade-offs:**+- Each card stays a contiguous, independently-revertable commit cluster โ matches the "one card at a time / each unit revertable" convention exactly.+- Data-loss fix reaches `main` first and stays green through every later commit; perf work rebases on a stabilized base, not vice-versa.+- Self-heal expansion rides with t_f27d80fe (its natural home) and stays decoupled from the reindex-forcing schema bump โ two independent safety/contract surfaces reviewed separately.+- The whole minor is gated behind one reindex because of C.2, so even pure-correctness fixes only take effect post-reindex for users โ acceptable for a minor, but not a fast path for the P4 data-loss fix.+- Longest single branch lifetime; later cards carry the most rebase surface.++**Complexity:** medium+**Risk:** low++## Variant 2 โ Two-release split: correctness patch (1.23.1) then perf+schema minor (1.24.0)++**Approach:** Cut two releases from the shared branch. First a patch: t_f27d80fe (data-loss) + the contract-neutral parts of t_2ba5f0c9 (B.2 surrogate, B.3 precision, B.4 audit redaction) โ no schema change, no reindex, ships immediately. Then a minor stacked on the patch: all of t_1224c740, C.2's schema bump, the Han-bigram tokenizer, and B.1's `created_at` tightening, all gated behind a single reindex.++**Trade-offs:**+- Gets the highest-priority silent-data-loss fix to users fastest, with zero reindex burden.+- Two version bumps + two CHANGELOG headings + two PRs on a protected `main` โ doubles the release ceremony that CLAUDE.md mandates per PR.+- Forces splitting t_2ba5f0c9 across releases: B.1 is a contract *tightening* (previously-accepted garbage โ `INVALID_PARAMS`) that belongs behind a minor, so the card can't land atomically โ violates card atomicity.+- Two reviewable surfaces are each smaller, but the stacked-PR dependency and cross-release version reconciliation add coordination risk.+- Best operational safety for the data-loss class; worst fit for the "drive cards one at a time, atomically" framing.++**Complexity:** large+**Risk:** medium++## Variant 3 โ Contract-surface-partitioned single minor (1.24.0)++**Approach:** Regroup the 13 fixes by whether they touch a frozen contract or require reindex, ignoring card boundaries for commit shaping. Land every contract-neutral fix first (t_f27d80fe lock, B.2/B.3/B.4, C.1/C.3โC.6), then a single terminal "contract + schema" commit that bumps `LATEST_SCHEMA_VERSION`, adds the `basename` column, tightens `created_at` (B.1), folds in the tokenizer reindex, expands the self-heal set, and carries the version bump โ one reindex, one concentrated contract-review target.++**Trade-offs:**+- Concentrates the entire frozen-contract blast radius into one clearly-labeled commit โ ideal review focus for a frozen-contract project.+- Fuses unrelated fixes (schema column, timestamp validation, tokenizer, self-heal) into one fat commit: sacrifices per-fix revertability and violates "each unit independently revertable / atomic conventional commits."+- Couples the `INDEX_UNREADABLE` self-heal safety change to the reindex-forcing schema bump โ a rollback of one drags the other.+- Cross-cutting commit ordering diverges from the "one card at a time" driving model, complicating per-card TDD attribution and merge-conflict reasoning against siblings.++**Complexity:** medium+**Risk:** high++**Recommended: Variant 1** โ it is the only shaping that honors every stated convention simultaneously: cards land one at a time as contiguous, independently-revertable commit clusters with per-fix TDD, the P4 data-loss fix stabilizes the base before perf work rebases onto it, and the single unavoidable reindex is anchored to C.2 as its own auditable schema commit โ keeping V3's contract-review clarity (schema bump is isolated) without fusing unrelated fixes or coupling the self-heal safety change to the reindex. The split in V2 buys faster data-loss delivery at the cost of doubled release ceremony on a protected `main` and a non-atomic t_2ba5f0c9, neither of which this repo's bump-in-PR discipline and card-atomic framing tolerate cleanly.diff --git a/docs/brainstorm/pr121-hardening-correctness/cli-output/prompt.md b/docs/brainstorm/pr121-hardening-correctness/cli-output/prompt.mdnew file mode 100644index 00000000..0ee6ea37--- /dev/null+++ b/docs/brainstorm/pr121-hardening-correctness/cli-output/prompt.md@@ -0,0 +1,145 @@+You are a senior software architect consulting on an Open Second Brain hardening release.++# Project++**Open Second Brain** โ an Obsidian-native memory layer for AI agents. Plain Markdown the user owns, in the vault they already use. The agent reads/writes a `Brain/` directory of `.md` files through deterministic CLI / MCP tools. No daemon, no hidden state outside the vault.++- **Runtime:** Bun + TypeScript (`"type": "module"`, ESM), targets Node. Tests run via `bun test` (script `scripts/test`).+- **Lint/format:** oxlint + oxfmt. Typecheck: `tsc --noEmit`. `bun run validate` = typecheck + lint + test.+- **Versioning:** SemVer with **frozen public contracts** since 1.0.0 (MCP tool surface: tool names + input schemas + response-field meanings; CLI verb tree; documented config keys/env mirrors). New optional params and new response fields are additive (minor); renaming/removing/restricting is breaking (major).+- **Stability policy:** search schema changes carry a `LATEST_SCHEMA_VERSION` bump and require a reindex. The kernel calls no LLM; embedding providers are pluggable.+- **Current version:** 1.23.0. This work targets a follow-up minor/patch release.++## Conventions (from prior suites, must be respected)++- TDD: failing test first, then implementation; characterization tests pin behavior before refactors.+- One feature branch, atomic conventional commits, full suite green between commits. Each unit independently revertable.+- SOLID / KISS / DRY. No misleading fallbacks (a degraded path must log that it degraded, never silently fake success). No hardcoding of paths, sizes, or magic numbers.+- All repo-facing strings (code, docs, commits, CHANGELOG, config) are **English-only**; logic is abstracted to be multi-language (CJK correctness matters โ there are dedicated Han/kana tests).+- NUL-byte sentinel files exist in `src/core/search/` (e.g. `search.ts`); plain grep silently skips them โ workers use `command grep -a` or codegraph.+- Existing patterns for file locks: a dedicated lockfile path (the in-flight fix for t_f27d80fe uses `acquireReindexLock`/`isReindexInProgress` โ a separate lockfile from the per-`Store.open` writer lock).++# Git context (recent history)++```+67bcb71c refactor: DRY and decomposition (Phases 0-2) (#121) <-- merge commit, branch base+b9bbcb16 feat(brain): semantic entity dedup and cross-encoder rerank (1.23.0) (#120)+b8d709ee fix: keep full MCP status output and normalize codegraph paths (#116)+a98bed1d feat(brain): retrieval precision and quality loop (v1.22.0) (#118)+42816058 feat(brain): integrity & safety hardening suite (1.21.0) (#115)+313d061e feat: configurable skills_dir + trigger-keyword auto-attach scoring (#114)+```++A validated local fix for **t_f27d80fe** already exists on branch `fix/index-swap-hardening` (commit `ec3bad27`, not yet pushed/PR'd) โ a dedicated reindex-level lock. The implementation phase may rebase/merge that onto this release branch rather than redo it.++# Scope: 4 tasks shipping together in one release++These four cards are driven **one at a time on a shared branch** `feat/pr121-hardening-correctness`. Each later card's worker must build on the commits the earlier cards landed and must not duplicate or conflict with siblings.++---++## TASK t_f27d80fe โ Index-swap hardening: fix reindex/crash-restore data-loss races (P4)++Two interlocking races around the search-index rebuild. **Silent data-loss class โ highest priority.**++**A.1 Concurrent `reindexVault` runs leave the live index unreadable.**+- Where: `src/core/search/indexer.ts:659,664` (pre-lock `tryUnlink(newPath)` + staging-db seed) and `src/core/search/store.ts:356-359`.+- Failure: run B deletes run A's in-progress staging DB before any lock, seeds a fresh empty staging DB at the same path. A finishes into the orphaned inode and its final `renameSync(newPath, dbPath)` renames B's EMPTY seed over the live index. An empty DB has no `index_state` table โ every subsequent open throws `INDEX_UNREADABLE`, which `openReadOrSelfHeal` (search.ts:210-226) deliberately does NOT self-heal. Search broken until manual `o2b search reindex`.+- Trigger: a schema-bump upgrade bumps `LATEST_SCHEMA_VERSION`; every concurrent search surface (CLI + long-lived MCP server) hits `SCHEMA_MISMATCH` at once and each invokes `reindexVault`.+- Fix direction: acquire the writer lock BEFORE touching `newPath` (unlink + seed move inside the lock); second contender waits-or-bails. Consider `INDEX_UNREADABLE` joining the self-heal set once the swap is safe.++**A.2 Crash-restore can clobber a freshly built index with a stale copy.**+- Where: `src/core/search/indexer.ts:680-682` (swap: `tryUnlink(bak); tryRename(db->bak); renameSync(new->db)`) vs `src/core/search/store.ts:295-306` (every `Store.open` runs a lockless crash-restore preamble: `existsSync(db)==false && existsSync(bak) -> renameSync(bak, db)`).+- Failure: a concurrent open in the window between `rename(db->bak)` and `rename(new->db)` restores the STALE pre-reindex snapshot, silently discarding the whole rebuild (incl. paid embeddings), with a misleading "restored" note.+- Fix direction: hold the writer lock across the swap AND make the crash-restore preamble probe the same lock; alternatively gate the restore on a marker file the swap removes last. Must keep genuine crash-recovery working (pin with a test).++**In-flight validated fix (commit ec3bad27):** dedicated reindex-level lock (`acquireReindexLock`/`isReindexInProgress` in store.ts, separate lockfile from per-Store.open writer lock) held across the entire reindexVault sequence โ staging build through final rename. Second concurrent reindexVault fails fast with existing typed `INDEX_LOCKED`. `Store.open`'s crash-restore preamble checks `isReindexInProgress` (non-blocking) and skips restoring while a reindex is actively running. 3 new regression tests: second reindexVault rejected while one in progress; Store.open does not restore `.bak` mid-reindex; genuine crash-restore still works.++## TASK t_2ba5f0c9 โ Validation/output hardening (P4): four small independent fixes++**B.1 `created_at` accepted with only a `YYYY-MM` prefix check.**+- Where: `src/mcp/brain/generation-tools.ts:88-90` โ `src/core/brain/generation-reports.ts:88,106-111` โ `src/core/brain/continuity/store.ts:42-45,176`. Latent hole via `host-memory-write.ts:103` and `session-summary.ts:95`.+- Failure: `"2026-13-01T00:00:00Z"` passes (junk `2026-13.jsonl` shard); `"2026-07-06T15:00:00+03:00"` passes then mis-sorts/mis-filters vs the fleet of `Z` timestamps (lexical compare at store.ts:213,222-223), shards into local-time month across boundaries.+- Fix: validate full timestamp at the continuity-store boundary (canonical UTC `Z` form, real calendar date), reject with structured error. Contract tightening: previously-accepted garbage โ INVALID_PARAMS.++**B.2 Card snippet truncation can split a surrogate pair.**+- Where: `src/core/search/search.ts:1121-1126` (`cardSnippet`: `collapsed.slice(0, CARD_SNIPPET_CHARS)`, `CARD_SNIPPET_CHARS=240`). File has a NUL-byte sentinel.+- Failure: UTF-16 units 239/240 straddling an astral char (emoji, rare CJK) โ lone surrogate rendered as U+FFFD, then "...". Flows into MCP `cards` disclosure surface.+- Fix: truncate on code points (`[...collapsed]` or `Intl.Segmenter`). Changes output bytes for affected inputs.++**B.3 Query-demand `since`/`until` precision-inconsistent filtering.**+- Where: `src/core/brain/query-demand.ts:238-239` (lexical compare) vs `:448-453` (`normalizeDemandTimestamp`, exists to fix this class); raw filter strings from `knowledge-gaps.ts:23` and `recall-tools.ts:291`.+- Failure: second-precision `--since 2026-07-01T00:00:00Z` lexically excludes every stored ms-precision record in `[00:00:00.000Z, 00:00:00.999Z]` because `.` sorts before `Z` โ up to one second of records wrongly dropped at each boundary.+- Fix: run filter bounds through `normalizeDemandTimestamp` before comparing.++**B.4 Secret-exec audit trail can persist a foreign bare secret.**+- Where: `src/core/brain/secrets/exec.ts:88,96-108` โ audit records log `command = argv.join(" ")`; `appendAuditRecord` (`reliability/audit.ts:22`) redacts with default key=value-shaped patterns only.+- Failure: agent passing some OTHER plaintext credential as a bare positional (`["mytool", "sk-abc123"]`) lands it in `Brain/log/secret-custody/*.jsonl` in cleartext; shaped patterns don't match bare tokens. Severity low but audit file is long-lived.+- Fix: redact the logged `command` with `redactInfra: true` and/or a high-entropy-token pass before persisting.++## TASK t_1224c740 โ Hot-path performance (P4): six items, each reviewable++**C.1 Entity registry re-read from disk on every search query.**+- Where: `src/core/search/entity-alias.ts:45` โ `src/core/brain/entities/index-builder.ts:80-106` (NUL file); second payer `src/core/search/query-expansion.ts:106`. Per-write rebuilds in `entities/registry.ts` are tolerable; per-QUERY rebuild is not.+- Cost: `readdirSync` + `readFileSync` + frontmatter parse of every file under `Brain/entities/` per query that names an entity. At 500-2000 entities that dwarfs the FTS/vector lookup.+- Fix: mtime-keyed memo (dir mtime probe per query, full rebuild on change). Staleness-window trade-off (write within mtime granularity โ one stale read).+- Test: invalidation test (write โ next query sees the new entity).++**C.2 `documents.basename` column to kill SUBSTR full scans.**+- Where: `src/core/search/store.ts:1064-1086` (`resolvedDocLinkPairs`) โ per unresolved link a COALESCE falls through to a `SUBSTR(d.path, ...)` suffix match plus an identical COUNT(*) uniqueness scan; neither can use `idx_documents_path`.+- Cost: dangling wikilinks never materialize, so every `brain_bridges`/cluster call re-pays 2 full `documents` scans per dangling link โ at 5k docs / 2-5k dangling links, tens of millions of row visits per call.+- Fix: add a `basename` column (populated at index time) with an index; rewrite suffix matches as equality joins. **Schema version bump + reindex required.**++**C.3 `buildBacklinkIndex` cross-call cost model.**+- Where: `src/core/brain/backlinks.ts:268-272` and per-tool-call callers (`query-tools.ts:250`, `resources.ts:369`, `doctor.ts:930`, `explorer.ts:88`, moc-audit, concept-cluster). Every call reads+parses the ENTIRE log history; grows unboundedly with brain age. Per-date dir rescan and digest double-build were fixed in PR #121 (no-behavior-change); this is the remaining cross-call recompute.+- Fix: mtime/generation-keyed memo (staleness trade-off, like C.1) OR incremental log cursor. "Recompute on demand" is a documented design choice โ revisit deliberately, not drive-by.++**C.4 Heal-enrich per-page regex rebuild.**+- Where: `src/core/brain/heal-run.ts:76-88` + `src/core/brain/heal-enrich.ts:42-58`. Per page, re-sorts all K known phrases and recompiles a K-branch alternation regex; at 5k pages ร 5k phrases = 5k compiles of a ~100-500KB pattern per dream heal run. Opt-in flag + dream-time-only already mitigate.+- Fix: sort once, compile once, post-filter self-matches per page. Also drop the duplicate frontmatter parse at heal-run:80.++**C.5 `links normalize --mode short` quadratic resolution.**+- Where: `src/cli/brain/verbs/links.ts:72-86`, `format-wikilink.ts:60-69,76-90`. O(files ร links ร pages) `endsWith` scans + fresh P-sized array allocation per link; minutes of wall clock at 5k pages / 100k links.+- Fix: build a basenameโpaths Map once per run; resolve+disambiguate against it. **Output must stay byte-identical โ pin with a fixture before optimizing.**++**C.6 Continuity store full-history reads for "find latest".**+- Where: `src/core/brain/continuity/store.ts:190-215` (`readAllRecords` parses every shard then filters); "latest"-seeking consumers at `session-summary.ts:182`, `pre-compact-extract.ts:148`, recall-telemetry/memory-cost/context-receipts full reads per call.+- Fix: date-named shards allow a `since`-aware file skip and reverse-chronological early exit for latest-record queries.++Sequencing note from the card: C.1 + C.2 are the biggest per-query wins; C.2 rides a schema bump which pairs naturally with a minor release. Remaining items opportunistic, each with a before/after measurement on a synthetic large vault.++## TASK t_3ccb1512 โ Han-bigram tokenizer: restrict bigrams to contiguous Han spans (P3)++- `tokenize()` in `src/core/surface/lexical-score.ts` emits overlapping 2-char bigrams for any token containing a Han char with length > 2. For a *mixed* token (`gbrainๅฎ็ฐ`, no separator) windows span the ASCII/Han boundary โ bigrams like `nๅฎ`: inert noise (match nothing real) but inflate term frequency/document length.+- A Han run of exactly 2 chars in a longer mixed token (`abๅฎ็ฐcd`) is gated by total length not run length โ inconsistent vs a standalone 2-char Han token.+- Desired: extract bigrams only from maximal contiguous Han spans, leaving non-Han segments untouched; full token still emitted. Removes cross-script noise; makes "Han" scope precise (docstring already says Han-only).+- Acceptance (TDD): `tokenize("gbrainๅฎ็ฐ")` emits `ๅฎ็ฐ` + full token, NO `nๅฎ`; `tokenize("abๅฎ็ฐcd")` emits `ๅฎ็ฐ`; pure-Han and pure-ASCII unchanged; no duplicate emission for standalone 2-char `ๅฎ็ฐ`.+- Out of scope: widening beyond Han (kana/Hangul/CJK extensions).++# Related source files (already inspected)++- `src/core/search/indexer.ts` (reindex swap sequence ~659-682), `src/core/search/store.ts` (crash-restore preamble ~293-306, schema-version gate ~340, `resolvedDocLinkPairs` ~1064-1086)+- `src/core/search/search.ts` (`cardSnippet` ~1121-1126, `openReadOrSelfHeal` ~210-226) โ NUL-byte sentinel+- `src/core/brain/continuity/store.ts` (created_at handling ~42-45,176,213,222-223; `readAllRecords` ~190-215)+- `src/core/brain/query-demand.ts` (since/until ~238-239, `normalizeDemandTimestamp` ~448-453)+- `src/core/brain/secrets/exec.ts` (~84-112), `src/core/brain/reliability/audit.ts` (~22)+- `src/core/brain/entities/index-builder.ts` (~80-106, NUL file), `src/core/search/entity-alias.ts:45`, `src/core/search/query-expansion.ts:106`+- `src/core/brain/backlinks.ts` (~268-272), `src/core/brain/heal-run.ts` (~76-88), `src/core/brain/heal-enrich.ts` (~42-58)+- `src/cli/brain/verbs/links.ts` (~72-86), `src/core/brain/link-graph/format-wikilink.ts` (~60-90)+- `src/core/surface/lexical-score.ts` (`tokenize`)++# Your task++Produce exactly THREE distinct architectural variants for delivering this four-task hardening+correctness+performance release on a single shared branch, then exactly one recommendation.++A "variant" is a *release-shaping* decision: how the 13 atomic fixes (2 in t_f27d80fe, 4 in t_2ba5f0c9, 6 in t_1224c740, 1 in t_3ccb1512) are grouped/sequenced/committed, how the schema-bump items (C.2) interact with the frozen search schema contract and the reindex requirement, how the in-flight validated fix (ec3bad27) is reconciled with a fresh implementation on the release branch, how staleness-window memoization (C.1, C.3) is exposed/contained, and how the output-byte-changing fixes (B.2, C.5 byte-identical pin) are de-risked. Consider: should everything ship as one minor, or split? Should the schema bump ride with t_f27d80fe's self-heal change or stay independent? Are the low-risk correctness fixes (B.1-B.4, t_3ccb1512) better landed first to stabilize, or last to avoid merge churn against the bigger perf work?++For EACH variant give:+- **Approach** (2-3 sentences)+- **Trade-offs** (bullets)+- **Complexity** (small | medium | large)+- **Risk** (low | medium | high)++Then exactly one line: **Recommended: Variant N** followed by a concise rationale (why it best fits the project's frozen-contract stability bar, TDD discipline, and reviewability).++Variants and recommendation ONLY. No code. No sections outside the three variants + the recommendation.diff --git a/docs/brainstorm/pr121-hardening-correctness/design.md b/docs/brainstorm/pr121-hardening-correctness/design.mdnew file mode 100644index 00000000..3019c116--- /dev/null+++ b/docs/brainstorm/pr121-hardening-correctness/design.md@@ -0,0 +1,99 @@+# PR #121 hardening + correctness โ design++**Status:** draft+**Author:** product-tech-lead (via Phase 0 brainstorm)+**Audience:** implementation+**Branch:** `feat/pr121-hardening-correctness` (from `main` @ `67bcb71c`, PR #121 merge)+**Scope source:** three-lens discovery pass (correctness/security/performance) on PR #121, 2026-07-06++## Problem statement++PR #121 (`refactor: DRY and decomposition`) landed as a pure no-behavior-change refactor. A three-lens discovery pass over its diff surfaced 13 concrete fixes across four classes that were deliberately deferred so they would not block the refactor:++- **Silent data loss** โ two interlocking races around the search-index rebuild (`reindexVault`): a concurrent run can swap an empty staging DB over the live index, and the crash-restore preamble can clobber a freshly built index with a stale `.bak`. Highest priority; realistic trigger is a schema-bump upgrade firing concurrent reindexes across every search surface.+- **Validation/output holes** โ `created_at` accepted with only a `YYYY-MM` prefix (junk shards, mis-sorted records); card snippet truncation splitting a UTF-16 surrogate pair (lone U+FFFD on the MCP `cards` surface); `since`/`until` precision-inconsistent filtering dropping up to a second of records at each boundary; the secret-exec audit trail persisting a foreign bare credential in cleartext.+- **Hot-path regressions** โ the entity index re-read from disk on every query; dangling-wikilink resolution doing full `documents` SUBSTR scans per link; unbounded backlink/heal/links-normalize/continuity recompute costs that scale with brain age.+- **CJK tokenizer noise** โ the Han-bigram pass emits cross-script bigrams (`nๅฎ`) for mixed tokens, inflating term frequencies despite the docstring promising Han-only scope.++All four are independently implementable; none were meta/umbrella; none have unmet parent dependencies.++## Scope++Four cards driven one at a time on the shared branch, each as a contiguous, independently-revertable commit cluster with per-fix TDD:++- `t_f27d80fe` (P4) โ index-swap hardening (2 fixes: concurrent-reindex race + crash-restore clobber). **Rebases the validated local fix `ec3bad27` (`acquireReindexLock`/`isReindexInProgress`) onto this branch, then extends it with the `INDEX_UNREADABLE`โself-heal change as a follow-on commit.**+- `t_2ba5f0c9` (P4) โ validation/output hardening (4 fixes: B.1 `created_at` tightening, B.2 surrogate-safe snippet, B.3 since/until precision, B.4 secret-exec audit redaction).+- `t_1224c740` (P4) โ hot-path performance (6 items: C.1 entity-index memo, C.2 `basename` column + schema bump, C.3 backlink memo, C.4 heal regex compile-once, C.5 links-normalize Map, C.6 continuity reverse-chron early exit).+- `t_3ccb1512` (P3) โ Han-bigram tokenizer restricted to contiguous Han spans.++Release target: **1.24.0** (minor) โ C.2 forces a `LATEST_SCHEMA_VERSION` bump + reindex, and B.1 is a contract tightening (previously-accepted garbage โ `INVALID_PARAMS`).++## Out of scope++- Widening the CJK tokenizer beyond Han (kana / Hangul / CJK extensions) โ separate enhancement, explicitly excluded by `t_3ccb1512`.+- The dependency-blocked high-value card `t_4678a91a` (blocked on out-of-scope `t_62363378`) โ deliberately excluded from this release.+- Registry/manifest-driven command frameworks or any new public MCP tools / CLI verbs / config keys beyond the schema bump.+- Auto-merging of semantic entity dedup (1.23.0's proposal-only surface stays proposal-only).+- Any change to the `o2b search reindex` trigger semantics; the reindex requirement itself is unchanged, only ridden once at release.++## Chosen approach++**Consultant Variant 1 โ Card-sequential single minor (1.24.0), risk-ordered.** Ship all 13 fixes as one minor on the shared branch, committed one card at a time in risk order:++1. `t_f27d80fe` first โ rebase `ec3bad27` (reindex-level lock) in as-is, then extend with the `INDEX_UNREADABLE` self-heal as a follow-on commit.+2. The low-risk correctness cluster โ `t_2ba5f0c9` (B.1โB.4) and `t_3ccb1512` (Han-bigram tokenizer).+3. Contract-neutral perf items โ C.1, C.3, C.4, C.5, C.6 of `t_1224c740`.+4. `t_1224c740` C.2 last โ the release-defining commit: `LATEST_SCHEMA_VERSION` bump, `basename` column + index, version + CHANGELOG bump. One reindex at release absorbs both C.2 and the tokenizer's re-tokenization.++It is the only shaping that honors every stated convention simultaneously: cards land one at a time as contiguous, independently-revertable commit clusters with per-fix TDD; the P4 data-loss fix stabilizes the base before perf work rebases onto it; and the single unavoidable reindex is anchored to C.2 as its own auditable schema commit without fusing unrelated fixes or coupling the self-heal safety change to the reindex.++## Design decisions++- **Reconcile `ec3bad27`, don't redo it.** The local fix branch already implements `acquireReindexLock`/`isReindexInProgress` (a separate lockfile from the per-`Store.open` writer lock) with 3 regression tests. The implementation phase rebases/merges it onto this release branch rather than reimplementing. The self-heal expansion (`INDEX_UNREADABLE` joining `INDEX_MISSING`/`SCHEMA_MISMATCH` in `openReadOrSelfHeal`) is a *separate follow-on commit* on top, gated on the swap being safe โ so the safety change stays decoupled from the lock and independently revertable.+- **Self-heal expansion stays decoupled from the schema bump.** `INDEX_UNREADABLE`โself-heal rides with `t_f27d80fe` (its natural home, reviewable as a safety fix), NOT bundled into C.2's reindex-forcing commit. A rollback of the schema bump must not drag the self-heal safety change with it.+- **One reindex, anchored to C.2.** C.2 is the only item that bumps `LATEST_SCHEMA_VERSION` (forcing a reindex). The Han-bigram tokenizer (`t_3ccb1512`) changes tokenization but does not bump the schema โ its effect on already-indexed content is absorbed by the single C.2-triggered reindex at release, so users pay one reindex, not two.+- **B.1 (`created_at`) is the contract-tightening gate for minor vs patch.** Previously-accepted garbage timestamps become `INVALID_PARAMS` โ a behavior change that belongs behind a minor, which is why this release is 1.24.0 and not 1.23.1. The other correctness fixes (B.2/B.3/B.4) are byte-changing but not contract-breaking and could in principle patch; they ride the minor because splitting the card across releases would violate card atomicity (Variant 2 rejected).+- **Staleness-window memos (C.1, C.3) are opt-in by construction, not config.** mtime/generation-keyed memoization introduces a bounded staleness window (a write within mtime granularity can serve one stale read). This is a *deliberate behavior trade-off*, documented at the call site, not a hidden flag โ matching the project's "no misleading fallbacks" rule (a memo hit must not silently lie about freshness; invalidation tests pin writeโread-visibility).+- **C.5 output is pinned byte-identical before optimization.** The `links normalize --mode short` rewrite (basenameโpaths Map) must not change user-facing output; a fixture captures current output first, then the optimization must reproduce it exactly.+- **B.2 snippet truncation changes output bytes for affected inputs โ by design.** Truncating on code points (`[...collapsed]` or `Intl.Segmenter`) fixes a lone-surrogate defect; the new bytes are correct where the old were corrupt. This is a fix, not a regression, and is covered by an emoji-at-boundary fixture.+- **No new dependencies.** All fixes use stdlib (`Array` spread, `Intl.Segmenter` where available, SQLite column + index, `Map`). The schema bump uses the existing migration path.+- **NUL-byte sentinel files respected.** `search.ts` and `index-builder.ts` contain NUL bytes; workers use `command grep -a` or codegraph, never plain grep (which silently skips matches).++## File changes++**`t_f27d80fe` (rebase `ec3bad27` + extend):**+- `src/core/search/store.ts` โ `acquireReindexLock`/`isReindexInProgress` (separate lockfile from per-`Store.open` writer lock); crash-restore preamble probes `isReindexInProgress` non-blocking.+- `src/core/search/indexer.ts` โ `reindexVault` acquires the reindex lock before touching `newPath` (unlink + seed move inside the lock).+- `src/core/search/search.ts` (NUL file) โ `openReadOrSelfHeal` self-heal set extended to include `INDEX_UNREADABLE`.+- `tests/core/search/` โ 3 regression tests from `ec3bad27` (second reindex rejected mid-flight; `Store.open` does not restore `.bak` mid-reindex; genuine crash-restore still works) + self-heal test for `INDEX_UNREADABLE`.++**`t_2ba5f0c9` (4 fixes):**+- `src/core/brain/continuity/store.ts` โ full timestamp validation at the boundary (canonical UTC `Z`, real calendar date); reject with structured `INVALID_PARAMS`.+- `src/core/search/search.ts` (NUL file) โ `cardSnippet` truncates on code points.+- `src/core/brain/query-demand.ts` โ filter bounds run through `normalizeDemandTimestamp` before lexical compare.+- `src/core/brain/secrets/exec.ts` โ logged `command` redacted with `redactRawOutput`/`redactInfra` and/or a high-entropy-token pass before it is handed to `appendAuditRecord`. The audit sink itself (`src/core/reliability/audit.ts`, imported via `../../reliability/audit.ts`) is unchanged; the fix lives at the call site in `exec.ts`, not in the audit module.+- `tests/` โ table-driven accept/reject for `created_at`; emoji-at-boundary snippet fixture; ms- vs second-precision since/until fixture; bare-`sk-` audit-redaction fixture.++**`t_1224c740` (6 items):**+- `src/core/brain/entities/index-builder.ts` (NUL file) + `src/core/search/entity-alias.ts` / `query-expansion.ts` โ mtime-keyed memo of `buildEntityIndex` (dir mtime probe per query).+- `src/core/search/store.ts` โ `documents.basename` column + index (populated at index time); `resolvedDocLinkPairs` rewritten as equality joins; `LATEST_SCHEMA_VERSION` bump.+- `src/core/brain/backlinks.ts` โ mtime/generation-keyed memo or incremental log cursor for `buildBacklinkIndex`.+- `src/core/brain/heal-run.ts` + `heal-enrich.ts` โ sort-once/compile-once regex; drop duplicate frontmatter parse.+- `src/cli/brain/verbs/links.ts` + `src/core/brain/link-graph/format-wikilink.ts` โ basenameโpaths Map (byte-identical output pinned first).+- `src/core/brain/continuity/store.ts` โ `since`-aware file skip + reverse-chron early exit for latest-record queries.+- `tests/` โ entity invalidation; link-resolution + synthetic 5k-doc scale smoke; backlink memo freshness; heal regex compile-count; links-normalize byte-identity fixture; continuity latest-record early-exit.++**`t_3ccb1512` (1 fix):**+- `src/core/surface/lexical-score.ts` โ `tokenize` extracts bigrams only from maximal contiguous Han spans.+- `tests/core/surface/lexical-score.test.ts` โ `gbrainๅฎ็ฐ`โ`ๅฎ็ฐ`+full (no `nๅฎ`); `abๅฎ็ฐcd`โ`ๅฎ็ฐ`; pure-Han/pure-ASCII unchanged; standalone 2-char `ๅฎ็ฐ` no duplicate.++**Release-wide:**+- `package.json` + `CHANGELOG.md` โ version bump to 1.24.0 (lands with C.2, the schema-bump commit).++## Risks++- **Schema bump + reindex on upgrade.** C.2 bumps `LATEST_SCHEMA_VERSION`; every existing install reindexes on next open. Mitigated by `t_f27d80fe` landing first (the reindex path is now race-safe), and by the single-reindex anchor (users pay once). Risk: the race fix and the schema bump are in the same release โ if the race fix regressed, the forced reindex would exercise it under load. Mitigation: the 3 regression tests + the integration overlap test pin the path before C.2 lands.+- **Staleness-window memos (C.1, C.3) can serve one stale read after a write.** Deliberate trade-off, pinned by invalidation tests (write โ next query sees the new entity / backlinks). Documented at the call site; not hidden behind a flag.+- **C.5 byte-identity regression.** The links-normalize rewrite must reproduce current output exactly. Mitigated by a characterization fixture captured before the rewrite.+- **B.2 / B.1 change observable bytes / accepted inputs.** B.2 fixes a corrupt lone surrogate (new bytes are correct); B.1 rejects previously-accepted garbage (the minor-version gate). Both covered by fixtures; neither is a silent regression.+- **`ec3bad27` rebase onto the release branch.** The fix was validated on `fix/index-swap-hardening` off an earlier base; rebasing onto `feat/pr121-hardening-correctness` (post-#121 merge) may surface context drift. Mitigation: rebased as the first commit cluster with the full suite green before any other card lands.diff --git a/docs/brainstorm/pr121-hardening-correctness/plan.md b/docs/brainstorm/pr121-hardening-correctness/plan.mdnew file mode 100644index 00000000..4b0a5db5--- /dev/null+++ b/docs/brainstorm/pr121-hardening-correctness/plan.md@@ -0,0 +1,103 @@+# PR #121 hardening + correctness โ implementation plan++One feature branch (`feat/pr121-hardening-correctness`), four cards driven **one at a time** in risk order, each a contiguous commit cluster with per-fix TDD (failing test first). Full suite green between cards. Each later card builds on the commits earlier cards landed โ do not duplicate or conflict with siblings.++Drive order (matches `design.md` Chosen approach):+1. `t_f27d80fe` โ index-swap hardening (stabilizes the base; the forced reindex from C.2 will exercise this path).+2. `t_2ba5f0c9` โ validation/output hardening (low-risk correctness).+3. `t_3ccb1512` โ Han-bigram tokenizer (low-risk CJK correctness; its tokenization change is absorbed by the C.2 reindex).+4. `t_1224c740` โ hot-path performance, ending with C.2 (the schema bump + version bump, the release-defining commit).++---++## Task 1: Index-swap hardening โ fix reindex/crash-restore data-loss races (kanban t_f27d80fe, P4)++Rebase the validated local fix `ec3bad27` (branch `fix/index-swap-hardening`) onto this release branch as the first commit cluster, then extend it with the `INDEX_UNREADABLE` self-heal as a separate follow-on commit.++**Files:**+- `src/core/search/store.ts` โ `acquireReindexLock` / `isReindexInProgress` (a dedicated lockfile, separate from the per-`Store.open` writer lock); `Store.open` crash-restore preamble (~293-306) probes `isReindexInProgress` non-blocking and skips restoring while a reindex is actively running.+- `src/core/search/indexer.ts` โ `reindexVault` acquires the reindex lock BEFORE touching `newPath` (the `tryUnlink(newPath)` + staging seed at ~659,664 move inside the lock); second concurrent `reindexVault` fails fast with the existing typed `INDEX_LOCKED`.+- `src/core/search/search.ts` (NUL-byte sentinel โ use `command grep -a`) โ `openReadOrSelfHeal` (~210-226) self-heal set extended to include `INDEX_UNREADABLE`.+- `tests/core/search/` โ the 3 regression tests from `ec3bad27` (second `reindexVault` rejected while one in progress; `Store.open` does not restore `.bak` mid-reindex; genuine crash-restore still works) + a new self-heal test asserting `INDEX_UNREADABLE` is recovered.++**Acceptance (a passing test):**+- Integration test that starts two overlapping `reindexVault` runs (the second entering during the first's build phase) and asserts the surviving index is complete and readable โ NOT an empty DB, NOT `INDEX_UNREADABLE`.+- `Store.open` does not restore `.bak` during an in-flight reindex (interleaved open-during-swap test).+- Genuine crash-restore path still green (the existing crash-restore test stays green).+- `INDEX_UNREADABLE` is now self-healed by `openReadOrSelfHeal` (new test).++**Depends on:** none (first card; stabilizes the base for the rest).++---++## Task 2: Validation/output hardening โ four independent fixes (kanban t_2ba5f0c9, P4)++Land as up to four commits (one per fix), each TDD. B.1 is the contract-tightening gate (previously-accepted garbage โ `INVALID_PARAMS`) โ it is why the release is a minor.++**B.1 โ `created_at` full-timestamp validation**+- **Files:** `src/core/brain/continuity/store.ts` (boundary ~42-45,176; lexical compare at 213,222-223); possibly a shared validator extracted from `normalizeDemandTimestamp` (DRY โ see B.3).+- **Acceptance:** table-driven accept/reject at the store boundary โ `"2026-13-01T00:00:00Z"` rejected (invalid month), `"2026-07-06T15:00:00+03:00"` rejected or canonicalized to `Z` (reject is simpler and matches the contract tightening); valid canonical `Z` timestamps accepted; one MCP-level case per exposed tool (`generation-tools.ts`, `host-memory-write.ts`, `session-summary.ts`).++**B.2 โ surrogate-safe card snippet**+- **Files:** `src/core/search/search.ts` (NUL file) โ `cardSnippet` (~1121-1126, `CARD_SNIPPET_CHARS=240`) truncates on code points (`[...collapsed].slice(0, N).join("")` or `Intl.Segmenter`).+- **Acceptance:** snippet fixture with an emoji astral char at the truncation boundary asserts no lone surrogate / no U+FFFD; the snippet ends on a complete code point followed by `...`.++**B.3 โ `since`/`until` precision-consistent filtering**+- **Files:** `src/core/brain/query-demand.ts` โ filter bounds (~238-239) run through `normalizeDemandTimestamp` (~448-453) before the lexical compare. Consider extracting a shared canonical-UTC normalizer reused by B.1 (DRY).+- **Acceptance:** boundary fixture โ a second-precision `--since 2026-07-01T00:00:00Z` does NOT exclude ms-precision records in `[00:00:00.000Z, 00:00:00.999Z]`.++**B.4 โ secret-exec audit redaction**+- **Files:** `src/core/brain/secrets/exec.ts` (~88,96-108) โ the logged `command = argv.join(" ")` is redacted (`redactInfra: true` and/or a high-entropy-token pass) before `appendAuditRecord` persists it.+- **Acceptance:** audit fixture asserting a bare `sk-`-shaped token passed as a positional argument is masked in `Brain/log/secret-custody/*.jsonl`.++**Depends on:** Task 1 (build on the stabilized base).++---++## Task 3: Han-bigram tokenizer โ restrict bigrams to contiguous Han spans (kanban t_3ccb1512, P3)++TDD-first per the card's acceptance. Its tokenization change is absorbed by the single C.2 reindex at release (no separate schema bump here).++**Files:**+- `src/core/surface/lexical-score.ts` โ `tokenize` extracts bigrams only from maximal contiguous Han spans within each token, leaving non-Han segments untouched; the full token is still emitted as today.+- `tests/core/surface/lexical-score.test.ts` โ the four acceptance cases below.++**Acceptance (a passing test):**+- `tokenize("gbrainๅฎ็ฐ")` emits the Han bigram `ๅฎ็ฐ` and the full token, and NO cross-boundary bigram (`nๅฎ`).+- `tokenize("abๅฎ็ฐcd")` emits `ๅฎ็ฐ`.+- Pure-Han and pure-ASCII tokenization unchanged (existing tests stay green).+- No duplicate emission for a standalone 2-char Han token (`tokenize("ๅฎ็ฐ")` stays `["ๅฎ็ฐ"]`).++**Depends on:** Task 2 (lands before the perf card; its tokenization change rides the C.2 reindex).++---++## Task 4: Hot-path performance โ six items (kanban t_1224c740, P4)++Land the five contract-neutral items first (C.1, C.3, C.4, C.5, C.6), then C.2 as the release-defining commit (schema bump + version bump). Each item carries a before/after measurement on a synthetic large vault where the card calls for it.++**C.1 โ entity-index mtime memo (biggest per-query win)**+- **Files:** `src/core/brain/entities/index-builder.ts` (NUL file, ~80-106); consumers `src/core/search/entity-alias.ts:45` and `src/core/search/query-expansion.ts:106`.+- **Acceptance:** an invalidation test โ write a new entity โ the next query that names an entity sees it (mtime probe triggers rebuild). Existing entity suites green. Documented staleness-window trade-off at the call site.++**C.2 โ `documents.basename` column (schema bump; release-defining commit)**+- **Files:** `src/core/search/store.ts` โ add `basename` column + index (populated at index time); `resolvedDocLinkPairs` (~1064-1086) rewritten as equality joins (no `SUBSTR` full scan); bump `LATEST_SCHEMA_VERSION`; `package.json` + `CHANGELOG.md` โ 1.24.0.+- **Acceptance:** link-resolution suites green; a scale smoke test on a synthetic 5k-doc / multi-k-dangling-link fixture shows the suffix scans eliminated (equality joins). The single release reindex absorbs C.2 and the Task 3 tokenizer change.++**C.3 โ `buildBacklinkIndex` cross-call memo**+- **Files:** `src/core/brain/backlinks.ts` (~268-272); callers `query-tools.ts:250`, `resources.ts:369`, `doctor.ts:930`, `explorer.ts:88`, moc-audit, concept-cluster.+- **Acceptance:** mtime/generation-keyed memo (or incremental log cursor) with a freshness test; full-history re-read eliminated on repeat calls within the same generation. "Recompute on demand" revisited deliberately, documented.++**C.4 โ heal-enrich compile-once regex**+- **Files:** `src/core/brain/heal-run.ts` (~76-88) + `src/core/brain/heal-enrich.ts` (~42-58). Sort once, compile once, post-filter self-matches per page; drop the duplicate frontmatter parse at `heal-run:80`.+- **Acceptance:** existing heal tests green; a test asserting the alternation regex is compiled once per run, not once per page (compile-count assertion or equivalent).++**C.5 โ `links normalize --mode short` Map-based resolution**+- **Files:** `src/cli/brain/verbs/links.ts` (~72-86) + `src/core/brain/link-graph/format-wikilink.ts` (~60-90).+- **Acceptance:** a characterization fixture captures current output BEFORE the rewrite; after the rewrite the output is byte-identical. Build a basenameโpaths Map once per run; resolve+disambiguate against it.++**C.6 โ continuity reverse-chron early exit**+- **Files:** `src/core/brain/continuity/store.ts` (~190-215, `readAllRecords`); consumers `session-summary.ts:182`, `pre-compact-extract.ts:148`, recall-telemetry/memory-cost/context-receipts.+- **Acceptance:** a `since`-aware file-skip test (shards outside the window are not parsed) and a latest-record reverse-chron early-exit test (stops at the first match, not after parsing all shards).++**Depends on:** Tasks 1โ3 (builds on the stabilized, correctness-fixed base; C.2's forced reindex exercises Task 1's race-safe reindex).diff --git a/docs/brainstorm/pr121-hardening-correctness/variants.md b/docs/brainstorm/pr121-hardening-correctness/variants.mdnew file mode 100644index 00000000..5fa5b954--- /dev/null+++ b/docs/brainstorm/pr121-hardening-correctness/variants.md@@ -0,0 +1,69 @@+# PR #121 hardening + correctness โ consultant variants++Verbatim output from the primary consultant (Claude, model `claude-opus-4-8`), 3 variants + recommendation. Primary succeeded (exit 0); no fallback run.++Source: `docs/brainstorm/pr121-hardening-correctness/cli-output/claude.md`++---++## Variant 1 โ Card-sequential single minor (1.24.0), risk-ordered++**Approach:** Ship all 13 fixes as one minor on the shared branch, committed one card at a time in risk order: land t_f27d80fe first (rebase `ec3bad27` in as-is, then extend it with the `INDEX_UNREADABLE`โself-heal change as a follow-on commit), then the low-risk correctness cluster (B.1โB.4 + the Han-bigram tokenizer), then the contract-neutral perf items (C.1, C.3โC.6), and finally C.2 as the release-defining commit carrying the `LATEST_SCHEMA_VERSION` bump, the `basename` column, and the version/CHANGELOG bump. One reindex at release absorbs both C.2 and the tokenizer's re-tokenization.++**Trade-offs:**+- Each card stays a contiguous, independently-revertable commit cluster โ matches the "one card at a time / each unit revertable" convention exactly.+- Data-loss fix reaches `main` first and stays green through every later commit; perf work rebases on a stabilized base, not vice-versa.+- Self-heal expansion rides with t_f27d80fe (its natural home) and stays decoupled from the reindex-forcing schema bump โ two independent safety/contract surfaces reviewed separately.+- The whole minor is gated behind one reindex because of C.2, so even pure-correctness fixes only take effect post-reindex for users โ acceptable for a minor, but not a fast path for the P4 data-loss fix.+- Longest single branch lifetime; later cards carry the most rebase surface.++**Complexity:** medium+**Risk:** low++---++## Variant 2 โ Two-release split: correctness patch (1.23.1) then perf+schema minor (1.24.0)++**Approach:** Cut two releases from the shared branch. First a patch: t_f27d80fe (data-loss) + the contract-neutral parts of t_2ba5f0c9 (B.2 surrogate, B.3 precision, B.4 audit redaction) โ no schema change, no reindex, ships immediately. Then a minor stacked on the patch: all of t_1224c740, C.2's schema bump, the Han-bigram tokenizer, and B.1's `created_at` tightening, all gated behind a single reindex.++**Trade-offs:**+- Gets the highest-priority silent-data-loss fix to users fastest, with zero reindex burden.+- Two version bumps + two CHANGELOG headings + two PRs on a protected `main` โ doubles the release ceremony that CLAUDE.md mandates per PR.+- Forces splitting t_2ba5f0c9 across releases: B.1 is a contract *tightening* (previously-accepted garbage โ `INVALID_PARAMS`) that belongs behind a minor, so the card can't land atomically โ violates card atomicity.+- Two reviewable surfaces are each smaller, but the stacked-PR dependency and cross-release version reconciliation add coordination risk.+- Best operational safety for the data-loss class; worst fit for the "drive cards one at a time, atomically" framing.++**Complexity:** large+**Risk:** medium++---++## Variant 3 โ Contract-surface-partitioned single minor (1.24.0)++**Approach:** Regroup the 13 fixes by whether they touch a frozen contract or require reindex, ignoring card boundaries for commit shaping. Land every contract-neutral fix first (t_f27d80fe lock, B.2/B.3/B.4, C.1/C.3โC.6), then a single terminal "contract + schema" commit that bumps `LATEST_SCHEMA_VERSION`, adds the `basename` column, tightens `created_at` (B.1), folds in the tokenizer reindex, expands the self-heal set, and carries the version bump โ one reindex, one concentrated contract-review target.++**Trade-offs:**+- Concentrates the entire frozen-contract blast radius into one clearly-labeled commit โ ideal review focus for a frozen-contract project.+- Fuses unrelated fixes (schema column, timestamp validation, tokenizer, self-heal) into one fat commit: sacrifices per-fix revertability and violates "each unit independently revertable / atomic conventional commits."+- Couples the `INDEX_UNREADABLE` self-heal safety change to the reindex-forcing schema bump โ a rollback of one drags the other.+- Cross-cutting commit ordering diverges from the "one card at a time" driving model, complicating per-card TDD attribution and merge-conflict reasoning against siblings.++**Complexity:** medium+**Risk:** high++---++**Recommended: Variant 1** โ it is the only shaping that honors every stated convention simultaneously: cards land one at a time as contiguous, independently-revertable commit clusters with per-fix TDD, the P4 data-loss fix stabilizes the base before perf work rebases onto it, and the single unavoidable reindex is anchored to C.2 as its own auditable schema commit โ keeping V3's contract-review clarity (schema bump is isolated) without fusing unrelated fixes or coupling the self-heal safety change to the reindex. The split in V2 buys faster data-loss delivery at the cost of doubled release ceremony on a protected `main` and a non-atomic t_2ba5f0c9, neither of which this repo's bump-in-PR discipline and card-atomic framing tolerate cleanly.++---++## Final decision rationale++**Accepted Variant 1 (no override).** It satisfies all four project-specific constraints that the other variants strain against:++1. **Card-atomic driving model** โ the orchestrator drives cards one at a time on a shared branch; Variant 1 is the only shaping whose commit clusters map 1:1 to cards, so per-card TDD attribution and merge-conflict reasoning stay clean. (Variant 3 ignores card boundaries; Variant 2 splits a card across releases.)+2. **Frozen-contract reviewability** โ the schema bump (`LATEST_SCHEMA_VERSION`, `basename` column) is isolated as its own auditable commit, not fused with unrelated correctness fixes (Variant 3's failure) or scattered across releases (Variant 2).+3. **Safety/contract decoupling** โ the `INDEX_UNREADABLE` self-heal rides with its natural home (t_f27d80fe) and stays decoupled from the reindex-forcing schema bump, so a rollback of one cannot drag the other.+4. **Single-reindex economy** โ C.2's unavoidable reindex absorbs the Han-bigram tokenizer's re-tokenization at no extra cost; users pay one reindex for the whole release.++The one real cost (the P4 data-loss fix is gated behind the C.2 reindex for users who upgrade to 1.24.0) is acceptable for a minor release and is itself mitigated by landing t_f27d80fe first โ the reindex path is race-safe before the schema bump forces load onto it.diff --git a/openclaw.plugin.json b/openclaw.plugin.jsonindex b6d916ef..bbfc3e2f 100644--- a/openclaw.plugin.json+++ b/openclaw.plugin.json@@ -2,7 +2,7 @@"id": "open-second-brain","name": "Open Second Brain","description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",- "version": "1.23.1",+ "version": "1.24.0","activation": { "onStartup": true },"skills": ["./skills"],"contracts": {diff --git a/package.json b/package.jsonindex ccef5e27..a1cf287e 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.23.1",+ "version": "1.24.0","private": false,"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.","keywords": [diff --git a/plugin.yaml b/plugin.yamlindex 4b312cc2..e803f447 100644--- a/plugin.yaml+++ b/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.23.1"+version: "1.24.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.jsonindex 421d0cf9..f0544400 100644--- a/plugins/codex/.codex-plugin/plugin.json+++ b/plugins/codex/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.23.1",+ "version": "1.24.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yamlindex 4b312cc2..e803f447 100644--- a/plugins/hermes/plugin.yaml+++ b/plugins/hermes/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.23.1"+version: "1.24.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/pyproject.toml b/pyproject.tomlindex eac77904..a29556b6 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"# CLI entry points (those moved to `package.json` `bin`).[project]name = "open-second-brain"-version = "1.23.1"+version = "1.24.0"description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."readme = "README.md"requires-python = ">=3.11"diff --git a/src/core/brain/continuity/store.ts b/src/core/brain/continuity/store.tsindex 802377b2..a1649717 100644--- a/src/core/brain/continuity/store.ts+++ b/src/core/brain/continuity/store.ts@@ -39,6 +39,58 @@ export interface ContinuityPaginationOptions extends ContinuityRecordFilter {const CONTINUITY_REL = `${BRAIN_LOG_REL}/continuity`;const CURSOR_PREFIX = "offset:";+/**+ * Canonical UTC ISO-8601 shape: `YYYY-MM-DDTHH:MM:SS[.sss]Z`. A trailing+ * `Z` is mandatory โ a numeric offset (`+03:00`) is rejected so every+ * stored `createdAt` sorts and filters lexically against its siblings+ * (readers compare the strings directly) and shards into the correct UTC+ * month. Fractional seconds are optional (0โ3 digits) to accept both the+ * second-precision and `toISOString()` millisecond forms already on disk.+ */+const CANONICAL_UTC_TIMESTAMP_RE =+ /^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})(?:\.\d{1,3})?Z$/;++/**+ * True when `value` is a canonical UTC timestamp with a real calendar+ * date/time. The regex enforces the `โฆZ` shape; the round-trip check+ * rejects out-of-range fields that `Date` silently rolls over (e.g.+ * `2026-02-30` โ Mar 2, `2026-13-โฆ` โ NaN), so `2026-13-01T00:00:00Z`+ * and `2026-02-30T00:00:00Z` are both caught.+ */+export function isCanonicalUtcTimestamp(value: unknown): value is string {+ if (typeof value !== "string") return false;+ const match = CANONICAL_UTC_TIMESTAMP_RE.exec(value);+ if (match === null) return false;+ const ms = Date.parse(value);+ if (!Number.isFinite(ms)) return false;+ const date = new Date(ms);+ return (+ date.getUTCFullYear() === Number(match[1]) &&+ date.getUTCMonth() + 1 === Number(match[2]) &&+ date.getUTCDate() === Number(match[3]) &&+ date.getUTCHours() === Number(match[4]) &&+ date.getUTCMinutes() === Number(match[5]) &&+ date.getUTCSeconds() === Number(match[6])+ );+}++/**+ * Guard every record's `createdAt` at the store boundary. Throws on any+ * non-canonical value so a malformed timestamp can never reach disk as a+ * junk shard (`2026-13.jsonl`) or a local-time record that mis-sorts+ * against the fleet of `Z` timestamps. All append paths funnel through+ * `buildRecord`, so this is the single choke point.+ */+export function assertCanonicalCreatedAt(createdAt: unknown): string {+ if (!isCanonicalUtcTimestamp(createdAt)) {+ throw new Error(+ `invalid continuity createdAt: expected a canonical UTC ISO-8601 timestamp ` ++ `(YYYY-MM-DDTHH:MM:SS[.sss]Z with a real calendar date), got ${JSON.stringify(createdAt)}`,+ );+ }+ return createdAt;+}+export function continuityLogPath(vault: string, month: string): string {if (!/^\d{4}-\d{2}$/.test(month)) throw new Error(`invalid continuity month: ${month}`);return ensureInsideVault(join(vault, CONTINUITY_REL, `${month}.jsonl`), vault);@@ -130,7 +182,7 @@ export function listContinuityRecords(vault: string,filter: ContinuityRecordFilter = {},): ReadonlyArray<ContinuityRecord> {- return Object.freeze(readAllRecords(vault).filter((record) => matches(record, filter)));+ return Object.freeze(readAllRecords(vault, filter).filter((record) => matches(record, filter)));}export function paginateContinuityRecords(@@ -155,6 +207,7 @@ function buildRecord(readonly payload: Readonly<Record<string, unknown>>;},): ContinuityRecord {+ assertCanonicalCreatedAt(input.createdAt);const payloadResult = safeContinuityPayload(input.payload ?? {});const sourceRefs = Object.freeze([...(input.sourceRefs ?? [])]);// `schema` stays OUT of recordId(): identical records must keep@@ -187,12 +240,36 @@ function appendRecord(vault: string, record: ContinuityRecord): ContinuityRecordreturn record;}-function readAllRecords(vault: string): ContinuityRecord[] {+/**+ * Read every stored record, optionally skipping whole month shards that+ * a `since`/`until` filter proves cannot contribute a matching record.+ *+ * Shards are named `YYYY-MM.jsonl`, so the shard month is a 7-char prefix+ * of every record's `createdAt` inside it. A shard whose month sorts+ * strictly before `since`'s month can only hold records with+ * `createdAt < since` (rejected by {@link matches}); one whose month sorts+ * strictly after `until`'s month can only hold `createdAt > until`. Both+ * are skipped without opening the file. The boundary months (equal to+ * `since`/`until`'s month) are still read in full, so the returned set is+ * byte-identical to reading everything and letting `matches` filter -+ * this only avoids reading shards that would contribute nothing. Grows+ * the read cost with the queried window, not the whole log history.+ */+function readAllRecords(vault: string, filter: ContinuityRecordFilter = {}): ContinuityRecord[] {const dir = ensureInsideVault(join(vault, CONTINUITY_REL), vault);if (!existsSync(dir)) return [];+ const sinceMonth = filter.since ? filter.since.slice(0, 7) : undefined;+ const untilMonth = filter.until ? filter.until.slice(0, 7) : undefined;const records: ContinuityRecord[] = [];for (const name of readdirSync(dir).toSorted()) {if (!name.endsWith(".jsonl")) continue;+ const month = name.slice(0, -".jsonl".length);+ // Only a canonical YYYY-MM shard has a month prefix we can range-skip+ // on; any other name is read in full to stay byte-identical.+ if (/^\d{4}-\d{2}$/.test(month)) {+ if (sinceMonth !== undefined && month < sinceMonth) continue;+ if (untilMonth !== undefined && month > untilMonth) continue;+ }const path = ensureInsideVault(join(dir, name), vault);let st;try {diff --git a/src/core/brain/entities/index-builder.ts b/src/core/brain/entities/index-builder.tsindex 2750d0ab..c6f90645 100644Binary files a/src/core/brain/entities/index-builder.ts and b/src/core/brain/entities/index-builder.ts differdiff --git a/src/core/brain/heal-enrich.ts b/src/core/brain/heal-enrich.tsindex 9261d11e..24712c35 100644--- a/src/core/brain/heal-enrich.ts+++ b/src/core/brain/heal-enrich.ts@@ -34,21 +34,50 @@ export function deriveTitleFromContent(markdown: string): string | null {}/**- * Wrap exact whole-token occurrences of any `known` title/alias in a- * wikilink. Text inside existing wikilinks or inline code is left- * untouched, so the function is idempotent. Returns the input unchanged- * when `known` is empty.+ * One known title/alias, prepared once: the raw form (for per-page+ * self-link exclusion, which matches on the exact string the caller+ * supplied) and its regex-escaped, trimmed form (for the alternation).*/-export function linkExactMentions(body: string, known: ReadonlyArray<string>): string {- const phrases = known- .map((k) => k.trim())- .filter((k) => k.length > 0)+interface PreparedPhrase {+ readonly raw: string;+ readonly escaped: string;+}++/**+ * The full known title/alias set, trimmed / dropped-if-empty / sorted+ * longest-first (lexicographic tie-break) / regex-escaped ONCE. The heal+ * runner shares this across every page instead of re-sorting and+ * re-escaping K phrases per page (the dominant per-page cost), then+ * excludes each page's own few terms via {@link linkExactMentionsPrepared}.+ */+export interface PreparedHealPhrases {+ readonly entries: ReadonlyArray<PreparedPhrase>;+}++/** Build the shared, sorted, escaped phrase set. Order-preserving under+ * later per-page filtering, so output stays identical to sorting the+ * already-filtered list. */+export function prepareHealPhrases(known: ReadonlyArray<string>): PreparedHealPhrases {+ const entries = known+ .map((raw) => ({ raw, trimmed: raw.trim() }))+ .filter((e) => e.trimmed.length > 0)// Longest first so a multi-word title wins over a contained shorter// one at the same position; lexicographic tie-break for determinism.- .toSorted((a, b) => b.length - a.length || (a < b ? -1 : a > b ? 1 : 0));- if (phrases.length === 0) return body;+ .toSorted(+ (a, b) =>+ b.trimmed.length - a.trimmed.length ||+ (a.trimmed < b.trimmed ? -1 : a.trimmed > b.trimmed ? 1 : 0),+ )+ .map((e) => Object.freeze({ raw: e.raw, escaped: escapeRegex(e.trimmed) }));+ return Object.freeze({ entries: Object.freeze(entries) });+}- const alternation = phrases.map(escapeRegex).join("|");+/** Compile the alternation from the ordered escaped forms and link every+ * whole-token match outside protected spans. Shared core of both public+ * entry points. */+function applyPhrases(body: string, escaped: ReadonlyArray<string>): string {+ if (escaped.length === 0) return body;+ const alternation = escaped.join("|");// Whole-token boundaries via Unicode letter/number lookarounds so the// match is language-agnostic (works for any script).const linkRe = new RegExp(`(?<![\\p{L}\\p{N}])(?:${alternation})(?![\\p{L}\\p{N}])`, "gu");@@ -62,6 +91,38 @@ export function linkExactMentions(body: string, known: ReadonlyArray<string>): sreturn parts.join("");}+/**+ * Wrap exact whole-token occurrences of any `known` title/alias in a+ * wikilink. Text inside existing wikilinks or inline code is left+ * untouched, so the function is idempotent. Returns the input unchanged+ * when `known` is empty.+ */+export function linkExactMentions(body: string, known: ReadonlyArray<string>): string {+ return applyPhrases(+ body,+ prepareHealPhrases(known).entries.map((e) => e.escaped),+ );+}++/**+ * Prepared-set variant of {@link linkExactMentions}: link every phrase in+ * `prepared` whose raw form is NOT in `exclude` (the page's own titles /+ * aliases). Filtering the pre-sorted set preserves order, so the+ * alternation - and therefore the output - is byte-identical to calling+ * `linkExactMentions` with the pre-excluded list.+ */+export function linkExactMentionsPrepared(+ body: string,+ prepared: PreparedHealPhrases,+ exclude: ReadonlySet<string>,+): string {+ const escaped: string[] = [];+ for (const entry of prepared.entries) {+ if (!exclude.has(entry.raw)) escaped.push(entry.escaped);+ }+ return applyPhrases(body, escaped);+}+/** A page view the heal planner reads. */export interface HealPageInput {readonly frontmatter: Record<string, unknown>;@@ -87,6 +148,27 @@ export function planHealEnrichment(page: HealPageInput,knownTitlesAndAliases: ReadonlyArray<string>,): HealPlan {+ return finishPlan(page, linkExactMentions(page.body, knownTitlesAndAliases));+}++/**+ * Prepared-set variant of {@link planHealEnrichment} for the heal runner:+ * links against the shared, pre-sorted phrase set minus this page's own+ * titles/aliases. Output is byte-identical to+ * `planHealEnrichment(page, [...known].filter((k) => !exclude.has(k)))`,+ * but skips re-sorting and re-escaping the whole phrase set per page.+ */+export function planHealEnrichmentPrepared(+ page: HealPageInput,+ prepared: PreparedHealPhrases,+ exclude: ReadonlySet<string>,+): HealPlan {+ return finishPlan(page, linkExactMentionsPrepared(page.body, prepared, exclude));+}++/** Shared tail: derive a missing title from the first H1 and package the+ * (possibly) relinked body into a plan. */+function finishPlan(page: HealPageInput, linked: string): HealPlan {const existingTitle = page.frontmatter["title"];const hasTitle = typeof existingTitle === "string" && existingTitle.trim().length > 0;@@ -96,7 +178,6 @@ export function planHealEnrichment(if (derived !== null) title = derived;}- const linked = linkExactMentions(page.body, knownTitlesAndAliases);const body = linked !== page.body ? linked : undefined;return {diff --git a/src/core/brain/heal-run.ts b/src/core/brain/heal-run.tsindex 4cf897df..4eb4ef66 100644--- a/src/core/brain/heal-run.ts+++ b/src/core/brain/heal-run.ts@@ -24,7 +24,7 @@ import {writeFrontmatterAtomic,} from "../vault.ts";import { BRAIN_ROOT_REL } from "./paths.ts";-import { planHealEnrichment } from "./heal-enrich.ts";+import { planHealEnrichmentPrepared, prepareHealPhrases } from "./heal-enrich.ts";export interface HealRunResult {/** Pages scanned (outside the Brain root). */@@ -72,6 +72,11 @@ export function runHealEnrichment(vault: string): HealRunResult {ownTokens.set(p.path, own);}+ // Sort + regex-escape the whole known set ONCE; per page we only exclude+ // that page's own few terms (see planHealEnrichmentPrepared) instead of+ // re-sorting and recompiling the K-phrase set from scratch.+ const prepared = prepareHealPhrases([...known]);+const changed: string[] = [];for (const p of pages) {let meta;@@ -84,8 +89,7 @@ export function runHealEnrichment(vault: string): HealRunResult {}// Never link a page to its own title or aliases.const own = ownTokens.get(p.path) ?? new Set<string>();- const others = [...known].filter((k) => !own.has(k));- const plan = planHealEnrichment({ frontmatter: meta, body }, others);+ const plan = planHealEnrichmentPrepared({ frontmatter: meta, body }, prepared, own);if (!plan.changed) continue;const newMeta = plan.title !== undefined ? { ...meta, title: plan.title } : meta;diff --git a/src/core/brain/link-graph/format-wikilink.ts b/src/core/brain/link-graph/format-wikilink.tsindex 1acb7b59..b22481b8 100644--- a/src/core/brain/link-graph/format-wikilink.ts+++ b/src/core/brain/link-graph/format-wikilink.ts@@ -51,6 +51,45 @@ function stripMd(path: string): string {return path.endsWith(".md") ? path.slice(0, -".md".length) : path;}+/**+ * `/`-aligned-suffix index over the known pages: every trailing-segment+ * suffix of every page maps to the (stripped) pages that carry it, in+ * `knownPaths` order. A target side `t` resolves to exactly the pages+ * where `page === t || page.endsWith("/" + t)` โ precisely the bucket+ * `bySuffix.get(t)`, so per-link resolution drops from an O(pages) scan+ * (plus a fresh pages-sized array per link in the old+ * {@link shortestUniqueSuffix}) to a single Map lookup.+ */+type SuffixIndex = ReadonlyMap<string, ReadonlyArray<string>>;++/**+ * Built once per run and memoized on the `knownPaths` array identity:+ * `links normalize` walks the vault once and passes the same array to+ * every {@link normalizeWikilinks} call, so the index is built one time+ * and reused across all files instead of O(files ร links) rebuilds.+ */+const SUFFIX_INDEX_MEMO = new WeakMap<ReadonlyArray<string>, SuffixIndex>();++function suffixIndex(knownPaths: ReadonlyArray<string>): SuffixIndex {+ const cached = SUFFIX_INDEX_MEMO.get(knownPaths);+ if (cached) return cached;+ const bySuffix = new Map<string, string[]>();+ for (const raw of knownPaths) {+ const page = stripMd(raw);+ const segments = page.split("/");+ // Each `take` is a distinct-length key, so a page is inserted at+ // most once per suffix (matching the old scan's single push).+ for (let take = 1; take <= segments.length; take += 1) {+ const key = segments.slice(segments.length - take).join("/");+ const list = bySuffix.get(key);+ if (list) list.push(page);+ else bySuffix.set(key, [page]);+ }+ }+ SUFFIX_INDEX_MEMO.set(knownPaths, bySuffix);+ return bySuffix;+}+/*** The shortest trailing-segment suffix of `path` that no other known* page shares. `Brain/notes/alpha` with no sibling `alpha` yields@@ -58,11 +97,15 @@ function stripMd(path: string): string {* `archive/beta`.*/export function shortestUniqueSuffix(path: string, knownPaths: ReadonlyArray<string>): string {+ const index = suffixIndex(knownPaths);const segments = path.split("/");- const others = knownPaths.map(stripMd).filter((p) => p !== path);for (let take = 1; take <= segments.length; take += 1) {const suffix = segments.slice(segments.length - take).join("/");- const clash = others.some((p) => p === suffix || p.endsWith("/" + suffix));+ // `path`'s own bucket entry is always present; a clash is any OTHER+ // page sharing the suffix โ identical to the old+ // `others.filter(p => p !== path).some(...)`.+ const bucket = index.get(suffix);+ const clash = bucket !== undefined && bucket.some((p) => p !== path);if (!clash) return suffix;}return path;@@ -79,11 +122,8 @@ function resolveTarget(): string | "ambiguous" | null {const target = stripMd(targetSide.trim());if (target === "") return null;- const matches: string[] = [];- for (const raw of knownPaths) {- const page = stripMd(raw);- if (page === target || page.endsWith("/" + target)) matches.push(page);- }+ const matches = suffixIndex(knownPaths).get(target);+ if (matches === undefined) return null;if (matches.length === 1) return matches[0]!;if (matches.length > 1) return "ambiguous";return null;diff --git a/src/core/brain/query-demand.ts b/src/core/brain/query-demand.tsindex 4e749293..b2f9860d 100644--- a/src/core/brain/query-demand.ts+++ b/src/core/brain/query-demand.ts@@ -224,13 +224,21 @@ export function readQueryDemand(): ReadonlyArray<QueryDemandRecord> {const path = queryDemandLogPath(vault);if (!existsSync(path)) return Object.freeze([]);+ // Normalize the filter bounds to the same millisecond-precision form the+ // stored `ts` already uses (records are normalized at write time). The+ // comparison below is lexical, so a second-precision `--since+ // 2026-07-01T00:00:00Z` would otherwise drop every ms-precision record in+ // `[00:00:00.000Z, 00:00:00.999Z]` โ `.` sorts before `Z`. Running the+ // bounds through the same normalizer keeps both sides at one precision.+ const since = filter.since !== undefined ? normalizeDemandTimestamp(filter.since) : undefined;+ const until = filter.until !== undefined ? normalizeDemandTimestamp(filter.until) : undefined;const out: QueryDemandRecord[] = [];for (const line of readFileSync(path, "utf8").split("\n")) {if (!line.trim()) continue;const record = coerceRecord(line);if (record === null) continue;- if (filter.since !== undefined && record.ts < filter.since) continue;- if (filter.until !== undefined && record.ts > filter.until) continue;+ if (since !== undefined && record.ts < since) continue;+ if (until !== undefined && record.ts > until) continue;out.push(record);}out.sort((a, b) => (a.ts < b.ts ? -1 : a.ts > b.ts ? 1 : 0));diff --git a/src/core/brain/secrets/exec.ts b/src/core/brain/secrets/exec.tsindex 3d5714fd..49765815 100644--- a/src/core/brain/secrets/exec.ts+++ b/src/core/brain/secrets/exec.ts@@ -86,6 +86,17 @@ export async function runWithSecret(): Promise<RunWithSecretResult> {if (argv.length === 0) throw new Error("secret run: a command is required after --");const command = argv.join(" ");+ // The audit trail is long-lived. The custody store's own secret never+ // reaches argv, but an agent may pass some OTHER credential as a bare+ // positional argument; scrub the logged command through the bare-token+ // and infra passes so a foreign secret is not persisted in cleartext.+ // The raw `command` still drives the allowlist match and the denial error+ // (the agent already holds its own input).+ const auditCommand = redactRawOutput(command, {+ redactTokens: true,+ redactInfra: true,+ maxInput: Number.POSITIVE_INFINITY,+ });const resolved = resolveSecretForExec(vault, name, ctx);if (!matchesAllowlist(resolved.allow, command)) {appendAuditRecord(join(brainDirs(vault).log, "secret-custody"), {@@ -94,7 +105,7 @@ export async function runWithSecret(action: "secret_exec_denied",target: resolved.name,ok: false,- details: { command, allow: resolved.allow },+ details: { command: auditCommand, allow: resolved.allow },});throw new SecretExecDeniedError(resolved.name, command, resolved.allow);}@@ -105,7 +116,7 @@ export async function runWithSecret(action: "secret_exec_started",target: resolved.name,ok: true,- details: { command, env_var: resolved.env_var },+ details: { command: auditCommand, env_var: resolved.env_var },});const childEnv: Record<string, string> = {};diff --git a/src/core/redactor.ts b/src/core/redactor.tsindex a9f41303..65648a03 100644--- a/src/core/redactor.ts+++ b/src/core/redactor.ts@@ -224,6 +224,48 @@ function isPrivateOrReservedIPv6(ip: string): boolean {return false;}+// ----- Bare high-entropy token detector (opt-in via `redactTokens`) ---------+//+// Credential tokens passed as bare positional values carry no key=value+// shape for the assignment passes to latch onto (e.g. an argv like+// `["mytool", "sk-abc123"]`). Two complementary shapes are scrubbed:+// well-known vendor-prefixed keys, and long mixed-class runs that look+// like an API key or hash. Every quantifier is bounded, so the pass stays+// linear and cannot be driven into catastrophic backtracking.++/**+ * Vendor-prefixed credential tokens (OpenAI/Stripe `sk-`/`sk_`/`rk_`/`pk_`,+ * GitHub `ghp_`/`gho_`/โฆ/`github_pat_`, Slack `xox?-`, AWS `AKIAโฆ`, Google+ * `AIzaโฆ`, GitLab `glpat-`). The recognizable prefix is what lets a short+ * token like `sk-abc123` be caught without a length gate that would also+ * hit ordinary words.+ */+const VENDOR_TOKEN_RE = new RegExp(+ [+ "\\b(?:sk|rk|pk)[-_][A-Za-z0-9._-]{3,200}",+ "\\b(?:ghp|gho|ghu|ghs|ghr)_[A-Za-z0-9]{6,255}",+ "\\bgithub_pat_[A-Za-z0-9_]{6,255}",+ "\\bxox[baprs]-[A-Za-z0-9-]{6,200}",+ "\\bAKIA[0-9A-Z]{16}\\b",+ "\\bAIza[0-9A-Za-z._-]{10,100}",+ "\\bglpat-[A-Za-z0-9_-]{6,100}",+ ].join("|"),+ "g",+);++/**+ * A long bare run that mixes letters and digits โ the shape of an+ * unprefixed API key, session id, or hash. Length-gated (โฅ 24) so ordinary+ * words and short ids are never touched, with bounded repetition so the+ * lookaheads stay linear.+ */+const HIGH_ENTROPY_TOKEN_RE =+ /\b(?=[A-Za-z0-9_-]{24,200}\b)(?=[A-Za-z0-9_-]{0,199}[A-Za-z])(?=[A-Za-z0-9_-]{0,199}\d)[A-Za-z0-9_-]{24,200}\b/g;++function redactBareTokens(text: string): string {+ return text.replace(VENDOR_TOKEN_RE, PLACEHOLDER).replace(HIGH_ENTROPY_TOKEN_RE, PLACEHOLDER);+}+function redactInfraTopology(text: string): string {let out = text.replace(BASIC_AUTH_URL_RE, (_m, scheme: string) => `${scheme}${PLACEHOLDER}@`);out = out.replace(IPV4_PORT_RE, PLACEHOLDER);@@ -308,6 +350,15 @@ export interface RedactRawOutputOptions {* where a bare coordinate is the likeliest topology leak.*/readonly redactInfra?: boolean;+ /**+ * When `true`, also scrub bare high-entropy credential tokens that carry+ * no key=value shape โ vendor-prefixed keys (`sk-โฆ`, `ghp_โฆ`, `AKIAโฆ`)+ * and long mixed letter+digit runs. Off by default (over-redacting prose+ * is worse than the narrow key/value passes). Enabled on the secret-exec+ * audit trail, whose long-lived log records a full argv that may carry a+ * foreign credential passed as a positional argument.+ */+ readonly redactTokens?: boolean;}export function redactRawOutput(text: string, opts: RedactRawOutputOptions = {}): string {@@ -361,6 +412,8 @@ export function redactRawOutput(text: string, opts: RedactRawOutputOptions = {})// Infra-topology pass last: it runs on values the key/value passes// already left untouched (bare coordinates), and any value they redacted// is now a placeholder with no IP/host shape left to match.+ if (opts.redactTokens) out = redactBareTokens(out);+if (opts.redactInfra) out = redactInfraTopology(out);return out;diff --git a/src/core/search/indexer.ts b/src/core/search/indexer.tsindex 6ccbf51e..bf98a97b 100644--- a/src/core/search/indexer.ts+++ b/src/core/search/indexer.ts@@ -46,7 +46,7 @@ import { parseFrontmatterText } from "../vault.ts";import { appendMetric } from "../brain/metrics.ts";import { throwIfAborted } from "../brain/safeguard.ts";import { extractEntities } from "./entities.ts";-import { Store } from "./store.ts";+import { acquireWriterLock, Store } from "./store.ts";import { LATEST_SCHEMA_VERSION } from "./schema.ts";import { SearchError } from "./types.ts";import { walkVault } from "./walker.ts";@@ -638,49 +638,67 @@ export async function reindexVault(mkdirSync(dirname(config.dbPath), { recursive: true });- // Build into the temp file with an override config.- const tempConfig: ResolvedSearchConfig = Object.freeze({- ...config,- dbPath: newPath,- });+ // Hold the writer lock on the LIVE index path for the whole rebuild ++ // swap. Keyed on `config.dbPath` (not the `.new` staging path), so a+ // second concurrent reindex โ the double-reindex a schema-bump upgrade+ // triggers when CLI and the long-lived MCP server both self-heal at+ // once โ waits-or-bails on the SAME lock instead of unlinking this run's+ // in-progress staging DB and then having its empty seed swapped over the+ // live index (INDEX_UNREADABLE, silent data loss). The staging-DB opens+ // below lock a different path (`.new`), so there is no self-deadlock.+ const release = await acquireWriterLock(config.dbPath);+ try {+ // Build into the temp file with an override config.+ const tempConfig: ResolvedSearchConfig = Object.freeze({+ ...config,+ dbPath: newPath,+ });- // Resumable staging (opt-in). A compatible in-progress `.new` from an- // interrupted run is resumed via the incremental fastpath instead of- // rebuilt from scratch; an incompatible or unreadable one is- // discarded. With the flag OFF, the temp file is always rebuilt fresh- // and no staging marker is written or read - byte-identical to the- // pre-suite path (no extra staging-DB open/close cycles).- let resume = false;- if (config.resumeReindex) {- const signature = reindexStagingSignature(config, opts?.embeddings === true);- if (existsSync(newPath)) {- resume = await stagingSignatureMatches(tempConfig, signature);+ // Resumable staging (opt-in). A compatible in-progress `.new` from an+ // interrupted run is resumed via the incremental fastpath instead of+ // rebuilt from scratch; an incompatible or unreadable one is+ // discarded. With the flag OFF, the temp file is always rebuilt fresh+ // and no staging marker is written or read - byte-identical to the+ // pre-suite path (no extra staging-DB open/close cycles).+ let resume = false;+ if (config.resumeReindex) {+ const signature = reindexStagingSignature(config, opts?.embeddings === true);+ if (existsSync(newPath)) {+ resume = await stagingSignatureMatches(tempConfig, signature);+ }+ if (!resume) tryUnlink(newPath);+ // Stamp the signature so an interruption leaves a build a later run+ // can recognise and resume.+ await withStagingStore(tempConfig, (store) =>+ store.setState(REINDEX_SIGNATURE_KEY, signature),+ );+ } else {+ tryUnlink(newPath);}- if (!resume) tryUnlink(newPath);- // Stamp the signature so an interruption leaves a build a later run- // can recognise and resume.- await withStagingStore(tempConfig, (store) => store.setState(REINDEX_SIGNATURE_KEY, signature));- } else {- tryUnlink(newPath);- }- // `force: false` on resume lets the fastpath skip the files the- // partial build already committed; a fresh build forces every file.- const stats = await indexVault(tempConfig, { ...opts, force: !resume });+ // `force: false` on resume lets the fastpath skip the files the+ // partial build already committed; a fresh build forces every file.+ const stats = await indexVault(tempConfig, { ...opts, force: !resume });- // Clear the marker so the swapped-in live index carries no staging- // state. Only present when resume was enabled.- if (config.resumeReindex) {- await withStagingStore(tempConfig, (store) => store.deleteState(REINDEX_SIGNATURE_KEY));- }+ // Clear the marker so the swapped-in live index carries no staging+ // state. Only present when resume was enabled.+ if (config.resumeReindex) {+ await withStagingStore(tempConfig, (store) => store.deleteState(REINDEX_SIGNATURE_KEY));+ }- // Same-directory rename swap. The two renames are each atomic on- // POSIX; the gap between them is the only crash window, which the- // .bak restore on Store.open handles.- tryUnlink(bakPath);- tryRename(config.dbPath, bakPath); // no-op (ENOENT) on fresh reindex- renameSync(newPath, config.dbPath); // must succeed โ `newPath` was just built- return stats;+ // Same-directory rename swap. The two renames are each atomic on+ // POSIX; the gap between them is the only crash window. A read that+ // opens in that window is held off by this same lock (see+ // restoreFromBakIfMissing in store.ts), so it cannot restore the stale+ // `.bak` over the freshly built index; a genuine crash leaves the lock+ // stale and the .bak restore on the next Store.open recovers.+ tryUnlink(bakPath);+ tryRename(config.dbPath, bakPath); // no-op (ENOENT) on fresh reindex+ renameSync(newPath, config.dbPath); // must succeed โ `newPath` was just built+ return stats;+ } finally {+ await release();+ }}/** index_state key carrying the staging-build compatibility signature. */diff --git a/src/core/search/schema.ts b/src/core/search/schema.tsindex ca3a95df..b55c268c 100644--- a/src/core/search/schema.ts+++ b/src/core/search/schema.ts@@ -16,7 +16,21 @@ import { SearchError } from "./types.ts";* this raises `SCHEMA_MISMATCH` on open โ the operator must reindex* with a newer binary.*/-export const LATEST_SCHEMA_VERSION = 7;+export const LATEST_SCHEMA_VERSION = 8;++/**+ * The wikilink-resolution basename of a stored document path: the final+ * `/`-segment with a trailing `.md` stripped. `notes/alpha.md` โ `alpha`,+ * top-level `alpha.md` โ `alpha`. Persisted in `documents.basename` so+ * dangling-link resolution can equality-join an index instead of+ * `SUBSTR`-scanning every path. Must stay in lockstep with the resolution+ * ladder in `store.ts` (`<target>.md` exact match, then basename suffix).+ */+export function documentBasename(path: string): string {+ const slash = path.lastIndexOf("/");+ const name = slash >= 0 ? path.slice(slash + 1) : path;+ return name.endsWith(".md") ? name.slice(0, -".md".length) : name;+}const DDL_V1 = `CREATE TABLE IF NOT EXISTS documents (@@ -325,6 +339,34 @@ export const MIGRATIONS: ReadonlyArray<Migration> = Object.freeze([`);},},+ {+ // v8 (hot-path-performance) - a persisted `documents.basename` column+ // (final path segment, `.md` stripped) with an index, so+ // dangling-wikilink resolution (`resolvedDocLinkPairs`,+ // `resolveAliasTargets`) can equality-join `idx_documents_basename`+ // instead of `SUBSTR`-scanning every `documents.path` twice per+ // unresolved link. The column is backfilled from the existing paths+ // in-place, so a migrated (not-yet-reindexed) index resolves links+ // identically at once; a full reindex repopulates it via+ // `upsertDocument`.+ version: 8,+ up(db) {+ const docCols = db.query<{ name: string }, []>("PRAGMA table_info(documents)").all();+ if (!docCols.some((c) => c.name === "basename")) {+ db.exec("ALTER TABLE documents ADD COLUMN basename TEXT");+ }+ db.exec("CREATE INDEX IF NOT EXISTS idx_documents_basename ON documents(basename)");+ const rows = db+ .query<{ id: number; path: string }, []>(+ "SELECT id, path FROM documents WHERE basename IS NULL",+ )+ .all();+ const update = db.prepare<undefined, [string, number]>(+ "UPDATE documents SET basename = ? WHERE id = ?",+ );+ for (const row of rows) update.run(documentBasename(row.path), row.id);+ },+ },]);/** Read the current schema version from `index_state`. Returns 0 if not yet recorded. */diff --git a/src/core/search/search.ts b/src/core/search/search.tsindex 94b9e8b1..6a7e4d90 100644--- a/src/core/search/search.ts+++ b/src/core/search/search.ts@@ -201,18 +201,22 @@ function addStructuredReasons(}/**- * Open the index for reading, self-healing a stale or absent index. After a- * plugin upgrade the on-disk index can be a different schema version- * (`SCHEMA_MISMATCH`) or not yet built (`INDEX_MISSING`); rather than forcing- * the user to run `o2b search reindex` / `o2b search index`, rebuild once and- * retry. `reindexVault` is imported lazily so the hot path never pulls in the- * indexer and there is no module cycle.+ * Open the index for reading, self-healing a stale, absent, or unreadable+ * index. After a plugin upgrade the on-disk index can be a different schema+ * version (`SCHEMA_MISMATCH`), not yet built (`INDEX_MISSING`), or corrupt+ * / truncated / non-OSB at the index path (`INDEX_UNREADABLE`); rather than+ * forcing the user to run `o2b search reindex` / `o2b search index`, rebuild+ * once and retry. `reindexVault` is imported lazily so the hot path never+ * pulls in the indexer and there is no module cycle.*/async function openReadOrSelfHeal(config: ResolvedSearchConfig): Promise<Store> {try {return await Store.open(config, { mode: "read" });} catch (e) {- if (e instanceof SearchError && (e.code === "INDEX_MISSING" || e.code === "SCHEMA_MISMATCH")) {+ if (+ e instanceof SearchError &&+ (e.code === "INDEX_MISSING" || e.code === "SCHEMA_MISMATCH" || e.code === "INDEX_UNREADABLE")+ ) {try {const { reindexVault } = await import("./indexer.ts");await reindexVault(config);@@ -1150,11 +1154,16 @@ function toSearchCard(result: BrainSearchResult): SearchCard {});}-function cardSnippet(content: string): string {+export function cardSnippet(content: string): string {const collapsed = content.replace(/\s+/g, " ").trim();- return collapsed.length <= CARD_SNIPPET_CHARS+ // Truncate on code points, not UTF-16 units: a raw `.slice` can cut an+ // astral character (emoji, rare CJK) mid-surrogate-pair, shipping a lone+ // surrogate that renders as U+FFFD. Spreading into an array iterates by+ // code point, so the cap never splits a character.+ const points = [...collapsed];+ return points.length <= CARD_SNIPPET_CHARS? collapsed- : `${collapsed.slice(0, CARD_SNIPPET_CHARS)}...`;+ : `${points.slice(0, CARD_SNIPPET_CHARS).join("")}...`;}/**diff --git a/src/core/search/store.ts b/src/core/search/store.tsindex acddc569..649d1af8 100644--- a/src/core/search/store.ts+++ b/src/core/search/store.ts@@ -25,6 +25,7 @@ import { SearchError } from "./types.ts";import type { ResolvedSearchConfig } from "./types.ts";import {applyMigrations,+ documentBasename,dropVecTable,ensureVecTable,LATEST_SCHEMA_VERSION,@@ -248,6 +249,75 @@ function acquireWriterLockSync(path: string): () => void {throw new SearchError("INDEX_LOCKED", `another writer holds the search index lock: ${msg}`);}+/**+ * Acquire the exclusive writer lock on the LIVE index path. Shared by+ * `Store.open({ mode: "write" })` and `reindexVault`'s rebuild+swap so both+ * serialise on the SAME lock (keyed on `dbPath`, not on the `.new` staging+ * path). Fast-fails to `INDEX_LOCKED` after a few retries rather than+ * blocking indefinitely, matching the module's fail-fast contention+ * contract. `realpath: false` lets the lock be taken before `dbPath` exists+ * (a fresh reindex has no live index yet); only the parent directory must+ * already exist.+ */+export async function acquireWriterLock(dbPath: string): Promise<() => Promise<void>> {+ try {+ return await lockfile.lock(dbPath, {+ retries: { retries: 3, factor: 1, minTimeout: 1000, maxTimeout: 1000 },+ stale: WRITER_LOCK_STALE_MS,+ // Explicit heartbeat: refresh the lock mtime mid-run so a long index+ // is never mistaken for a stale lock and taken over.+ update: WRITER_LOCK_HEARTBEAT_MS,+ realpath: false,+ });+ } catch (e) {+ const msg = e instanceof Error ? e.message : String(e);+ throw new SearchError("INDEX_LOCKED", `another writer holds the search index lock: ${msg}`);+ }+}++/**+ * Crash-recovery preamble shared by every {@link Store.open}: if the live+ * index file is absent but a `.bak` from a `reindex` rename swap is present,+ * restore it. Guarded by the SAME writer lock `reindexVault` holds across+ * its swap, so a read that opens during the swap's brief db-absent window+ * cannot mistake it for a crashed reindex and clobber the freshly built+ * index with the stale `.bak` (the silent-data-loss race, A.2).+ *+ * Lock semantics distinguish the two cases without a heuristic:+ * - a live reindex mid-swap holds a heartbeated (non-stale) lock, so we+ * block briefly, then find `dbPath` present and skip the restore;+ * - a genuine crash leaves a stale lock (no heartbeat) that is taken over+ * within the stale window, after which we restore.+ * If the lock cannot be taken at all we skip the restore rather than risk a+ * clobber; the caller's open path then re-evaluates existence and reports+ * honestly (INDEX_MISSING). The `.bak` restore of a genuine crash is+ * unaffected: with no live holder the lock is acquired immediately.+ */+function restoreFromBakIfMissing(dbPath: string): void {+ const bak = dbPath + ".bak";+ if (existsSync(dbPath) || !existsSync(bak)) return;+ let release: (() => void) | null = null;+ try {+ release = acquireWriterLockSync(dbPath);+ } catch {+ // A live writer holds the lock (mid-swap). Do not restore โ the swap+ // will place the fresh index; the open path re-checks existence.+ return;+ }+ try {+ // Re-check under the lock: the swap may have completed while we waited,+ // or another opener may have already restored.+ if (existsSync(dbPath) || !existsSync(bak)) return;+ renameSync(bak, dbPath);+ // eslint-disable-next-line no-console+ console.error(`restored search index from ${bak} (previous reindex crash)`);+ } catch {+ /* fall through โ open path below will report INDEX_MISSING */+ } finally {+ release();+ }+}+/*** Canonical alias/lookup-key normalisation for `doc_aliases` (v7):* trim, NFC-normalise, lower-case - the exact rule@@ -290,20 +360,11 @@ export class Store {const loadVec = opts.loadVec !== false;// Crash recovery: if the main index is missing but `.bak` is present- // from a failed `reindex` rename window, restore it. Stderr notice- // (not a thrown error) so existing tooling keeps working.- if (!existsSync(config.dbPath)) {- const bak = config.dbPath + ".bak";- if (existsSync(bak)) {- try {- renameSync(bak, config.dbPath);- // eslint-disable-next-line no-console- console.error(`restored search index from ${bak} (previous reindex crash)`);- } catch {- /* fall through โ open path below will report INDEX_MISSING */- }- }- }+ // from a failed `reindex` rename window, restore it. Lock-guarded so a+ // read opening during a live reindex's swap window cannot clobber the+ // freshly built index with the stale `.bak`. Stderr notice (not a thrown+ // error) so existing tooling keeps working.+ restoreFromBakIfMissing(config.dbPath);if (opts.mode === "read") {if (!existsSync(config.dbPath)) {@@ -358,20 +419,7 @@ export class Store {seed.close();}- let release: () => Promise<void>;- try {- release = await lockfile.lock(config.dbPath, {- retries: { retries: 3, factor: 1, minTimeout: 1000, maxTimeout: 1000 },- stale: WRITER_LOCK_STALE_MS,- // Explicit heartbeat: refresh the lock mtime mid-run so a long- // index is never mistaken for a stale lock and taken over.- update: WRITER_LOCK_HEARTBEAT_MS,- realpath: false,- });- } catch (e) {- const msg = e instanceof Error ? e.message : String(e);- throw new SearchError("INDEX_LOCKED", `another writer holds the search index lock: ${msg}`);- }+ const release = await acquireWriterLock(config.dbPath);let db: Database;try {@@ -566,11 +614,23 @@ export class Store {const row = this.db.query<{ id: number },- [string, string | null, string, number, number, string | null, string, string, string]+ [+ string,+ string,+ string | null,+ string,+ number,+ number,+ string | null,+ string,+ string,+ string,+ ]>(- "INSERT INTO documents(path, title, content_hash, mtime, size, page_type, created_at, updated_at, indexed_at) " +- "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) " ++ "INSERT INTO documents(path, basename, title, content_hash, mtime, size, page_type, created_at, updated_at, indexed_at) " ++ "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) " +"ON CONFLICT(path) DO UPDATE SET " ++ " basename = excluded.basename, " +" title = excluded.title, " +" content_hash = excluded.content_hash, " +" mtime = excluded.mtime, " +@@ -582,6 +642,7 @@ export class Store {).get(doc.path,+ documentBasename(doc.path),doc.title,doc.contentHash,doc.mtime,@@ -1021,9 +1082,12 @@ export class Store {"JOIN documents d ON d.id = a.document_id " +"WHERE a.alias = ? ORDER BY d.path ASC LIMIT 1",);- const basenameExists = this.db.prepare<{ id: number }, [string, string, string]>(- "SELECT id FROM documents WHERE path = ? " +- "OR SUBSTR(path, -(LENGTH(?) + 4)) = '/' || ? || '.md' LIMIT 1",+ // A real document basename owns the target (top-level `target.md` or+ // any nested `.../target.md`): both collapse to `basename = target`,+ // an index lookup instead of the old `path = target.md OR SUBSTR(...)`+ // full scan.+ const basenameExists = this.db.prepare<{ id: number }, [string]>(+ "SELECT id FROM documents WHERE basename = ? LIMIT 1",);const update = this.db.prepare<undefined, [number, string]>("UPDATE links SET target_document_id = ? " +@@ -1037,7 +1101,7 @@ export class Store {const key = normalizeAlias(row.target_path);if (key.length === 0) continue;// Skip targets a real document basename owns.- if (basenameExists.get(`${row.target_path}.md`, row.target_path, row.target_path)) {+ if (basenameExists.get(row.target_path)) {continue;}const owner = aliasOwner.get(key);@@ -1064,14 +1128,21 @@ export class Store {resolvedDocLinkPairs(): Array<{ readonly source: number; readonly target: number }> {const rows = this.db.query<{ source: number; target: number | null }, []>(+ // Resolution ladder: materialized id, then `<target>.md` exact+ // (path is UNIQUE + indexed), then an UNAMBIGUOUS nested-basename+ // match. The basename branch equality-joins idx_documents_basename+ // and mirrors the old `SUBSTR(path) = '/'||target||'.md'` scan+ // exactly: a `/`-aligned basename suffix is precisely+ // `basename = target AND path is nested` (top-level `target.md`+ // has no leading slash and is owned by the exact branch above)."SELECT DISTINCT l.source_document_id AS source, " +" COALESCE(" +" l.target_document_id, " +" (SELECT d.id FROM documents d WHERE d.path = l.target_path || '.md'), " +" (SELECT d.id FROM documents d " +- " WHERE SUBSTR(d.path, -(LENGTH(l.target_path) + 4)) = '/' || l.target_path || '.md' " ++ " WHERE d.basename = l.target_path AND instr(d.path, '/') > 0 " +" AND 1 = (SELECT COUNT(*) FROM documents d2 " +- " WHERE SUBSTR(d2.path, -(LENGTH(l.target_path) + 4)) = '/' || l.target_path || '.md'))" ++ " WHERE d2.basename = l.target_path AND instr(d2.path, '/') > 0))" +" ) AS target " +"FROM links l WHERE l.target_path IS NOT NULL",)diff --git a/src/core/surface/lexical-score.ts b/src/core/surface/lexical-score.tsindex 9d56cd05..c6c7ab70 100644--- a/src/core/surface/lexical-score.ts+++ b/src/core/surface/lexical-score.ts@@ -40,17 +40,29 @@ const DEFAULTS = Object.freeze({ nameWeight: 3, tagWeight: 2, k1: 1.2, b: 0.75 }* is provably unchanged, and the only consumer of scoreDescriptors today* is the skill-attach path. The range covers Han Unified Ideographs only;* kana / Hangul / CJK extensions are not bigram-split.+ *+ * Bigrams are extracted only from each maximal contiguous Han span within a+ * token, so a mixed token like "gbrain..." yields the inner Han bigram but+ * never a cross-script window (e.g. an ASCII+Han pair) that would match+ * nothing yet inflate term frequency / document length. Scope is thus the+ * Han run, not the total token length: an embedded 2-char Han run still+ * emits its bigram.*/+const HAN_SPAN = /[ไธ-้ฟฟ]+/gu;+export function tokenize(text: string): string[] {const out: string[] = [];for (const raw of text.toLowerCase().split(/[^\p{L}\p{N}]+/u)) {if (raw.length < 2) continue;- // Han bigram pass: a token with a Han character and length > 2 also- // emits overlapping 2-char windows so "ๅฎ็ฐๆนๅผ" and "็ๅฎ็ฐๆนๅผ"- // share the bigram "ๅฎ็ฐ".- if (/[\u4e00-\u9fff]/.test(raw) && raw.length > 2) {- for (let i = 0; i < raw.length - 1; i++) {- out.push(raw.slice(i, i + 2));+ // Han bigram pass: emit overlapping 2-char windows from each maximal Han+ // span so a spaceless run and a query prefixed run share their inner+ // bigrams, without crossing into adjacent ASCII.+ for (const span of raw.match(HAN_SPAN) ?? []) {+ for (let i = 0; i < span.length - 1; i++) {+ const bigram = span.slice(i, i + 2);+ // Skip a bigram equal to the whole token (a standalone 2-char Han+ // token) so it is not emitted twice.+ if (bigram !== raw) out.push(bigram);}}out.push(raw);diff --git a/src/mcp/brain/generation-tools.ts b/src/mcp/brain/generation-tools.tsindex 84895f8a..6a0d1e89 100644--- a/src/mcp/brain/generation-tools.ts+++ b/src/mcp/brain/generation-tools.ts@@ -26,6 +26,7 @@ import {type GenerationReportFilter,type GenerationUsage,} from "../../core/brain/generation-reports.ts";+import { isCanonicalUtcTimestamp } from "../../core/brain/continuity/store.ts";import type { ContinuitySourceRef } from "../../core/brain/continuity/types.ts";import { INVALID_PARAMS, MCPError } from "../protocol.ts";import type { ServerContext, ToolDefinition } from "../tools.ts";@@ -56,6 +57,19 @@ function recordAction(ctx: ServerContext, args: Record<string, unknown>): Recordconst agent = requiredStringArg(TOOL, args, "agent");const prompt = requiredStringArg(TOOL, args, "prompt");+ // Validate `created_at` here rather than relying on the store guard:+ // `emitGenerationReport` is fail-open (a throwing build returns null),+ // so a malformed timestamp would silently report `recorded: false`+ // instead of the INVALID_PARAMS the caller needs.+ const createdAt = optionalStringArg(TOOL, args, "created_at");+ if (createdAt !== undefined && !isCanonicalUtcTimestamp(createdAt)) {+ throw new MCPError(+ INVALID_PARAMS,+ `${TOOL}: created_at must be a canonical UTC ISO-8601 timestamp ` ++ `(e.g. 2026-07-06T15:00:00Z or 2026-07-06T15:00:00.000Z)`,+ );+ }+// Opt-in: per-call enable wins, else the config gate. Off => nothing written.const enabled =args["enable"] === true || resolveGenerationTraceEnabled(ctx.configPath ?? undefined);diff --git a/tests/core/brain/continuity-store.test.ts b/tests/core/brain/continuity-store.test.tsindex efe38ce7..4e848548 100644--- a/tests/core/brain/continuity-store.test.ts+++ b/tests/core/brain/continuity-store.test.ts@@ -9,6 +9,7 @@ import {appendContinuityRecords,appendContinuitySourceInvalidation,continuityLogPath,+ isCanonicalUtcTimestamp,listContinuityRecords,paginateContinuityRecords,} from "../../../src/core/brain/continuity/store.ts";@@ -123,6 +124,33 @@ describe("continuity store", () => {});expect(records.map((record) => record.kind)).toEqual(["session_turn", "source_invalidation"]);});++ test("since/until shard-skip returns the same records as an unfiltered window", () => {+ const months = ["2026-03", "2026-05", "2026-07", "2026-09"];+ for (const month of months) {+ appendContinuityRecord(vault, {+ kind: "context_receipt",+ createdAt: `${month}-15T12:00:00Z`,+ sourceRefs: [{ id: `src-${month}` }],+ payload: { query: month },+ });+ }++ // A bounded window skips the 2026-03 and 2026-09 shards but returns+ // exactly the in-window records, in ascending createdAt order.+ const windowed = listContinuityRecords(vault, {+ since: "2026-05-01T00:00:00Z",+ until: "2026-07-31T23:59:59Z",+ });+ expect(windowed.map((r) => r.payload["query"])).toEqual(["2026-05", "2026-07"]);++ // Boundary months are read in full: since exactly on a record's month.+ const openEnded = listContinuityRecords(vault, { since: "2026-05-15T12:00:00Z" });+ expect(openEnded.map((r) => r.payload["query"])).toEqual(["2026-05", "2026-07", "2026-09"]);++ // No filter still reads every shard.+ expect(listContinuityRecords(vault)).toHaveLength(4);+ });});describe("continuity batch append", () => {@@ -212,3 +240,53 @@ describe("continuity batch append", () => {expect(existsSync(continuityLogPath(vault, "2026-06"))).toBe(false);});});++describe("createdAt validation at the store boundary", () => {+ const ACCEPT: string[] = [+ "2026-05-31T12:00:00Z", // second precision+ "2026-06-15T10:00:00.000Z", // millisecond precision+ "2026-06-15T10:00:00.5Z", // sub-second, โค3 digits+ "2024-02-29T00:00:00Z", // real leap day+ "2026-12-31T23:59:59.999Z",+ ];+ const REJECT: Array<[string, unknown]> = [+ ["month out of range shards into junk", "2026-13-01T00:00:00Z"],+ ["day out of range for the month", "2026-02-30T00:00:00Z"],+ ["Feb 29 on a non-leap year", "2026-02-29T00:00:00Z"],+ ["numeric offset instead of Z mis-sorts", "2026-07-06T15:00:00+03:00"],+ ["no zone designator", "2026-07-06T15:00:00"],+ ["date only", "2026-07-06"],+ ["month prefix only", "2026-07"],+ ["free text", "not-a-timestamp"],+ ["empty string", ""],+ ["lowercase z", "2026-07-06T15:00:00z"],+ ["non-string number", 1_783_350_000_000],+ ["null", null],+ ];++ test.each(ACCEPT)("accepts %s", (value) => {+ expect(isCanonicalUtcTimestamp(value)).toBe(true);+ const record = appendContinuityRecord(vault, {+ kind: "session_turn",+ createdAt: value,+ sourceRefs: [{ id: "ok" }],+ payload: { snippet: "valid" },+ });+ expect(record.createdAt).toBe(value);+ });++ test.each(REJECT)("rejects %s", (_label, value) => {+ expect(isCanonicalUtcTimestamp(value)).toBe(false);+ expect(() =>+ appendContinuityRecord(vault, {+ kind: "session_turn",+ // eslint-disable-next-line @typescript-eslint/no-explicit-any+ createdAt: value as any,+ sourceRefs: [{ id: "bad" }],+ payload: { snippet: "invalid" },+ }),+ ).toThrow(/invalid continuity createdAt/);+ // Nothing was written: no shard exists for a rejected timestamp.+ expect(listContinuityRecords(vault, {})).toHaveLength(0);+ });+});diff --git a/tests/core/brain/entities.registry.test.ts b/tests/core/brain/entities.registry.test.tsindex 321b5397..ab9a7dc7 100644--- a/tests/core/brain/entities.registry.test.ts+++ b/tests/core/brain/entities.registry.test.ts@@ -337,4 +337,44 @@ describe("buildEntityIndex", () => {const index = buildEntityIndex(vault);expect(index.entities).toHaveLength(1);});++ test("memoizes: an unchanged registry returns the same index object", () => {+ seedAda();+ const first = buildEntityIndex(vault);+ const second = buildEntityIndex(vault);+ expect(second).toBe(first); // same reference: no rebuild when nothing changed+ });++ test("invalidates the memo when a new entity is written", () => {+ seedAda();+ const before = buildEntityIndex(vault);+ expect(before.byKey.has("projects:open second brain")).toBe(false);+ upsertEntity(vault, {+ category: "projects",+ name: "Open Second Brain",+ agent: "claude-dev-agent",+ now: LATER,+ });+ const after = buildEntityIndex(vault);+ expect(after).not.toBe(before); // rebuilt after the write+ expect(after.byKey.get("projects:open second brain")?.id).toBe(+ "ent-projects-open-second-brain",+ );+ });++ test("invalidates the memo when an entity file is edited in place", () => {+ seedAda();+ expect(buildEntityIndex(vault).byAlias.has("newnym")).toBe(false);+ // Idempotent upsert rewrites the SAME file (same path) with an added+ // alias; the memo must notice via mtime/size and not serve the stale+ // index despite the unchanged file set.+ upsertEntity(vault, {+ category: "people",+ name: "Ada",+ aliases: ["ะะดะฐ", "S.E.", "Newnym"],+ agent: "claude-dev-agent",+ now: LATER,+ });+ expect(buildEntityIndex(vault).byAlias.get("newnym")?.id).toBe("ent-people-ada");+ });});diff --git a/tests/core/brain/format-wikilink.test.ts b/tests/core/brain/format-wikilink.test.tsindex 8665dbdf..a6054d23 100644--- a/tests/core/brain/format-wikilink.test.ts+++ b/tests/core/brain/format-wikilink.test.ts@@ -33,6 +33,22 @@ test("shortestUniqueSuffix grows until the suffix is unambiguous", () => {expect(shortestUniqueSuffix("Brain/archive/beta", PAGES)).toBe("archive/beta");});+test("shortestUniqueSuffix: memoized index stays correct across repeated calls and a 3-way basename clash", () => {+ const pages = [+ "a/x/beta",+ "b/y/beta",+ "c/z/beta", // three pages share basename `beta`; each needs 2 segments+ "solo/gamma",+ ];+ // Repeated calls reuse the memoized suffix index built from `pages`.+ expect(shortestUniqueSuffix("a/x/beta", pages)).toBe("x/beta");+ expect(shortestUniqueSuffix("b/y/beta", pages)).toBe("y/beta");+ expect(shortestUniqueSuffix("a/x/beta", pages)).toBe("x/beta"); // no corruption on reuse+ expect(shortestUniqueSuffix("solo/gamma", pages)).toBe("gamma");+ // A fresh array with identical content rebuilds the index and agrees.+ expect(shortestUniqueSuffix("c/z/beta", [...pages])).toBe("z/beta");+});+// โโ formatWikilinkBody โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโtest("full mode rewrites a unique basename to its full key path", () => {diff --git a/tests/core/brain/heal-enrich.test.ts b/tests/core/brain/heal-enrich.test.tsindex 2530b0b3..e5a6a27c 100644--- a/tests/core/brain/heal-enrich.test.ts+++ b/tests/core/brain/heal-enrich.test.ts@@ -14,7 +14,10 @@ import { describe, expect, test } from "bun:test";import {deriveTitleFromContent,linkExactMentions,+ linkExactMentionsPrepared,planHealEnrichment,+ planHealEnrichmentPrepared,+ prepareHealPhrases,} from "../../../src/core/brain/heal-enrich.ts";describe("deriveTitleFromContent", () => {@@ -101,3 +104,51 @@ describe("planHealEnrichment", () => {expect(plan.body).toBeUndefined();});});++describe("prepared-set path is byte-identical to the excluded-list path", () => {+ // Full known set; several phrases where one page's own multi-word title+ // ("Foo Bar") contains another known phrase ("Bar") as a whole sub-token+ // - the exact case where a naive compile-once-then-post-filter would+ // diverge. The prepared path must match the excluded-list reference.+ const known = ["Bar", "Foo Bar", "Acme Corp", "Acme", "Widget"];+ const bodies = [+ "Foo Bar mentions Bar and Widget here",+ "Acme Corp and Acme both appear, plus `Widget` in code",+ "[[Foo Bar]] already linked; Bar again",+ "nothing relevant",+ "Bar Foo Bar Bar",+ ];+ const excludeSets: ReadonlyArray<ReadonlyArray<string>> = [+ ["Foo Bar"], // page owns the multi-word title containing "Bar"+ ["Acme Corp"],+ [],+ ["Widget"],+ ["Bar"],+ ];++ const prepared = prepareHealPhrases(known);++ for (const body of bodies) {+ for (const exclude of excludeSets) {+ test(`link "${body.slice(0, 20)}" excluding [${exclude.join(",")}]`, () => {+ const reference = linkExactMentions(+ body,+ known.filter((k) => !new Set(exclude).has(k)),+ );+ const viaPrepared = linkExactMentionsPrepared(body, prepared, new Set(exclude));+ expect(viaPrepared).toBe(reference);+ });+ }+ }++ test("planHealEnrichmentPrepared matches planHealEnrichment on the excluded list", () => {+ const page = { frontmatter: { title: "Foo Bar" }, body: "Foo Bar links Bar and Widget" };+ const exclude = new Set(["Foo Bar"]);+ const reference = planHealEnrichment(+ page,+ known.filter((k) => !exclude.has(k)),+ );+ const viaPrepared = planHealEnrichmentPrepared(page, prepared, exclude);+ expect(viaPrepared).toEqual(reference);+ });+});diff --git a/tests/core/brain/query-demand.test.ts b/tests/core/brain/query-demand.test.tsindex 0c373e23..d8e5d1f4 100644--- a/tests/core/brain/query-demand.test.ts+++ b/tests/core/brain/query-demand.test.ts@@ -112,6 +112,38 @@ describe("readQueryDemand", () => {test("empty when the log does not exist", () => {expect(readQueryDemand(vault)).toEqual([]);});++ test("second-precision since/until bounds compare consistently against ms records", () => {+ // Records are stored at millisecond precision. A raw lexical compare+ // against a second-precision bound misfires because `.` (0x2E) sorts+ // before `Z` (0x5A): `since 00:00:00Z` wrongly DROPS the whole+ // [.000, .999] window, and `until 00:00:00Z` wrongly KEEPS it.+ recordQueryDemand(vault, { terms: ["before"], resultCount: 1, at: "2026-06-30T23:59:59.500Z" });+ recordQueryDemand(vault, {+ terms: ["lowedge"],+ resultCount: 1,+ at: "2026-07-01T00:00:00.000Z",+ });+ recordQueryDemand(vault, { terms: ["midlow"], resultCount: 1, at: "2026-07-01T00:00:00.500Z" });+ recordQueryDemand(vault, {+ terms: ["highedge"],+ resultCount: 1,+ at: "2026-07-01T00:00:00.999Z",+ });+ recordQueryDemand(vault, { terms: ["after"], resultCount: 1, at: "2026-07-01T00:00:01.000Z" });++ // `since 00:00:00Z` (normalized to .000Z) keeps the whole second and the+ // record after it; the pre-fix lexical compare dropped lowedge/midlow/+ // highedge because `.NNNZ` < `Z`.+ const fromSecond = readQueryDemand(vault, { since: "2026-07-01T00:00:00Z" });+ expect(fromSecond.map((r) => r.terms[0])).toEqual(["lowedge", "midlow", "highedge", "after"]);++ // `until 00:00:00Z` (normalized to .000Z) is inclusive of exactly .000Z+ // and excludes the rest of the second; the pre-fix lexical compare+ // wrongly kept midlow/highedge (`.NNNZ` < `Z` โ not `> until`).+ const untilSecond = readQueryDemand(vault, { until: "2026-07-01T00:00:00Z" });+ expect(untilSecond.map((r) => r.terms[0])).toEqual(["before", "lowedge"]);+ });});describe("aggregateQueryDemand", () => {diff --git a/tests/core/brain/secrets/exec.test.ts b/tests/core/brain/secrets/exec.test.tsindex 740dba74..056bf48c 100644--- a/tests/core/brain/secrets/exec.test.ts+++ b/tests/core/brain/secrets/exec.test.ts@@ -121,6 +121,42 @@ describe("runWithSecret", () => {}});+ test("a foreign bare secret in argv is redacted in the persisted audit trail", async () => {+ setSecret(vault, {+ name: "api-key",+ value: "own-custody-secret",+ envVar: "MY_API_KEY",+ allow: ["bun *"],+ agent: "tester",+ now: NOW,+ });+ // The agent passes some OTHER credential as a bare positional argument โ+ // it never enters the custody store, so it is not a known literal, but it+ // must not land in the long-lived audit log in cleartext.+ const foreignToken = "sk-foreign-abc123def456";+ const result = await runWithSecret(+ vault,+ "api-key",+ ["bun", "-e", "process.exit(0)", foreignToken],+ CTX,+ );+ expect(result.exitCode).toBe(0);++ const auditDir = join(vault, "Brain", "log", "secret-custody");+ const raw = readdirSync(auditDir)+ .map((f) => readFileSync(join(auditDir, f), "utf8"))+ .join("");+ expect(raw).not.toContain(foreignToken);+ const started = raw+ .split("\n")+ .filter((l) => l.trim().length > 0)+ .map((l) => JSON.parse(l) as { action: string; details?: { command?: string } })+ .find((l) => l.action === "secret_exec_started");+ expect(started).toBeDefined();+ expect(started!.details!.command).toContain("***REDACTED***");+ expect(started!.details!.command).not.toContain(foreignToken);+ });+test("the subprocess exit code propagates", async () => {setSecret(vault, {name: "api-key",diff --git a/tests/core/search/card-snippet.test.ts b/tests/core/search/card-snippet.test.tsnew file mode 100644index 00000000..83aca10f--- /dev/null+++ b/tests/core/search/card-snippet.test.ts@@ -0,0 +1,49 @@+/**+ * `cardSnippet` truncates a layer-1 card snippet on CODE POINTS, not+ * UTF-16 units. A raw `.slice(0, 240)` can cut an astral character (emoji,+ * rare CJK) mid-surrogate-pair, shipping a lone surrogate that renders as+ * U+FFFD to the MCP `cards` disclosure surface. These fixtures place an+ * astral character exactly on the boundary and assert it survives whole.+ */++import { describe, expect, test } from "bun:test";++import { cardSnippet } from "../../../src/core/search/search.ts";++const CARD_SNIPPET_CHARS = 240;+const REPLACEMENT_CHAR = "๏ฟฝ";++describe("cardSnippet", () => {+ test("does not split a surrogate pair straddling the char boundary", () => {+ // 239 ASCII chars, then an emoji whose two UTF-16 units are at index+ // 239 and 240 โ a raw UTF-16 slice(0, 240) would keep only the high+ // surrogate. The emoji is followed by more text so truncation fires.+ const content = "a".repeat(CARD_SNIPPET_CHARS - 1) + "๐" + "b".repeat(50);+ const snippet = cardSnippet(content);++ expect(snippet.endsWith("...")).toBe(true);+ expect(snippet).not.toContain(REPLACEMENT_CHAR);+ // The whole emoji is retained (both code units present, none dangling).+ expect(snippet).toContain("๐");+ // Round-tripping through the code-point iterator is lossless โ no lone+ // surrogate survived (a lone surrogate re-encodes to U+FFFD).+ expect([...snippet].every((cp) => cp !== REPLACEMENT_CHAR)).toBe(true);+ // Kept 240 code points of body + the "..." marker.+ expect([...snippet].length).toBe(CARD_SNIPPET_CHARS + 3);+ });++ test("returns short content unchanged (no marker, whitespace collapsed)", () => {+ expect(cardSnippet(" hello world ")).toBe("hello world");+ expect(cardSnippet("plain ๐ emoji")).toBe("plain ๐ emoji");+ });++ test("measures the cap in code points, not UTF-16 units", () => {+ // 240 emoji = 480 UTF-16 units but exactly 240 code points, so it is at+ // the cap and must pass through untruncated.+ const content = "๐".repeat(CARD_SNIPPET_CHARS);+ const snippet = cardSnippet(content);+ expect(snippet.endsWith("...")).toBe(false);+ expect([...snippet].length).toBe(CARD_SNIPPET_CHARS);+ expect(snippet).not.toContain(REPLACEMENT_CHAR);+ });+});diff --git a/tests/core/search/reindex-swap-races.test.ts b/tests/core/search/reindex-swap-races.test.tsnew file mode 100644index 00000000..370f68c4--- /dev/null+++ b/tests/core/search/reindex-swap-races.test.ts@@ -0,0 +1,140 @@+/**+ * Index-swap hardening (t_f27d80fe). Two interlocking races around the+ * `reindexVault` rebuild + rename swap, both silent-data-loss class:+ *+ * A.1 โ two concurrent reindex runs. The pre-fix code unlinked and+ * re-seeded the shared `.new` staging path with NO lock held, so a+ * second run could destroy the first's in-progress staging DB and+ * then have its empty seed swapped over the live index+ * (INDEX_UNREADABLE, search stays broken until a manual reindex).+ *+ * A.2 โ a read opening in the swap window between `rename(db -> bak)` and+ * `rename(new -> db)`. The lockless crash-restore preamble on+ * `Store.open` saw "db missing, bak present", concluded a prior+ * reindex crashed, and restored the STALE `.bak` over the fresh+ * index โ discarding the whole rebuild.+ *+ * The fix serialises reindexes on the LIVE db-path writer lock and gates+ * the crash-restore on that same lock. The genuine crash-recovery path+ * (no live holder) is exercised by indexer.test.ts's ".bak auto-restore".+ */+import { afterEach, beforeEach, expect, test } from "bun:test";+import { cpSync, existsSync, rmSync } from "node:fs";++import { indexVault, reindexVault } from "../../../src/core/search/indexer.ts";+import { acquireWriterLock, Store } from "../../../src/core/search/store.ts";+import { SearchError } from "../../../src/core/search/types.ts";+import type { IndexStats } from "../../../src/core/search/types.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";++let vault: string;+let dbPath: string;+let cleanup: () => void;++beforeEach(() => {+ const v = createTempVault("reindex-swap-races");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+});++afterEach(() => {+ cleanup();+});++test("A.1: two overlapping reindex runs leave a complete, readable index", async () => {+ const docs = 8;+ for (let i = 0; i < docs; i++) writeMd(vault, `n${i}.md`, `# N${i}\n\nbody number ${i} words`);+ const cfg = makeConfig({ vault, dbPath });+ await indexVault(cfg); // seed a live index so the swap / .bak path engages++ // Launch a second reindex DURING the first's build phase (while it holds+ // the live-path writer lock). With the fix it serialises on that lock;+ // without it, its synchronous pre-lock `tryUnlink(.new)` destroyed the+ // first run's staging DB mid-build.+ //+ // The contender promise is held in a single-cell container rather than a+ // bare `let` because TypeScript's control-flow analysis does not track+ // mutations performed inside the `onFile` closure โ a bare `let second =+ // null` is narrowed to `null` at the post-`await` read site, breaking the+ // downstream `instanceof Error` narrowing (TS2358).+ const second: Array<Promise<IndexStats | Error>> = [];+ const firstStats = await reindexVault(cfg, {+ onFile: () => {+ if (second.length === 0) {+ second.push(+ reindexVault(cfg).catch((e: unknown) => (e instanceof Error ? e : new Error(String(e)))),+ );+ }+ },+ });+ const secondResult: IndexStats | Error | null =+ second.length > 0 ? ((await second[0]) ?? null) : null;++ expect(firstStats.added).toBeGreaterThanOrEqual(docs);++ // The second contender either rebuilt cleanly (took the lock once the+ // first released) or fast-failed with INDEX_LOCKED โ never corruption.+ if (secondResult instanceof Error) {+ expect(secondResult).toBeInstanceOf(SearchError);+ expect((secondResult as SearchError).code).toBe("INDEX_LOCKED");+ }++ // The surviving live index is complete and READABLE โ no empty seed was+ // swapped in, so no INDEX_UNREADABLE.+ const store = await Store.open(cfg, { mode: "read" });+ try {+ expect(store.listDocuments().size).toBe(docs);+ } finally {+ await store.close();+ }+ // Staging file cleaned up by the swap.+ expect(existsSync(`${dbPath}.new`)).toBe(false);+});++test("A.2: a read during the swap window does not restore stale .bak over the fresh index", async () => {+ writeMd(vault, "a.md", "# A\n\nalpha content");+ const cfg = makeConfig({ vault, dbPath });+ await indexVault(cfg); // live index present+ // Fabricate a `.bak` snapshot, as a prior reindex swap would leave.+ cpSync(dbPath, `${dbPath}.bak`);++ // Simulate a reindex mid-swap: hold the LIVE-path writer lock with the+ // live db momentarily absent โ the exact window between rename(db->bak)+ // and rename(new->db).+ const release = await acquireWriterLock(dbPath);+ try {+ rmSync(dbPath);++ // A concurrent read must NOT restore `.bak` while a writer holds the+ // lock: doing so would clobber the about-to-be-swapped-in fresh index.+ // It skips the restore and reports the transient missing state honestly.+ await expect(Store.open(cfg, { mode: "read" })).rejects.toMatchObject({+ code: "INDEX_MISSING",+ });++ // Crucially, `.bak` was left intact โ not consumed by a spurious restore.+ expect(existsSync(`${dbPath}.bak`)).toBe(true);+ expect(existsSync(dbPath)).toBe(false);+ } finally {+ await release();+ }+});++test("A.2: crash-restore still fires when no writer holds the lock (genuine crash)", async () => {+ writeMd(vault, "a.md", "# A\n\nalpha content");+ const cfg = makeConfig({ vault, dbPath });+ await indexVault(cfg);+ cpSync(dbPath, `${dbPath}.bak`);++ // No live writer: db absent + bak present is a genuine crashed-reindex+ // signature, so the next open restores immediately.+ rmSync(dbPath);+ const store = await Store.open(cfg, { mode: "read" });+ try {+ expect(existsSync(dbPath)).toBe(true);+ expect(existsSync(`${dbPath}.bak`)).toBe(false); // consumed by the restore+ } finally {+ await store.close();+ }+});diff --git a/tests/core/search/schema-migration-v8.test.ts b/tests/core/search/schema-migration-v8.test.tsnew file mode 100644index 00000000..c0d8410a--- /dev/null+++ b/tests/core/search/schema-migration-v8.test.ts@@ -0,0 +1,92 @@+/**+ * Schema migration v7 -> v8: the `documents.basename` column (hot-path+ * performance). v8 adds a `basename` column + `idx_documents_basename`+ * and backfills it from the existing `path` values so a migrated (not+ * yet reindexed) index resolves dangling wikilinks immediately.+ */++import { test, expect, beforeEach, afterEach } from "bun:test";+import { Database } from "bun:sqlite";+import { mkdtempSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ applyMigrations,+ documentBasename,+ LATEST_SCHEMA_VERSION,+ readSchemaVersion,+} from "../../../src/core/search/schema.ts";++let tmp: string;+let dbPath: string;++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "osb-mig-v8-"));+ dbPath = join(tmp, "test.sqlite");+});+afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++function hasColumn(db: Database, table: string, name: string): boolean {+ return db+ .query<{ name: string }, []>(`PRAGMA table_info(${table})`)+ .all()+ .some((c) => c.name === name);+}++function hasIndex(db: Database, name: string): boolean {+ return (+ db+ .query<{ name: string }, [string]>(+ "SELECT name FROM sqlite_master WHERE type='index' AND name = ?",+ )+ .get(name) !== null+ );+}++test("documentBasename strips the final path segment and .md", () => {+ expect(documentBasename("Brain/notes/alpha.md")).toBe("alpha");+ expect(documentBasename("alpha.md")).toBe("alpha");+ expect(documentBasename("a/b/c.md")).toBe("c");+ expect(documentBasename("no-extension")).toBe("no-extension");+});++test("fresh migration reaches latest with basename column and index", () => {+ const db = new Database(dbPath);+ expect(applyMigrations(db)).toBe(LATEST_SCHEMA_VERSION);+ expect(hasColumn(db, "documents", "basename")).toBe(true);+ expect(hasIndex(db, "idx_documents_basename")).toBe(true);+ db.close();+});++test("a v7 index upgrades to v8 and backfills basename from path", () => {+ const db = new Database(dbPath);+ applyMigrations(db); // -> latest+ // Seed rows, then simulate an older v7 index: drop the column + index+ // and rewind the version so the v8 migration re-runs the backfill.+ db.run(+ "INSERT INTO documents(id, path, basename, title, content_hash, mtime, size, created_at, updated_at, indexed_at) " ++ "VALUES (1, 'notes/alpha.md', NULL, 'A', 'h', 1, 1, 't', 't', 't')",+ );+ db.run(+ "INSERT INTO documents(id, path, basename, title, content_hash, mtime, size, created_at, updated_at, indexed_at) " ++ "VALUES (2, 'top.md', NULL, 'T', 'h', 1, 1, 't', 't', 't')",+ );+ db.run("DROP INDEX idx_documents_basename");+ db.run("UPDATE documents SET basename = NULL");+ db.run("UPDATE index_state SET value = '7' WHERE key = 'schema_version'");+ expect(readSchemaVersion(db)).toBe(7);++ expect(applyMigrations(db)).toBe(LATEST_SCHEMA_VERSION);+ expect(hasIndex(db, "idx_documents_basename")).toBe(true);+ const rows = db+ .query<{ id: number; basename: string }, []>("SELECT id, basename FROM documents ORDER BY id")+ .all();+ expect(rows).toEqual([+ { id: 1, basename: "alpha" },+ { id: 2, basename: "top" },+ ]);+ db.close();+});diff --git a/tests/core/search/self-heal.test.ts b/tests/core/search/self-heal.test.tsindex 89a02f0b..7474acf2 100644--- a/tests/core/search/self-heal.test.ts+++ b/tests/core/search/self-heal.test.ts@@ -1,8 +1,10 @@/*** Lazy self-heal on the search read path: after a plugin upgrade the on-disk- * index can be a stale schema (SCHEMA_MISMATCH) or absent (INDEX_MISSING). A- * search must transparently rebuild once and return results instead of forcing- * the user to run `o2b search reindex` / `o2b search index`.+ * index can be a stale schema (SCHEMA_MISMATCH), absent (INDEX_MISSING), or+ * unreadable (INDEX_UNREADABLE โ a corrupt or truncated file, or a non-OSB+ * sqlite db at the index path). A search must transparently rebuild once and+ * return results instead of forcing the user to run `o2b search reindex` /+ * `o2b search index`.*/import { test, expect, beforeEach, afterEach } from "bun:test";import { Database } from "bun:sqlite";@@ -10,6 +12,7 @@ import { existsSync, rmSync } from "node:fs";import { indexVault } from "../../../src/core/search/indexer.ts";import { search } from "../../../src/core/search/search.ts";+import { Store } from "../../../src/core/search/store.ts";import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";let vault: string;@@ -51,6 +54,33 @@ test("INDEX_MISSING on read self-heals: search builds the index on first use", aexpect(existsSync(dbPath)).toBe(true); // built by the self-heal});+test("INDEX_UNREADABLE on read self-heals: search rebuilds a corrupt index", async () => {+ const config = makeConfig({ vault, dbPath });+ await indexVault(config); // seed a real index so the path exists++ // Corrupt the index's metadata so a read open fails AFTER the file opens+ // successfully but DURING schema probing: drop the `index_state` table,+ // which makes `readSchemaVersion` throw a raw sqlite error that the read+ // path wraps as INDEX_UNREADABLE (the typed code for "index present but+ // cannot be read"). This is the realistic corruption shape โ a partial+ // write or a half-migrated file โ as opposed to total byte-level damage.+ const corruptor = new Database(dbPath);+ corruptor.run("DROP TABLE index_state");+ corruptor.close();++ // Sanity: the corruption really does surface as INDEX_UNREADABLE on a+ // raw read open (the pre-self-heal condition).+ await expect(Store.open(config, { mode: "read" })).rejects.toMatchObject({+ code: "INDEX_UNREADABLE",+ });++ // Without self-heal this INDEX_UNREADABLE escapes to the caller; with it,+ // the read path rebuilds once and returns results.+ const outcome = await search(config, { query: "fox" });+ expect(outcome.results.length).toBeGreaterThan(0);+ expect(existsSync(dbPath)).toBe(true); // rebuilt by the self-heal+});+test("a genuinely empty vault returns empty results, not an error", async () => {rmSync(`${vault}/Notes/foo.md`, { force: true });const config = makeConfig({ vault, dbPath });diff --git a/tests/core/search/store-aliases.test.ts b/tests/core/search/store-aliases.test.tsindex 67780358..30f31791 100644--- a/tests/core/search/store-aliases.test.ts+++ b/tests/core/search/store-aliases.test.ts@@ -53,8 +53,8 @@ function targetDocId(store: Store, sourceId: number): number | null {return row?.target_document_id ?? null;}-test("schema version is 7", () => {- expect(LATEST_SCHEMA_VERSION).toBe(7);+test("schema version is 8", () => {+ expect(LATEST_SCHEMA_VERSION).toBe(8);});test("normalizeAlias lower-cases, trims, and NFC-normalises", () => {@@ -191,3 +191,36 @@ describe("resolveAliasTargets", () => {}});});++describe("resolvedDocLinkPairs basename resolution (v8)", () => {+ test("exact top-level path wins over a nested basename; unique nested resolves; ambiguous nested is dropped", async () => {+ const store = await Store.open(config, { mode: "write" });+ try {+ // Top-level `alpha.md` and nested `notes/alpha.md` both exist: the+ // exact `<target>.md` branch must pick the top-level doc, not treat+ // `alpha` as an ambiguous basename.+ const topAlpha = doc(store, "alpha.md");+ doc(store, "notes/alpha.md");+ // Unique nested basename.+ const beta = doc(store, "notes/beta.md");+ // Ambiguous nested basename: two `.../gamma.md`.+ doc(store, "x/gamma.md");+ doc(store, "y/gamma.md");++ const sAlpha = doc(store, "src/s-alpha.md");+ const sBeta = doc(store, "src/s-beta.md");+ const sGamma = doc(store, "src/s-gamma.md");+ link(store, sAlpha, "alpha");+ link(store, sBeta, "beta");+ link(store, sGamma, "gamma");++ const pairs = store.resolvedDocLinkPairs();+ const bySource = new Map(pairs.map((p) => [p.source, p.target]));+ expect(bySource.get(sAlpha)).toBe(topAlpha); // exact top-level wins+ expect(bySource.get(sBeta)).toBe(beta); // unique nested resolves+ expect(bySource.has(sGamma)).toBe(false); // ambiguous nested dropped+ } finally {+ await store.close();+ }+ });+});diff --git a/tests/core/surface/lexical-score.test.ts b/tests/core/surface/lexical-score.test.tsindex ccb0bef4..b2beaabd 100644--- a/tests/core/surface/lexical-score.test.ts+++ b/tests/core/surface/lexical-score.test.ts@@ -39,6 +39,19 @@ test("a 2-char CJK token is kept whole without extra bigrams", () => {expect(tokenize("ๅฎ็ฐ")).toEqual(["ๅฎ็ฐ"]);});+test("bigrams come only from the Han span of a mixed token, not across the boundary", () => {+ // A spaceless mixed token like "gbrainๅฎ็ฐ" should share the Han bigram+ // "ๅฎ็ฐ" with a query, but must NOT emit the cross-script window "nๅฎ"+ // (inert noise that inflates term frequency / document length).+ expect(tokenize("gbrainๅฎ็ฐ")).toEqual(["ๅฎ็ฐ", "gbrainๅฎ็ฐ"]);+});++test("a 2-char Han run embedded in a longer mixed token still emits its bigram", () => {+ // Gated by the Han run, not the total token length: "abๅฎ็ฐcd" yields "ๅฎ็ฐ"+ // even though the token as a whole is > 2 chars of mostly ASCII.+ expect(tokenize("abๅฎ็ฐcd")).toEqual(["ๅฎ็ฐ", "abๅฎ็ฐcd"]);+});+test("ASCII tokenization is unaffected by the CJK bigram pass", () => {expect(tokenize("Hello World")).toEqual(["hello", "world"]);});diff --git a/tests/mcp/generation-reports-tool.test.ts b/tests/mcp/generation-reports-tool.test.tsindex 9e4270ab..8a31bad9 100644--- a/tests/mcp/generation-reports-tool.test.ts+++ b/tests/mcp/generation-reports-tool.test.ts@@ -140,6 +140,32 @@ describe("brain_generation_reports tool", () => {).toEqual([report["id"]]);});+ test("rejects a malformed created_at with INVALID_PARAMS", async () => {+ const server = new MCPServer({ vault, configPath });+ await initialize(server);++ const bad = (await server.handleRequest({+ jsonrpc: JSONRPC_VERSION,+ id: 11,+ method: "tools/call",+ params: {+ name: "brain_generation_reports",+ arguments: {+ action: "record",+ handoff_kind: "write_session",+ ref: "ws-bad",+ agent: "tester",+ enable: true,+ prompt: "do the thing",+ created_at: "2026-13-01T00:00:00Z",+ },+ },+ })) as { error?: { message?: string }; result?: { isError?: boolean } };+ expect(bad.error ?? bad.result?.isError).toBeTruthy();+ // Nothing was written โ the invalid timestamp is rejected before the emit.+ expect(listGenerationReports(vault)).toHaveLength(0);+ });+test("rejects an unknown action and an unknown handoff_kind", async () => {const server = new MCPServer({ vault, configPath });await initialize(server);