Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript

feat: near-duplicate defense across the fact lifecycle (v1.74.0) (#237)

Serg Eroshe0cbb60b85ec0
diff --git a/.apb/playbooks/osb-feature-release/1.4.0/playbook.yaml b/.apb/playbooks/osb-feature-release/1.4.0/playbook.yaml
new file mode 100644
index 00000000..9f02bf80
--- /dev/null
+++ b/.apb/playbooks/osb-feature-release/1.4.0/playbook.yaml
@@ -0,0 +1,656 @@
+schema: 2
+id: osb-feature-release
+name: Open Second Brain feature release wave
+description: 'Run one full release wave for Open Second Brain: scan the kanban triage column, propose
+ argued scope options to the operator, then create an isolated git worktree for the chosen scope (sibling
+ directory `../<repo>-worktrees/<slug>` on branch `feat/<slug>`, published as the run working tree so
+ every later node and both sub-playbooks run there and the project root checkout is never switched or
+ branched), then drive the feature-release-playbook in order - reconnaissance and design (phases 0-1,
+ code search with zg), TDD implementation (phase 2), PR preparation as the osb-pr-prepare sub-playbook
+ (phases 3-7: self-review, test and security audit of the diff, docs and the version bump, QA gates,
+ the code-ranker gate against the main baseline, an OpenCodeReview delegation review, the gates again,
+ then the PR with a verified mermaid, CI and bot review; never merges), the tracker tick that moves the
+ shipped cards to done as soon as the PR is green (the only node that writes to the board), the operator
+ merge gate, and the GitHub release as the osb-release sub-playbook (phases 8-9, with its own approval
+ gate before publishing). The operator brief rides as a default parameter any run may override.'
+version: 1.4.0
+params:
+- name: brief
+ type: text
+ label: Standing engineering brief (override to change the rules of the cycle)
+ options: null
+ default: 'Work as a senior backend developer: SOLID, KISS, DRY, repeated values hoisted into named constants.
+ Forbidden: fallbacks that silently do nothing and mislead (errors surface explicitly, by name), stubs,
+ hardcoded values that belong in constants, and hardcoded natural-language phrases in any language
+ - everything is English and other-language cases are handled abstractly. Follow the feature-release-playbook
+ skill through ALL its phases in order; merging or skipping phases is forbidden. Phases 0-2 run in
+ this playbook; phases 3-7 (self-review against the base, QA, docs and the version bump inside the
+ PR per CLAUDE.md, the PR with a verified mermaid, CI and review handling) run in the osb-pr-prepare
+ sub-playbook; phases 8-9 (the GitHub release) run in the osb-release sub-playbook; phase 10 (the tracker
+ tick) runs here as soon as the PR is green, before the merge gate and the release. Every node after
+ the workspace node works in the run working tree (the worktree the workspace node created); never
+ create, switch or commit branches in the project root checkout. Code search uses zg (`zg query --mode
+ direct --refresh wait "<question>"` for semantic hits, `--fts <Symbol>` for exact symbols, `zg query
+ --rg -n -w <symbol> src tests` for exhaustive callers); no code-graph tool is used. Never move the
+ selected triage cards before the PR is green (moving them early triggers the hermes agents); once
+ osb-pr-prepare reports the PR with green CI, move them to done with handoff summaries that point at
+ the PR. Reply to the operator in the operator''s chat language; write every artifact in English. A
+ node attempt is one uninterrupted job: never end your reply while a command you started is still running
+ - wait for it and finish the whole task; a partial progress report is not a node result.'
+- name: cycle_focus
+ type: text
+ label: What this cycle should optimize for (override per run)
+ options: null
+ default: Pick one or several RELATED triage cards by priority, reading the validator comments on each.
+ Propose two or three options and argue them properly; usefulness to the project is the supreme criterion.
+ A pull request around 50-70 changed files with at most 10 features is the target shape, and hitting
+ that target is the weakest signal of all - never trade usefulness for it.
+- name: board_access
+ type: text
+ label: How this machine reaches the board the kanban script talks to (operator setup, shown to the agent)
+ default: The script reaches the board endpoint configured in it. A connection error means that endpoint
+ is unreachable from this machine, not that the board is empty.
+worktree: '{{nodes.workspace.output.working_tree}}'
+defaults:
+ profile:
+ name: osb-developer
+ scope: auto
+ max_retries: null
+ timeout_seconds: null
+supervisor: null
+trigger:
+ when:
+ - start a new open-second-brain release cycle from the kanban board
+ - implement a batch of related triage tasks end to end and release them
+ - run the feature-release-playbook as a full wave with an operator scope gate
+ avoid_when:
+ - validating or triaging cards without implementing (use osb-triage-validator)
+ - a single trivial fix that does not warrant a release wave
+ examples:
+ - начинаем новый цикл
+ - take a big scope of related low-priority tasks and ship a release
+requires:
+ files:
+ - .ai-notes/kanban.ts
+ - CLAUDE.md
+ commands:
+ - bun
+ - git
+ - gh
+ - zg
+ - code-ranker
+ - python3
+ - curl
+goal:
+ statement: A scope the operator chose is implemented, its PR is green and every shipped card on the
+ kanban board reads done with a handoff summary pointing at that PR, then the PR is merged into main
+ and released on GitHub with an interaction-diagram image.
+ criteria:
+ - description: Every phase of the feature-release-playbook ran in order, none merged and none skipped;
+ phases 3-7 ran as osb-pr-prepare (self-review of the full branch against main included) and phases
+ 8-9 as osb-release.
+ check:
+ type: manual
+ - description: The version was bumped inside the feature PR, the shipped cards were moved to done as
+ soon as the PR was green (refused or reshaped cards carry their verdicts instead), and the release
+ is published with a rendered interaction diagram.
+ check:
+ type: manual
+effects:
+- fs_read
+- fs_write
+- network
+- external
+- irreversible
+nodes:
+- id: start
+ title: Start
+ expected_duration: null
+ inputs: null
+ outputs: null
+ cache: null
+ type: start
+- id: scope_scan
+ title: Scan the board and build scope options
+ expected_duration: 20m
+ inputs: null
+ outputs:
+ files: []
+ extract: node_output
+ cache: null
+ type: agent_task
+ prompt: "Standing brief: {{params.brief}}\nCycle focus: {{params.cycle_focus}}\n\nRead the board with\
+ \ the operator-local, credential-bearing kanban\nscript: run it, never print, copy or quote it, its\
+ \ headers or any\ntoken. Board access: {{params.board_access}}\n\nWork in a run-local temporary directory,\
+ \ never in the shared /tmp,\nand remove it before you reply, on success and on failure alike:\n\n\
+ \ k=.ai-notes/kanban.ts\n [ -f \"$k\" ] || k=\"$(git worktree list --porcelain | sed -n '1s/^worktree\
+ \ //p')/.ai-notes/kanban.ts\"\n mkdir -p .apb/tmp && d=$(mktemp -d \"$PWD/.apb/tmp/scope-scan.XXXXXX\"\
+ )\n bun \"$k\" triage --json >\"$d/triage.json\" 2>\"$d/err\"\n\n(.apb/tmp/ is git-ignored.) The\
+ \ first fallback path is the main\ncheckout of the repository, for runs started in a worktree that\n\
+ has no .ai-notes/ of its own. If the command fails with an\n`HTTP 5xx` line in \"$d/err\" (a transient\
+ \ server error: the script\nfails whole on one failed comment fetch), wait five seconds and\nrun it\
+ \ exactly once more. Fail this node, instead of proposing\noptions from memory, when:\n- it fails\
+ \ again with HTTP 5xx: \"board API returned <status> twice\";\n- it fails with a connection error\
+ \ (refused, timeout, DNS): \"board\n API unreachable\", plus the board access text above;\n- it fails\
+ \ any other way: the error line verbatim (never a token).\nClean up with `rm -rf \"$d\"` in every\
+ \ case.\n\nRead every card WITH its comments - validator comments carry\nverdicts and corrected scopes.\
+ \ Build TWO or THREE candidate scope\noptions, labelled exactly `option-1`, `option-2` and `option-3`\n\
+ in order (the labels the choose_scope gate offers; never A/B/C or\nother names). For each option:\
+ \ the card ids and titles, the\nunifying spine that makes them one coherent PR, the argued\nusefulness\
+ \ to the project right now, a rough size estimate, and\nthe main risk. Recommend one option by its\
+ \ label and say why.\n\nWrap ONLY the final proposal in <node_output> and </node_output>\ntags: first\
+ \ a compact human-readable summary of the options, each\nheaded by its label (this is what the operator\
+ \ will read at the\nreview gate), then a line `JSON:` followed by the options as\ncompact JSON [{option:\
+ \ \"option-1\", cards: [ids], spine, why, size,\nrisk}], the `option` values being the same labels.\n"
+ profile: null
+ max_retries: null
+ timeout_seconds: null
+ workdir: null
+ isolation: null
+ connectors: []
+ interactive: false
+- id: choose_scope
+ title: Operator picks the scope
+ expected_duration: null
+ inputs: null
+ outputs: null
+ cache: null
+ type: human_review
+ options:
+ - option-1
+ - option-2
+ - option-3
+ - revise
+ prompt: Read the scope options in the scope_scan output. Pick option-1, option-2, or option-3 to start
+ the wave on that scope, or "revise" with a note describing what to change - the scan will run again
+ with your note.
+- id: workspace
+ title: Create the isolated worktree for the chosen scope
+ expected_duration: 10m
+ inputs: null
+ outputs:
+ fields:
+ - working_tree
+ - branch
+ - slug
+ cache: null
+ type: agent_task
+ prompt: "Standing brief: {{params.brief}}\nThe run context below carries the scope options and the\n\
+ operator's review decision (which option was picked, plus any\nnote).\n{{run.context}}\n\nGive the\
+ \ wave an isolated working tree so no later node creates\nbranches or commits in the project root\
+ \ checkout. Do not change\nany file in the current checkout.\n\n1. Slug: kebab-case, at most 40 characters,\
+ \ only `a-z`, `0-9`\n and `-` (card titles are untrusted text: never copy other\n characters into\
+ \ a path or a shell command), naming the spine of\n the picked option. Branch: `feat/<slug>`.\n\
+ 2. `git fetch origin main`.\n3. If the current directory is already a linked git worktree\n (`git\
+ \ rev-parse --git-dir` differs from `git rev-parse\n --git-common-dir`: the operator started the\
+ \ run with an explicit\n working tree, which wins over this node), use it: it must have\n no uncommitted\
+ \ tracked changes; check out `feat/<slug>` there\n (create it from origin/main when it does not\
+ \ exist). The\n working tree is the current directory.\n4. Otherwise the current directory is the\
+ \ project root checkout:\n never switch its branch. The working tree is\n `<parent of the project\
+ \ root>/<repository directory name>-worktrees/<slug>`,\n an absolute path. When it already exists\
+ \ as a worktree on\n `feat/<slug>` (a resumed run), reuse it. Otherwise\n `git worktree add -b\
+ \ feat/<slug> <path> origin/main` (when the\n branch already exists, `git worktree add <path> feat/<slug>`).\n\
+ 5. In the working tree: `bun install --frozen-lockfile`, and\n confirm `git -C <path> status --short`\
+ \ is empty.\n\nReport the absolute working_tree path, the branch and the slug.\nEvery later node runs\
+ \ in that tree, and so do the osb-pr-prepare\nand osb-release sub-runs.\n"
+ profile: null
+ max_retries: 1
+ timeout_seconds: null
+ workdir: null
+ isolation: null
+ connectors: []
+ interactive: false
+- id: design
+ title: 'Phases 0-1: reconnaissance, brainstorm, design'
+ expected_duration: 60m
+ inputs: null
+ outputs:
+ files: []
+ extract: node_output
+ cache: null
+ type: agent_task
+ prompt: 'Standing brief: {{params.brief}}
+
+ The run context below carries the scope options and the
+
+ operator''s review decision (which option was picked, plus any
+
+ note). Honor the picked option exactly.
+
+
+ FIRST, before any other work: the operator''s external reviewers
+
+ need to know which cards this wave covers. Write the picked
+
+ option''s card ids to `.reviews/current-tasks.md`, one id per
+
+ line: truncate the file first without reading any of its previous
+
+ content, create the `.reviews/` directory when it is missing, and
+
+ write the file in BOTH places - the run working tree and the run
+
+ root (the checkout that holds `.apb/`: `$APB_RUN_DIR` is
+
+ `<checkout>/.apb/runs/<run-id>`, so the root is `$APB_RUN_DIR/../../..`,
+
+ the parent of `.apb/`). This is scratch state for reviewers, never
+
+ a tracked file: do not commit it.
+
+ {{run.context}}
+
+
+ Invoke the feature-release-playbook skill and run its phases 0
+
+ and 1 for the chosen option, with one strengthening this project
+
+ has proven necessary: BEFORE the consultant brainstorm, verify
+
+ every selected card''s premise against the live source (open the
+
+ anchors, search with zg for mechanisms the card claims are missing, check
+
+ the kernel invariants). Past waves found a third to two thirds
+
+ of premises corrected or refuted by this reading. A refuted card
+
+ gets its verdict recorded in the design doc rather than being
+
+ built.
+
+
+ Work in the run working tree
+
+ ({{nodes.workspace.output.working_tree}}, the current directory)
+
+ on branch {{nodes.workspace.output.branch}}, which the workspace
+
+ node created; never create or switch branches yourself. Premise
+
+ checks and reconnaissance use zg: build or refresh the local
+
+ index once with `sh .apb/playbooks/osb-pr-prepare/$(cat
+
+ .apb/playbooks/osb-pr-prepare/current)/scripts/zg-index.sh`, then
+
+ `zg query --mode direct --refresh wait "<mechanism>"` for
+
+ semantic hits and `zg query --rg -n -w <symbol> src tests` for
+
+ exhaustive callers. No code-graph tool is used.
+
+
+ Deliverables committed on that branch, slug
+
+ {{nodes.workspace.output.slug}}: the
+
+ docs/brainstorm/<slug>/ set (design.md, plan.md, variants.md,
+
+ cli-output/) per the playbook''s own templates, with the plan
+
+ broken into TDD-ready tasks.
+
+
+ Wrap ONLY a completion summary in <node_output> and
+
+ </node_output> tags: branch name, slug, task list from plan.md,
+
+ any premise corrections or refusals, commit sha.
+
+ '
+ profile: null
+ max_retries: null
+ timeout_seconds: null
+ workdir: null
+ isolation: null
+ connectors: []
+ interactive: false
+- id: implement
+ title: 'Phase 2: TDD implementation'
+ expected_duration: 4h
+ inputs: null
+ outputs:
+ files: []
+ extract: node_output
+ cache: null
+ type: agent_task
+ prompt: 'Standing brief: {{params.brief}}
+
+ Design summary: {{nodes.design.output}}
+
+
+ Run phase 2 of the feature-release-playbook in the run working
+
+ tree ({{nodes.workspace.output.working_tree}}) on branch
+
+ {{nodes.workspace.output.branch}}; never create or switch
+
+ branches. Code search with zg as in the brief: implement plan.md task by task, strict
+
+ TDD - failing test first, run it and see the expected failure,
+
+ implement to green, refactor green, one conventional commit per
+
+ atomic unit, formatted and linted before every commit. Test runs
+
+ use a throwaway HOME (`env HOME=$(mktemp -d) bun test <paths>`).
+
+ Improve defects you find in code you touch and name them in the
+
+ commit bodies. Stage files by name, never `git add -A` or
+
+ `git add .`: ignored runtime state (.apb/runs/, .zvec-grep/) and
+
+ the committed .apb/ playbooks and profiles are not part of any
+
+ feature.
+
+
+ Wrap ONLY a completion summary in <node_output> and
+
+ </node_output> tags: commits (sha + title), per-task status,
+
+ failing-first evidence snippets, suites run with counts.
+
+ '
+ profile: null
+ max_retries: null
+ timeout_seconds: null
+ workdir: null
+ isolation: null
+ connectors: []
+ interactive: false
+- id: pr_prepare
+ title: 'Phases 3-7: prepare the PR (osb-pr-prepare)'
+ type: playbook
+ playbook: osb-pr-prepare
+ expected_duration: 6h30m
+ instruction: 'Kind: feature, or fix when every change in the design summary repairs
+
+ existing behavior without adding capability. Base: main. The PR ships a
+
+ release (the kind decides the bump). The implementation is committed on
+
+ branch {{nodes.workspace.output.branch}} in the run working tree
+
+ {{nodes.workspace.output.working_tree}}, which this sub-run works in.
+
+ Kanban cards in scope are context only: do not move, comment on or
+
+ mention them in public text. GitHub issues, if the design names any,
+
+ are referenced in the PR.
+
+ Operator scope decision and note: {{nodes.choose_scope.review_note}}
+
+ Design summary: {{nodes.design.output}}
+
+ Implementation summary: {{nodes.implement.output}}
+
+ '
+- id: pr_blocked
+ title: Operator decides on a blocked PR preparation
+ type: human_review
+ options:
+ - retry
+ - abandon
+ prompt: The osb-pr-prepare sub-run ended without a PR with green CI (its run answer names the node,
+ the reason and the next step). Fix what it names if needed, then pick "retry" to run the PR preparation
+ again on the same branch (it updates the same PR), or "abandon" to stop the wave without merging or
+ releasing.
+- id: tracker
+ title: 'Phase 10: tracker tick at the green PR'
+ expected_duration: 20m
+ inputs: null
+ outputs:
+ files: []
+ extract: node_output
+ cache: null
+ type: agent_task
+ prompt: 'Standing brief: {{params.brief}}
+
+ PR preparation answer (first line `PR: #<n> <url>`, then the CI
+
+ state): {{nodes.pr_prepare.output}}
+
+ Design summary (for refusals): {{nodes.design.output}}
+
+
+ Run phase 10 now, as soon as the PR is green: after osb-pr-prepare
+
+ and before the merge gate and the release. This is the only node
+
+ that writes to the board. The kanban script is git-ignored and
+
+ lives in the main checkout, not in the run working tree:
+
+ `k="$(git worktree list --porcelain | sed -n ''1s/^worktree //p'')/.ai-notes/kanban.ts"`.
+
+ Never print, copy or quote it or any token. Move every shipped card
+
+ to done via `bun "$k" done <id> "<handoff summary pointing at the
+
+ PR URL from the first line above and its green CI state>"`. Cards
+
+ whose premise was refused or reshaped get a verdict comment and an
+
+ appropriate parked priority instead of a dishonest done. Nothing in
+
+ the scope is left open without either a done or a recorded verdict.
+
+ Re-read the board after writing and report the actual status of
+
+ every card in scope.
+
+
+ Wrap ONLY the tracker summary in <node_output> and </node_output>
+
+ tags: per-card disposition as read back from the board, the PR
+
+ number and URL, its CI state, the version the PR ships, and the run
+
+ working tree ({{nodes.workspace.output.working_tree}}) with the
+
+ command that removes it once the wave has ended and the operator no
+
+ longer needs it (`git worktree remove <path>`); do not remove it
+
+ yourself: the release still runs there.
+
+ '
+ profile: null
+ max_retries: null
+ timeout_seconds: null
+ workdir: null
+ isolation: null
+ connectors: []
+ interactive: false
+- id: merge_gate
+ title: Merge the green PR (authorized since 2026-10-06)
+ type: agent_task
+ prompt: 'Merge the PR this wave prepared. The operator authorized agent merges on 2026-10-06 ("merging without approval is allowed; the rights are in place now"): no human gate here.
+
+
+ Read the PR number and URL from the first line of the PR preparation answer in the run context (PR:
+ #N <url>). First run `gh pr view <n> --json state,mergeCommit`: if the state is already MERGED (the
+ operator merged it themselves), accept it: report the merge commit and succeed without merging again,
+ skipping every step below.
+
+
+ Otherwise verify with gh before merging, in this order:
+
+ 1. `gh pr view <n> --json state,headRefOid,baseRefName,mergeable` - the PR is OPEN and targets the
+ base branch.
+
+ 2. `gh pr checks <n>` - every check is green; if any check is pending, WAIT with one blocking re-check
+ after a few minutes (never a polling loop). Fail the node when a check is red, and also when a check
+ is still pending after that one re-check (name the pending checks); never merge on a pending check.
+
+ 3. The PR head equals the local branch HEAD in the run working tree (`git rev-parse HEAD`).
+
+ 4. Auto-merge is not armed (`gh pr view <n> --json autoMergeRequest`).
+
+
+ Merge with squash and admin (the repository requires it for feature branches):
+
+ `body=$(mktemp "${TMPDIR:-/tmp}/osb-pr-body.XXXXXX")`, then `gh pr view <n> --json body -q .body >
+ "$body"`, then
+
+ `gh pr merge <n> --squash --admin --match-head-commit <head> --subject "<PR title> (#<n>)" --body-file
+ "$body" && rm -f "$body"` in the same shell call (a private file per run, so parallel waves never
+ share it, and the file is removed after the merge).
+
+ The subject is the PR title exactly as GitHub shows it with the (#N) suffix; the body is the PR description,
+ so no commit spam reaches main.
+
+
+ Read back `gh pr view <n> --json state,mergeCommit`: it must be MERGED; report the squash sha in your
+ reply. On any refusal (red checks, merge conflict, rights), fail the node with the verbatim error:
+ the wave ends without a release and the done cards stay done.'
+ expected_duration: 10m
+ effects:
+ - irreversible
+- id: release
+ title: 'Phases 8-9: publish the release (osb-release)'
+ type: playbook
+ playbook: osb-release
+ expected_duration: 1h15m
+ instruction: 'Target: the squash-merge commit of the PR named in the first line of
+
+ the PR preparation answer below (`gh pr view <n> --json
+
+ state,mergeCommit`; it must be MERGED); fail if it is not merged.
+
+ The release covers everything merged since the last published
+
+ release. Kanban cards are not yours to touch.
+
+ Merge node report: {{nodes.merge_gate.output}}
+
+ PR preparation answer: {{nodes.pr_prepare.output}}
+
+ Design summary (for the release narrative): {{nodes.design.output}}
+
+ '
+- id: done
+ title: null
+ expected_duration: null
+ inputs: null
+ outputs: null
+ cache: null
+ type: finish
+ outcome: success
+ prompt: null
+ profile: null
+- id: abandoned
+ title: null
+ expected_duration: null
+ inputs: null
+ outputs: null
+ cache: null
+ type: finish
+ outcome: failure
+ prompt: null
+ profile: null
+- id: release_failed
+ title: null
+ expected_duration: 2m
+ type: finish
+ outcome: failure
+ prompt: 'The osb-release sub-run did not publish a release (refused, aborted at
+
+ its gate, or failed). Say so plainly and name the PR that merged
+
+ ({{nodes.pr_prepare.output}}). The shipped cards were already moved to
+
+ done when that PR went green (the tracker output has each card''s
+
+ disposition); they stay there, and the release is still owed.
+
+ '
+ profile: null
+edges:
+- from: start
+ to: scope_scan
+- from: scope_scan
+ to: choose_scope
+- from: choose_scope
+ to: workspace
+ condition:
+ type: review_status
+ equals: option-1
+- from: choose_scope
+ to: workspace
+ condition:
+ type: review_status
+ equals: option-2
+- from: choose_scope
+ to: workspace
+ condition:
+ type: review_status
+ equals: option-3
+- from: workspace
+ to: design
+- from: choose_scope
+ to: scope_scan
+ condition:
+ type: review_status
+ equals: revise
+ max_traversals: 3
+- from: choose_scope
+ to: abandoned
+ fallback: true
+- from: design
+ to: implement
+- from: implement
+ to: pr_prepare
+- from: pr_prepare
+ to: tracker
+ condition:
+ type: node_status
+ node: pr_prepare
+ equals: success
+- from: pr_prepare
+ to: pr_blocked
+ condition:
+ type: node_status
+ node: pr_prepare
+ equals: failure
+- from: pr_blocked
+ to: pr_prepare
+ condition:
+ type: review_status
+ equals: retry
+ max_traversals: 2
+- from: pr_blocked
+ to: abandoned
+ condition:
+ type: review_status
+ equals: abandon
+- from: pr_blocked
+ to: abandoned
+ fallback: true
+- from: merge_gate
+ to: release
+ condition:
+ type: node_status
+ node: merge_gate
+ equals: success
+- from: merge_gate
+ to: abandoned
+ fallback: true
+- from: release
+ to: done
+ condition:
+ type: node_status
+ node: release
+ equals: success
+- from: release
+ to: release_failed
+ condition:
+ type: node_status
+ node: release
+ equals: failure
+- from: tracker
+ to: merge_gate
diff --git a/.apb/playbooks/osb-feature-release/current b/.apb/playbooks/osb-feature-release/current
index f0bb29e7..88c5fb89 100644
--- a/.apb/playbooks/osb-feature-release/current
+++ b/.apb/playbooks/osb-feature-release/current
@@ -1 +1 @@
-1.3.0
+1.4.0
diff --git a/.apb/playbooks/osb-pr-prepare/1.2.0/playbook.yaml b/.apb/playbooks/osb-pr-prepare/1.2.0/playbook.yaml
new file mode 100644
index 00000000..fd43be0a
--- /dev/null
+++ b/.apb/playbooks/osb-pr-prepare/1.2.0/playbook.yaml
@@ -0,0 +1,864 @@
+schema: 2
+id: osb-pr-prepare
+name: Open Second Brain PR prepare
+description: 'Turn an Open Second Brain branch whose implementation is done into an open pull request
+ that is ready for the operator to merge. Works for every change kind (feature, fix, refactor, chore,
+ docs, deps). Steps: intake and commit of any uncommitted work, a local zg index, a self-review against
+ the base with zg reconnaissance, a test-audit of the tests the diff adds or changes, a focused security
+ review of the diff, docs plus the version and CHANGELOG decision the change kind drives (CLAUDE.md),
+ then before the push, in this order and each with its fix loop: the CI QA gates run locally exactly
+ as CI runs them, the code-ranker architecture gate against the main baseline (zero new violations, else
+ blocked), an OpenCodeReview review of the branch diff in delegation mode on a fixed model (profile osb-reviewer)
+ that fixes the confirmed findings, and the QA and architecture gates again when that review changed
+ anything; then push, open the PR with a verified mermaid diagram, one blocking CI wait that ends only
+ when every check has finished, and a bounded bot-review loop. It never merges and never arms auto-merge.
+ The run input (instruction) carries the change kind, the intent, the related issues or cards and the
+ base branch. The finish answer reports the PR URL, the CI state, a summary and whatever is left for
+ a human.'
+version: 1.2.0
+params:
+- name: brief
+ type: text
+ label: Standing brief for PR preparation (override to change the rules)
+ default: 'Open Second Brain is a PUBLIC repository. Every artifact (code, commits, PR text, CHANGELOG,
+ comments) is English, uses the full name "Open Second Brain" in prose, and carries no exclamation
+ marks. Engineering bar: SOLID, KISS, DRY, repeated literals hoisted into named constants, no fallbacks
+ that silently do nothing, no stubs, errors surface explicitly and by name. Commits: conventional (`type(scope):
+ imperative summary`), one atomic unit each, formatted and linted before every commit (`bun run fmt`,
+ `bun run lint`; the repo pre-commit hook enforces fmt, mirrors and lint), files staged by name - never
+ `git add -A` or `git add .` (the ignored runtime state in .apb/runs/, .apb/tmp/ and .zvec-grep/ is
+ never part of a change, and the committed .apb/ playbooks and profiles change only when the change
+ is about them). When the commit_trailer parameter is non-empty, every commit message ends with a blank
+ line and that trailer; when it is empty (the default), commit messages carry no trailer and no AI-authorship
+ marker. Tests run with a throwaway HOME: `env HOME=$(mktemp -d) bun test <paths>`. Code search: zg
+ (`zg query --mode direct --refresh wait "<question>"` for semantic and hybrid hits, `--fts <Symbol>`
+ for exact symbols, `zg query --rg -n -w <symbol> src tests` for exhaustive callers); CodeGraph is
+ not used. Never merge a PR, never arm auto-merge, never push to the base branch, never force-push
+ over commits you did not write. Never touch the kanban board: card ids are context, not something
+ to move or comment. Security details never go into public text; see the security_review node. Long
+ commands: wait with ONE blocking call, never a polling loop; never end a reply while a command you
+ started is still running, and never end with a progress report - the reply is the node result.'
+- name: kind
+ type: enum
+ label: Change kind (auto reads it from the instruction, then the branch prefix)
+ options:
+ - auto
+ - feature
+ - fix
+ - refactor
+ - chore
+ - docs
+ - deps
+ default: auto
+- name: base
+ type: text
+ label: Base branch (an explicit base in the instruction wins)
+ default: main
+- name: release
+ type: enum
+ label: Version bump (auto derives it from the kind, and for deps from the runtime dependencies; see
+ the intake node)
+ options:
+ - auto
+ - major
+ - minor
+ - patch
+ - none
+ default: auto
+- name: commit_trailer
+ type: text
+ label: Trailer that ends every commit message (empty for none)
+ default: ''
+- name: pr_footer
+ type: text
+ label: Last line of every PR description (empty for none)
+ default: ''
+defaults:
+ profile: osb-developer
+ on_failure: blocked
+trigger:
+ when:
+ - prepare a pull request for an open-second-brain branch whose implementation is done
+ - review, gate, version and open a PR for an open-second-brain change of any kind
+ - get an open-second-brain branch to an open PR with green CI and addressed bot review
+ avoid_when:
+ - the change is not implemented yet (implement it first, or use osb-feature-release)
+ - merging or releasing (the operator merges; osb-release publishes)
+ examples:
+ - prepare the PR for this branch, it is a fix for issue 190
+ - open a docs PR from the current branch
+requires:
+ files:
+ - CLAUDE.md
+ commands:
+ - zg
+ - code-ranker
+ - bun
+ - git
+ - gh
+ - ocr
+goal:
+ statement: The branch is open as a pull request against the base, with green CI, bot review findings
+ addressed or answered, the version and CHANGELOG handled as CLAUDE.md prescribes for the change kind,
+ and nothing merged.
+ criteria:
+ - description: The PR is open, not merged, auto-merge is not armed, and its head is the reviewed and
+ gated commit.
+ check:
+ type: manual
+ - description: Self-review, the diff-scoped test-audit and security review, the local CI gates, the
+ architecture gate with zero new violations against the main baseline and the OpenCodeReview delegation
+ review ran before the push, with the gates re-run after any review fix; CI is green on the final
+ head, or the finish answer names exactly what is left for a human.
+ check:
+ type: manual
+ - description: A release-bearing kind carries its version bump, CHANGELOG entry with link reference
+ and a mermaid diagram that renders in the PR; a no-release kind carries none of the version changes.
+ check:
+ type: manual
+effects:
+- fs_read
+- fs_write
+- network
+- external
+nodes:
+- id: start
+ type: start
+- id: intake
+ title: Resolve the PR context and commit pending work
+ type: agent_task
+ expected_duration: 15m
+ prompt: "Standing brief: {{params.brief}}\nCommit trailer: {{params.commit_trailer}}\nParameters: kind={{params.kind}},\
+ \ base={{params.base}}, release={{params.release}}\nRun input: {{run.instruction}}\n\nYou prepare\
+ \ the context every later node relies on. Work in the\nrepository root; do not change code beyond\
+ \ committing work that is\nalready there.\n\n1. Base: an explicit base branch in the run input wins,\
+ \ else the\n base parameter. `git fetch origin <base>`.\n2. Kind (feature|fix|refactor|chore|docs|deps):\
+ \ the run input wins,\n then a kind parameter other than auto, then the branch prefix\n (feat/\
+ \ fix/ refactor/ chore/ docs/ deps/ or chore/deps). If none\n of these decides it, fail naming the\
+ \ ambiguity; do not guess.\n3. Release bump: a release parameter other than auto wins (the run\n \
+ \ input may state one too). Otherwise the kind decides, following\n CLAUDE.md (\"for any feature/fix\
+ \ PR that ships a release\") and the\n history of main: feature -> minor, fix -> patch, refactor,\
+ \ chore,\n docs -> none (no version bump and no CHANGELOG entry, like #175,\n #142 and #121 on\
+ \ main). deps -> none when only development\n dependencies or tooling move (package.json devDependencies,\n\
+ \ bun.lock entries only they pull in, CI actions); deps -> patch\n when the diff adds, removes\
+ \ or changes a RUNTIME dependency\n (package.json dependencies, optionalDependencies or\n peerDependencies,\
+ \ or pyproject.toml project dependencies),\n because that changes what users install. Read the diff\
+ \ of those\n files to decide; say which rule applied in the intent.\n4. Branch: if HEAD is on the\
+ \ base branch with uncommitted work, create\n `<prefix>/<slug>` (prefix from the kind: feat, fix,\
+ \ refactor,\n chore, docs, deps) and move the work there. A detached HEAD, or\n a branch with\
+ \ no change against origin/<base> and nothing\n uncommitted, is a failure with that reason.\n5.\
+ \ Uncommitted implementation work: read it, then commit it as\n atomic conventional commits per\
+ \ the brief (formatted and linted,\n staged by name). Leave the ignored runtime state (.apb/runs/,\
+ \ .zvec-grep/) alone. If a\n change looks unrelated to the stated intent, leave it\n uncommitted\
+ \ and fail, naming the files.\n6. Record the context for the script nodes:\n `d=$(git rev-parse\
+ \ --path-format=absolute --git-path osb-pr-prepare); mkdir -p \"$d\"`\n and write \"$d/context.env\"\
+ \ with exactly these shell assignments:\n OSB_BASE=<base>, OSB_KIND=<kind>, OSB_RELEASE=<major|minor|patch|none>,\n\
+ \ OSB_BRANCH=<branch>. Remove a stale \"$d/pr.env\" only if its PR is\n not open for this branch\
+ \ (`gh pr view <n> --json state,headRefName`);\n keep it when it is, so a re-run updates the same\
+ \ PR.\n\nWrite your verdict to $APB_STATUS_FILE with outputs\n{kind, release, base, branch, intent\
+ \ (one paragraph, English),\nissues (GitHub issue numbers the change addresses, or []),\ncards (kanban\
+ \ card ids from the run input, or []),\ncommits_added (list of \"sha title\")}.\n"
+ success_check: scripts/check-context.sh
+ max_retries: 1
+- id: zg_index
+ title: Ensure the local zg index
+ type: script
+ script: scripts/zg-index.sh
+ runner: sh
+ timeout_seconds: 1800
+ expected_duration: 5m
+- id: self_review
+ title: Self-review against the base (zg reconnaissance)
+ type: agent_task
+ expected_duration: 75m
+ prompt: 'Standing brief: {{params.brief}}
+
+ Commit trailer: {{params.commit_trailer}}
+
+ Context: {{nodes.intake.output}}
+
+ Index step: {{nodes.zg_index.output}}
+
+
+ Review EVERYTHING on the branch against the merge base with
+
+ origin/<base>, committed work included (`git diff $(git merge-base
+
+ origin/<base> HEAD)...HEAD`), as a fresh reviewer who did not write
+
+ it. Use the superpowers skills requesting-code-review and
+
+ verification-before-completion.
+
+
+ Reconnaissance and impact with zg, not CodeGraph. For every changed
+
+ exported symbol, config key, CLI verb or MCP tool: find its callers
+
+ and consumers (`zg query --rg -n -w <name> src tests plugins scripts
+
+ hooks skills`), related mechanisms (`zg query --mode direct --refresh
+
+ wait "<what the change does>"`), and sibling implementations that
+
+ should have changed alike (Python plugin mirror, Codex mirrors,
+
+ docs). If the index step failed, use `zg query --rg` only and say so.
+
+
+ Check correctness, error paths, the engineering bar in the brief,
+
+ tests that pin the behavior, and docs that describe it. Fix real
+
+ findings in dedicated conventional commits with the smallest diff;
+
+ a finding you reject is named with its reason. Do not bump the
+
+ version or touch CHANGELOG here.
+
+
+ Wrap ONLY the result in <node_output> and </node_output>: findings
+
+ by severity with disposition (fixed in <sha> | rejected: reason),
+
+ the zg queries that decided the impact analysis, remaining known
+
+ issues (none, or argued).
+
+ '
+ outputs:
+ extract: node_output
+- id: test_audit
+ title: Test-audit of the tests this diff adds or changes
+ type: agent_task
+ expected_duration: 40m
+ prompt: 'Standing brief: {{params.brief}}
+
+ Commit trailer: {{params.commit_trailer}}
+
+ Context: {{nodes.intake.output}}
+
+
+ Apply the test-audit skill (load it by name),
+
+ authoring-gate mode, to exactly the test files this branch adds or
+
+ changes: `git diff --name-only --diff-filter=AMR $(git merge-base
+
+ origin/<base> HEAD) HEAD -- tests ''*.test.ts'' ''*_test.py''
+
+ ''test_*.py''`. Tests the diff does not touch are out of scope: no
+
+ sweep, no campaign, no deletion outside the diff.
+
+
+ For each added or changed test: the four gate questions, the junk
+
+ patterns, and the retention bar, plus one gate question of this
+
+ project: does the test compare output that embeds a timestamp or a
+
+ generated id (for example a second-granular run id), so that two
+
+ runs differ by timing alone? Such a comparison is normalised (the
+
+ volatile part masked or pinned) or the test is rewritten. Rewrite or
+
+ remove a test that fails the gate, in the same branch.
+
+
+ Regression tests (kind fix, or any test that pins a bug): prove each
+
+ one fails on the pre-fix code for the intended reason. Take the
+
+ merge base in a throwaway worktree (`git worktree add --detach
+
+ "$(mktemp -d)" <merge-base>`), copy in only the new or changed test
+
+ file, run it there (`env HOME=$(mktemp -d) bun test <file>`),
+
+ record the failure line, then remove the worktree. A regression
+
+ test that passes on the pre-fix code is rewritten until it fails
+
+ there.
+
+
+ The skill''s Validation and Landing sections name OpenClaw tooling
+
+ (vitest, crabbox, scripts/pr); here use the Open Second Brain commands from the
+
+ brief. Commit fixes as `test(<scope>): ...`.
+
+
+ Wrap ONLY the result in <node_output> and </node_output>: per test
+
+ file the verdict (kept | rewritten | removed) with the reason, and
+
+ for each regression test the pre-fix failure line.
+
+ '
+ outputs:
+ extract: node_output
+- id: security_review
+ title: Focused security review of the diff
+ type: agent_task
+ expected_duration: 40m
+ prompt: "Standing brief: {{params.brief}}\nCommit trailer: {{params.commit_trailer}}\nContext: {{nodes.intake.output}}\n\
+ \nApply the security-audit skill (load it by name)\nin GUIDANCE mode as a focused review. Never the\
+ \ full audit workflow:\nno six phases, no output directory, no hunter or verifier fan-out,\nno report\
+ \ artifacts. Scope: the diff against the merge base with\norigin/<base> and the trust boundaries it\
+ \ touches (read callers with\n`zg query --rg` where a changed function is reached from an MCP\ntool,\
+ \ CLI verb, hook, HTTP transport, file import or config read).\nFollow its core principles: a finding\
+ \ needs a lower-trust principal,\nthe crossed boundary and a concrete result; source-first; no live\n\
+ probing; run only the repository's own tests locally.\n\nDispositions:\n- A confirmed issue the branch\
+ \ introduces (never released): fix it\n in the branch with a regression test, as an ordinary\n conventional\
+ \ commit.\n- A confirmed issue in code already released on the base that the\n diff touches: fix\
+ \ it in THIS public PR, worded neutrally (below),\n AND open a private DRAFT security advisory (below)\
+ \ that carries\n the details. Both, always: the fix does not wait for the advisory,\n and the advisory\
+ \ is not skipped because the fix ships.\n- A confirmed issue in released code the diff does not touch:\
+ \ do\n not fix it here; open a private draft advisory only.\nPrivate advisory, created as a draft\
+ \ and never published by you:\n`gh api repos/itechmeat/open-second-brain/security-advisories -X POST\n\
+ -f summary=<short> -f description=<details and fix> -f severity=<low|medium|high|critical>`\nplus\
+ \ the required vulnerabilities field per the GitHub REST docs.\nThe advisory is the ONLY place the\
+ \ exploit details, the attack\nnarrative and the severity are written.\n\nThe repository is PUBLIC.\
+ \ Public artifacts - commit messages, code\ncomments, test names, the PR title and body, PR comments\
+ \ and\nreplies, CHANGELOG, docs and release notes - never carry exploit\ndetails: they describe a\
+ \ security fix only in neutral behavioral\nterms (\"validate X before Y\", \"refuse paths outside\
+ \ the vault\"),\nwith no exploit, no attack narrative, no severity, no advisory id\nand no words like\
+ \ vulnerability or exploit.\n\nWrap ONLY the result in <node_output> and </node_output>: confirmed\n\
+ findings with disposition (fixed in <sha>, advisory <GHSA id or\ndraft URL>), needs-validation items\
+ \ with the missing fact, and the\nboundaries reviewed. This output stays in the run; it is not public.\n"
+ outputs:
+ extract: node_output
+- id: arch_gate
+ title: Architecture gate (code-ranker, regressions only)
+ type: script
+ script: scripts/arch-gate.sh
+ runner: sh
+ timeout_seconds: 900
+ expected_duration: 2m
+- id: arch_check
+ title: Architecture verdict
+ type: condition
+ max_loops: 3
+- id: arch_fix
+ title: Fix the architecture regressions
+ type: agent_task
+ expected_duration: 30m
+ prompt: 'Standing brief: {{params.brief}}
+
+ Commit trailer: {{params.commit_trailer}}
+
+ Gate output: {{nodes.arch_gate.output}}
+
+
+ The code-ranker gate found violations this branch introduces against
+
+ the merge base (pre-existing debt is excluded by the baseline).
+
+ Before touching code, read `code-ranker docs ts <PRINCIPLE>` for each
+
+ finding (for example ADP for a cycle, HK for coupling), then apply
+
+ the smallest structural fix it prescribes, keeping behavior and
+
+ public APIs intact. Re-run the gate yourself with
+
+ `sh .apb/playbooks/osb-pr-prepare/$(cat .apb/playbooks/osb-pr-prepare/current)/scripts/arch-gate.sh`
+
+ (the script node runs it again after you).
+
+ Run the affected tests, commit as `refactor(<scope>): ...`.
+
+
+ Wrap ONLY the result in <node_output> and </node_output>: each
+
+ violation and the commit that removed it.
+
+ '
+ outputs:
+ extract: node_output
+- id: docs_version
+ title: Docs, version and CHANGELOG per the change kind
+ type: agent_task
+ expected_duration: 30m
+ prompt: "Standing brief: {{params.brief}}\nCommit trailer: {{params.commit_trailer}}\nContext: {{nodes.intake.output}}\n\
+ Review summary: {{nodes.self_review.output}}\n\nFollow CLAUDE.md \"Versioning\" exactly. The release\
+ \ bump in the\ncontext (OSB_RELEASE in <git-dir>/osb-pr-prepare/context.env) decides:\n\n- major|minor|patch\
+ \ (the PR ships a release):\n 1. CHANGELOG.md: a new `## [X.Y.Z] - <today, YYYY-MM-DD>` entry at\n\
+ \ the top in the house voice (read the last three entries first:\n an opening paragraph, then\
+ \ Fixed / Changed / Added with bold\n lead-ins, capability-first, true to the diff). An existing\n\
+ \ `## [Unreleased]` section becomes that entry rather than\n staying above it. Add the link\
+ \ reference\n `[X.Y.Z]: https://github.com/itechmeat/open-second-brain/compare/vPREV...vX.Y.Z`\n\
+ \ above the previous one. Security fixes are worded neutrally.\n 2. package.json `version` only,\
+ \ then `bun run scripts/sync-version.ts`\n and `bun run scripts/sync-version.ts --check`. Never\
+ \ hand-edit\n the mirrored manifests.\n- none: no version bump, no CHANGELOG entry.\n\nFor every\
+ \ kind: README \"What is new\" or the docs the change affects\n(install/, docs/, hooks/README.md,\
+ \ skills/) are updated when the\nchange alters what they describe; after changing skills/ or\nhooks/hooks.json\
+ \ run `bun run sync-plugin-mirrors`. Commit as\n`chore(release): vX.Y.Z` for the bump and `docs(<scope>):\
+ \ ...` for\ndocs.\n\nWrap ONLY the result in <node_output> and </node_output>: the bump\n(PREV ->\
+ \ X.Y.Z, or none and why), the CHANGELOG heading, the docs\ntouched, commit shas.\n"
+ success_check: scripts/check-release.sh
+ max_retries: 1
+ outputs:
+ extract: node_output
+- id: qa_gates
+ title: CI gates, run locally as CI runs them
+ type: script
+ script: scripts/qa-gates.sh
+ runner: sh
+ timeout_seconds: 3600
+ expected_duration: 25m
+- id: qa_check
+ title: QA verdict
+ type: condition
+ max_loops: 4
+- id: qa_fix
+ title: Fix the red QA gates
+ type: agent_task
+ expected_duration: 40m
+ prompt: 'Standing brief: {{params.brief}}
+
+ Commit trailer: {{params.commit_trailer}}
+
+ Gate output: {{nodes.qa_gates.output}}
+
+
+ Make every red gate green in the branch. Full logs are in
+
+ <git-dir>/osb-pr-prepare/qa-logs/<step>.log. The usual repairs:
+
+ `bun run fmt`, `bun run lint:fix`, `bun run sync-plugin-mirrors`,
+
+ `bun run scripts/sync-version.ts`, and for the openclaw-bundle gate
+
+ a rebuild with the CI Bun: `PATH=~/.cache/osb-pr-prepare/bun-1.4.0/bun-linux-x64:$PATH
+
+ bun run build:openclaw` (never with the local newer Bun). Fix
+
+ failing tests at their cause, not by weakening assertions.
+
+
+ A failing test that is environment-only (it fails on the merge base
+
+ too, in this environment, for a reason outside the branch, such as
+
+ the WSL loopback timeout in capability-tier) is not fixed here:
+
+ prove it by running that one test on the merge base in a throwaway
+
+ worktree, then append its exact `bun test` name (as printed after
+
+ "(fail) ", without the timing) to
+
+ <git-dir>/osb-pr-prepare/known-test-failures.txt and name it with
+
+ the evidence in your output. Never add a test that fails only on
+
+ the branch.
+
+
+ Commit fixes per the brief. Wrap ONLY the result in <node_output>
+
+ and </node_output>: each red gate, the cause, the fix commit or the
+
+ known-failure evidence.
+
+ '
+ outputs:
+ extract: node_output
+- id: ocr_review
+ title: OpenCodeReview of the branch diff (delegation mode)
+ type: agent_task
+ profile: osb-reviewer
+ expected_duration: 45m
+ prompt: "Standing brief: {{params.brief}}\nCommit trailer: {{params.commit_trailer}}\nContext: {{nodes.intake.output}}\n\
+ Self-review summary: {{nodes.self_review.output}}\n\nReview the whole branch diff against the merge\
+ \ base with\norigin/<base> with OpenCodeReview (`ocr`) in delegation mode: load\nthe open-code-review\
+ \ delegate-review skill by name\n(`open-code-review:delegate-review` as a plugin skill, or\n`open-code-review-delegate`)\
+ \ and follow its workflow. In delegation\nmode OCR calls no LLM: it selects the reviewable files and\
+ \ resolves\nthe review rules, and you perform the review. The review model is\ntherefore this node's\
+ \ executor, pinned by the osb-reviewer profile\n(claude-opus-5-5, no fallback executor); `ocr review`,\
+ \ `ocr scan`\nand any LLM endpoint or model configured for OCR itself are not\nused.\n\n1. `ocr delegate\
+ \ preview --format json --from origin/<base> --to HEAD`\n (the output names the merge base and the\
+ \ reviewable and excluded\n files).\n2. `ocr delegate rule --format json --from origin/<base> --to\
+ \ HEAD <reviewable paths>`.\n3. For each reviewable file, keyed by (path, status): its diff\n (`git\
+ \ diff <merge_base>..HEAD -- <path>`), its rule group, and\n the context you need (read callers\
+ \ with `zg query --rg -n -w\n <symbol> src tests plugins scripts hooks skills`). Mark it\n reviewed,\
+ \ or skipped with a concrete reason. Review large diffs\n in batches by rule group; do not stop\
+ \ at the first finding.\n4. Verify every finding against the current code with the\n receiving-code-review\
+ \ discipline. Confirmed: fix it with the\n smallest diff, add or adjust a test when behavior changes,\
+ \ run\n the affected tests (`env HOME=$(mktemp -d) bun test <paths>`),\n and commit per the brief\
+ \ (`fix(<scope>): ...` or\n `refactor(<scope>): ...`). Rejected or a likely false positive:\n \
+ \ one-sentence technical reason. A security-relevant finding\n follows the security_review dispositions\
+ \ and public-wording\n rules of this playbook.\n5. Do not push and do not touch the version or CHANGELOG:\
+ \ the QA\n and architecture gates run again after you when you committed.\n\nWrap ONLY the result\
+ \ in <node_output> and </node_output>. Required\nfirst line: `OCR-REVIEW: done (<reviewed>/<total>\
+ \ files reviewed,\n<confirmed> confirmed, <fixed> fixed)`. Then total_files,\nreviewed_files, skipped_files\
+ \ with reasons, coverage_rate, and the\nfindings by severity with disposition (fixed in <sha> | rejected:\n\
+ reason). A preview or rule command that fails is a node failure\nnaming the error, never a review\
+ \ without OCR.\n"
+ success_check:
+ marker: 'OCR-REVIEW: done'
+ max_retries: 1
+ outputs:
+ extract: node_output
+- id: final_gates
+ title: QA and architecture gates again after the review
+ type: script
+ script: scripts/final-gates.sh
+ runner: sh
+ timeout_seconds: 4500
+ expected_duration: 27m
+- id: final_check
+ title: Final gate verdict
+ type: condition
+ max_loops: 3
+- id: final_fix
+ title: Fix the red final gates
+ type: agent_task
+ expected_duration: 40m
+ prompt: 'Standing brief: {{params.brief}}
+
+ Commit trailer: {{params.commit_trailer}}
+
+ Gate output: {{nodes.final_gates.output}}
+
+ Review summary: {{nodes.ocr_review.output}}
+
+
+ The QA or the architecture gate turned red after the OpenCodeReview
+
+ pass. Make every red gate green in the branch without undoing a
+
+ confirmed review fix: for QA, the repairs and the known-failure rule
+
+ of the qa_fix node (full logs in <git-dir>/osb-pr-prepare/qa-logs/);
+
+ for architecture, the code-ranker rule of the arch_fix node (read
+
+ `code-ranker docs ts <PRINCIPLE>` first, smallest structural fix).
+
+ Commit per the brief. The script node runs both gates again after
+
+ you.
+
+
+ Wrap ONLY the result in <node_output> and </node_output>: each red
+
+ gate, the cause, and the fix commit.
+
+ '
+ outputs:
+ extract: node_output
+- id: push_pr
+ title: Push and open (or update) the PR
+ type: agent_task
+ expected_duration: 25m
+ prompt: "Standing brief: {{params.brief}}\nPR footer (the exact last line of the description): {{params.pr_footer}}\n\
+ Context: {{nodes.intake.output}}\nReview summary: {{nodes.self_review.output}}\nOpenCodeReview summary:\
+ \ {{nodes.ocr_review.output}}\nDocs and version: {{nodes.docs_version.output}}\n\n1. Push. If the\
+ \ diff against origin/<base> touches .github/workflows/,\n push over SSH so no token scope can refuse\
+ \ it:\n `git push [email protected]:itechmeat/open-second-brain.git HEAD:refs/heads/<branch>`\n then\
+ \ `git branch --set-upstream-to=origin/<branch>` after a\n `git fetch origin <branch>`; otherwise\
+ \ `git push -u origin <branch>`.\n Never push to the base, never force-push commits you did not\
+ \ write.\n2. PR text, in the voice of the recent PRs (`gh pr view 193` and\n `gh pr view 191` are\
+ \ the reference): title as a conventional\n summary, with ` (vX.Y.Z)` appended when the PR ships\
+ \ a release\n (`feat: <capability tagline> (v1.58.0)`, `fix(<scope>): ... (v1.57.2)`).\n Body:\
+ \ Summary (capability-first), a mermaid diagram of the\n process-level benefit (required when the\
+ \ PR ships a release,\n optional otherwise), What changed, Verification (the gates that\n ran,\
+ \ with counts; when the diff against origin/<base> exceeds\n 100 files, one line naming the bot\
+ \ review as skipped, since\n CodeRabbit skips a change set over its 100-file limit), and\n `Closes\
+ \ #N` / `Refs #N` for the GitHub issues in the context.\n Kanban card ids stay out of the public\
+ \ text. Security fixes in\n neutral terms only. When the PR footer above is non-empty, the\n description\
+ \ ends with it verbatim; when it is empty, the\n description carries no footer and no AI-authorship\
+ \ marker.\n3. Mermaid: try a local parse first when it runs here\n (`bunx -p @mermaid-js/mermaid-cli\
+ \ mmdc -i <file>.mmd -o <file>.svg`;\n agent-browser's Chrome can serve as PUPPETEER_EXECUTABLE_PATH).\n\
+ \ The check that counts is the PR page: once the PR exists, open it\n with agent-browser and confirm\
+ \ the diagram rendered as a diagram\n (no \"Unable to render rich display\" box). Fix and\n `gh\
+ \ pr edit --body-file` until it does.\n4. Open or update: if an open PR for this branch exists\n \
+ \ (`gh pr view --json number,state`), update its title and body;\n else `gh pr create --base <base>\
+ \ --head <branch> --title ...\n --body-file ...`. Never `gh pr merge`, never `--auto`.\n5. Record\
+ \ `OSB_PR_NUMBER=<n>` and `OSB_PR_URL=<url>` in\n <git-dir>/osb-pr-prepare/pr.env.\n\nWrap ONLY\
+ \ the result in <node_output> and </node_output>. Required\nfirst line: `PR: #<number> <url>`. Then\
+ \ the pushed head sha and the\nmermaid evidence (local render, and what the PR page showed).\n"
+ success_check: scripts/check-pr.sh
+ max_retries: 2
+ outputs:
+ extract: node_output
+- id: ci_wait
+ title: Wait for CI (one blocking watch)
+ type: script
+ script: scripts/ci-wait.sh
+ runner: sh
+ timeout_seconds: 6600
+ expected_duration: 35m
+- id: ci_wait_final
+ title: Wait for CI after the budget-capped bot review pass
+ type: script
+ script: scripts/ci-wait.sh
+ runner: sh
+ timeout_seconds: 6600
+ expected_duration: 35m
+- id: ci_check
+ title: CI verdict
+ type: condition
+ max_loops: 4
+- id: ci_fix
+ title: Fix the red CI checks and push
+ type: agent_task
+ expected_duration: 40m
+ prompt: 'Standing brief: {{params.brief}}
+
+ Commit trailer: {{params.commit_trailer}}
+
+ CI output: {{nodes.ci_wait.output}}
+
+
+ Diagnose each failed check from the job logs above (`gh run view
+
+ --job <id> --log` for more). Reproduce it locally when the platform
+
+ allows (the windows job cannot run here: reason from the log and
+
+ the Windows notes in install/windows.md and CLAUDE.md). A flaky
+
+ timing failure unrelated to the branch is re-run once with
+
+ `gh run rerun <run-id> --failed` instead of a code change, and
+
+ named as such; a rerun is only for failures outside the
+
+ repository''s code. An HTTP 429 or 5xx from an external service
+
+ inside a repository script is fixed at the cause: a bounded retry
+
+ in that script (a fixed attempt count and backoff, then the error
+
+ by name), not a rerun. Otherwise fix at the cause, run the
+
+ affected local gates, commit per the brief and push the same way push_pr did
+
+ (SSH when .github/workflows/ changed). Do not edit the PR text
+
+ unless the fix changes what it claims.
+
+
+ Wrap ONLY the result in <node_output> and </node_output>: each failed
+
+ check, the cause, and the fix commit or the rerun.
+
+ '
+ success_check: scripts/check-pr.sh
+ outputs:
+ extract: node_output
+- id: bot_review
+ title: Handle the bot review (CodeRabbit)
+ type: agent_task
+ expected_duration: 30m
+ prompt: "Standing brief: {{params.brief}}\nCommit trailer: {{params.commit_trailer}}\nPR: read the number\
+ \ from <git-dir>/osb-pr-prepare/pr.env.\n\nThis node starts IMMEDIATELY after the push, while CI is\
+ \ still running.\nDo not wait for the checks to conclude and do not watch them; the\nci_wait node\
+ \ after you waits for the checks. Start the review work now.\n\nFirst wait for CodeRabbit's review\
+ \ to be POSTED: it usually arrives\nminutes after the push, long before the checks end. Poll\n`gh\
+ \ api repos/<owner>/<repo>/pulls/<n>/reviews` and\n`.../pulls/<n>/comments` in one blocking loop (sleep\
+ \ 60 between calls,\nat most 10 minutes), then read what it said: `gh pr view <n>\n--comments`. If\
+ \ no CodeRabbit review appears in that window, say so in\nthe reply and continue. Handle only comments\
+ \ you have not already\nanswered in an earlier pass of this node.\n\nThe operator may hand external\
+ \ review documents to the orchestrator\nmid-wave (paths under .reviews/ or pasted text). When this\
+ \ task's\nresponse carries them, treat every finding in them exactly like a\nCodeRabbit finding: verify\
+ \ against the current code, then fix or\nanswer. Such findings are worked on immediately, never held\
+ \ back for\nCI.\n\n- Skipped (for example \"files exceed the limit\"): post one PR\n comment saying\
+ \ which internal review compensated (the\n self-review, test-audit, security review and OpenCodeReview\
+ \ pass\n of this run), in neutral terms.\n- Each finding: verify it against the current code with\
+ \ the\n receiving-code-review discipline. Valid: fix with the smallest\n diff. Not valid or no longer\
+ \ applicable: reply on the thread with\n a one-sentence technical reason. Valid fixes get a reply\
+ \ naming\n the commit.\n- Commit fixes per the brief (for example `fix: address CodeRabbit\n review\
+ \ feedback`) and push the same way push_pr did. A push here\n restarts CI; that is expected, because\
+ \ the ci_wait node after you\n waits for the new run.\nOne pass per call; do not wait for a second\
+ \ bot pass.\n\nWrite your verdict to $APB_STATUS_FILE with outputs\n{pushed: \"yes\" if you pushed\
+ \ commits, else \"no\", findings: <count>,\nfixed: [shas], answered: <count>, skipped_reason: <text\
+ \ or \"\">}."
+ success_check: scripts/check-pr.sh
+ max_retries: 1
+- id: ready
+ type: finish
+ outcome: success
+ expected_duration: 3m
+ prompt: 'Compose the final answer for the operator from the run context:
+
+ first line `PR: #<number> <url>`; then CI state as
+
+ `gh pr checks <n>` reports it NOW (run it once; say green, or name
+
+ what is red or pending); the kind, the version bump or none; a
+
+ five-line summary of what the PR ships; what the self-review, the
+
+ test-audit, the security review (only "clean" or the count of fixed
+
+ items and advisory drafts, never details), the architecture gate,
+
+ the OpenCodeReview pass (coverage, confirmed and fixed counts) and
+
+ the final gates did; and a "Left for a human" list: gates that ended on their
+
+ loop budget, known environment-only test failures, bot findings
+
+ answered but not fixed, advisory drafts to publish, the merge
+
+ itself. Plain text, English.
+
+ '
+- id: blocked
+ type: finish
+ outcome: failure
+ expected_duration: 3m
+ prompt: 'The PR preparation stopped before a PR with green CI existed.
+
+ Compose the answer from the run context: the node that stopped it
+
+ and its last output (the reason, verbatim where short), whether a
+
+ PR exists (`PR: #<n> <url>` or `PR: none`), what was already
+
+ committed and pushed, and the exact next step a human should take.
+
+ Plain text, English. No security details.
+
+ '
+edges:
+- from: start
+ to: intake
+- from: intake
+ to: zg_index
+- from: zg_index
+ to: self_review
+ condition:
+ type: node_status
+ node: zg_index
+ equals: success
+- from: zg_index
+ to: self_review
+ condition:
+ type: node_status
+ node: zg_index
+ equals: failure
+- from: self_review
+ to: test_audit
+- from: test_audit
+ to: security_review
+- from: security_review
+ to: docs_version
+- from: docs_version
+ to: qa_gates
+- from: qa_gates
+ to: qa_check
+- from: qa_check
+ to: arch_gate
+ condition:
+ type: node_status
+ node: qa_gates
+ equals: success
+- from: qa_check
+ to: qa_fix
+ condition:
+ type: output_match
+ node: qa_gates
+ pattern: 'QA-GATE: red'
+- from: qa_check
+ to: blocked
+ fallback: true
+- from: qa_fix
+ to: qa_gates
+- from: arch_gate
+ to: arch_check
+- from: arch_check
+ to: ocr_review
+ condition:
+ type: node_status
+ node: arch_gate
+ equals: success
+- from: arch_check
+ to: arch_fix
+ condition:
+ type: output_match
+ node: arch_gate
+ pattern: 'ARCH-GATE: violations'
+- from: arch_check
+ to: blocked
+ fallback: true
+- from: arch_fix
+ to: arch_gate
+- from: ocr_review
+ to: final_gates
+- from: final_gates
+ to: final_check
+- from: final_check
+ to: push_pr
+ condition:
+ type: node_status
+ node: final_gates
+ equals: success
+- from: final_check
+ to: final_fix
+ condition:
+ type: output_match
+ node: final_gates
+ pattern: 'FINAL-GATES: red'
+- from: final_check
+ to: blocked
+ fallback: true
+- from: final_fix
+ to: final_gates
+- from: push_pr
+ to: bot_review
+- from: ci_wait
+ to: ci_check
+- from: ci_check
+ to: ci_fix
+ condition:
+ type: output_match
+ node: ci_wait
+ pattern: 'CI-GATE: red'
+- from: ci_check
+ to: blocked
+ condition:
+ type: node_status
+ node: ci_wait
+ equals: failure
+- from: ci_check
+ to: bot_review
+ condition:
+ type: output_field
+ node: bot_review
+ field: pushed
+ equals: 'yes'
+- from: ci_check
+ to: ready
+ condition:
+ type: output_field
+ node: bot_review
+ field: pushed
+ equals: 'no'
+- from: ci_check
+ to: blocked
+ fallback: true
+- from: ci_fix
+ to: ci_wait
+- from: bot_review
+ to: ci_wait
+ max_traversals: 2
+- from: bot_review
+ to: ci_wait_final
+ fallback: true
+- from: ci_wait_final
+ to: ready
+ condition:
+ type: node_status
+ node: ci_wait_final
+ equals: success
+- from: ci_wait_final
+ to: blocked
+ fallback: true
diff --git a/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/arch-gate.sh b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/arch-gate.sh
new file mode 100755
index 00000000..3afdba90
--- /dev/null
+++ b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/arch-gate.sh
@@ -0,0 +1,51 @@
+#!/bin/sh
+# Architecture gate: code-ranker in baseline mode. The merge base with
+# origin/<base> is analyzed as the baseline and the committed HEAD as the
+# candidate (both from `git archive`, so untracked files and local tool
+# directories never enter the graph); only violations the branch introduces
+# fail the gate. Pre-existing debt on the base is not this PR's to fix.
+#
+# Exit 0: no new violation. Exit 1: new violations (the output starts with
+# `ARCH-GATE: violations`, followed by code-ranker's fix prompt). Exit 2: the
+# tool itself failed (`ERROR:`), which no code change in the branch can fix.
+[ -n "${BASH_VERSION:-}" ] || exec bash "$0" "$@"
+set -euo pipefail
+. "$(dirname "$0")/lib.sh"
+
+root=$(repo_root)
+cd "$root"
+load_ctx
+command -v code-ranker >/dev/null 2>&1 || die "code-ranker is not on PATH"
+
+mb=$(git merge-base "origin/$OSB_BASE" HEAD) || die "no merge base with origin/$OSB_BASE"
+work=$(mktemp -d "${TMPDIR:-/tmp}/osb-arch-gate.XXXXXX")
+trap 'rm -rf "$work"' EXIT
+mkdir -p "$work/base" "$work/head"
+git archive "$mb" | tar -x -C "$work/base"
+git archive HEAD | tar -x -C "$work/head"
+
+# Explicit plugins so base and head are analyzed with the same languages.
+plugins=(--plugins ts --plugins python --plugins md)
+cd "$work" # code-ranker writes its default HTML viewer under the cwd
+code-ranker report base "${plugins[@]}" --output.mode quiet \
+ --output.json.path "$work/base.json" --output.html.path "$work/base.html" >/dev/null 2>"$work/report.err" ||
+ die "code-ranker report on the merge base failed: $(cat "$work/report.err")"
+
+set +e
+code-ranker check head "${plugins[@]}" --baseline "$work/base.json" \
+ --output-format prompt --output.mode summary >"$work/check.out" 2>&1
+rc=$?
+set -e
+
+if [ "$rc" -eq 0 ]; then
+ cat "$work/check.out"
+ echo "ARCH-GATE: pass (no new violation against $mb)"
+ exit 0
+fi
+if grep -q 'new violation(s) vs baseline' "$work/check.out"; then
+ echo "ARCH-GATE: violations (new against merge base $mb)"
+ sed "s#$work/##g" "$work/check.out"
+ exit 1
+fi
+cat "$work/check.out"
+die "code-ranker check failed without a violation verdict (exit $rc)"
diff --git a/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/check-context.sh b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/check-context.sh
new file mode 100755
index 00000000..cba70848
--- /dev/null
+++ b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/check-context.sh
@@ -0,0 +1,24 @@
+#!/bin/sh
+# success_check of `intake`: the recorded PR context is complete and true.
+# apb runs every script through `sh`; re-exec under bash for pipefail.
+[ -n "${BASH_VERSION:-}" ] || exec bash "$0" "$@"
+set -euo pipefail
+. "$(dirname "$0")/lib.sh"
+
+cd "$(repo_root)"
+load_ctx
+
+case " $OSB_KINDS " in *" $OSB_KIND "*) ;; *) die "OSB_KIND=$OSB_KIND is not one of: $OSB_KINDS" ;; esac
+case " $OSB_RELEASES " in *" $OSB_RELEASE "*) ;; *) die "OSB_RELEASE=$OSB_RELEASE is not one of: $OSB_RELEASES" ;; esac
+
+branch=$(git symbolic-ref --quiet --short HEAD) || die "HEAD is detached; the PR needs a branch"
+[ "$branch" = "$OSB_BRANCH" ] || die "checked-out branch $branch differs from OSB_BRANCH=$OSB_BRANCH"
+[ "$branch" != "$OSB_BASE" ] || die "the PR branch is the base branch $OSB_BASE"
+git rev-parse --verify --quiet "origin/$OSB_BASE" >/dev/null || die "origin/$OSB_BASE does not exist; fetch it"
+mb=$(git merge-base "origin/$OSB_BASE" HEAD) || die "no merge base between origin/$OSB_BASE and HEAD"
+git diff --quiet "$mb" HEAD && die "the branch carries no change against origin/$OSB_BASE"
+git diff --quiet || die "tracked files have unstaged changes; the intake commits the implementation first"
+git diff --cached --quiet || die "the index holds staged, uncommitted changes"
+
+echo "context ok: kind=$OSB_KIND release=$OSB_RELEASE base=$OSB_BASE branch=$OSB_BRANCH merge-base=$mb"
+echo "commits: $(git rev-list --count "$mb"..HEAD), files: $(git diff --name-only "$mb" HEAD | wc -l)"
diff --git a/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/check-pr.sh b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/check-pr.sh
new file mode 100755
index 00000000..2450e8be
--- /dev/null
+++ b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/check-pr.sh
@@ -0,0 +1,27 @@
+#!/bin/sh
+# success_check of `push_pr` (and of the fix nodes that push): the PR recorded
+# in pr.env is open, not merged, targets the base branch, and its head is
+# exactly the local HEAD, which the remote branch also carries.
+[ -n "${BASH_VERSION:-}" ] || exec bash "$0" "$@"
+set -euo pipefail
+. "$(dirname "$0")/lib.sh"
+
+cd "$(repo_root)"
+load_ctx
+load_pr
+
+head=$(git rev-parse HEAD)
+git diff --quiet && git diff --cached --quiet || die "uncommitted tracked changes: commit and push them"
+remote=$(git ls-remote origin "refs/heads/$OSB_BRANCH" | cut -f1)
+[ "$remote" = "$head" ] || die "origin/$OSB_BRANCH is '${remote:-absent}', local HEAD is $head: push"
+
+info=$(gh pr view "$OSB_PR_NUMBER" --json state,headRefOid,baseRefName,headRefName,url \
+ -q '[.state,.headRefOid,.baseRefName,.headRefName,.url]|@tsv') || die "gh pr view $OSB_PR_NUMBER failed"
+IFS=$'\t' read -r state pr_head pr_base pr_branch url <<<"$info"
+[ "$state" = OPEN ] || die "PR #$OSB_PR_NUMBER is $state, expected OPEN (this playbook never merges)"
+[ "$pr_base" = "$OSB_BASE" ] || die "PR #$OSB_PR_NUMBER targets $pr_base, expected $OSB_BASE"
+[ "$pr_branch" = "$OSB_BRANCH" ] || die "PR #$OSB_PR_NUMBER is from $pr_branch, expected $OSB_BRANCH"
+[ "$pr_head" = "$head" ] || die "PR #$OSB_PR_NUMBER head is $pr_head, local HEAD is $head"
+auto=$(gh pr view "$OSB_PR_NUMBER" --json autoMergeRequest -q '.autoMergeRequest != null')
+[ "$auto" = false ] || die "PR #$OSB_PR_NUMBER has auto-merge armed; merging is the operator's call: gh pr merge --disable-auto $OSB_PR_NUMBER"
+echo "PR ok: #$OSB_PR_NUMBER $url open on $OSB_BASE at $head"
diff --git a/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/check-release.sh b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/check-release.sh
new file mode 100755
index 00000000..262f5f97
--- /dev/null
+++ b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/check-release.sh
@@ -0,0 +1,51 @@
+#!/bin/sh
+# success_check of `docs_version`: the version and CHANGELOG state match what
+# the change kind decided (CLAUDE.md "Versioning").
+#
+# Release-bearing PR (OSB_RELEASE major|minor|patch):
+# - package.json carries the expected bump of the base version;
+# - the first `## [` heading of CHANGELOG.md is `## [X.Y.Z] - YYYY-MM-DD`
+# (no `[Unreleased]` left above it);
+# - the `[X.Y.Z]: .../compare/vPREV...vX.Y.Z` link reference exists;
+# - `sync-version --check` passes and everything is committed.
+# No-release PR (OSB_RELEASE none): package.json still carries the base version.
+[ -n "${BASH_VERSION:-}" ] || exec bash "$0" "$@"
+set -euo pipefail
+. "$(dirname "$0")/lib.sh"
+
+cd "$(repo_root)"
+load_ctx
+
+prev=$(git show "origin/$OSB_BASE:package.json" | json_version)
+now=$(json_version <package.json)
+
+if [ "$OSB_RELEASE" = none ]; then
+ [ "$now" = "$prev" ] || die "kind $OSB_KIND ships no release, but package.json moved $prev -> $now"
+ echo "release check ok: no release, version stays $now"
+ exit 0
+fi
+
+expected=$(python3 - "$prev" "$OSB_RELEASE" <<'PY'
+import sys
+major, minor, patch = (int(p) for p in sys.argv[1].split("."))
+bump = sys.argv[2]
+if bump == "major":
+ major, minor, patch = major + 1, 0, 0
+elif bump == "minor":
+ minor, patch = minor + 1, 0
+else:
+ patch += 1
+print(f"{major}.{minor}.{patch}")
+PY
+)
+[ "$now" = "$expected" ] || die "package.json is $now; a $OSB_RELEASE bump of $prev is $expected"
+
+first=$(grep -m1 -E '^## \[' CHANGELOG.md || true)
+printf '%s\n' "$first" | grep -qE "^## \[$expected\] - [0-9]{4}-[0-9]{2}-[0-9]{2}\$" ||
+ die "the first CHANGELOG heading is '$first', expected '## [$expected] - <date>'"
+grep -qE "^\[$expected\]: https://github\.com/itechmeat/open-second-brain/compare/v$prev\.\.\.v$expected\$" CHANGELOG.md ||
+ die "CHANGELOG.md lacks the link reference [$expected]: https://github.com/itechmeat/open-second-brain/compare/v$prev...v$expected"
+
+bun run scripts/sync-version.ts --check >/dev/null 2>&1 || die "sync-version --check fails: run bun run scripts/sync-version.ts"
+git diff --quiet && git diff --cached --quiet || die "version or CHANGELOG changes are not committed"
+echo "release check ok: $prev -> $expected ($OSB_RELEASE), CHANGELOG heading and link reference present"
diff --git a/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/ci-wait.sh b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/ci-wait.sh
new file mode 100755
index 00000000..131add6e
--- /dev/null
+++ b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/ci-wait.sh
@@ -0,0 +1,69 @@
+#!/bin/sh
+# Waits for the PR's checks on the current head with ONE blocking command,
+# `gh pr checks --watch` (no polling loop of our own), then prints the verdict.
+# The checks include CodeRabbit's status, so a green result also means the bot
+# review has finished (or was skipped).
+#
+# The watch runs WITHOUT --fail-fast: the verdict waits until every check has a
+# terminal state. A fast failure (a scan failing in seconds) must not start the
+# fix round while a slow job (windows) is still running, or a late failure of
+# that job needs a second fix round. The red verdict lists every failed check.
+#
+# Before the watch, the script makes sure the CI workflow run for THIS head
+# commit exists. Right after a push, `gh pr checks` can still show the previous
+# commit's finished checks and would return at once with a stale verdict. That
+# registration wait is bounded (REGISTRATION_TRIES x REGISTRATION_SLEEP) and
+# ends as soon as GitHub lists the run.
+#
+# Exit 0 green, 1 red (failed-job logs follow), 2 setup error.
+[ -n "${BASH_VERSION:-}" ] || exec bash "$0" "$@"
+set -euo pipefail
+. "$(dirname "$0")/lib.sh"
+
+REGISTRATION_TRIES=18
+REGISTRATION_SLEEP=10
+WATCH_TIMEOUT=6000 # the windows job alone has taken 45 minutes on a slow runner
+WATCH_INTERVAL=60 # gh's own refresh period while it blocks
+LOG_TAIL=250
+CI_WORKFLOW=ci.yml
+
+cd "$(repo_root)"
+load_ctx
+load_pr
+
+head=$(git rev-parse HEAD)
+pr_head=$(gh pr view "$OSB_PR_NUMBER" --json headRefOid -q .headRefOid)
+[ "$pr_head" = "$head" ] || die "PR #$OSB_PR_NUMBER head $pr_head is not local HEAD $head: push first"
+
+run_id=""
+for _ in $(seq "$REGISTRATION_TRIES"); do
+ run_id=$(gh run list --commit "$head" --workflow "$CI_WORKFLOW" --limit 1 --json databaseId -q '.[0].databaseId // empty')
+ [ -n "$run_id" ] && break
+ sleep "$REGISTRATION_SLEEP"
+done
+[ -n "$run_id" ] || die "no $CI_WORKFLOW run registered for $head after $((REGISTRATION_TRIES * REGISTRATION_SLEEP))s"
+
+set +e
+timeout "$WATCH_TIMEOUT" gh pr checks "$OSB_PR_NUMBER" --watch --interval "$WATCH_INTERVAL" >/dev/null 2>&1
+rc=$?
+set -e
+echo "checks for PR #$OSB_PR_NUMBER at $head (CI run $run_id):"
+gh pr checks "$OSB_PR_NUMBER" 2>&1 || true
+
+if [ "$rc" -eq 0 ]; then
+ echo "CI-GATE: green"
+ exit 0
+fi
+if [ "$rc" -eq 124 ]; then
+ echo "CI-GATE: red (checks still pending after ${WATCH_TIMEOUT}s)"
+ exit 1
+fi
+failed=$(gh pr checks "$OSB_PR_NUMBER" --json name,bucket -q '[.[] | select(.bucket == "fail" or .bucket == "cancel") | .name] | join(", ")' 2>/dev/null || true)
+echo "CI-GATE: red (failed: ${failed:-the check list could not be read, see the checks above})"
+# Per failed job, not `--log-failed` on the run: the job logs stay readable
+# even when another workflow run on the same head is still in progress.
+for job in $(gh run view "$run_id" --json jobs -q '.jobs[] | select(.conclusion == "failure") | .databaseId'); do
+ echo "--- failed job $job of run $run_id (log tail) ---"
+ gh run view --job "$job" --log 2>&1 | tail -n "$LOG_TAIL" || true
+done
+exit 1
diff --git a/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/final-gates.sh b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/final-gates.sh
new file mode 100755
index 00000000..70e51eb3
--- /dev/null
+++ b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/final-gates.sh
@@ -0,0 +1,43 @@
+#!/bin/sh
+# Final gates before the push: the QA gates (qa-gates.sh) and the
+# architecture gate (arch-gate.sh) again, after the OpenCodeReview pass may
+# have committed fixes. When HEAD is still the commit the QA gates last
+# passed on, nothing changed since the green QA and architecture gates (the
+# architecture gate runs after the QA gates and any fix it needed moves
+# HEAD), so the re-run is skipped.
+#
+# Exit 0: both green, or skipped (`FINAL-GATES: pass`). Exit 1: a red gate
+# (`FINAL-GATES: red`, followed by the markers and output of the red gates).
+# Exit 2: a setup or tool error (`ERROR:`), which no code change can fix.
+[ -n "${BASH_VERSION:-}" ] || exec bash "$0" "$@"
+set -uo pipefail
+here=$(dirname "$0")
+. "$here/lib.sh"
+
+cd "$(repo_root)"
+load_ctx
+
+head=$(git rev-parse HEAD)
+green="$CTX_DIR/qa-green-head"
+if [ -f "$green" ] && [ "$(cat "$green")" = "$head" ]; then
+ echo "FINAL-GATES: pass (HEAD $head unchanged since the green QA and architecture gates)"
+ exit 0
+fi
+
+qa_out=$(sh "$here/qa-gates.sh" 2>&1)
+qa_rc=$?
+arch_out=$(sh "$here/arch-gate.sh" 2>&1)
+arch_rc=$?
+
+if [ "$qa_rc" -eq 0 ] && [ "$arch_rc" -eq 0 ]; then
+ printf '%s\n%s\n' "$qa_out" "$arch_out"
+ echo "FINAL-GATES: pass (QA and architecture gates green at $head)"
+ exit 0
+fi
+if [ "$qa_rc" -eq 2 ] || [ "$arch_rc" -eq 2 ]; then
+ printf '%s\n%s\n' "$qa_out" "$arch_out"
+ die "a gate could not run (qa exit $qa_rc, arch exit $arch_rc)"
+fi
+echo "FINAL-GATES: red (qa exit $qa_rc, arch exit $arch_rc)"
+printf '%s\n%s\n' "$qa_out" "$arch_out"
+exit 1
diff --git a/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/lib.sh b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/lib.sh
new file mode 100755
index 00000000..287d0c71
--- /dev/null
+++ b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/lib.sh
@@ -0,0 +1,95 @@
+# Shared helpers for the osb-pr-prepare script nodes. Sourced, never run.
+#
+# apb hands a script node no parameters and no environment of its own, so the
+# per-run facts (base branch, change kind, release bump, PR number) travel in
+# files under the repository's git directory, which git never tracks:
+#
+# $(git rev-parse --git-path osb-pr-prepare)/context.env written by `intake`
+# $(git rev-parse --git-path osb-pr-prepare)/pr.env written by `push_pr`
+# $(git rev-parse --git-path osb-pr-prepare)/known-test-failures.txt
+# written by `qa_fix`
+#
+# A missing file is an explicit error naming the node that should have written
+# it, never a silent default.
+
+# The CI toolchain pin (.github/workflows/ci.yml, `bun-version`).
+OSB_CI_BUN_VERSION=1.4.0
+# The CI Python pin (.github/workflows/ci.yml, `python-version`).
+OSB_CI_PYTHON_VERSION=3.11
+# The only local embedding model the zg index uses: no remote provider, no key.
+OSB_ZG_MODEL=local/potion-code-16m-v2
+OSB_KINDS="feature fix refactor chore docs deps"
+OSB_RELEASES="major minor patch none"
+
+die() {
+ printf 'ERROR: %s\n' "$*"
+ exit 2
+}
+
+repo_root() {
+ git rev-parse --show-toplevel 2>/dev/null || die "not inside a git work tree"
+}
+
+ctx_dir() {
+ git rev-parse --path-format=absolute --git-path osb-pr-prepare
+}
+
+load_ctx() {
+ CTX_DIR=$(ctx_dir)
+ [ -f "$CTX_DIR/context.env" ] ||
+ die "$CTX_DIR/context.env is missing: the intake node did not record the PR context"
+ # shellcheck disable=SC1091
+ . "$CTX_DIR/context.env"
+ for v in OSB_BASE OSB_KIND OSB_RELEASE OSB_BRANCH; do
+ eval "val=\${$v:-}"
+ [ -n "$val" ] || die "$v is not set in $CTX_DIR/context.env"
+ done
+}
+
+load_pr() {
+ [ -f "$CTX_DIR/pr.env" ] ||
+ die "$CTX_DIR/pr.env is missing: the push_pr node did not record the PR number"
+ # shellcheck disable=SC1091
+ . "$CTX_DIR/pr.env"
+ [ -n "${OSB_PR_NUMBER:-}" ] || die "OSB_PR_NUMBER is not set in $CTX_DIR/pr.env"
+}
+
+json_version() {
+ # Reads the "version" field of a package.json given on stdin.
+ python3 -c 'import json,sys; print(json.load(sys.stdin)["version"])'
+}
+
+# Puts the CI-pinned Bun first on PATH. The Linux release zip is cached under
+# ~/.cache/osb-pr-prepare; the download happens once and needs no sudo. Local
+# Bun (1.4.x newer than the pin) bundles openclaw/index.js differently, so the
+# bundle-in-sync gate is only meaningful with the pinned build.
+use_ci_bun() {
+ cache="${XDG_CACHE_HOME:-$HOME/.cache}/osb-pr-prepare/bun-$OSB_CI_BUN_VERSION"
+ case "$(uname -m)" in
+ x86_64) asset=bun-linux-x64 ;;
+ aarch64 | arm64) asset=bun-linux-aarch64 ;;
+ *) die "no Bun $OSB_CI_BUN_VERSION release asset known for $(uname -m)" ;;
+ esac
+ bin="$cache/$asset/bun"
+ if [ ! -x "$bin" ]; then
+ mkdir -p "$cache"
+ url="https://github.com/oven-sh/bun/releases/download/bun-v$OSB_CI_BUN_VERSION/$asset.zip"
+ curl -fsSL -o "$cache/$asset.zip" "$url" || die "download failed: $url"
+ python3 -m zipfile -e "$cache/$asset.zip" "$cache" || die "unzip failed: $cache/$asset.zip"
+ chmod +x "$bin"
+ fi
+ got=$("$bin" --version) || die "cached Bun at $bin does not run"
+ [ "$got" = "$OSB_CI_BUN_VERSION" ] || die "cached Bun at $bin reports $got, expected $OSB_CI_BUN_VERSION"
+ PATH="$(dirname "$bin"):$PATH"
+ export PATH
+}
+
+# Prints the interpreter path of the CI-pinned Python, installing it through uv
+# (user-level, no sudo) when missing.
+ci_python() {
+ command -v uv >/dev/null 2>&1 || die "uv is not on PATH; it provides Python $OSB_CI_PYTHON_VERSION"
+ uv python find "$OSB_CI_PYTHON_VERSION" 2>/dev/null && return 0
+ uv python install "$OSB_CI_PYTHON_VERSION" >/dev/null 2>&1 ||
+ die "uv could not install Python $OSB_CI_PYTHON_VERSION"
+ uv python find "$OSB_CI_PYTHON_VERSION"
+}
diff --git a/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/qa-gates.sh b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/qa-gates.sh
new file mode 100755
index 00000000..dc1c6d66
--- /dev/null
+++ b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/qa-gates.sh
@@ -0,0 +1,132 @@
+#!/bin/sh
+# Open Second Brain's own QA gates, as the CI `validate` job runs them
+# (.github/workflows/ci.yml), in the same order and with the same toolchain
+# pins: Bun 1.4.0 (cached release binary) and Python 3.11 (uv). Every step
+# runs even after a failure, so one pass shows the whole picture; the full
+# log of each step lands in <git-dir>/osb-pr-prepare/qa-logs/<step>.log and
+# only the tail of a failing step is printed.
+#
+# Known environment-only test failures: a test named in
+# <git-dir>/osb-pr-prepare/known-test-failures.txt (one exact `bun test` name
+# per line) does not fail the gate. Only `qa_fix` adds a name there, and only
+# after proving the same test fails on the merge base in this environment.
+#
+# On green, the tested HEAD is recorded in <git-dir>/osb-pr-prepare/qa-green-head
+# for final-gates.sh.
+#
+# The CI `windows` job cannot run here; CI covers it after the push.
+[ -n "${BASH_VERSION:-}" ] || exec bash "$0" "$@"
+set -uo pipefail
+. "$(dirname "$0")/lib.sh"
+
+root=$(repo_root)
+cd "$root"
+load_ctx
+command -v bun >/dev/null 2>&1 || die "bun is not on PATH"
+use_ci_bun
+py=$(ci_python) || exit 2
+
+logs="$CTX_DIR/qa-logs"
+rm -rf "$logs"
+mkdir -p "$logs"
+known="$CTX_DIR/known-test-failures.txt"
+test_home=$(mktemp -d "${TMPDIR:-/tmp}/osb-qa-home.XXXXXX")
+bundle=$(mktemp "${TMPDIR:-/tmp}/osb-openclaw-rebuild.XXXXXX.js")
+trap 'rm -rf "$test_home" "$bundle"' EXIT
+TAIL_LINES=80
+failed=()
+
+step() {
+ # step <name> <command...>: runs one gate, records PASS/FAIL.
+ name=$1
+ shift
+ if "$@" >"$logs/$name.log" 2>&1; then
+ echo "PASS $name"
+ else
+ echo "FAIL $name (log: $logs/$name.log)"
+ tail -n "$TAIL_LINES" "$logs/$name.log" | sed 's/^/ /'
+ failed+=("$name")
+ fi
+}
+
+report_only() {
+ name=$1
+ shift
+ "$@" >"$logs/$name.log" 2>&1
+ echo "INFO $name (report-only, exit $?; log: $logs/$name.log)"
+}
+
+openclaw_bundle_in_sync() {
+ bun build src/openclaw/index.ts --outfile "$bundle" --target=node --format=esm \
+ --external openclaw/plugin-sdk/plugin-entry &&
+ diff -q "$bundle" openclaw/index.js
+}
+
+bun_test() {
+ # Throwaway HOME, as the project's test discipline requires.
+ HOME="$test_home" bun test >"$logs/bun-test.raw" 2>&1
+ rc=$?
+ cat "$logs/bun-test.raw"
+ [ "$rc" -eq 0 ] && return 0
+ sed -n 's/^(fail) \(.*\) \[[0-9.]*m\{0,1\}s\]$/\1/p; t; s/^(fail) \(.*\)$/\1/p' "$logs/bun-test.raw" |
+ LC_ALL=C sort -u >"$logs/bun-test.failed"
+ [ -s "$logs/bun-test.failed" ] || {
+ echo "bun test exited $rc without a (fail) line: a crash, not a test failure"
+ return 1
+ }
+ if [ -f "$known" ]; then
+ unknown=$(LC_ALL=C sort -u "$known" | LC_ALL=C comm -23 "$logs/bun-test.failed" -)
+ else
+ unknown=$(cat "$logs/bun-test.failed")
+ fi
+ if [ -z "$unknown" ]; then
+ echo "only known environment-only failures failed:"
+ sed 's/^/ known: /' "$logs/bun-test.failed"
+ return 0
+ fi
+ echo "failing tests not on the known list:"
+ printf '%s\n' "$unknown" | sed 's/^/ /'
+ return 1
+}
+
+python_tests() {
+ PATH="$root/scripts:$PATH" HOME="$test_home" "$py" -m unittest discover -s tests/python -v
+}
+
+anti_drift_ran() {
+ (
+ cd tests/python &&
+ PATH="$root/scripts:$PATH" HOME="$test_home" "$py" -m unittest test_static_schemas -v >"$logs/anti-drift.raw" 2>&1
+ cat "$logs/anti-drift.raw"
+ grep -q "test_static_schemas_match_live_tools_list" "$logs/anti-drift.raw" || exit 1
+ if grep -qE 'test_static_schemas_match_live_tools_list.* \.\.\. skipped' "$logs/anti-drift.raw"; then
+ echo "anti-drift skipped itself; the live server was not reachable"
+ exit 1
+ fi
+ )
+}
+
+echo "QA gates for $OSB_BRANCH ($(git rev-parse --short HEAD)), Bun $(bun --version), $("$py" --version)"
+step install bun install --frozen-lockfile
+step sync-version bun run sync-version:check
+step sync-plugin-mirrors bun run sync-plugin-mirrors:check
+step openclaw-bundle openclaw_bundle_in_sync
+step link-ratchet bun run link-ratchet:check
+report_only check-paths bun run check:paths
+step fmt bun run fmt:check
+step lint bun run lint
+step typecheck bun run typecheck
+step bun-test bun_test
+step python-tests python_tests
+step anti-drift anti_drift_ran
+step python-compile "$py" -m compileall -q plugins/hermes
+step hermes-scan bun run check:hermes-scan
+
+if [ "${#failed[@]}" -eq 0 ]; then
+ # final-gates.sh skips its re-run while HEAD is still this commit.
+ git rev-parse HEAD >"$CTX_DIR/qa-green-head"
+ echo "QA-GATE: green"
+ exit 0
+fi
+echo "QA-GATE: red (${failed[*]})"
+exit 1
diff --git a/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/zg-index.sh b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/zg-index.sh
new file mode 100755
index 00000000..3236afa1
--- /dev/null
+++ b/.apb/playbooks/osb-pr-prepare/1.2.0/scripts/zg-index.sh
@@ -0,0 +1,20 @@
+#!/bin/sh
+# Idempotent: makes sure the workspace has a ready zg (zvec-grep) index built
+# with a LOCAL embedding model. A missing index is created, an existing one is
+# refreshed incrementally and keeps its stored model (the env default only
+# applies to a new index). No remote provider, no credential, no daemon
+# (`--mode direct`). The index lives in <root>/.zvec-grep, which the repository
+# .gitignore lists, so neither git nor oxfmt sees it.
+[ -n "${BASH_VERSION:-}" ] || exec bash "$0" "$@"
+set -euo pipefail
+. "$(dirname "$0")/lib.sh"
+
+cd "$(repo_root)"
+command -v zg >/dev/null 2>&1 || die "zg is not on PATH"
+git check-ignore -q .zvec-grep/ || die ".zvec-grep/ is not ignored: add it to the repository .gitignore"
+
+export ZVEC_GREP_EMBEDDING="$OSB_ZG_MODEL"
+export ZVEC_GREP_DEVICE=cpu
+zg index --mode direct 2>&1
+zg status --mode direct --check-ready 2>&1
+echo "ZG-INDEX: ready"
diff --git a/.apb/playbooks/osb-pr-prepare/current b/.apb/playbooks/osb-pr-prepare/current
index 9084fa2f..26aaba0e 100644
--- a/.apb/playbooks/osb-pr-prepare/current
+++ b/.apb/playbooks/osb-pr-prepare/current
@@ -1 +1 @@
-1.1.0
+1.2.0
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json
index 9c608992..5b228014 100644
--- a/.claude-plugin/plugin.json
+++ b/.claude-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.73.1",
+ "version": "1.74.0",
"description": "Plugin-first second brain package for AI agents and humans.",
"author": {
"name": "Open Second Brain contributors"
diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json
index d74f86cc..2bcd535a 100644
--- a/.codex-plugin/plugin.json
+++ b/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.73.1",
+ "version": "1.74.0",
"description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
"author": {
"name": "Open Second Brain contributors",
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b1eb6392..f968c0fd 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,29 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+## [1.74.0] - 2026-10-07
+
+Open Second Brain now defends against near-duplicate memory at the points a fact enters or leaves the vault: one reach-aware near-duplicate kernel scores the note-write receipt hint, the feedback advisory and a new retire-sibling list, so a context-driven retire names the active paraphrases that would otherwise survive it; the receipt hint can look beyond the written page's directory through the keyword index; every receipt lint finding answers at the caller's reach; extract-signals grounds time bounds in each turn's timestamp and refuses a payload that repeats a topic; and route metrics time each write stage without carrying any content.
+
+### Added
+
+- **A reach-aware near-duplicate kernel.** One pure lookup (`src/core/brain/near-duplicate.ts`) scores the write-receipt hint, the `brain_feedback` write-conflict advisory and the retire siblings over one named threshold table. Its `readable` predicate is required and runs before anything else, so a page the caller may not read is neither scored nor counted.
+- **Retire siblings.** A context-driven retire (`superseded-by-context`, `rebutted`, `quarantine-violated`, `user-rejected`) now nominates the active preferences that resemble it, at 0.7 Jaccard or above on the principle. `brain_review_candidates` returns them as an additive `retire_siblings` array of `{ retiring_id, sibling_id, score, method }` pairs, the dream run summary carries the same key, and `o2b brain reject` lists the siblings of the rejected preference with the command that rejects each (`--json` adds `retire_siblings`). The list is advisory and accept-gated: nothing is retired by it, and each pair is accepted through `brain_apply_evidence` with `result: outdated` or `o2b brain reject`. Decay retires, merges and retires the confirmed-evidence gate holds back nominate nothing, and a pair is kept only when the caller may read both preferences.
+- **A stored-vector tier for retire siblings.** With a search config, `brain_review_candidates` also compares the stored content vectors of the retiring and sibling preferences and adds pairs at 0.92 cosine or above as `method: "embedding"`, with zero embedding spend. `retire_siblings_semantic` reports the tier's outcome (`used`, `index_missing`, `vec_unavailable`, `not_embedded`, `model_mismatch`), and the tier probes only the retires the answer keeps.
+- **`near_duplicate_retire_siblings_enabled`.** The key (env `OPEN_SECOND_BRAIN_NEAR_DUPLICATE_RETIRE_SIBLINGS_ENABLED`, default off) gates every retire-sibling surface; off, the dream summary, `brain_review_candidates` and `o2b brain reject` are unchanged.
+- **A wider near-duplicate receipt hint.** With `near_duplicate_write_widening_enabled` (env `OPEN_SECOND_BRAIN_NEAR_DUPLICATE_WRITE_WIDENING_ENABLED`, default off), the advisory near-duplicate hint on note-write receipts also compares the written pages with candidates from other directories, pulled from the keyword index (the top 20 chunks per written page) and re-read from disk, at no embedding spend. An index that cannot be opened or queried never fails the write: the lint report carries `widening: "index_unavailable"` and a `widening_detail` code naming the failure, and the same-directory hint still runs.
+- **Turn timestamps in extract-signals.** Every `turns_mined` entry carries the turn's stored `timestamp` (`""` when the source has none), and each prompt line names it. The envelope asks for a time bound as an ISO 8601 date or interval resolved against the stating turn's timestamp, in language-neutral terms, and the dream pass turns that bound into the preference's `valid_from` / `valid_until`.
+- **Write-stage timings on route metrics.** `mcp_route_latency` records gain an optional `stages` array of `{ name, ms }` entries from a closed allowlist of nine stage names (`validate`, `idempotency_lookup`, `near_duplicate_lookup`, `document_write`, `idempotency_remember`, `log_append`, `preference_write`, `write_receipt`, `lint`); any other name and any negative or non-finite value is dropped, so no content reaches the record. Stages appear only in the continuity record, never in a tool response, and `brain_route_metrics` `summary` rolls them up per route (count, average and p95 per stage).
+
+### Changed
+
+- **Wave playbooks.** The feature-release playbook (1.4.0) records the picked cards for the wave's reviewers and merges the green pull request itself, squash and verified head, under the operator's 2026-10-06 merge authorization; the pr-prepare playbook (1.2.0) handles the bot review while CI runs, waits out its bot budget with one final CI watch, and reaches ready on a green gate when the bot review pushed nothing.
+- **Extract payloads that repeat a topic are refused.** Two items sharing a `topic` refuse the whole payload, naming both items, and nothing is written.
+- **Docs:** `docs/mcp.md` gains the retire siblings, the receipt widening and reach, the `turns_mined` timestamps and the write stages, `docs/observability.md` the `stages` field, its allowlist and the routes that emit it, and `docs/cli-reference.md` the extract-signals timestamps and topic refusal; the `o2b brain reject` help names the retire siblings.
+
+### Fixed
+
+- **Every receipt lint finding answers at the caller's reach.** Note-write receipts used to apply the caller's reach to the near-duplicate check only; the broken-wikilink and merged-link checks now treat a withheld page as absent too, so a wikilink to it reads as broken and a merge chain ends at its first withheld hop.
## [1.73.1] - 2026-10-07
Sturdier input handling in three places, and a faster CI.
@@ -8246,6 +8269,7 @@ plugin config (vault field)`, and exits with a clear
- Sandbox vault and plugin manifest fixtures for tests.
- GitHub release workflow for tag-based and manually dispatched releases.
+[1.74.0]: https://github.com/itechmeat/open-second-brain/compare/v1.73.1...v1.74.0
[1.73.1]: https://github.com/itechmeat/open-second-brain/compare/v1.73.0...v1.73.1
[1.73.0]: https://github.com/itechmeat/open-second-brain/compare/v1.72.0...v1.73.0
[1.72.0]: https://github.com/itechmeat/open-second-brain/compare/v1.71.0...v1.72.0
diff --git a/bun.lock b/bun.lock
index 9d605efd..ad2ec6e7 100644
--- a/bun.lock
+++ b/bun.lock
@@ -11,8 +11,8 @@
"@types/bun": "^1.4.2",
"@types/node": "^22.20.4",
"@types/proper-lockfile": "^4.1.4",
- "oxfmt": "0.71.0",
- "oxlint": "1.86.0",
+ "oxfmt": "0.70.0",
+ "oxlint": "1.85.0",
"typescript": "^5.5.0",
},
"optionalDependencies": {
@@ -51,81 +51,81 @@
"@node-rs/jieba-win32-x64-msvc": ["@node-rs/[email protected]", "", { "os": "win32", "cpu": "x64" }, "sha512-m9elDBggy4aCLN3MkZlwIzAHctU8G2/F2KfbEQ15FDwACfvVWlzXb8YXc38o845P+dKFttag2Znv0R66tFOH2A=="],
- "@oxfmt/binding-android-arm-eabi": ["@oxfmt/[email protected]", "", { "os": "android", "cpu": "arm" }, "sha512-l31EyBfB4egJeFua11aQFEUE9nnDtfpWCadqQovm0WL30t/yUHtdFl/5HyFXvd5xT8Ewh3U4TI9f30GowrQ1fw=="],
+ "@oxfmt/binding-android-arm-eabi": ["@oxfmt/[email protected]", "", { "os": "android", "cpu": "arm" }, "sha512-Xd7YO4/T2axEj6FTLcj4Why3mTBqFMg+x24xtorT4Lb2+1g82090GH0a/4U1m0pABGYiix2bq1pqkYrmV3f0Sw=="],
- "@oxfmt/binding-android-arm64": ["@oxfmt/[email protected]", "", { "os": "android", "cpu": "arm64" }, "sha512-E8q39SUZzSXQ08oZoFoTkxRADkdBNgL6De4qgZYr2O4JCunwqrZwAyCebxIF3yVHBafHM30WUkhTkDi+jNYZOw=="],
+ "@oxfmt/binding-android-arm64": ["@oxfmt/[email protected]", "", { "os": "android", "cpu": "arm64" }, "sha512-x9rlMYyKXdgKdYyUJzGsK1ZV8P4di/J32ipzcS6Jet6p9r9UAh28neXIMtdlSaJJycdi61Z4YkcLKLpk8ueFjg=="],
- "@oxfmt/binding-darwin-arm64": ["@oxfmt/[email protected]", "", { "os": "darwin", "cpu": "arm64" }, "sha512-pTteTrN88DicrmJ3DocBmpNDa6Umfh0iveEG8rnAlDFQpclmtDS7hnIvZbIEV9mDO0Ot1U0sjtoJQ1itd/ZMhQ=="],
+ "@oxfmt/binding-darwin-arm64": ["@oxfmt/[email protected]", "", { "os": "darwin", "cpu": "arm64" }, "sha512-IUTUPvrBVYy7POh4stXzRdz4IVC/1QSaviCWoyenSlOhGu0X9j5K07vCTM9biLjAA2Zs31l0Rj5vvRpj9n95wA=="],
- "@oxfmt/binding-darwin-x64": ["@oxfmt/[email protected]", "", { "os": "darwin", "cpu": "x64" }, "sha512-G5melRC3IiNEgXlNKJhSExJYQGAQIppHFp2cl/wFrqQcEwmc4k4TkrHaFTR29SDudjoW5AZq5cXPGJDqtlH/Yg=="],
+ "@oxfmt/binding-darwin-x64": ["@oxfmt/[email protected]", "", { "os": "darwin", "cpu": "x64" }, "sha512-vw745q870oTd6J517O24asoX4/E+eK0nxYIFoedSLqgJ+nI5En7+ZS82iZSHZ69zevQrnOXiyHP01dA+t8xD8w=="],
- "@oxfmt/binding-freebsd-x64": ["@oxfmt/[email protected]", "", { "os": "freebsd", "cpu": "x64" }, "sha512-Hb1P5yKb9aiH2VSbm3Ml37mKjzqpunYet9krw/YK7V5JsAUJDQlju3Yk/dEqQvJD/usBvpi01Vbk9ZDNo0a8Cw=="],
+ "@oxfmt/binding-freebsd-x64": ["@oxfmt/[email protected]", "", { "os": "freebsd", "cpu": "x64" }, "sha512-NO14EgSM9dFkcg+MfGPxvsKqXYs9LKaxPrOKXpv1R0rLokGGFDcCq6dBMq18dE4wlpFOovX0UZY2uh1P30O7QA=="],
- "@oxfmt/binding-linux-arm-gnueabihf": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "arm" }, "sha512-r7n4JEO+rkp0kaP3ia+0hv6FzK3PQIHqaF0jxGuhjJ4JHVX9ZVRbgpDVDW+OloyuJsRWpWFDKIHe8gfl1p/4mQ=="],
+ "@oxfmt/binding-linux-arm-gnueabihf": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "arm" }, "sha512-139OEhHarj9CYoJ/i9gXlPv4KLBGtLj2toseOWYFf09QwlhklaZk+wW3aOvlqoeZtuayvkoSNVWra7WJ21s3VQ=="],
- "@oxfmt/binding-linux-arm-musleabihf": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "arm" }, "sha512-EEynTF4nakgqDhVCmd3cHQCmTcw0BM8ea6eHigTdtZ3yxdsBNv+dYkt+OpDDZx1n/jzjI7J+Ht7IgkqFWuJUxQ=="],
+ "@oxfmt/binding-linux-arm-musleabihf": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "arm" }, "sha512-GEh2PY3IWTE0M24eNhTduountANSbWyDmMnzFSQE/nGg/bjPugbUgiGuFu+xdqcQSd/HKSwH80/F2yVVD48yhA=="],
- "@oxfmt/binding-linux-arm64-gnu": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "arm64" }, "sha512-7VgJIrywCwR/G6YMv+HqsccUt8Z4q4mbM2xR+Ry/6AeMk4pCmZ9aO4Srd3mmrvVu8PsCZGP7oHEHSXWTmyloaQ=="],
+ "@oxfmt/binding-linux-arm64-gnu": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "arm64" }, "sha512-En5i+UJmZSPxuSf47F2Hl5YOzKB0bicQLnGQkeTCMQ35cWLtbrSwACJKfiLqRZrk05DwSnsJkhBRaM3OURtIaA=="],
- "@oxfmt/binding-linux-arm64-musl": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "arm64" }, "sha512-AOCaminv/+fhinUXKvtPZT4POhNYmN9GfvaJdgJa1AbvuzClWGwDIXfJg8eu3sYaVL2B68jNvmS7LZGAYa0z3A=="],
+ "@oxfmt/binding-linux-arm64-musl": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "arm64" }, "sha512-WWOoV5W9Im3flVwOVrWn/2DUlOF8v5vcCip+kcNuaMpulRCh6nzzt1Su2vcL2F908YJIXNV3HvegbBHuyLwKHg=="],
- "@oxfmt/binding-linux-ppc64-gnu": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "ppc64" }, "sha512-0SHxuaa4QRLM7jfsd8wc6L7KdLg+FR6/ah9k1eAvcrK/Sni7oxchMvsjwUsyakVAta+wBVDkjJTr7/RFU0uUGA=="],
+ "@oxfmt/binding-linux-ppc64-gnu": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "ppc64" }, "sha512-YUouneIqW+5n7aE8xx/zeZ6/utr/KH7oykcGoFyd8Uz8uh591T1oKlnoWA3BsRq/ZR42oY1w4MUYvS/0e/MQOA=="],
- "@oxfmt/binding-linux-riscv64-gnu": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "none" }, "sha512-UNKj1+rPorTK6uSlQzpN96vDNbHbL/63zcpznweo0ImQdkDdtQCS7G3puxX++2cESPKT2B56Y6DICUmFTJde5A=="],
+ "@oxfmt/binding-linux-riscv64-gnu": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "none" }, "sha512-iEnMf21S5aGVa4hViDGY8sAQ/AHyCu2JPyrQF8P06wtHhSkD1YJBeT4m/KiGewgf7+a5XCYSCRIPcRQa1xwEoQ=="],
- "@oxfmt/binding-linux-riscv64-musl": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "none" }, "sha512-5cVXKj7a0mWu7GU6/bZgNyv+KtLgM5S9cnT26Os2/abj8pK/bklHsbdYEHGLLniGjNYkyzwem8mZciPCaDtilw=="],
+ "@oxfmt/binding-linux-riscv64-musl": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "none" }, "sha512-91Sdniaj20fQzyMeCxMDzTP4c9s4RB8dGQ308xHhDR0n6U7+1Xq7N9klE7mfXq8iV3lRmIGSXi5X23Hn/0XX/g=="],
- "@oxfmt/binding-linux-s390x-gnu": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "s390x" }, "sha512-lNpcMuZvyU1V/CY5h19hac03m4E4PLSYRckcxZOynSPojI4Mwt6nEu9wT/gVf0bclH3xGEWjwqApnSTgjkd2oA=="],
+ "@oxfmt/binding-linux-s390x-gnu": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "s390x" }, "sha512-uUV30M6E+2TKKGMaKiwfeL4RZrviHXlUxsrYJ/jFBb+1EZy+pnFT+hF73eeWdzh5NqPOAnw0iiMAIqjqiLZPFg=="],
- "@oxfmt/binding-linux-x64-gnu": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "x64" }, "sha512-5/Z6pUewQpknXqC4/ykK6Zc6RiteAnPem1Ci7K1RZLVF6w6MMjwHjR4vsjijW4Czidgv7HKeVglGjElADliT9w=="],
+ "@oxfmt/binding-linux-x64-gnu": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "x64" }, "sha512-ivMcX6kNDPhqtbOaBt/ItFlLlTlXNHLgRuNmxP6Na6UuYXRT10llpJcAPbGeRgjjb3Qzv4jwPp3fB0hui50WNQ=="],
- "@oxfmt/binding-linux-x64-musl": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "x64" }, "sha512-uVdG2N/4GEbOeljpQ+xv+NeEwJWJGj0WaxSiSYnoiqIYy3RWrWd3rGUmxWXP1A8+ferNvvwFoDAtvgsDUvBuSw=="],
+ "@oxfmt/binding-linux-x64-musl": ["@oxfmt/[email protected]", "", { "os": "linux", "cpu": "x64" }, "sha512-w+S+fERxYmlZSyZlJK/U292FjyBoH8cCEj21/tYJX6atX5kNSn+HDkhlFQKT2zcMwUW0uAtUL/bOrlwJZwqRdA=="],
- "@oxfmt/binding-openharmony-arm64": ["@oxfmt/[email protected]", "", { "os": "none", "cpu": "arm64" }, "sha512-1LwEZmSDRFKeBaGd/Hgj2xI3SmgGFpF4YtgkSx0sTYdTiZfHWbcFvRIBjL1YiDNdQd9Khda3Ij8CoqBULjskDg=="],
+ "@oxfmt/binding-openharmony-arm64": ["@oxfmt/[email protected]", "", { "os": "none", "cpu": "arm64" }, "sha512-Zlom1Xkx257R8bk4ZI4zJsrGno2opknz1+5v5baka3nn4FPyvNSdh8JUL4CdN1S1OWRMvJ9UJQ+RfIqGGCUEfA=="],
- "@oxfmt/binding-win32-arm64-msvc": ["@oxfmt/[email protected]", "", { "os": "win32", "cpu": "arm64" }, "sha512-5tplmkeMXmza3PUptJQy25wCPJbIAg5g5r7zs+DI+7eEYOqbY+7roMdSH2zbDvv1Pb4oTDJ6NWYx9IRxBxXwhQ=="],
+ "@oxfmt/binding-win32-arm64-msvc": ["@oxfmt/[email protected]", "", { "os": "win32", "cpu": "arm64" }, "sha512-FQgPW5R17vzt7cgrJ8eG/dqX00o2xHsqFeLfw4xzA9FRHpN/DjFo9YDonvIIXGxiEuS9F/jZPnGO+KHNKuCo4Q=="],
- "@oxfmt/binding-win32-ia32-msvc": ["@oxfmt/[email protected]", "", { "os": "win32", "cpu": "ia32" }, "sha512-T2LWgh3Vx4PYDjtcqRRGDyCsVxeF/VE5xTpUQiCm65bNLwMpRnJYj6Hlkf6qRM3+jxCIOqHD1kVHNQCCiAVMNQ=="],
+ "@oxfmt/binding-win32-ia32-msvc": ["@oxfmt/[email protected]", "", { "os": "win32", "cpu": "ia32" }, "sha512-ZfZublNhZ+XBndMiXhkiLlPE+XyGRDa0CweeTL6t1fZypfCh1LTg7e5CvnOeTBunq15MskOcRempumSPGAaCQA=="],
- "@oxfmt/binding-win32-x64-msvc": ["@oxfmt/[email protected]", "", { "os": "win32", "cpu": "x64" }, "sha512-L06SVp7Hu1EQTkh3+PHnKK26dxVbEpOG2InMwCooTE4MA437b5FgfTleCCiGn4ykISHnHU6UHaIg7SiMO7rBHA=="],
+ "@oxfmt/binding-win32-x64-msvc": ["@oxfmt/[email protected]", "", { "os": "win32", "cpu": "x64" }, "sha512-HlIZEn+WzLQL0DszNzldiRl/DPRCX5R0Vkt6qeUPR1YHwy52hZZo4x6HoTOVmKRP2wUiwPGtKsihNY/f8KRaBg=="],
- "@oxlint/binding-android-arm-eabi": ["@oxlint/[email protected]", "", { "os": "android", "cpu": "arm" }, "sha512-63Ozq6yQn79B2UxjSZ9huJmI+3/ZPI7hxAe6TyFh9oQL447KnIMoffOdtog+h4KILbQr61ZRvFPW5gYbAUgpgQ=="],
+ "@oxlint/binding-android-arm-eabi": ["@oxlint/[email protected]", "", { "os": "android", "cpu": "arm" }, "sha512-q2KO/Zso9UT+OMn0NF9ywn4E4t0MI3yxiDhNyhsQ7DyQJrC4FhFE4TXOi4bktFnOWXTMds8qZSbpv2XwRaNOBg=="],
- "@oxlint/binding-android-arm64": ["@oxlint/[email protected]", "", { "os": "android", "cpu": "arm64" }, "sha512-JdLGp44phbf/0Du236Dt06yfK6Qw8xhTFeDvHJdjk167r59OfP5+BL31mgMYVnswS8e+j6tFuccSY0QDC/lgCw=="],
+ "@oxlint/binding-android-arm64": ["@oxlint/[email protected]", "", { "os": "android", "cpu": "arm64" }, "sha512-SxLN3ALjoT9NNdvpjEevGeHvfzTAFrF0NBYB5tzK7/GtCKMze3j1e/m/X2ozqGj2U9hfGG/dg/OG8vpVK4PiDA=="],
- "@oxlint/binding-darwin-arm64": ["@oxlint/[email protected]", "", { "os": "darwin", "cpu": "arm64" }, "sha512-h+vkOr4ik6KLFCXdtNVEj9xfnXfNUpPoF95QjORdhiSpLqQePzcN81kGHjAgsw1/G7WBWK5KsFKbI2UK8EyzMA=="],
+ "@oxlint/binding-darwin-arm64": ["@oxlint/[email protected]", "", { "os": "darwin", "cpu": "arm64" }, "sha512-Y/Sup/J4f0f9UGsSd/xyCNTeWL+gepO63GBdEDAfue9nBsnk9zMmnIXx1O6b1V8C90vB5nucYNZ0pbMXAp8zJA=="],
- "@oxlint/binding-darwin-x64": ["@oxlint/[email protected]", "", { "os": "darwin", "cpu": "x64" }, "sha512-wbglRuH5eKsp68keTOOeKDHU1astWSRYFsh2bwKQTAPb44rD6kVgzYI1ZrAJRTc6/7PH55ZXFCQx0rQXrj0UdA=="],
+ "@oxlint/binding-darwin-x64": ["@oxlint/[email protected]", "", { "os": "darwin", "cpu": "x64" }, "sha512-ApOSNC04ynpDTwvBD+//0wyfODRSbEzvRoKpX8teffmc27z8AockwSNeMXGJXn5KP85eahDgR/2llICWLkzcnw=="],
- "@oxlint/binding-freebsd-x64": ["@oxlint/[email protected]", "", { "os": "freebsd", "cpu": "x64" }, "sha512-yTQ7QSOlNfQsrfbeVweTiegQJLKPjUKgXp/JpGiFzza6Hp29S9LAoH77TM3L0RjK39iQRZB5lk5iq2t241Mfag=="],
+ "@oxlint/binding-freebsd-x64": ["@oxlint/[email protected]", "", { "os": "freebsd", "cpu": "x64" }, "sha512-bNrVrCOA/kHky3Tu79IXWXe5bhIgLXfUuUEDHlAGOHUk96MkvDZ1ecaQF19rwstrnaqfP1o9nBTqzIr9+ZHkUg=="],
- "@oxlint/binding-linux-arm-gnueabihf": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "arm" }, "sha512-QYJGy0E2Tv8GAayyBIeHFTwQb8u+zrvrOYrifuDGZDr9ad5XsDhiMScpZymhS9fwWjqrgG2ZUwT4vpY0ISqp4Q=="],
+ "@oxlint/binding-linux-arm-gnueabihf": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "arm" }, "sha512-NUrzOJ1s/EqsVvfn2L/1D8Wro2LPIZUbihL8kOJLh5fEdGEN3rdOGUYq3HwnUIL8sjpoP+4N6RaGrgmMJnaMPw=="],
- "@oxlint/binding-linux-arm-musleabihf": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "arm" }, "sha512-oQo83CafLT8p3qh0AGkN0tSUIh5Hm63mfDsoqUmBQUBnSZLNBAbyRaAftfGY5a0cKZ34dG6jAtMYgcY5kZqt5g=="],
+ "@oxlint/binding-linux-arm-musleabihf": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "arm" }, "sha512-UJXrAT3E/RWkEqXLIs2ehETja1qfgkPb+5gwLIIS+o/6cf+grHvoOXTa5997a/YNQfcJS0DRBTOfZt95cvOI1g=="],
- "@oxlint/binding-linux-arm64-gnu": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "arm64" }, "sha512-EM6wy5c2UM12qPb7iDJDBAZvHyfEJGH6iosFgaFiaRZpirz99//EHcUyD2KAskz07x1TULuiAtsojf4nvWWksw=="],
+ "@oxlint/binding-linux-arm64-gnu": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "arm64" }, "sha512-lK40QLjI0HxigO7CjDDshEtfYIeiYS0020v5BHFPqN4uuQBQxd2K9LNom2dW15o9F1937quSCRVp4ZsVhdbYdg=="],
- "@oxlint/binding-linux-arm64-musl": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "arm64" }, "sha512-vCiUQb9ZNzZalxYRU4mlZwUqT5f+c7Sk9DwUOplWHtik3Dl4r3DVnUB5tBeRuTAUe6MUVCTylNW7TkkN8J1oTw=="],
+ "@oxlint/binding-linux-arm64-musl": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "arm64" }, "sha512-c2zbdBwGKreHXwRx3gWBuFGJxLhxgsg6YlZ+3H+RgRusU/UEV9jNwJ3HGYK+nRo0LvBa7mt6Kj86xoVotUo8cw=="],
- "@oxlint/binding-linux-ppc64-gnu": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "ppc64" }, "sha512-T0AzE89yjTYmOhB/hb6i+rWTpe2mYt+xiguBhs9DEPIJTZ1H8OYqsE4VxJHHw2Wlq8Mztwvp2P8NV87ZOuhPRw=="],
+ "@oxlint/binding-linux-ppc64-gnu": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "ppc64" }, "sha512-tlt/Hy8lZ97/lCPmCgw/B3k/mwh+BzaIPbPkldZEly7TwLmx0xe2CQcaW2g/rR0dOgS9JNGCZsMEqLhUNMGvaw=="],
- "@oxlint/binding-linux-riscv64-gnu": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "none" }, "sha512-s49r0K2f5Oc7fJoMUO1WkdhYK5Ob79uYRwWQ/XNoGysaOq3RUGmGUbHSuXiyydTTA+P9xEdDEnnlZVXiaSJhlw=="],
+ "@oxlint/binding-linux-riscv64-gnu": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "none" }, "sha512-3tNR9Xey82X0zKuY1d8hJ6Rc9gwRDurmqGLnQZa5xqOXy8/YyiqFXjAtugkKLY82obOlpK1eSiDRlgcNPuxtIg=="],
- "@oxlint/binding-linux-riscv64-musl": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "none" }, "sha512-JIa2tYR8ayb3nvcMSZzn4O/xTy13S7MHllu7cBD+B5qDQXXSRy/Vwk1aNpiTjZmBJGwKCHOy/NvnY5Xcwg27uQ=="],
+ "@oxlint/binding-linux-riscv64-musl": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "none" }, "sha512-wbGRd5PqCcjkJFHhZuZ2OBSUQY9czlQsoA/cQQB9JK/L9mC5MQgGoKAh+xd8QjA5V+0D3j+Qd1lAWn1I8zlelA=="],
- "@oxlint/binding-linux-s390x-gnu": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "s390x" }, "sha512-XwXMyBsAuWZkhZU74UNcfQKr1CJJAKnS80dZzOnjl6A4fKcf91IjFgidegQugMHA5SN8ulwDXEDV1yPntjXgQQ=="],
+ "@oxlint/binding-linux-s390x-gnu": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "s390x" }, "sha512-3Sn0kSrE4DPZCWV/8o+n4x3aFZxI9ulMnkYlwCbJ8eUVkwRK2IerohE/A/z3SNbCwoPFOCJmGE5Avrq0rrvdvQ=="],
- "@oxlint/binding-linux-x64-gnu": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "x64" }, "sha512-C1WjukSyMnr66b+w1/tV8RFVv6d9v0MzDf4p9IxVXknqgmTHBgZh1pccN1eHzFr0b9Tbb3OXoPsAAdAuHAQfeA=="],
+ "@oxlint/binding-linux-x64-gnu": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "x64" }, "sha512-JY2pxxYfB62bAGfejljVCqc44etItehPuAyaeSAdMuEMtwNA00ggMnS66lC1oIhos6oOXUkuU6mZ9bpFh3BqWg=="],
- "@oxlint/binding-linux-x64-musl": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "x64" }, "sha512-ap6KLmvC38c6MdYzsIh25cXQupqYvjd37tMNftzrX1DCtkX1Gcf2B+S2B17dD4lKa5c5gJog7DQJyDo82PBKyw=="],
+ "@oxlint/binding-linux-x64-musl": ["@oxlint/[email protected]", "", { "os": "linux", "cpu": "x64" }, "sha512-5k74vZ6qJBjBHEOlBk9B/iv68Yu0F1Afw/vvT2ar6OGCqEeXLaSjXz2n/IPCbhLG22UoKoYEJTzpYraRdcp6PA=="],
- "@oxlint/binding-openharmony-arm64": ["@oxlint/[email protected]", "", { "os": "none", "cpu": "arm64" }, "sha512-996Q9w/2GLwNyy5Djs39Jt09aFlr4b7TPhoj7dvo5FHmcyiF68nPz73HNc5UAYKIYeK+M9JzPQ4Wg1r1dXvP6A=="],
+ "@oxlint/binding-openharmony-arm64": ["@oxlint/[email protected]", "", { "os": "none", "cpu": "arm64" }, "sha512-GbAl5qt5TCkPLXTaIISZJnugrcBhra6rodcXc9jYt620UtdsTt71NlNmJmm0frxzFpd54x/G+MkitEJA8I/BoA=="],
- "@oxlint/binding-win32-arm64-msvc": ["@oxlint/[email protected]", "", { "os": "win32", "cpu": "arm64" }, "sha512-9uTCHYTknNkug5+46ViSoUMuOuGWK61D87QnblU6CXaWKJC630Z7dbls61vdHxya/cR2ycQZ6Mm6ckgF6Cag5g=="],
+ "@oxlint/binding-win32-arm64-msvc": ["@oxlint/[email protected]", "", { "os": "win32", "cpu": "arm64" }, "sha512-kjmws5MK0et2swk4ND85D7NVQyDHw162i6whtZDLUA/lo6FQyBZDcmMRCMcVZcNrAhIaftVb00x9ChGDOjjNJA=="],
- "@oxlint/binding-win32-ia32-msvc": ["@oxlint/[email protected]", "", { "os": "win32", "cpu": "ia32" }, "sha512-a47ga7EKVfMU8rO0i1m3mGwfNRj3FhQyt1d/ukO/E9u1HHF3KpR3tyb3VhuJBGEVPF8uR4exSyHUyx7xbE+uxw=="],
+ "@oxlint/binding-win32-ia32-msvc": ["@oxlint/[email protected]", "", { "os": "win32", "cpu": "ia32" }, "sha512-eSsIJx9n4yxvOqYTZyPEMyEXRmE60XH7xGAU7i0Qbsn1lf6Za3CWJ9aRd82oSFKXaxhp+sA6/yMJVRIpLpna6A=="],
- "@oxlint/binding-win32-x64-msvc": ["@oxlint/[email protected]", "", { "os": "win32", "cpu": "x64" }, "sha512-az4dzifCRzES1crwu6pDFXzJTIanrdxXFL9TbiJWrx9LKcuacbR83aPvxDA1LKLNswi/RoqitC7Afcflhu3zBQ=="],
+ "@oxlint/binding-win32-x64-msvc": ["@oxlint/[email protected]", "", { "os": "win32", "cpu": "x64" }, "sha512-pBebIPUpKKhWrhSMWhy8TdAZBewiXnfxmaAGxhzxM1068GagqFaTwgKlU6e+UyJ2sPR+VoHouhXuGJkQjsrDvA=="],
"@types/bun": ["@types/[email protected]", "", { "dependencies": { "bun-types": "1.4.2" } }, "sha512-GimotNn7+ZV0uVArItBbriZsR1oNf0+WTzPkdcFrzShI7k2norL0uzEaJT8T33dWr7O/c9ZDuAFQrctKCi72oQ=="],
@@ -139,9 +139,9 @@
"graceful-fs": ["[email protected]", "", {}, "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ=="],
- "oxfmt": ["[email protected]", "", { "dependencies": { "tinypool": "2.2.0" }, "optionalDependencies": { "@oxfmt/binding-android-arm-eabi": "0.71.0", "@oxfmt/binding-android-arm64": "0.71.0", "@oxfmt/binding-darwin-arm64": "0.71.0", "@oxfmt/binding-darwin-x64": "0.71.0", "@oxfmt/binding-freebsd-x64": "0.71.0", "@oxfmt/binding-linux-arm-gnueabihf": "0.71.0", "@oxfmt/binding-linux-arm-musleabihf": "0.71.0", "@oxfmt/binding-linux-arm64-gnu": "0.71.0", "@oxfmt/binding-linux-arm64-musl": "0.71.0", "@oxfmt/binding-linux-ppc64-gnu": "0.71.0", "@oxfmt/binding-linux-riscv64-gnu": "0.71.0", "@oxfmt/binding-linux-riscv64-musl": "0.71.0", "@oxfmt/binding-linux-s390x-gnu": "0.71.0", "@oxfmt/binding-linux-x64-gnu": "0.71.0", "@oxfmt/binding-linux-x64-musl": "0.71.0", "@oxfmt/binding-openharmony-arm64": "0.71.0", "@oxfmt/binding-win32-arm64-msvc": "0.71.0", "@oxfmt/binding-win32-ia32-msvc": "0.71.0", "@oxfmt/binding-win32-x64-msvc": "0.71.0" }, "peerDependencies": { "svelte": "^5.0.0", "vite-plus": "*" }, "optionalPeers": ["svelte", "vite-plus"], "bin": { "oxfmt": "bin/oxfmt" } }, "sha512-lUPUl0d/+Io5pDrsPXWs6rB4N/bpB78oj9CTDpnbulfDz+0r3XXcHPlQ7kRPJ2GjIT4nX+/mcqunOeP9BvsEtg=="],
+ "oxfmt": ["[email protected]", "", { "dependencies": { "tinypool": "2.1.2" }, "optionalDependencies": { "@oxfmt/binding-android-arm-eabi": "0.70.0", "@oxfmt/binding-android-arm64": "0.70.0", "@oxfmt/binding-darwin-arm64": "0.70.0", "@oxfmt/binding-darwin-x64": "0.70.0", "@oxfmt/binding-freebsd-x64": "0.70.0", "@oxfmt/binding-linux-arm-gnueabihf": "0.70.0", "@oxfmt/binding-linux-arm-musleabihf": "0.70.0", "@oxfmt/binding-linux-arm64-gnu": "0.70.0", "@oxfmt/binding-linux-arm64-musl": "0.70.0", "@oxfmt/binding-linux-ppc64-gnu": "0.70.0", "@oxfmt/binding-linux-riscv64-gnu": "0.70.0", "@oxfmt/binding-linux-riscv64-musl": "0.70.0", "@oxfmt/binding-linux-s390x-gnu": "0.70.0", "@oxfmt/binding-linux-x64-gnu": "0.70.0", "@oxfmt/binding-linux-x64-musl": "0.70.0", "@oxfmt/binding-openharmony-arm64": "0.70.0", "@oxfmt/binding-win32-arm64-msvc": "0.70.0", "@oxfmt/binding-win32-ia32-msvc": "0.70.0", "@oxfmt/binding-win32-x64-msvc": "0.70.0" }, "peerDependencies": { "svelte": "^5.0.0", "vite-plus": "*" }, "optionalPeers": ["svelte", "vite-plus"], "bin": { "oxfmt": "bin/oxfmt" } }, "sha512-IsHxZ4y0wQLLMhnrJblBJgZsLDzfULrJnAw5j/QqsTlMa/m3AqsbToi+W71uhBGaqlqq/PbbjvHc09TJwdv3Tw=="],
- "oxlint": ["[email protected]", "", { "optionalDependencies": { "@oxlint/binding-android-arm-eabi": "1.86.0", "@oxlint/binding-android-arm64": "1.86.0", "@oxlint/binding-darwin-arm64": "1.86.0", "@oxlint/binding-darwin-x64": "1.86.0", "@oxlint/binding-freebsd-x64": "1.86.0", "@oxlint/binding-linux-arm-gnueabihf": "1.86.0", "@oxlint/binding-linux-arm-musleabihf": "1.86.0", "@oxlint/binding-linux-arm64-gnu": "1.86.0", "@oxlint/binding-linux-arm64-musl": "1.86.0", "@oxlint/binding-linux-ppc64-gnu": "1.86.0", "@oxlint/binding-linux-riscv64-gnu": "1.86.0", "@oxlint/binding-linux-riscv64-musl": "1.86.0", "@oxlint/binding-linux-s390x-gnu": "1.86.0", "@oxlint/binding-linux-x64-gnu": "1.86.0", "@oxlint/binding-linux-x64-musl": "1.86.0", "@oxlint/binding-openharmony-arm64": "1.86.0", "@oxlint/binding-win32-arm64-msvc": "1.86.0", "@oxlint/binding-win32-ia32-msvc": "1.86.0", "@oxlint/binding-win32-x64-msvc": "1.86.0" }, "peerDependencies": { "oxlint-tsgolint": ">=7.0.2003", "vite-plus": "*" }, "optionalPeers": ["oxlint-tsgolint", "vite-plus"], "bin": { "oxlint": "bin/oxlint" } }, "sha512-og0lhgvZfgGF//gOOmZXvtr+GmBbAGEnbEhv/QUg7UW2Wi4wHMnJbnMD+zuHgPdJxdklfgpPupdlAaAySyxrZg=="],
+ "oxlint": ["[email protected]", "", { "optionalDependencies": { "@oxlint/binding-android-arm-eabi": "1.85.0", "@oxlint/binding-android-arm64": "1.85.0", "@oxlint/binding-darwin-arm64": "1.85.0", "@oxlint/binding-darwin-x64": "1.85.0", "@oxlint/binding-freebsd-x64": "1.85.0", "@oxlint/binding-linux-arm-gnueabihf": "1.85.0", "@oxlint/binding-linux-arm-musleabihf": "1.85.0", "@oxlint/binding-linux-arm64-gnu": "1.85.0", "@oxlint/binding-linux-arm64-musl": "1.85.0", "@oxlint/binding-linux-ppc64-gnu": "1.85.0", "@oxlint/binding-linux-riscv64-gnu": "1.85.0", "@oxlint/binding-linux-riscv64-musl": "1.85.0", "@oxlint/binding-linux-s390x-gnu": "1.85.0", "@oxlint/binding-linux-x64-gnu": "1.85.0", "@oxlint/binding-linux-x64-musl": "1.85.0", "@oxlint/binding-openharmony-arm64": "1.85.0", "@oxlint/binding-win32-arm64-msvc": "1.85.0", "@oxlint/binding-win32-ia32-msvc": "1.85.0", "@oxlint/binding-win32-x64-msvc": "1.85.0" }, "peerDependencies": { "oxlint-tsgolint": ">=7.0.2001", "vite-plus": "*" }, "optionalPeers": ["oxlint-tsgolint", "vite-plus"], "bin": { "oxlint": "bin/oxlint" } }, "sha512-bc26s97nuvPj1ViyPsqmKecVkUWFMEdtayO8MaQ6oiLfs1pj94cQlZZhrh4BPNlr9HQosjhIlwgZKsfcwmcNgg=="],
"proper-lockfile": ["[email protected]", "", { "dependencies": { "graceful-fs": "^4.2.4", "retry": "^0.12.0", "signal-exit": "^3.0.2" } }, "sha512-TjNPblN4BwAWMXU8s9AEz4JmQxnD1NNL7bNOY/AKUzyamc379FWASUhc/K1pL2noVb+XmZKLL68cjzLsiOAMaA=="],
@@ -163,7 +163,7 @@
"tiny-segmenter": ["[email protected]", "", {}, "sha512-m+aTJQ/CUBKurLaJRpLmJiwcL+Gpkzft5ZYnRU9AkuO45Y/k/2iJmuLEbN1XLrq6N3kDVyIUCCeqRzQx0feBag=="],
- "tinypool": ["[email protected]", "", {}, "sha512-jBrmx4lYmaC9k/mgPbylxs7kBUxHtD8256up+HjLaDFfXScKJQyil+SWXSvhAtT7XHo+yTVlpB81PGmHP8oLSQ=="],
+ "tinypool": ["[email protected]", "", {}, "sha512-9YodfrxS9g9IbFr/KOjE5bAeJ0p61n3bW6mqvy0jtoeKd1kTW1Cxm0oulm6KX2lyM9Gl6WIe8nEbY7LWv5ZJww=="],
"typescript": ["[email protected]", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],
diff --git a/docs/brainstorm/near-duplicate-defense/cli-output/claude.md b/docs/brainstorm/near-duplicate-defense/cli-output/claude.md
new file mode 100644
index 00000000..2096b9a2
--- /dev/null
+++ b/docs/brainstorm/near-duplicate-defense/cli-output/claude.md
@@ -0,0 +1,147 @@
+# Near-duplicate defense: architectural variants
+
+I read all four premise reports. Every variant below is built on their corrections:
+
+- The forget-side scan hooks the dream retire plan (`dream.ts:297`) and the `reject` verb. It does not hook hygiene `forget`.
+- The v1.64.0 receipt hint is extended. It is not rebuilt.
+- Date grounding means a timestamp on `MinedTurn` plus a prompt instruction to write ISO dates.
+- Stage timings ride on `mcp_route_latency` and use stage names the write tools actually have.
+
+Two facts hold for every variant:
+
+- **The shipped write hint leaks.** `lintWrittenPages` has no reach predicate (`page-lint.ts:583`, `notes-tools.ts:487`), so a receipt can name a page the caller can't read. Every variant fixes this first, unconditionally.
+- **The timing premise is slightly off.** It says `brain_feedback` has no similarity stage. But `adviseIncomingFeedback` (`write-advisory.ts:163`) already runs a Jaccard pass on that route. This wave adds more lookups, so a `near_duplicate_lookup` stage becomes real on several routes. That is exactly the cost the timing card should measure.
+
+---
+
+## Variant A: "One lexical kernel, zero new spend"
+
+**Approach.** One new pure module, `src/core/brain/near-duplicate.ts`, built on `tokenise`/`jaccard`. It takes:
+- a probe;
+- a candidate source, injected as a small interface: directory siblings, a cross-directory search-index pull, active preferences, or the other items in the same payload;
+- a reach predicate, applied before scoring and before counting;
+- a cap.
+
+It returns `{ candidates: [{ ref, score, method: "lexical" }], census: { compared, capped } }`. All the thresholds sit in one named table: write hint 0.8, retire sibling 0.7, in-payload 0.8 for items with the same signal.
+
+Four callers use it:
+- **The retire-sibling projection.** It is computed after `planAutoRetires`, checked against the retires that are gated at apply time (`gatedRetireSlugs`), and shown as `retire_siblings` in `brain_review_candidates`. Retiring a sibling is accepted through the existing `apply_evidence outdated` or `reject`.
+- **The receipt lint.** It gets the reach fix, plus an optional cross-directory pull from the existing keyword (BM25) search index. That pull sits behind a config key that is off by default.
+- **The extract-signals check.** One `crossItem` check is added to the existing registration.
+- **`adviseIncomingFeedback`.** It is refactored onto the same core.
+
+Stage timings come from extending the decision-latency storage scope (an AsyncLocalStorage) into one shared route scope. Stage names are a closed allowlist that `emitMcpRouteLatency` enforces. They are written only to the continuity record, never to the tool response.
+
+**Pros**
+- No embedding spend and no egress.
+- Fully deterministic, and the dream dry run stays write-free.
+- One threshold table replaces three copies that are drifting apart (0.6, 0.7, 0.8).
+- It fits the budget, at roughly 40–45 files.
+- Low risk on Windows.
+
+**Cons**
+- Real paraphrases often score below a 0.8 Jaccard. Cross-topic comparison of short principles is noisy, which needs a minimum token count.
+- The "embeddings" half of the write-hint card stays unbuilt.
+
+**Cuts or defers**
+- The semantic tier.
+- Model-generated rewordings. "Reword" becomes "show the pair, and the operator rewrites it with the existing verbs."
+- Hook capture timing.
+- Acceptance-inside-a-proposal in extract-signals.
+
+---
+
+## Variant B: "Lexical kernel plus a vector tier that reuses paid embeddings" (two tiers)
+
+**Approach.** Same module, contract and callers as A. The difference is a second tier, `method: "embedding"`. It only does vector-to-vector cosine over vectors already stored in the search index, so it never calls `provider.embed`.
+
+- **Forget side.** The retiring preference and its siblings are both already indexed, so the tier costs nothing at the margin and runs whenever vectors exist.
+- **Write side.** The new body is not indexed yet. The tier is either skipped and disclosed as `semantic: not_indexed`, or, behind a key that is off by default, it embeds the body through `prepareQueryEmbed` with full spend reporting.
+
+The orphan `detectSemanticDedup` is reworked onto the stored-vector reader, so it no longer calls embed directly without a spend gate.
+
+**Pros**
+- Delivers the residual value the write-hint card actually has: catching paraphrases across directories and topics.
+- On the forget side it does so with no new spend.
+- Respects the v1.72.0/v1.73.0 spend gates by construction.
+- The `method` label in the receipt and in review candidates tells the operator which tier fired.
+
+**Cons**
+- It needs a reader that fetches stored vectors by page, which may not exist yet. The reader also has to match the index's chunking (page vector vs. chunk max).
+- Stored vectors go stale between reindexes.
+- About 50–55 files, at the top of the budget.
+- Promoting the write-side embed to on-by-default needs a labelled pair evaluation that this wave doesn't build.
+
+**Cuts or defers**
+- Write-time embedding stays opt-in and unevaluated.
+- Model rewordings.
+- Hook timing.
+
+---
+
+## Variant C: "Out-of-band candidate ledger"
+
+**Approach.** The write path keeps only the reach-fixed lexical lint for directory siblings. Every wider lookup moves off the hot path: cross-directory, cross-type, embedding-based and retire-sibling scans all run in the dream or hygiene pass after reindex. Results go to a persisted `near_duplicate` candidate ledger. `brain_review_candidates` and the next receipt that touches the page show the pending pairs.
+
+**Pros**
+- No latency added to the synchronous write tools.
+- The embedding tier always uses paid vectors.
+- One review queue covers the write side and the forget side.
+
+**Cons**
+- The hint arrives late. The writer who created the duplicate never hears about it in the same turn, which is what the write-hint card was for.
+- It adds a persisted store with its own reach filtering, expiry and staleness rules, which is new kernel state.
+- Extract-signals still needs a separate in-payload check.
+- The stage-timing card measures much less, because the expensive stage leaves the routes being timed.
+
+**Cuts or defers**
+- Same-turn cross-directory hints.
+- Rewordings.
+
+---
+
+## Variant D: "Model-assisted rewording proposals"
+
+**Approach.** Variant A's kernel finds the candidates. Then a new `needs_llm` step, registered in the existing step registry with response-shape and semantic checks, asks the agent for a verdict on each retire/sibling pair (same / related / different) and, for `related`, a proposed rewording. Proposals are written as pending review items. They are applied only when the operator accepts them through a new accept verb. The pair-verdict machinery (`runPairVerdicts`, `advisoryUseActive`) is reused when a decision model is configured.
+
+**Pros**
+- Matches the card's literal "accept-gated rewording proposals".
+- The kernel stays model-free; the model runs only behind the step registry and the egress registry.
+
+**Cons**
+- It adds a new executor that writes, an accept verb, a pending-proposal store and a new response surface, all on top of the kernel. That is about 55–65 files, which breaks the budget.
+- More egress of preference principles.
+- The weakest correctness story of the four. The upstream project measured a rise in wrong merges even from a plain hint.
+
+**Cuts or defers**
+- It has to drop either the stage-timing card or the vector tier to fit.
+
+---
+
+## Cross-cutting choices (the same in A, B and C)
+
+- **Write receipt.** Extend the hint; don't replace it. Keep the code `near-duplicate`, add `method`, and keep the receipt byte-identical when there is no finding. Withheld siblings are dropped before they are counted.
+- **Defaults.** Cross-directory widening and retire-sibling surfacing are behind config keys that are off by default. Only the reach fix is on unconditionally.
+- **Stage names.** A closed set:
+ - `validate`
+ - `idempotency_lookup`
+ - `near_duplicate_lookup`
+ - `document_write`
+ - `idempotency_remember`
+ - `log_append`
+ - `preference_write`
+ - `write_receipt`
+ - `lint`
+
+ Timing uses separate sync and async helpers with `try/finally`. Stages are rounded to 0.1 ms. The docs list which routes emit stages, so a missing array reads as "not instrumented" rather than "zero cost". Stages are never written to the tool response.
+- **Extract-signals.** Turn timestamps are rendered verbatim, and the line drops the `@` marker when the timestamp is empty. Hygiene rules go in as language-neutral prompt text. Two `crossItem` refusals are added: duplicate `topic` in one payload, and principles that are too similar within the same `signal`, using the shared kernel's threshold.
+
+---
+
+## Recommendation: Variant B, built in stages
+
+Build A's kernel and callers first; they carry the whole wave. Then add only the stored-vector tier, active on the forget side and used opt-in on the write side.
+
+B is the only variant that delivers the write-hint card's real remaining value (catching paraphrases across directories and across facts) without breaking the spend gates. It reuses vectors that are already paid for and keeps the write-time embed off by default. It also keeps the hint in the same turn, unlike C, and avoids a new executor that writes, unlike D. The rewording half becomes an operator action on a surfaced pair, which meets "accept-gated" without a model.
+
+If the stored-vector reader turns out not to exist, or not to line up with the index's chunking, B drops back cleanly to A by turning off one tier. Nothing about the contract or the callers changes.
diff --git a/docs/brainstorm/near-duplicate-defense/cli-output/codex.md b/docs/brainstorm/near-duplicate-defense/cli-output/codex.md
new file mode 100644
index 00000000..e69de29b
diff --git a/docs/brainstorm/near-duplicate-defense/cli-output/opencode.md b/docs/brainstorm/near-duplicate-defense/cli-output/opencode.md
new file mode 100644
index 00000000..e69de29b
diff --git a/docs/brainstorm/near-duplicate-defense/cli-output/prompt.md b/docs/brainstorm/near-duplicate-defense/cli-output/prompt.md
new file mode 100644
index 00000000..e0470489
--- /dev/null
+++ b/docs/brainstorm/near-duplicate-defense/cli-output/prompt.md
@@ -0,0 +1,37 @@
+You are brainstorming architectural variants for an Open Second Brain release wave. Do not write code. Do not write a final design. Only produce variants and a recommendation.
+
+# Task
+
+Epic "Near-Duplicate Defense Across the Fact Lifecycle" for Open Second Brain. Four kanban cards ship as one release (about 40-55 changed files):
+
+1. t_acab97de - Surface near-duplicate facts when forgetting, with accept-gated rewording proposals.
+2. t_fda66477 - Hint an existing similar page when a write creates a near-duplicate.
+3. t_b915b9cc - Derivation rules that ground dates, skip chit-chat, and dedupe extracted facts.
+4. t_72e93e18 - Stage-level timing breakdown within the memory write path.
+
+Spine: one near-duplicate candidate lookup built on the existing similarity machinery, wired into every path a fact enters or leaves the vault, plus content-free per-stage timings so the added cost is measured.
+
+# Verified premise reports (read all four files before answering)
+
+- <wave-cache>/design/premise-t_acab97de.md
+- <wave-cache>/design/premise-t_fda66477.md
+- <wave-cache>/design/premise-t_b915b9cc.md
+- <wave-cache>/design/premise-t_72e93e18.md
+
+Key verified corrections to respect:
+- The hygiene apply "forget" action exists but NO detector ever proposes it, so a sibling scan hooked into the apply path would never fire; retirement actually happens through merge/archive/recompile today.
+- A Jaccard-0.8 same-directory near-duplicate hint has shipped in write receipts since v1.64.0; the card's residual value is cross-cutting similarity (embeddings, cross-directory, facts not just pages) and better surfacing, not the hint itself.
+- The extract lane mines taste rules, not facts; date grounding needs the turn timestamp carried into MinedTurn plus an ISO-instruction, reusing the dream pass temporal extraction; no natural-language date parser (kernel language-independence rule).
+- Lifecycle hooks already run capture detached from the agent turn; stage timings on mcp_route_latency cover only MCP tools/call and the CLI bridge, and the card's stage names must be remapped to what the write tools actually do.
+
+# Constraints
+
+- Engineering rules in CLAUDE.md at the repo root (read it): SOLID, KISS, DRY; no silent fallbacks; no hardcoded natural-language phrases; errors surface by name.
+- Nothing model-driven enters the kernel: model-invoking steps live behind the existing step registries.
+- New similarity spend must respect the embedding spend gates from v1.72.0/v1.73.0; reuse already-paid embeddings where possible.
+- Features that propose retire/reword must be accept-gated and off by default (the upstream project measured wrong-merge churn with a default-on write hint).
+- Windows CI must stay green (no POSIX-only assumptions in new tests); test runs use a throwaway HOME.
+
+# Deliverable
+
+3-5 named architectural variants. For each: the approach in two or three sentences, pros, cons, and what it cuts or defers. End with one recommended variant and why in at most five sentences. Consider at least: where the shared near-duplicate lookup lives and what it returns; how the forget-side scan reaches real retirements; whether the existing write-receipt hint is extended or replaced; how stage timings are named and attached without bloating every response.
diff --git a/docs/brainstorm/near-duplicate-defense/cli-output/second-claude.md b/docs/brainstorm/near-duplicate-defense/cli-output/second-claude.md
new file mode 100644
index 00000000..92a1d9d8
--- /dev/null
+++ b/docs/brainstorm/near-duplicate-defense/cli-output/second-claude.md
@@ -0,0 +1,111 @@
+# Adversarial review: Near-Duplicate Defense wave
+
+## 1. Most likely failure modes
+
+**F1. Wrong-merge or wrong-retire churn from cross-bucket lexical matching.** This is the most likely way the wave goes wrong. The obvious shared design is one candidate lookup, cross-directory and cross-topic, maybe with embeddings, wired into retire, write and extract. Cross-bucket Jaccard on short principles is noisy. "use tabs" and "never use tabs" score high, and the premise reports already warn about stop-token overlap. Two parts make it worse:
+- On the retire side, one `outdated` event (`dream-refresh.ts:215-231`, which already ignores pin) produces a sibling list. Operators or agents who mechanically apply `outdated` to every listed sibling cause a cascade.
+- On the extract side, an in-payload Jaccard refusal rejects the whole payload (refusal, not filtering). One false positive therefore drops every valid signal in that run, silently from the operator's point of view.
+
+Upstream measured wrong merges rising from 2.50% to 3.75% with a default-on hint. This vault has more short, polarity-sensitive rules than upstream's pages.
+
+**F2. Visibility leak that grows with the reach of the lookup.** The shipped `near-duplicate` lint already leaks: `lintWrittenPages` has no `readable` predicate (`page-lint.ts:583`, called at `notes-tools.ts:487`). It discloses withheld sibling paths and scores, and leaks through the `candidates_skipped` and `candidates_unreadable` counts. A shared lookup that goes vault-wide (FTS widening, embeddings, cross-topic retire siblings) turns a same-directory leak into a vault-wide one. The leak then appears on four surfaces at once: write receipts, `brain_review_candidates`, the extract envelope and the dream summary. Calling this "recall-quality regression" understates it. It is a reach-invariant break, and it is the most damaging outcome if it ships.
+
+**F3. Spend and latency leaks hidden by the instrumentation meant to measure them.** Two risks:
+- **Spend.** A semantic layer on the write path costs one embed per note write, multiplied by `brain_write_batch`. The tempting reuse, `detectSemanticDedup`, calls `provider.embed` directly with no gate (`dedup.ts:154`).
+- **Timing.** The stage card measures only MCP `tools/call`. Its natural stage set has no `similarity_lookup` stage, because none exists today. If the lookup lands without a stage name in the closed allowlist, the added cost is invisible, which defeats the wave's own spine claim. If stage names are built from input, or the `stages` array is emitted on every record including the gate-off path, records bloat and the content-free claim breaks.
+
+Secondary hazards:
+- **Windows CI.** Asserting positive or ordered sub-millisecond durations, or mtime-ordered candidate fixtures, will flake on Windows timer and filesystem resolution. Locale-formatted timestamps in the transcript line will differ by OS.
+- **Dry-run versus real-run drift.** Siblings of retires that are gated at apply time (`dream-apply.ts:298-311`) get reported as if they were retiring.
+
+## 2. Counter-variants
+
+### C1 "Fix the leak, measure, then defend" (minimal honest scope)
+Ship the reach fix for the existing lint, stage timings and the language-neutral extract rules. Ship the retire-sibling scan as a read-only, lexical, same-signal projection in `brain_review_candidates`. There is no shared vault-wide lookup, no embeddings and no write-side widening. The shared primitive is a small pure helper over `tokenise`/`jaccard`: `nearDuplicateCandidates(subject, pool, {threshold, minTokens, readable})`. It returns `{id, score, method: "lexical"}[]`, filters by the reach predicate before scoring and counting, and is called by the retire projection, the extract cross-item check and the lint.
+
+- **Pros**
+ - Zero new spend or egress.
+ - Closes a real shipped defect.
+ - Every new output is advisory or read-only.
+ - Each new behaviour can be traced through the timings.
+ - Roughly 30-38 files.
+- **Cons**
+ - The cross-directory and paraphrase cases from t_fda66477 stay unsolved.
+ - The card's headline ("hint an existing similar page") ships as "the hint is now safe", not "the hint is smarter".
+- **Cuts and defers**
+ - FTS-backed widening.
+ - The semantic layer and its labelled-pair evaluation.
+ - Decision-model pair verdicts on retire siblings.
+ - Rewording proposals, which are cut outright and not just deferred.
+ - Hook-capture timing.
+ - Assistant-turn acceptance.
+
+### C2 "Shared lookup, lexical lanes only, widening behind a flag"
+Same as C1, plus an index-backed lexical candidate pull (FTS/BM25 top-k, rescored with Jaccard) for the write lint. It is behind a config key that defaults to off, with a time and size bound and a disclosed skip when a bound trips.
+
+- **Pros**
+ - Catches cross-folder duplicates at zero embedding spend.
+ - Proves the shared lookup on a second candidate source.
+- **Cons**
+ - Adds a search dependency on the hot write path.
+ - Index staleness: the newly written page is not indexed yet, and recently written siblings may not be either.
+ - Needs its own cost test.
+ - Ships a default-off feature with no evaluation to justify ever turning it on.
+- **Cuts and defers**
+ - Embeddings.
+ - Pair verdicts.
+ - Rewording.
+ - The promotion evaluation.
+
+### C3 "Retire side only, write side is a bug fix" (narrowest)
+Ship only the retire-sibling projection and the reach fix. Defer t_b915b9cc and t_72e93e18 to the next wave.
+
+- **Pros**
+ - Smallest diff, about 20 files.
+ - A single new behaviour to review.
+- **Cons**
+ - Drops the spine's "measured cost" leg.
+ - The timing card is the cheapest and safest of the four, so cutting it removes risk nobody was carrying.
+ - The date-grounding part of t_b915b9cc is a free win: the downstream ISO extraction already exists.
+- **Cuts and defers**
+ - Two whole cards.
+
+**Attack on the ambitious first-consultant shape** (one embedding-backed lookup across all four paths): it fails all three tests in section 1 at once. It adds new egress and spend on the write path, a vault-wide reach surface and a whole-payload refusal triggered by semantic similarity. It also has no evaluation that can measure any of this, because `rerank-eval-gate.ts` measures recall hit@k and MRR, not duplicate-catch or wrong-hint rates. Reject any variant that turns on embeddings without a labelled-pair evaluation landing in the same PR.
+
+## 3. Minimal honest scope (my recommendation: C1)
+
+**Ship in one PR:**
+1. **t_fda66477, reach fix only.** Add a `readable` predicate to `lintWrittenPages` and `collectNearDuplicateCandidates`, threaded from `noteWriteResult` so it covers all four note-write tools. Withheld siblings are neither scored nor counted. Add a visibility-surface registry row for the note-write lint and a byte-identical clean-receipt test.
+2. **t_acab97de, a read-only `retire_siblings` field.** Compute it after `planAutoRetires` (`dream.ts:297`), and add it to `DreamRunSummary` and `buildReviewCandidates`.
+ - **Triggers:** superseded-by-context, rebutted, quarantine-violated, and user-rejected. Decay retires and merges are excluded.
+ - **Matching:** lexical, threshold ≥0.7, a minimum token count, same `signal` sign only, capped like `DEDUP_CANDIDATE_CAP`.
+ - **Reconciliation:** check against gated retires.
+ - **Acceptance:** only through existing verbs (`outdated` evidence, `reject`). No new executor. Nothing retires automatically.
+3. **t_b915b9cc.** Carry the turn timestamp into `MinedTurn`. Render it verbatim, and leave it out when empty, never substituting `now`. Add the ISO-bound instruction and the prompt-only hygiene text. The cross-item check refuses duplicate `topic` values only. Ship the Jaccard cross-item refusal only if fixtures prove zero false refusals on short same-signal principles; otherwise defer it.
+4. **t_72e93e18.** Use one shared ALS scope that carries both `decision_ms` and `stages`, so there is no second store. Stage names form a closed allowlist enforced in `emitMcpRouteLatency`, and it includes `near_duplicate_lookup` for the lint. The `stages` key appears only when at least one stage was noted, and only when the existing default-off gate is on. Durations use 0.1 ms rounding. Docs list which routes emit stages and state that hooks are out of scope.
+
+**Explicitly left undone, and stated in CHANGELOG and docs:**
+- Rewording proposals. A model-generated reword is a write of new content, so it is not "accept-gated surfacing".
+- Any embedding or semantic layer, and the labelled-pair evaluation it needs.
+- FTS cross-directory widening.
+- Decision-model verdicts on siblings.
+- Hook-capture stage timing.
+- Assistant-turn acceptance.
+- Any change to `computeDedupHash`.
+- The hygiene `forget` anchor, which stays untouched.
+
+## 4. Gates that must hard-fail the build
+
+| Gate | Must fail when |
+|---|---|
+| **Visibility-surface census** (`tests/core/architecture/visibility-surface-census.test.ts`) plus a reach test | Any surface returns a sibling id, path, score or count without a reach predicate. This is the gate that matters most for this wave. |
+| **Egress census** (`egress-census.test.ts`) | Any new network path appears. Under C1 the expected delta is zero, so any delta means the scope slipped. |
+| **Embedding spend counters / `planEmbeddingSpend` path** | Any `provider.embed` call is reachable from a dream, write or extract path without going through `prepareQueryEmbed`. Under C1 that call count must also be zero. Add an assertion that `detectSemanticDedup` still has no production caller. |
+| **Architecture census, specifically write-site and destructive-site** (`write-site-census`, `destructive-site-census`) | A new write or a new caller of `moveToRetired` appears. The retire projection must add neither, since it is accept-gated by construction. |
+| **Route-metrics payload safety** | A stage name outside the allowlist reaches disk. Test with a hostile name on a persisted record. |
+| **Response-check registry** | A second `registerResponseCheck` on the extract surface is registered (it already throws), or a violation code appears outside the closed set. |
+| **`sync-version --check`, `sync-plugin-mirrors:check`, Windows job** | These standing gates are already blocking. |
+
+**The rerank evaluation gate should not be a gate for this wave.** It measures something else, and citing it as cover would be a false assurance. If a later wave adds a semantic hint, its own duplicate-catch/wrong-hint evaluation must hard-fail before the config default can change.
+
+**Why C1:** the wave can close a real shipped reach leak, measure its own cost, and make retirements visible without adding any spend, egress or automatic action. Every ambitious variant depends on a precision claim (cross-bucket or semantic near-duplicate detection) that no gate in the repo can currently test. The near-duplicate helper stays small and reach-aware, so a later semantic lane plugs into it rather than forking a second threshold. Leaving rewording and embeddings undone is honest, since nothing here could catch it if they were wrong.
diff --git a/docs/brainstorm/near-duplicate-defense/design.md b/docs/brainstorm/near-duplicate-defense/design.md
new file mode 100644
index 00000000..1acda6f8
--- /dev/null
+++ b/docs/brainstorm/near-duplicate-defense/design.md
@@ -0,0 +1,449 @@
+# Near-Duplicate Defense Across the Fact Lifecycle - one reach-aware near-duplicate lookup at every point a fact enters or leaves the vault, with the added cost measured per stage
+
+**Status:** draft
+**Author:** claude-devbox-agent (via feature-release-playbook)
+**Audience:** implementation
+**Release:** v1.74.0, one PR on `feat/near-duplicate-defense`
+**Cards:** t_acab97de (forget side), t_fda66477 (write side), t_b915b9cc (extract rules), t_72e93e18 (stage timings)
+
+## Problem statement
+
+A paraphrased duplicate of a fact can enter the vault and outlive the
+original. Three gaps cause this:
+
+1. Nothing looks for paraphrased siblings when a preference is retired.
+2. The write-receipt near-duplicate hint only compares pages in the same
+ directory. It also leaks: it names sibling pages the caller cannot read.
+3. The extract lane hands the agent turns without their timestamps and with
+ no hygiene rules.
+
+Separately, the write path reports one total duration per call. So the cost
+this wave adds, and every cost before it, cannot be attributed to a stage.
+
+## Verified premise corrections
+
+The four premise reports were verified against live source at `3d49820c`.
+Where they disagree with the kanban cards, they win. This design is built on
+the following corrections.
+
+1. **The forget hook goes where retirement actually happens (t_acab97de).**
+ - **The hygiene path the card names never runs.** Hygiene apply `forget`
+ (`src/core/brain/hygiene/apply.ts:123`) exists, but no detector ever
+ proposes it. The detectors only produce `review`, `merge` and
+ `recompile` actions, and usefulness findings are kept out of apply
+ plans (`hygiene/plan.ts:49`). A sibling scan hooked there would never
+ fire, and reviewers must reject that placement.
+ - **Where retirement really happens:**
+ - Dream-pass retires, planned in `dream-refresh.ts:226`, `:265`,
+ `dream-plan-topics.ts:532` and `dream-plan-retires.ts:55`, `:95`, and
+ executed by `moveToRetired` (`dream-apply.ts:318`).
+ - Operator `reject` (`src/cli/brain/verbs/reject.ts:55`).
+ - Merge (`merge.ts:243`). Its sibling is the page that is kept, so merge
+ is out of scope.
+ - **Where the scan hooks:** it runs on the dream retire set after
+ `planAutoRetires` (`dream.ts:297`), is checked against the gated
+ retires, and also runs on `reject`.
+ - **Which retires trigger it:** only those caused by context, namely
+ `superseded-by-context`, `rebutted`, `quarantine-violated` and
+ `user-rejected`. Decay retires (`stale-no-evidence`,
+ `expired-unconfirmed`) mean nobody used the rule, not that the rule is
+ wrong, so they do not trigger it.
+2. **The write hint already exists (t_fda66477).** A Jaccard-0.8,
+ same-directory, same-scope-bucket `near-duplicate` lint has shipped in
+ every note-write receipt since v1.64.0 (commit `4499698c`,
+ `src/core/brain/page-lint.ts:389-414`, `src/mcp/brain/notes-tools.ts:481-488`).
+ - **What the card still adds:** the extension beyond that hint (reaching
+ other directories, scoring by embedding, comparing facts and not just
+ pages) and safer surfacing.
+ - **The shipped hint has a reach leak:**
+ - `lintWrittenPages` takes no reach predicate (`page-lint.ts:583`), and
+ `noteWriteResult` calls it without the caller's reach
+ (`notes-tools.ts:487`).
+ - A receipt can therefore name a withheld sibling and its score, and
+ the `candidates_skipped` and `candidates_unreadable` counts can
+ disclose one.
+ - Fixing this is unconditional and lands first.
+ - **The card's anchors do not hold:**
+ - `page-dedup.ts` is the merge tool for preferences.
+ - `create-note.ts:121` is the policy for an occupied path.
+ - `rerank-eval-gate.ts` measures recall hit@k and MRR, not
+ duplicate-catch or wrong-hint rates, so it cannot gate any default.
+3. **Date grounding needs data, not only a prompt (t_b915b9cc).**
+ - The extract lane mines taste rules, not facts
+ (`extract-signals.ts:422-423`).
+ - `MinedTurn` keeps only `turnId` and `text` (`:186-189`), although
+ `ExpandedRawTurn.timestamp` is available (`session-recall.ts:176`).
+ `MinedTurn` must carry the turn timestamp, and the transcript line must
+ show it.
+ - The extract instruction must require that any time bound is written as
+ an ISO date or interval.
+ - The dream pass already turns ISO tokens in a principle into
+ `valid_from` / `valid_until` (`dream-plan-topics.ts:604-615` through
+ `extractTemporalConstraints`, `temporal-extract.ts:45`), and that
+ temporal extraction is reused unchanged.
+ - No natural-language date parser and no phrase lists are added, because
+ language independence is a kernel rule (`temporal-extract.ts:5-8`,
+ `fact-extract.ts:9-13`).
+ - Acceptance inside a question or proposal cannot be supported. Only user
+ turns reach the prompt (`MINED_TURN_ROLE = "user"`, `:96`), so it is
+ deferred.
+4. **Stage timings have a narrower reach than the card says (t_72e93e18).**
+ - **Where they attach:** `mcp_route_latency` is emitted from one seam,
+ `MCPServer.invokeToolHandler` (`src/mcp/server.ts:280-322`). That seam
+ serves the JSON-RPC `tools/call` path and the CLI `callTool` bridge
+ (`server.ts:255`, `src/cli/main.ts:1113`), so stage timings attach
+ there and only there.
+ - **Hooks are out of scope.** Lifecycle hooks call
+ `captureSessionLifecycleEvent` directly (`hooks/session-capture.ts:144`)
+ and hand the dedup walk to a detached worker (`:82-95`).
+ - **Stage names follow what the write tools actually do:**
+ - signal validation;
+ - the idempotency-key lookup (`signal.ts:357`);
+ - the atomic document write (`:382-399`, `create-note.ts:570`);
+ - remembering the key (`signal.ts:412`);
+ - the log append (`feedback-tools.ts:224`);
+ - the forced preference write (`:270`);
+ - the write receipt (`create-note.ts:594`, `write-batch.ts:901`);
+ - the receipt lint;
+ - the near-duplicate lookup.
+ - **The card names stages that do not exist on these routes:**
+ "snapshot" means the write receipt, and "index or ledger writes" are
+ not on these routes at all.
+ - **One premise claim was wrong.** The timing premise said `brain_feedback`
+ has no similarity stage. `adviseIncomingFeedback`
+ (`write-advisory.ts:163`) already runs a Jaccard pass on that route, so
+ `near_duplicate_lookup` is a real stage there today.
+5. **Preferences carry no sign.** `BrainPreference` (`src/core/brain/types.ts:875`)
+ has no `signal` field, so a "same sign only" guard on retire siblings
+ cannot be built. The precision guards are a minimum token count, the
+ accept gate, and an off-by-default key.
+
+## Scope
+
+- **Shared kernel.** One pure near-duplicate lookup,
+ `src/core/brain/near-duplicate.ts`, built on `tokenise` / `jaccard`. It
+ applies a reach predicate before any scoring or counting, holds one named
+ threshold table, requires a minimum token count, caps its work, and labels
+ every match with a `method`.
+- **Forget side (t_acab97de).** A read-only retire-sibling projection on the
+ dream retire set and on `reject`.
+ - It surfaces in `DreamRunSummary.retire_siblings` and in
+ `brain_review_candidates.retire_siblings`, and `o2b brain reject` prints
+ it.
+ - It runs a lexical tier always. It also runs a stored-vector tier, which
+ reuses embeddings already paid for in the search index and makes zero
+ provider calls.
+ - Acceptance goes only through existing verbs: `brain_apply_evidence
+ result: outdated` or `o2b brain reject`.
+ - It is off by default.
+- **Write side (t_fda66477).**
+ - The reach fix for the shipped `near-duplicate` lint, on unconditionally.
+ - A widening beyond the directory, off by default. It pulls BM25
+ candidates from the existing keyword index, rescores them through the
+ kernel, and has zero embedding spend.
+ - `adviseOnIncoming` scoring moves onto the kernel, with its threshold
+ source unchanged, so the write side has one scorer.
+- **Extract rules (t_b915b9cc).**
+ - `MinedTurn.timestamp`, and a transcript line `[turnId @ <timestamp>]`.
+ - Language-neutral hygiene rules in the prompt.
+ - A refusal of duplicate `topic` values in one payload, under
+ `crossItem`.
+ - An additive `timestamp` on the `turns_mined` projections.
+- **Stage timings (t_72e93e18).**
+ - One route scope that carries decision time and stages. It replaces the
+ decision-only AsyncLocalStorage store.
+ - A closed allowlist of stage names, enforced where the record is
+ emitted.
+ - An optional `stages` array on `mcp_route_latency`, 0.1 ms precision.
+ - A per-stage roll-up in the `brain_route_metrics` summary.
+ - Instrumentation on `brain_feedback` and on the four note-write tools.
+- **Release.** CHANGELOG `[1.74.0]`, `docs/mcp.md`, `docs/observability.md`,
+ `docs/cli-reference.md`, and the version bump inside this PR.
+
+## Out of scope
+
+- **Model-generated rewordings.** A reworded fact is new content written by
+ a model, which is not accept-gated surfacing. Under this design,
+ "rewording" means the operator rewrites the fact using the existing
+ verbs, after seeing the surfaced pair.
+- **Write-time embedding.** The new body is not indexed at write time, so a
+ semantic write hint means one paid embed per write, more for batches. It
+ also needs a labelled pair evaluation (duplicates caught against wrong
+ hints) before any default could change. Both are deferred to their own
+ wave.
+- **Decision-model pair verdicts on retire siblings.** `runPairVerdicts` /
+ `advisoryUseActive` could add them later, behind the advisory egress path.
+- **A Jaccard refusal across items in extract-signals.** Refusal rejects the
+ whole payload, and two short principles with opposite meanings ("use
+ tabs", "never use tabs") score high, so one false positive would drop
+ every valid signal in the run. Only the exact duplicate-topic rule ships.
+- **Hook capture timing** and **acceptance inside an assistant turn.**
+- **Changes elsewhere.** No change to `computeDedupHash`, to hygiene
+ `forget`, to `moveToRetired` or to `detectSemanticDedup`, which stays
+ without a production caller.
+
+## Chosen approach
+
+This is the first consultant's Variant B ("lexical kernel plus a vector
+tier that reuses paid embeddings"), built in the staged form it
+recommended. Three guards from the adversarial consultant's C1 are folded
+in:
+
+1. The reach fix is unconditional and lands first.
+2. Every new surface is advisory, read-only and off by default.
+3. Nothing in this wave calls an embedding provider.
+
+**The kernel.** `findNearDuplicates(probe, pool, opts)` is pure. It
+receives:
+- a probe;
+- a candidate pool, which the caller has already projected into tokens;
+- a required `readable` predicate;
+- a threshold from `NEAR_DUPLICATE_THRESHOLDS`;
+- `minTokens` and `cap`.
+
+Withheld candidates are dropped before they are scored and before they are
+counted, so neither a match nor a census number can disclose them. The
+kernel returns matches labelled `method: "lexical"` and a census that
+counts only readable entries.
+
+**The vector tier.** It lives in a separate module that reads the index,
+`src/core/brain/near-duplicate-vectors.ts`. It follows the precedent of
+`src/core/brain/surprisal.ts`:
+- It opens the store read-only.
+- It reads vectors that are already stored, through
+ `storedEmbeddingsForDocument` (`src/core/search/store/vectors.ts:262`).
+- It compares only vectors whose `model` and `dimension` match.
+- It never imports a provider.
+- It reports its outcome as a named status (`used`, `index_missing`,
+ `vec_unavailable`, `model_mismatch`). It never degrades silently.
+
+The kernel never sees vectors. The projection merges the vector scores
+into matches labelled `method: "embedding"`.
+
+**The forget side.** `planRetireSiblings` is pure, like `planAutoRetires`,
+so the dream dry run stays write-free and the dry and real runs agree.
+
+1. It runs inside `dream()` once the retire plan is complete.
+2. The summary builder drops siblings of the retires that were gated at
+ apply time (`gated_retires`).
+3. `buildReviewCandidates` projects the result and, when a search config is
+ present, adds the vector tier (the same `opts.searchConfig` path that
+ `scoreSignalNovelty` already uses).
+4. `review-tools.ts` filters both ids of every pair through the caller's
+ reach view before counting.
+5. `reject` prints the siblings of the preference it just retired, at the
+ CLI's operator reach.
+
+**The write side.**
+- **Reach.** `lintWrittenPages` gains a required options object with
+ `readable`, and `noteWriteResult` threads `readableAtContextReach(ctx)`
+ through, which covers all four note-write tools at once.
+- **Widening.** When the widening key is on, an async pre-step in
+ `notes-tools.ts` (`collectWideningCandidates` in the new
+ `src/core/brain/page-lint-widening.ts`) builds an FTS5 query with
+ `buildFtsMatch` from the written body. It pulls the BM25 top-k chunks
+ through `keywordTopK`, maps them to documents, re-reads each candidate
+ from disk, and hands it to the lint as an extra candidate. The scope-bucket
+ rule and the reach rule are unchanged. The census reports the widening
+ status, and that field is absent when the key is off, so a clean receipt
+ stays byte-identical.
+
+**Extract.** The extract lane reaches no Jaccard code:
+- it adds the timestamp field and its rendering;
+- it adds the prompt rules;
+- it extends the existing `registerResponseCheck` closure with an exact
+ duplicate-topic rule.
+
+**Stage timings.** A new `src/core/route-scope.ts` takes over the
+AsyncLocalStorage store from `src/core/decision-model/latency.ts`. That
+file shrinks to a re-export of `noteDecisionLatency`, so `run.ts` is
+untouched. `invokeToolHandler` opens one `createRouteScope()` instead of
+`createDecisionLatencyScope()`. The write tools wrap their stages in
+`timeStageSync` or `timeStage`, which use `try/finally` and never swallow
+an error. `emitMcpRouteLatency` then:
+- drops any name outside `ROUTE_STAGE_NAMES`;
+- sums repeated names, keeping the order each name was first seen;
+- rounds to 0.1 ms;
+- omits `stages` when nothing was noted.
+
+With the gate off, no scope is open, and every stage helper costs one
+`getStore()` that returns `undefined`.
+
+## Design decisions
+
+- **One threshold table, values unchanged where a behaviour already
+ ships.** The table is `NEAR_DUPLICATE_THRESHOLDS`:
+
+ | Key | Value | Source |
+ |---|---|---|
+ | `writeHint` | 0.8 | `NEAR_DUPLICATE_JACCARD`, which becomes an alias |
+ | `retireSiblingLexical` | 0.7 | the doctor lint bar |
+ | `retireSiblingEmbedding` | 0.92 | the entity semantic-dedup bar, `entities/semantic-dedup.ts:39-41` |
+
+ The feedback advisory keeps reading
+ `BRAIN_HEALTH_DEFAULTS.contradiction_jaccard`, which operators can
+ configure. Only its scoring loop moves onto the kernel.
+- **`readable` is required, not optional.** If the predicate were
+ optional, the leak this wave fixes could come back at a new call site. An
+ operator-reach caller, such as the `reject` verb or a CLI dream, passes
+ `READ_ALL_REFS` explicitly, so every unfiltered call is visible in the
+ diff.
+- **A minimum of 4 tokens for both the probe and the candidate
+ (`NEAR_DUPLICATE_MIN_TOKENS`).** Short principles share most of their
+ tokens by accident. Below that count the kernel does not score at all,
+ and the census reports `below_min_tokens`.
+- **Off-by-default keys, read through `resolveConfigFlag`:**
+ - `near_duplicate_retire_siblings_enabled`, env
+ `OPEN_SECOND_BRAIN_NEAR_DUPLICATE_RETIRE_SIBLINGS_ENABLED`;
+ - `near_duplicate_write_widening_enabled`, env
+ `OPEN_SECOND_BRAIN_NEAR_DUPLICATE_WRITE_WIDENING_ENABLED`.
+
+ Stage timings reuse the existing `mcp_route_metrics_enabled` and add no
+ key. The reach fix and the extract changes have no key: the first is a
+ defect fix, the second a contract tightening.
+- **The vector tier has no key of its own.** It runs only when
+ retire-siblings is on and stored vectors exist. It costs zero provider
+ calls and zero new egress, so a separate key would be configuration
+ without a decision behind it. The status field makes it visible whether
+ the tier ran.
+- **Spend gates by construction.** Neither `near-duplicate-vectors.ts` nor
+ `page-lint-widening.ts` imports from `src/core/search/embeddings/`. A
+ test that scans the source of both modules pins this, and also pins that
+ `detectSemanticDedup` still has no production caller. The egress census
+ delta for this wave is zero.
+- **Retire siblings live in the plan, not the apply step.**
+ `moveToRetired` and the write-site and destructive-site censuses are
+ unchanged. An accepted sibling retires on the next dream pass through
+ the existing superseded-by-context path, or at once through `reject`.
+ Siblings are never cascaded: a sibling of a sibling is not computed.
+- **The pool for retire siblings** is every active preference (unconfirmed,
+ confirmed or quarantine) except:
+ - the retiring set itself;
+ - pages that have already been merged, using `isMergeResolved`, as
+ `hygiene/detectors/dedup.ts:96-97` does.
+
+ The pool crosses topic and scope buckets, because a later paraphrase
+ usually lands under another topic slug. The cost is O(retires x active)
+ and is capped by `NEAR_DUPLICATE_CANDIDATE_CAP` (200, matching
+ `NEAR_DUPLICATE_MAX_CANDIDATES`).
+- **Widening is bounded.** `NEAR_DUPLICATE_WIDENING_TOP_K` is 20 chunks. A
+ candidate already present from the same directory is not counted twice.
+ An index that cannot be opened, or has no FTS table, reports
+ `widening: "index_unavailable"` and never fails the write. The write path
+ stays fail-open, as the existing lint does.
+- **The extract timestamp is rendered verbatim.** It is the stored string,
+ never a locale-formatted `Date`, so Windows and Linux output stay
+ byte-identical. An empty timestamp drops the `@` part, and `now` is never
+ substituted. The resolved date stays in the principle text, and the dream
+ pass derives the window. The extract lane never writes `valid_from`,
+ because `signal.ts:147-152` reserves it for event-time backfill.
+- **One store for decision time and stages.** Two AsyncLocalStorage stores
+ would double the wrapping in `server.ts`. Stage names are code constants,
+ and the allowlist is enforced in `emitMcpRouteLatency`, not at the call
+ sites, so a hostile name cannot reach disk. That property gets its own
+ test against a persisted record.
+- **No timing in tool responses.** Stages live only in the continuity
+ record. `docs/observability.md` lists the routes that emit stages, so an
+ absent array reads as "not instrumented", not as "zero cost".
+
+## Kernel invariants this design respects
+
+1. **Language independence.** There are no word lists, phrase lists or
+ relative-date parsers. Similarity is `tokenise` / `jaccard` over Unicode
+ letter and number runs. The hygiene rules are prompt text that applies
+ to any language.
+2. **No model in the kernel.** The kernel, the planner, the widening step
+ and the stage scope are all deterministic. The only model-facing change
+ is envelope text that the calling agent executes, behind the existing
+ `buildNeedsLlmStep` registry.
+3. **Embedding spend gates (v1.72.0 / v1.73.0).** The wave makes zero
+ `provider.embed` calls. The vector tier reads only stored vectors, and
+ the widening step is BM25. Nothing bypasses `prepareQueryEmbed`, because
+ nothing needs it.
+4. **Accept-gated, off-by-default proposals.** Retire siblings are a
+ projection. A sibling retires only when the operator calls an existing
+ verb. Both new behaviour keys default to off, and with them off every
+ receipt, summary and review projection is byte-identical to v1.73.0.
+5. **Reach and visibility.** Matches are filtered before scoring and
+ counting on every surface: receipts, review candidates, the dream
+ summary and the reject output. The visibility-surface registry gains a
+ row for the note-write lint and a note on the `brain_review_candidates`
+ row.
+6. **Determinism and dry-run purity.** `dream({ dryRun: true })` stays
+ write-free. `planRetireSiblings` is pure and sorts its output by
+ `(retiring_id, -score, sibling_id)`.
+7. **Fail-closed registry.** The extract check extends the single existing
+ registration (`extract-signals.ts:302-329`) and uses the closed
+ `SEMANTIC_VIOLATION_CODES.crossItem` code.
+
+## File changes
+
+New:
+- `src/core/brain/near-duplicate.ts`
+- `src/core/brain/near-duplicate-vectors.ts`
+- `src/core/brain/retire-siblings.ts`
+- `src/core/brain/page-lint-widening.ts`
+- `src/core/route-scope.ts`
+- the tests named in `plan.md`
+
+Modified:
+- **Forget side:** `src/core/config.ts`, `src/core/brain/dream.ts`,
+ `src/core/brain/dream-types.ts`, `src/core/brain/review-candidates.ts`,
+ `src/mcp/brain/review-tools.ts`, `src/cli/brain/verbs/reject.ts`.
+- **Write side:** `src/core/brain/page-lint.ts`,
+ `src/mcp/brain/notes-tools.ts`, `src/core/brain/write-advisory.ts`,
+ `src/core/brain/health/contradiction.ts`,
+ `src/core/search/visibility-surface-registry.ts`.
+- **Extract:** `src/core/brain/extract-signals.ts`,
+ `src/mcp/brain/extract-tools.ts`, `src/cli/brain/verbs/extract-signals.ts`.
+- **Stage timings:** `src/core/decision-model/latency.ts`,
+ `src/core/brain/mcp-route-metrics.ts`, `src/mcp/server.ts`,
+ `src/core/brain/signal.ts`, `src/mcp/brain/feedback-tools.ts`,
+ `src/core/brain/notes/create-note.ts`, `src/core/brain/write-batch.ts`,
+ `src/mcp/brain/recall-tools.ts`.
+- **Docs:** `CHANGELOG.md`, `docs/mcp.md`, `docs/observability.md`,
+ `docs/cli-reference.md`.
+- **Version:** `package.json` and the mirrors that
+ `scripts/sync-version.ts` rewrites.
+
+The count per lane is in `plan.md`: 51 files written by hand, plus 9
+generated by `sync-version.ts`. No skill or `hooks/hooks.json` change is
+planned, so `sync-plugin-mirrors` is not triggered. If a skill line is
+edited after all, the integrator runs `bun run sync-plugin-mirrors`.
+
+## Risks and open questions
+
+- **Cross-topic false positives on retire siblings.** These are mitigated by
+ the 0.7 / 0.92 bars, the 4-token minimum, the off-by-default key and the
+ accept gate. There is no cascade, because siblings are never computed
+ for siblings.
+- **Stale vectors.** Stored vectors trail the last reindex. The tier
+ compares only vectors from the same model and dimension. A page that
+ changed since its last embedding is still scored on its old vector,
+ which is acceptable for an advisory, and the docs say so.
+- **Index staleness for widening.** A page that was just written, or a
+ sibling written recently, may not be indexed yet. Widening is
+ best-effort beyond the same-directory walk, which still reads the disk.
+ Candidates are re-read from disk, so a deleted page in the index cannot
+ produce a hint.
+- **Latency on the write path.** Widening adds one read-only store open and
+ one FTS query per note-write call. The `near_duplicate_lookup` stage
+ measures it, which is the reason this wave ships timings with the lookup.
+- **Windows CI.**
+ - Tests assert only that stages are present, that their names are
+ allowed, and that their values are finite and non-negative. They never
+ assert a positive duration or an order.
+ - Fixtures set mtimes explicitly.
+ - Paths stay vault-relative POSIX through `canonicalPage`.
+ - Timestamps are rendered verbatim.
+- **Overlap with card t_cabe84aa (supersede sweep).** That card also extends
+ `DreamRunSummary`. This wave adds only the optional `retire_siblings`
+ field, so a later supersede field does not collide with it.
+- **Decided by the design writer.** These were open in the premises:
+ 1. Vector tier on the forget side only. There is no write-time embed.
+ 2. The Jaccard refusal in extract-signals is deferred, and only
+ duplicate-topic ships.
+ 3. Repeated stages are summed per name.
+ 4. Stage precision is 0.1 ms.
+ 5. `readable` is required in the kernel.
+ 6. Docs and the version bump belong to the integrator, not to a lane.
diff --git a/docs/brainstorm/near-duplicate-defense/plan.md b/docs/brainstorm/near-duplicate-defense/plan.md
new file mode 100644
index 00000000..97f969e6
--- /dev/null
+++ b/docs/brainstorm/near-duplicate-defense/plan.md
@@ -0,0 +1,744 @@
+# Near-Duplicate Defense Across the Fact Lifecycle - implementation plan
+
+This plan has four implementation lanes (A, B, C, D) and one integrator step:
+
+- Feature branch: `feat/near-duplicate-defense`, one worktree.
+- Every lane commits to this branch in place. The integrator ships one PR,
+ which releases v1.74.0.
+- Every task is TDD:
+ 1. Write the failing test first.
+ 2. Make it pass.
+ 3. Make one conventional commit.
+- Design and rationale: `design.md`.
+
+## 0. Pinned cross-lane contract
+
+Everything two lanes share is decided here. Lanes implement against this
+section as written. A lane that needs to change it stops and asks the
+integrator. It does not edit another lane's file.
+
+### 0.1 `src/core/brain/near-duplicate.ts` (owner: lane A, task A1)
+
+```ts
+export type NearDuplicateMethod = "lexical" | "embedding";
+export type ReadableRef = (ref: string) => boolean;
+/** Explicit operator-reach predicate; every unfiltered call site names it. */
+export const READ_ALL_REFS: ReadableRef;
+
+export const NEAR_DUPLICATE_THRESHOLDS: Readonly<{
+ writeHint: 0.8; // == page-lint NEAR_DUPLICATE_JACCARD
+ retireSiblingLexical: 0.7;
+ retireSiblingEmbedding: 0.92;
+}>;
+export const NEAR_DUPLICATE_MIN_TOKENS = 4;
+export const NEAR_DUPLICATE_CANDIDATE_CAP = 200;
+export const NEAR_DUPLICATE_WIDENING_TOP_K = 20;
+
+export interface NearDuplicateProbe {
+ readonly ref: string; // vault-relative path or pref id
+ readonly tokens: ReadonlySet<string>; // from similarity.ts tokenise
+}
+export interface NearDuplicatePoolEntry {
+ readonly ref: string;
+ readonly tokens: ReadonlySet<string>;
+ /** Optional bucket; when the options carry `bucket`, entries must match it. */
+ readonly bucket?: string;
+}
+export interface NearDuplicateOptions {
+ readonly threshold: number;
+ readonly readable: ReadableRef; // required, applied before scoring and counting
+ readonly minTokens?: number; // default NEAR_DUPLICATE_MIN_TOKENS
+ readonly cap?: number; // default NEAR_DUPLICATE_CANDIDATE_CAP
+ readonly bucket?: string;
+}
+export interface NearDuplicateMatch {
+ readonly ref: string;
+ readonly score: number; // rounded to 3 decimals
+ readonly method: NearDuplicateMethod;
+}
+export interface NearDuplicateScan {
+ readonly compared: number; // readable entries actually scored
+ readonly capped: number; // readable entries left out by the cap
+ readonly below_min_tokens: number; // readable entries skipped as too short
+}
+export interface NearDuplicateResult {
+ readonly matches: ReadonlyArray<NearDuplicateMatch>; // sorted by -score, then ref
+ readonly scan: NearDuplicateScan;
+}
+export function findNearDuplicates(
+ probe: NearDuplicateProbe,
+ pool: ReadonlyArray<NearDuplicatePoolEntry>,
+ opts: NearDuplicateOptions,
+): NearDuplicateResult;
+```
+
+`findNearDuplicates` follows these rules:
+
+1. It is pure: no I/O, no clock, no model.
+2. It excludes the probe's own `ref`.
+3. It never throws on well-formed input.
+4. Unreadable entries never appear in `matches` or in any `scan` counter.
+
+### 0.2 `src/core/config.ts` additions (owner: lane A, task A1)
+
+```ts
+export function resolveNearDuplicateRetireSiblingsEnabled(configPath?: string): boolean;
+// key near_duplicate_retire_siblings_enabled
+// env OPEN_SECOND_BRAIN_NEAR_DUPLICATE_RETIRE_SIBLINGS_ENABLED
+export function resolveNearDuplicateWriteWideningEnabled(configPath?: string): boolean;
+// key near_duplicate_write_widening_enabled
+// env OPEN_SECOND_BRAIN_NEAR_DUPLICATE_WRITE_WIDENING_ENABLED
+```
+
+Both go through `resolveConfigFlag` and default to off. No other config key
+is added in this wave. Stage timings reuse `mcp_route_metrics_enabled`.
+
+### 0.3 Retire-sibling shapes (owner: lane A)
+
+```ts
+// src/core/brain/retire-siblings.ts
+export const RETIRE_SIBLING_TRIGGER_REASONS: ReadonlySet<BrainRetiredReason>;
+// superseded-by-context, rebutted, quarantine-violated, user-rejected
+export interface RetireSibling {
+ readonly retiring_id: string; // pref-* id being retired
+ readonly sibling_id: string; // active pref-* id that resembles it
+ readonly score: number;
+ readonly method: NearDuplicateMethod;
+}
+export function planRetireSiblings(
+ active: ReadonlyArray<{ id: string; principle: string }>,
+ retiring: ReadonlyArray<{ id: string; principle: string; reason: BrainRetiredReason }>,
+ opts: { readonly readable: ReadableRef; readonly gated: ReadonlySet<string> },
+): ReadonlyArray<RetireSibling>;
+
+// src/core/brain/dream-types.ts, DreamRunSummary
+readonly retire_siblings?: ReadonlyArray<RetireSibling>; // absent when the key is off or the list is empty
+
+// src/core/brain/near-duplicate-vectors.ts
+export type StoredVectorStatus = "used" | "index_missing" | "vec_unavailable" | "model_mismatch";
+export interface StoredVectorSimilarity {
+ readonly status: StoredVectorStatus;
+ readonly scores: ReadonlyMap<string, number>; // vault-relative candidate path -> max chunk-pair cosine
+}
+export function storedVectorSimilarity(
+ config: ResolvedSearchConfig,
+ probePath: string,
+ candidatePaths: ReadonlyArray<string>,
+): Promise<StoredVectorSimilarity>;
+
+// review projection: ReviewCandidatesReport gains
+readonly retire_siblings?: ReadonlyArray<RetireSibling>;
+readonly retire_siblings_semantic?: StoredVectorStatus; // present only when retire_siblings is computed with a search config
+```
+
+### 0.4 Write-side shapes (owner: lane B)
+
+```ts
+// src/core/brain/page-lint.ts
+export interface LintWrittenPagesOptions {
+ readonly readable: ReadableRef; // required
+ readonly extraCandidates?: ReadonlyArray<NearDuplicateCandidate>; // from widening
+ readonly widening?: NearDuplicateWideningStatus; // absent when the key is off
+}
+export function lintWrittenPages(
+ vault: string,
+ pages: ReadonlyArray<string>,
+ opts: LintWrittenPagesOptions,
+): PageLintReport;
+export type NearDuplicateWideningStatus = "used" | "index_unavailable";
+// NearDuplicateCensus gains: readonly widening?: NearDuplicateWideningStatus
+
+// src/core/brain/page-lint-widening.ts
+export interface WideningResult {
+ readonly status: NearDuplicateWideningStatus;
+ readonly candidates: ReadonlyArray<NearDuplicateCandidate>;
+}
+export function collectWideningCandidates(
+ config: ResolvedSearchConfig,
+ vault: string,
+ pages: ReadonlyArray<string>,
+ readable: ReadableRef,
+): Promise<WideningResult>;
+```
+
+The finding code stays `near-duplicate`. A widened finding carries the same
+code, and its message names the method: `method=lexical`.
+
+### 0.5 Extract shapes (owner: lane C)
+
+```ts
+// src/core/brain/extract-signals.ts
+export interface MinedTurn {
+ readonly turnId: string;
+ readonly text: string;
+ readonly timestamp: string; // verbatim stored string; "" when the source has none
+}
+// transcript line: `[${turnId} @ ${timestamp}] ${text}`, or `[${turnId}] ${text}` when the timestamp is ""
+// crossItem refusal message names both indices: `items[i] and items[j] share topic "<topic>"`
+// turns_mined entries (MCP + CLI) gain: timestamp: string
+```
+
+### 0.6 `src/core/route-scope.ts` (owner: lane D, task D1)
+
+```ts
+export const ROUTE_STAGE: Readonly<{
+ validate: "validate";
+ idempotencyLookup: "idempotency_lookup";
+ nearDuplicateLookup: "near_duplicate_lookup";
+ documentWrite: "document_write";
+ idempotencyRemember: "idempotency_remember";
+ logAppend: "log_append";
+ preferenceWrite: "preference_write";
+ writeReceipt: "write_receipt";
+ lint: "lint";
+}>;
+export type RouteStageName = (typeof ROUTE_STAGE)[keyof typeof ROUTE_STAGE];
+export const ROUTE_STAGE_NAMES: ReadonlySet<RouteStageName>;
+export function isRouteStageName(value: unknown): value is RouteStageName;
+export interface RouteStageTiming { readonly name: RouteStageName; readonly ms: number }
+export function timeStageSync<T>(name: RouteStageName, fn: () => T): T; // try/finally, rethrows
+export function timeStage<T>(name: RouteStageName, fn: () => Promise<T>): Promise<T>; // try/finally, rethrows
+export function noteDecisionLatency(ms: number): void; // moved from latency.ts
+export interface RouteScope {
+ run<T>(fn: () => Promise<T>): Promise<T>;
+ decisionMs(): number | undefined;
+ stages(): ReadonlyArray<RouteStageTiming> | undefined; // summed per name, first-seen order, 0.1 ms; undefined when none
+}
+export function createRouteScope(): RouteScope;
+```
+
+With no scope open, `timeStageSync` and `timeStage` reduce to `fn()`.
+`src/core/decision-model/latency.ts` re-exports `noteDecisionLatency` and
+keeps `createDecisionLatencyScope` as a thin wrapper over
+`createRouteScope`, so `run.ts` and its tests stay untouched.
+
+### 0.7 Record and log kinds
+
+1. **No new continuity kind.** `mcp_route_latency` gains the optional
+ `stages: Array<{ name: RouteStageName; ms: number }>`, and
+ `McpRouteLatencyInput` gains `stages?: ReadonlyArray<RouteStageTiming>`.
+2. **No new Brain log event kind.** The `write-conflict-advisory` kind keeps
+ its existing payload.
+3. **No new MCP tool.** No new `PAGE_LINT_SKIP_REASON`.
+4. **New closed string sets:** `StoredVectorStatus`,
+ `NearDuplicateWideningStatus` and `ROUTE_STAGE`.
+
+### 0.8 Who imports whom
+
+| Consumer | Imports | From (lane, task) |
+|---|---|---|
+| B3, B4 | `findNearDuplicates`, `NEAR_DUPLICATE_*`, `ReadableRef`, `READ_ALL_REFS` | A1 |
+| B3 | `resolveNearDuplicateWriteWideningEnabled` | A1 |
+| B5 | `timeStageSync`, `timeStage`, `ROUTE_STAGE` | D1 |
+| A2-A6 | own modules only | - |
+| C1-C5 | nothing from another lane | - |
+| D2-D6 | own modules only | - |
+
+The two substrate tasks, A1 and D1, are each lane's first task. They are
+small: one module, one test file and the contract signatures above. The
+tasks that consume them (B3, B4 and B5) sit after lane B's tasks that do not
+depend on them.
+
+## 1. Lanes, owned files and start conditions
+
+Each lane edits only the files listed for it. The integrator owns
+`CHANGELOG.md`, `docs/mcp.md` and the version files.
+
+### Lane A: kernel and forget side (t_acab97de). Safe to start immediately.
+
+Owned files (16):
+
+| Source | Tests |
+|---|---|
+| `src/core/brain/near-duplicate.ts` (new) | `tests/core/brain/near-duplicate.test.ts` (new) |
+| `src/core/config.ts` | `tests/core/config-near-duplicate.test.ts` (new) |
+| `src/core/brain/retire-siblings.ts` (new) | `tests/core/brain/retire-siblings.test.ts` (new) |
+| `src/core/brain/dream.ts` | |
+| `src/core/brain/dream-types.ts` | |
+| `src/core/brain/near-duplicate-vectors.ts` (new) | `tests/core/brain/near-duplicate-vectors.test.ts` (new) |
+| `src/core/brain/review-candidates.ts` | `tests/core/brain/review-candidates.test.ts` |
+| `src/mcp/brain/review-tools.ts` | `tests/mcp/review-tools-reach.test.ts` |
+| `src/cli/brain/verbs/reject.ts` | `tests/cli/brain-reject-siblings.test.ts` (new) |
+
+### Lane B: write side (t_fda66477). Safe to start immediately.
+
+B1 and B2 have no cross-lane import. B3 and B4 start once A1 is committed;
+the contract in 0.1 and 0.2 is fixed, so their tests can be written before
+that. B5 starts once D1 is committed.
+
+Owned files (12):
+
+| Source | Tests |
+|---|---|
+| `src/core/brain/page-lint.ts` | `tests/core/brain/page-lint.test.ts` |
+| `src/mcp/brain/notes-tools.ts` | `tests/mcp/note-write-reach.test.ts`, `tests/mcp/brain-create-note.test.ts` |
+| `src/core/brain/page-lint-widening.ts` (new) | `tests/core/brain/page-lint-widening.test.ts` (new) |
+| `src/core/search/visibility-surface-registry.ts` | `tests/core/architecture/visibility-surface-census.test.ts` |
+| `src/core/brain/health/contradiction.ts` | `tests/core/brain/write-advisory.test.ts` |
+| `src/core/brain/write-advisory.ts` | |
+
+### Lane C: extract rules (t_b915b9cc). Safe to start immediately.
+
+Lane C has no cross-lane import.
+
+Owned files (8):
+
+| Source | Tests and docs |
+|---|---|
+| `src/core/brain/extract-signals.ts` | `tests/core/brain/extract-signals.test.ts`, `tests/core/brain/extract-signals-prefilter.test.ts` (edited only if it pins the old line format) |
+| `src/mcp/brain/extract-tools.ts` | `tests/mcp/extract-signals-tool.test.ts` |
+| `src/cli/brain/verbs/extract-signals.ts` | `tests/core/brain/extract-signals-temporal.test.ts` (new) |
+| | `docs/cli-reference.md` |
+
+### Lane D: stage timings (t_72e93e18). Safe to start immediately.
+
+Owned files (13):
+
+| Source | Tests and docs |
+|---|---|
+| `src/core/route-scope.ts` (new) | `tests/core/route-scope.test.ts` (new) |
+| `src/core/decision-model/latency.ts` | |
+| `src/core/brain/mcp-route-metrics.ts` | `tests/core/brain/mcp-route-metrics.test.ts` |
+| `src/mcp/server.ts` | `tests/mcp/route-metrics-tool.test.ts` |
+| `src/core/brain/signal.ts` | |
+| `src/mcp/brain/feedback-tools.ts` | |
+| `src/core/brain/notes/create-note.ts` | |
+| `src/core/brain/write-batch.ts` | |
+| `src/mcp/brain/recall-tools.ts` | `docs/observability.md` |
+
+### Integrator (after all lanes): 2 files, plus 9 generated
+
+- `CHANGELOG.md` and `docs/mcp.md`.
+- `package.json`. Then run `bun run scripts/sync-version.ts`, which rewrites
+ `plugin.yaml`, `plugins/hermes/plugin.yaml`, `.claude-plugin/plugin.json`,
+ `.codex-plugin/plugin.json`, `plugins/codex/.codex-plugin/plugin.json`,
+ `openclaw.plugin.json`, `pyproject.toml` and `uv.lock`.
+
+### Touched-file estimate
+
+| Lane | Files |
+|---|---|
+| A | 16 |
+| B | 12 |
+| C | 8 |
+| D | 13 |
+| Integrator | 2 |
+| **Total written by hand** | **51** |
+| Generated by `sync-version.ts` | 9 (`package.json` plus 8 mirrors) |
+
+The 51 files written by hand fall within the 40-55 budget.
+
+## 2. Tasks
+
+The test command for every task, from the worktree root:
+
+```
+env HOME=$(mktemp -d) bun test <paths>
+```
+
+### Lane A
+
+**A1. Near-duplicate kernel and config keys (substrate)**
+- Test first:
+ - `tests/core/brain/near-duplicate.test.ts`:
+ - an unreadable entry never appears in `matches` or in any `scan` counter;
+ - an entry under `minTokens` counts in `below_min_tokens` and is not scored;
+ - the cap counts as `capped`;
+ - the probe's own `ref` is excluded;
+ - sorting is deterministic;
+ - multilingual tokens (Cyrillic, CJK) score through the shared
+ `tokenise`;
+ - `NEAR_DUPLICATE_THRESHOLDS.writeHint === NEAR_DUPLICATE_JACCARD`.
+ - `tests/core/config-near-duplicate.test.ts`: both keys default to off, and
+ the config key and the env var each turn them on.
+- Implement: `src/core/brain/near-duplicate.ts` exactly as in contract 0.1;
+ the two resolvers in `src/core/config.ts` (0.2).
+- Commit: `feat(brain): add a reach-aware near-duplicate kernel and its config keys`
+
+**A2. Retire-sibling planner**
+- Test first: `tests/core/brain/retire-siblings.test.ts`:
+ - only the four trigger reasons nominate;
+ - decay reasons (`stale-no-evidence`, `expired-unconfirmed`) and
+ `merged-into` never nominate;
+ - siblings of a gated id are dropped;
+ - merge-resolved pages are excluded;
+ - preferences from other topics and scopes are compared;
+ - a sibling of a sibling is never computed;
+ - the output is stable across runs.
+- Implement: `src/core/brain/retire-siblings.ts`, a pure planner over
+ `findNearDuplicates`, using the threshold `retireSiblingLexical`.
+- Commit: `feat(brain): plan retire siblings for context-driven retires`
+
+**A3. Dream summary wiring**
+- Test first: `tests/core/brain/retire-siblings.test.ts` (dream section):
+ - with the key on, `dream({ dryRun: true })` and a real run report the
+ same `retire_siblings`;
+ - the dry run writes nothing (the vault tree is compared before and
+ after);
+ - with the key off, the field is absent and the summary is byte-identical
+ to the baseline;
+ - a sibling of a gated retire does not appear.
+- Implement:
+ - `src/core/brain/dream-types.ts`: add the optional field.
+ - `src/core/brain/dream.ts`: compute the siblings once the retire plan is
+ complete, and reconcile them against `gated_retires` in the summary
+ builder.
+- Commit: `feat(dream): report retire siblings in the run summary`
+
+**A4. Stored-vector tier**
+- Test first: `tests/core/brain/near-duplicate-vectors.test.ts`. Each status
+ is driven by a fixture index or a missing index:
+ - `index_missing`;
+ - `vec_unavailable`, when `vecLoaded()` is false;
+ - `model_mismatch`, when the stored model or dimension differs;
+ - `used`, with the max chunk-pair cosine as the score.
+
+ A source-scan test asserts that neither this module nor
+ `src/core/brain/page-lint-widening.ts` (once it exists) imports from
+ `src/core/search/embeddings/`. It also asserts that `detectSemanticDedup`
+ has no caller outside `tests/`.
+- Implement: `src/core/brain/near-duplicate-vectors.ts` (contract 0.3). It
+ opens the store read-only, uses `storedEmbeddingsForDocument`, and makes
+ zero provider calls.
+- Commit: `feat(brain): score retire siblings on stored vectors at zero embedding spend`
+
+**A5. Review projection and MCP reach**
+- Test first:
+ - `tests/core/brain/review-candidates.test.ts`: `retire_siblings` and
+ `retire_siblings_semantic` are projected. Embedding matches merge in
+ with `method: "embedding"` at or above `retireSiblingEmbedding`. With
+ the key off, both fields are absent.
+ - `tests/mcp/review-tools-reach.test.ts`: a pair whose retiring id or
+ sibling id is outside the caller's reach is neither listed nor counted.
+ Both the `pref-` and `ret-` spellings are checked.
+- Implement:
+ - `src/core/brain/review-candidates.ts`: project the field and run the
+ vector tier on the existing `opts.searchConfig` path.
+ - `src/mcp/brain/review-tools.ts`: filter through `reviewView` before any
+ counting, and update the output schema and description.
+- Commit: `feat(mcp): surface reach-filtered retire siblings in brain_review_candidates`
+
+**A6. Siblings on `reject`**
+- Test first: `tests/cli/brain-reject-siblings.test.ts`:
+ - with the key on, `o2b brain reject <id>` prints each sibling with its
+ score and the accept command;
+ - with the key off, the output is byte-identical to before;
+ - nothing besides the rejected preference is retired.
+- Implement: `src/cli/brain/verbs/reject.ts`. Call `planRetireSiblings` with
+ `READ_ALL_REFS` (operator reach) and the reason `user-rejected`, then
+ print the result and update the help text.
+- Commit: `feat(cli): list retire siblings after brain reject`
+
+### Lane B
+
+**B1. Reach fix for the shipped hint**
+- Test first:
+ - `tests/core/brain/page-lint.test.ts`: a withheld sibling is neither
+ scored, named, nor counted in `candidates_skipped` or
+ `candidates_unreadable`. A clean write still produces no `lint` key.
+ - `tests/mcp/note-write-reach.test.ts`: `brain_create_note`,
+ `brain_update_note`, `brain_append_note` and `brain_write_batch` never
+ name a withheld sibling.
+ - `tests/core/architecture/visibility-surface-census.test.ts`: the new
+ registry row is pinned.
+- Implement:
+ - `src/core/brain/page-lint.ts`: `LintWrittenPagesOptions` with
+ `readable` (contract 0.4), applied in `collectNearDuplicateCandidates`
+ before the census.
+ - `src/mcp/brain/notes-tools.ts`: `noteWriteResult` passes
+ `readableAtContextReach(ctx)`.
+ - `src/core/search/visibility-surface-registry.ts`: add a row for the
+ note-write lint, and a note on the `brain_review_candidates` row for
+ `retire_siblings`.
+
+ Until A1 lands, page-lint declares its own local predicate type with the
+ same shape as `ReadableRef`. B3 switches it to the kernel import.
+- Commit: `fix(brain): filter near-duplicate receipt candidates by caller reach`
+
+**B2. Keyword-index widening collector**
+- Test first: `tests/core/brain/page-lint-widening.test.ts`, using a fixture
+ index built by the indexer. Cases:
+ - a near-identical page in another directory is returned;
+ - a withheld page is dropped before it is returned;
+ - a page deleted from disk but still in the index is dropped;
+ - a missing index returns `status: "index_unavailable"` and empty
+ candidates;
+ - the top-k bound holds.
+- Implement: `src/core/brain/page-lint-widening.ts`:
+ 1. build the query with `buildFtsMatch` over the written body;
+ 2. pull hits with `keywordTopK` (limit `NEAR_DUPLICATE_WIDENING_TOP_K`,
+ declared locally until A1 lands);
+ 3. map them to document paths, apply `readable`, and re-read each page
+ from disk;
+ 4. project each page into a `NearDuplicateCandidate`.
+
+ It must not import from `src/core/search/embeddings/`.
+- Commit: `feat(brain): collect cross-directory near-duplicate candidates from the keyword index`
+
+**B3. Kernel scoring and opt-in widening on the receipt** (after A1)
+- Test first:
+ - `tests/core/brain/page-lint.test.ts`: findings and messages from the
+ same directory are byte-identical to v1.73.0. With
+ `extraCandidates`, a cross-directory finding has the same code, and its
+ message carries `method=lexical`. The census `widening` field is
+ present only when it is passed.
+ - `tests/mcp/brain-create-note.test.ts`: with
+ `near_duplicate_write_widening_enabled` on, a create in another folder
+ reports the earlier page. With the key off, the receipt is unchanged.
+- Implement:
+ - `nearDuplicateFindings` scores through `findNearDuplicates`, using the
+ threshold `writeHint` and the scope key as `bucket`.
+ - `NEAR_DUPLICATE_JACCARD` becomes an alias of the table value.
+ - `notes-tools.ts` runs `collectWideningCandidates` only when
+ `resolveNearDuplicateWriteWideningEnabled()` is on.
+- Commit: `feat(mcp): widen the near-duplicate receipt hint beyond the directory behind a flag`
+
+**B4. Feedback advisory on the kernel** (after A1)
+- Test first: `tests/core/brain/write-advisory.test.ts`. The advisory output
+ and the `write-conflict-advisory` log line are byte-identical for the
+ existing fixtures, and the threshold still comes from
+ `BRAIN_HEALTH_DEFAULTS.contradiction_jaccard`. The only allowed behaviour
+ change is that a principle under `NEAR_DUPLICATE_MIN_TOKENS` no longer
+ advises; the test pins that case.
+- Implement: `src/core/brain/health/contradiction.ts`, where the
+ `adviseOnIncoming` scoring loop calls `findNearDuplicates`. Reach stays
+ where `write-advisory.ts` applies it today, through the `readable`
+ argument.
+- Commit: `refactor(brain): score the feedback write advisory through the shared kernel`
+
+**B5. Stage instrumentation on write-side lookups** (after D1)
+- Test first:
+ - `tests/mcp/brain-create-note.test.ts`: with route metrics on, a create
+ record carries `lint` and `near_duplicate_lookup` stages with finite,
+ non-negative values. Only presence is asserted, never order.
+ - `tests/core/brain/write-advisory.test.ts`: a feedback write records
+ `near_duplicate_lookup`.
+- Implement:
+ - `notes-tools.ts`: wrap the lint in `timeStageSync(ROUTE_STAGE.lint, ...)`,
+ and widening plus scoring in `ROUTE_STAGE.nearDuplicateLookup`.
+ - `write-advisory.ts`: wrap the advisory computation in
+ `ROUTE_STAGE.nearDuplicateLookup`.
+- Commit: `feat(mcp): time the near-duplicate lookup and receipt lint stages`
+
+### Lane C
+
+**C1. Turn timestamp in the envelope**
+- Test first: `tests/core/brain/extract-signals.test.ts`:
+ - `MinedTurn.timestamp` equals the stored string verbatim;
+ - the transcript line is `[turnId @ <ts>] text`;
+ - an empty timestamp renders `[turnId] text`, and `now` is never
+ substituted;
+ - assistant turns stay excluded.
+
+ If `tests/core/brain/extract-signals-prefilter.test.ts` pins the old line
+ format, update it in this commit.
+- Implement: `src/core/brain/extract-signals.ts`, at `MinedTurn` (`:186-189`),
+ its construction (`:385`) and `buildMiningStep` (`:415-417`).
+- Commit: `feat(brain): carry the turn timestamp into mined extract turns`
+
+**C2. Language-neutral hygiene rules in the prompt**
+- Test first: `tests/core/brain/extract-signals.test.ts`. The envelope
+ instruction contains the five rule clauses:
+ 1. ISO bounds;
+ 2. skipping conversational mechanics;
+ 3. one rule per item;
+ 4. conditions kept;
+ 5. no restatements.
+
+ The test also asserts that the envelope carries no per-language example
+ words beyond what is there today.
+- Implement: the `buildMiningStep` prompt and `schema_hints`.
+- Commit: `feat(brain): add date-grounding and hygiene rules to the extract envelope`
+
+**C3. Duplicate-topic refusal**
+- Test first: `tests/core/brain/extract-signals.test.ts`:
+ - two items with the same `topic` are refused whole under `crossItem`,
+ and the message names both indices;
+ - distinct topics pass;
+ - the refusal happens before any write and before `computeDedupHash`.
+- Implement: extend the single `registerResponseCheck` closure
+ (`:302-329`).
+- Commit: `feat(brain): refuse extract payloads that repeat a topic`
+
+**C4. `turns_mined` timestamp projection**
+- Test first: `tests/mcp/extract-signals-tool.test.ts`. The `turns_mined`
+ entries carry `timestamp`, and the CLI JSON output carries the same field,
+ asserted through the MCP bridge fixture.
+- Implement: `src/mcp/brain/extract-tools.ts:61` and
+ `src/cli/brain/verbs/extract-signals.ts:86`.
+- Commit: `feat(mcp): expose the turn timestamp in turns_mined`
+
+**C5. Downstream temporal proof and docs**
+- Test first: `tests/core/brain/extract-signals-temporal.test.ts`. An
+ extract payload whose principle carries an ISO bound produces a signal.
+ A dream pass over that signal yields `valid_until` through the existing
+ `extractTemporalConstraints`, with no new parser.
+- Implement: `docs/cli-reference.md`, covering the envelope rules, the line
+ format and the new refusal. No source change is expected. If the test
+ exposes a gap, fix it in `extract-signals.ts` only.
+- Commit: `test(brain): prove extract date grounding reaches the dream temporal window`
+
+### Lane D
+
+**D1. Route scope (substrate)**
+- Test first: `tests/core/route-scope.test.ts`:
+ - with no scope open, both helpers are a no-op and return the value of
+ `fn`;
+ - inside `run`, a sync stage is recorded;
+ - repeated names are summed in first-seen order;
+ - values are rounded to 0.1 ms;
+ - an error inside a stage still records and rethrows;
+ - `decisionMs()` matches the old behaviour;
+ - `isRouteStageName` rejects names outside the set.
+- Implement:
+ - `src/core/route-scope.ts` (contract 0.6).
+ - `src/core/decision-model/latency.ts` becomes a re-export plus a thin
+ wrapper. Run `tests/core/decision-model/run.test.ts` unchanged to prove
+ that nothing regressed.
+- Commit: `feat(core): add a shared route scope for decision time and write stages`
+
+**D2. Stages on the route-latency payload**
+- Test first: `tests/core/brain/mcp-route-metrics.test.ts`:
+ - `stages` is written only when it is non-empty;
+ - an unknown name, a negative value and a non-finite value are dropped;
+ - a hostile stage name, such as a path or a topic, never reaches the
+ persisted continuity file;
+ - records without stages are byte-identical.
+- Implement: `src/core/brain/mcp-route-metrics.ts`, in
+ `McpRouteLatencyInput` and in the payload build next to the `decision_ms`
+ guard.
+- Commit: `feat(brain): carry allowlisted write stages on mcp_route_latency`
+
+**D3. Server wiring**
+- Test first: `tests/mcp/route-metrics-tool.test.ts`:
+ - with the gate on, a `brain_feedback` call through `callTool` yields a
+ record with `stages`;
+ - with the gate off, no record is written and no scope is opened;
+ - `decision_ms` behaviour is unchanged.
+- Implement: `src/mcp/server.ts`. `invokeToolHandler` uses
+ `createRouteScope()` and spreads `stages` next to `decisionMs`. The reach
+ gates stay before the timer.
+- Commit: `feat(mcp): open one route scope per tool call and emit its stages`
+
+**D4. Feedback-path stages**
+- Test first: `tests/mcp/route-metrics-tool.test.ts`. A `brain_feedback`
+ record carries `validate`, `idempotency_lookup`, `document_write`,
+ `idempotency_remember` and `log_append`, and with `force_confirmed` it
+ also carries `preference_write`. The idempotency mismatch error still
+ propagates.
+- Implement:
+ - `src/core/brain/signal.ts`, at `:357`, `:382-399` and `:412`, without
+ reordering the hash computation;
+ - `src/mcp/brain/feedback-tools.ts`, at `:224` and `:270`.
+- Commit: `feat(brain): time the signal write stages`
+
+**D5. Note-write stages**
+- Test first: `tests/mcp/route-metrics-tool.test.ts`. A `brain_create_note`
+ record carries `document_write` and `write_receipt`, and a
+ `brain_write_batch` record carries `write_receipt`.
+- Implement:
+ - `src/core/brain/notes/create-note.ts`, at `:570` and `:594`;
+ - `src/core/brain/write-batch.ts`, at `:901`.
+- Commit: `feat(brain): time the note write and receipt stages`
+
+**D6. Per-stage roll-up and docs**
+- Test first: `tests/core/brain/mcp-route-metrics.test.ts`. The summary
+ gives each route a per-stage count, average and p95. A route without
+ stages has no `stages` key in its summary.
+- Implement:
+ - `summarizeMcpRouteLatency` in `mcp-route-metrics.ts`;
+ - the `brain_route_metrics` description in `src/mcp/brain/recall-tools.ts`;
+ - `docs/observability.md`: the payload rows, the list of instrumented
+ routes, the 0.1 ms precision, and a note that hook capture is not an MCP
+ route.
+- Commit: `feat(mcp): roll write stages up in brain_route_metrics summary`
+
+### Integrator
+
+**I1. Docs, changelog and version**
+- Run the full targeted test set of all four lanes, then `bun run typecheck`
+ (or the repo's equivalent script), then `oxlint` and `oxfmt --check` over
+ the whole tree.
+- `docs/mcp.md`:
+ - the receipt hint: reach, widening and the `method` text;
+ - `brain_review_candidates` `retire_siblings`;
+ - the extract envelope;
+ - the route-metrics row.
+- `CHANGELOG.md`: a `## [1.74.0] - 2026-10-07` entry and its
+ `[1.74.0]: https://github.com/itechmeat/open-second-brain/compare/v1.73.0...v1.74.0`
+ link reference. The entry states the deferred items: rewordings,
+ write-time embedding, the extract Jaccard refusal, hook timing and
+ assistant-turn acceptance.
+- Bump `package.json` to `1.74.0`, run `bun run scripts/sync-version.ts`,
+ then `bun run scripts/sync-version.ts --check` and
+ `bun run sync-plugin-mirrors:check`.
+- Commit: `chore(release): v1.74.0 near-duplicate defense`
+
+## 3. Integration rule
+
+1. All lanes commit to `feat/near-duplicate-defense` in this worktree. A
+ lane edits only its own files (section 1).
+2. **Contract changes.**
+ - A missing contract symbol is a blocker for the integrator. It is never
+ a reason to edit another lane's file.
+ - If a lane finds the contract wrong, it stops and reports. The
+ integrator amends section 0, and the owning lane adapts.
+3. **Commit order.** A1 and D1 should land first; both are small. A
+ consuming task (B3, B4, B5) does not commit until the substrate it
+ imports is committed and its tests pass on the branch.
+4. **Integrator duties.**
+ 1. Run each lane's test files together, then the architecture censuses:
+ `tests/core/architecture/visibility-surface-census.test.ts`,
+ `egress-census.test.ts`, `write-site-census.test.ts` and
+ `destructive-site-census.test.ts`. The expected delta is zero for
+ egress, write sites and destructive sites.
+ 2. Finish I1.
+ 3. Run the full gates once.
+5. **Hard-fail gates.** The PR is not opened while any of these fail:
+ 1. a surface lists or counts a near-duplicate candidate without a reach
+ predicate;
+ 2. a provider embed call is reachable from a dream, write or extract
+ path;
+ 3. a stage name outside `ROUTE_STAGE_NAMES` reaches disk;
+ 4. a new `moveToRetired` caller appears;
+ 5. `sync-version --check` or `sync-plugin-mirrors:check` fails.
+
+## 4. Commit discipline (every lane, every commit)
+
+1. **Stage by name only.**
+ - For a new file, register it first with `git add -N <new-file>`.
+ - Commit with `git commit --only <path> [<path> ...] -m "<subject>"`.
+ - Never use `git add -A`, `git add .`, `git commit -a`, or a pathspec
+ that covers another lane's files.
+2. **Format and lint your own files before every commit.**
+
+ ```
+ ./node_modules/.bin/oxfmt <your files>
+ ./node_modules/.bin/oxlint -c oxlint.json <your files>
+ ```
+
+ Both must be clean before you commit. The repo formats `.ts`, `.js`
+ and `.json` (the `fmt` script in `package.json`). Markdown is not
+ formatted, so don't pass `.md` files to oxfmt.
+3. **Run your task's tests first.**
+
+ ```
+ env HOME=$(mktemp -d) bun test <paths>
+ ```
+
+ They must be green before you commit.
+4. **If the pre-commit hook rejects the commit.** The hook checks the whole
+ tree, so another lane's work in progress can fail it. Wait 30 seconds and
+ retry, up to 5 times.
+
+ Only after the fifth rejection may the lane commit with `--no-verify`.
+ It must first prove its own files clean: rerun `oxfmt --check` and
+ `oxlint` over exactly its committed paths, run its tests, and quote both
+ outputs in the lane report. Never use `--no-verify` for a failure in the
+ lane's own files.
+5. **Write conventional subjects exactly as listed in section 2.** There is
+ no AI attribution trailer or marker in any commit.
+6. **The rest of the tree.**
+ - Never commit `.reviews/`, scratch files or another lane's paths.
+ - Never switch branches.
+ - Never use bare `git stash`.
diff --git a/docs/brainstorm/near-duplicate-defense/variants.md b/docs/brainstorm/near-duplicate-defense/variants.md
new file mode 100644
index 00000000..798fbda2
--- /dev/null
+++ b/docs/brainstorm/near-duplicate-defense/variants.md
@@ -0,0 +1,348 @@
+# Near-Duplicate Defense Across the Fact Lifecycle - brainstorm audit trail
+
+## Consultants
+
+The prompt is `cli-output/prompt.md`. Raw outputs are in `cli-output/`.
+
+1. **Primary: Claude Code** (`claude -p`). It exited 0 and returned four
+ variants (A-D) and a recommendation. Raw output: `cli-output/claude.md`.
+2. **Codex: not available.** The run stopped with
+ "You've hit your usage limit ... try again at Oct 11th, 2026 3:03 AM".
+ `cli-output/codex.md` is empty.
+3. **opencode: not available.** It rejected its command-line arguments and
+ printed its usage text instead of running. `cli-output/opencode.md` is
+ empty.
+4. **Second consultant: a second Claude Code instance**, briefed with an
+ adversarial lens. Its job was to attack the obvious shared design, find
+ the most likely failure modes, and propose counter-variants and the
+ minimal honest scope. It returned three counter-variants (C1-C3), a
+ recommendation (C1) and a gate list. Raw output:
+ `cli-output/second-claude.md`.
+
+Both transcripts are reproduced below verbatim.
+
+## Primary consultant (Claude Code) - verbatim
+
+# Near-duplicate defense: architectural variants
+
+I read all four premise reports. Every variant below is built on their corrections:
+
+- The forget-side scan hooks the dream retire plan (`dream.ts:297`) and the `reject` verb. It does not hook hygiene `forget`.
+- The v1.64.0 receipt hint is extended. It is not rebuilt.
+- Date grounding means a timestamp on `MinedTurn` plus a prompt instruction to write ISO dates.
+- Stage timings ride on `mcp_route_latency` and use stage names the write tools actually have.
+
+Two facts hold for every variant:
+
+- **The shipped write hint leaks.** `lintWrittenPages` has no reach predicate (`page-lint.ts:583`, `notes-tools.ts:487`), so a receipt can name a page the caller can't read. Every variant fixes this first, unconditionally.
+- **The timing premise is slightly off.** It says `brain_feedback` has no similarity stage. But `adviseIncomingFeedback` (`write-advisory.ts:163`) already runs a Jaccard pass on that route. This wave adds more lookups, so a `near_duplicate_lookup` stage becomes real on several routes. That is exactly the cost the timing card should measure.
+
+---
+
+## Variant A: "One lexical kernel, zero new spend"
+
+**Approach.** One new pure module, `src/core/brain/near-duplicate.ts`, built on `tokenise`/`jaccard`. It takes:
+- a probe;
+- a candidate source, injected as a small interface: directory siblings, a cross-directory search-index pull, active preferences, or the other items in the same payload;
+- a reach predicate, applied before scoring and before counting;
+- a cap.
+
+It returns `{ candidates: [{ ref, score, method: "lexical" }], census: { compared, capped } }`. All the thresholds sit in one named table: write hint 0.8, retire sibling 0.7, in-payload 0.8 for items with the same signal.
+
+Four callers use it:
+- **The retire-sibling projection.** It is computed after `planAutoRetires`, checked against the retires that are gated at apply time (`gatedRetireSlugs`), and shown as `retire_siblings` in `brain_review_candidates`. Retiring a sibling is accepted through the existing `apply_evidence outdated` or `reject`.
+- **The receipt lint.** It gets the reach fix, plus an optional cross-directory pull from the existing keyword (BM25) search index. That pull sits behind a config key that is off by default.
+- **The extract-signals check.** One `crossItem` check is added to the existing registration.
+- **`adviseIncomingFeedback`.** It is refactored onto the same core.
+
+Stage timings come from extending the decision-latency storage scope (an AsyncLocalStorage) into one shared route scope. Stage names are a closed allowlist that `emitMcpRouteLatency` enforces. They are written only to the continuity record, never to the tool response.
+
+**Pros**
+- No embedding spend and no egress.
+- Fully deterministic, and the dream dry run stays write-free.
+- One threshold table replaces three copies that are drifting apart (0.6, 0.7, 0.8).
+- It fits the budget, at roughly 40–45 files.
+- Low risk on Windows.
+
+**Cons**
+- Real paraphrases often score below a 0.8 Jaccard. Cross-topic comparison of short principles is noisy, which needs a minimum token count.
+- The "embeddings" half of the write-hint card stays unbuilt.
+
+**Cuts or defers**
+- The semantic tier.
+- Model-generated rewordings. "Reword" becomes "show the pair, and the operator rewrites it with the existing verbs."
+- Hook capture timing.
+- Acceptance-inside-a-proposal in extract-signals.
+
+---
+
+## Variant B: "Lexical kernel plus a vector tier that reuses paid embeddings" (two tiers)
+
+**Approach.** Same module, contract and callers as A. The difference is a second tier, `method: "embedding"`. It only does vector-to-vector cosine over vectors already stored in the search index, so it never calls `provider.embed`.
+
+- **Forget side.** The retiring preference and its siblings are both already indexed, so the tier costs nothing at the margin and runs whenever vectors exist.
+- **Write side.** The new body is not indexed yet. The tier is either skipped and disclosed as `semantic: not_indexed`, or, behind a key that is off by default, it embeds the body through `prepareQueryEmbed` with full spend reporting.
+
+The orphan `detectSemanticDedup` is reworked onto the stored-vector reader, so it no longer calls embed directly without a spend gate.
+
+**Pros**
+- Delivers the residual value the write-hint card actually has: catching paraphrases across directories and topics.
+- On the forget side it does so with no new spend.
+- Respects the v1.72.0/v1.73.0 spend gates by construction.
+- The `method` label in the receipt and in review candidates tells the operator which tier fired.
+
+**Cons**
+- It needs a reader that fetches stored vectors by page, which may not exist yet. The reader also has to match the index's chunking (page vector vs. chunk max).
+- Stored vectors go stale between reindexes.
+- About 50–55 files, at the top of the budget.
+- Promoting the write-side embed to on-by-default needs a labelled pair evaluation that this wave doesn't build.
+
+**Cuts or defers**
+- Write-time embedding stays opt-in and unevaluated.
+- Model rewordings.
+- Hook timing.
+
+---
+
+## Variant C: "Out-of-band candidate ledger"
+
+**Approach.** The write path keeps only the reach-fixed lexical lint for directory siblings. Every wider lookup moves off the hot path: cross-directory, cross-type, embedding-based and retire-sibling scans all run in the dream or hygiene pass after reindex. Results go to a persisted `near_duplicate` candidate ledger. `brain_review_candidates` and the next receipt that touches the page show the pending pairs.
+
+**Pros**
+- No latency added to the synchronous write tools.
+- The embedding tier always uses paid vectors.
+- One review queue covers the write side and the forget side.
+
+**Cons**
+- The hint arrives late. The writer who created the duplicate never hears about it in the same turn, which is what the write-hint card was for.
+- It adds a persisted store with its own reach filtering, expiry and staleness rules, which is new kernel state.
+- Extract-signals still needs a separate in-payload check.
+- The stage-timing card measures much less, because the expensive stage leaves the routes being timed.
+
+**Cuts or defers**
+- Same-turn cross-directory hints.
+- Rewordings.
+
+---
+
+## Variant D: "Model-assisted rewording proposals"
+
+**Approach.** Variant A's kernel finds the candidates. Then a new `needs_llm` step, registered in the existing step registry with response-shape and semantic checks, asks the agent for a verdict on each retire/sibling pair (same / related / different) and, for `related`, a proposed rewording. Proposals are written as pending review items. They are applied only when the operator accepts them through a new accept verb. The pair-verdict machinery (`runPairVerdicts`, `advisoryUseActive`) is reused when a decision model is configured.
+
+**Pros**
+- Matches the card's literal "accept-gated rewording proposals".
+- The kernel stays model-free; the model runs only behind the step registry and the egress registry.
+
+**Cons**
+- It adds a new executor that writes, an accept verb, a pending-proposal store and a new response surface, all on top of the kernel. That is about 55–65 files, which breaks the budget.
+- More egress of preference principles.
+- The weakest correctness story of the four. The upstream project measured a rise in wrong merges even from a plain hint.
+
+**Cuts or defers**
+- It has to drop either the stage-timing card or the vector tier to fit.
+
+---
+
+## Cross-cutting choices (the same in A, B and C)
+
+- **Write receipt.** Extend the hint; don't replace it. Keep the code `near-duplicate`, add `method`, and keep the receipt byte-identical when there is no finding. Withheld siblings are dropped before they are counted.
+- **Defaults.** Cross-directory widening and retire-sibling surfacing are behind config keys that are off by default. Only the reach fix is on unconditionally.
+- **Stage names.** A closed set:
+ - `validate`
+ - `idempotency_lookup`
+ - `near_duplicate_lookup`
+ - `document_write`
+ - `idempotency_remember`
+ - `log_append`
+ - `preference_write`
+ - `write_receipt`
+ - `lint`
+
+ Timing uses separate sync and async helpers with `try/finally`. Stages are rounded to 0.1 ms. The docs list which routes emit stages, so a missing array reads as "not instrumented" rather than "zero cost". Stages are never written to the tool response.
+- **Extract-signals.** Turn timestamps are rendered verbatim, and the line drops the `@` marker when the timestamp is empty. Hygiene rules go in as language-neutral prompt text. Two `crossItem` refusals are added: duplicate `topic` in one payload, and principles that are too similar within the same `signal`, using the shared kernel's threshold.
+
+---
+
+## Recommendation: Variant B, built in stages
+
+Build A's kernel and callers first; they carry the whole wave. Then add only the stored-vector tier, active on the forget side and used opt-in on the write side.
+
+B is the only variant that delivers the write-hint card's real remaining value (catching paraphrases across directories and across facts) without breaking the spend gates. It reuses vectors that are already paid for and keeps the write-time embed off by default. It also keeps the hint in the same turn, unlike C, and avoids a new executor that writes, unlike D. The rewording half becomes an operator action on a surfaced pair, which meets "accept-gated" without a model.
+
+If the stored-vector reader turns out not to exist, or not to line up with the index's chunking, B drops back cleanly to A by turning off one tier. Nothing about the contract or the callers changes.
+
+## Second consultant (second Claude Code instance, adversarial lens) - verbatim
+
+# Adversarial review: Near-Duplicate Defense wave
+
+## 1. Most likely failure modes
+
+**F1. Wrong-merge or wrong-retire churn from cross-bucket lexical matching.** This is the most likely way the wave goes wrong. The obvious shared design is one candidate lookup, cross-directory and cross-topic, maybe with embeddings, wired into retire, write and extract. Cross-bucket Jaccard on short principles is noisy. "use tabs" and "never use tabs" score high, and the premise reports already warn about stop-token overlap. Two parts make it worse:
+- On the retire side, one `outdated` event (`dream-refresh.ts:215-231`, which already ignores pin) produces a sibling list. Operators or agents who mechanically apply `outdated` to every listed sibling cause a cascade.
+- On the extract side, an in-payload Jaccard refusal rejects the whole payload (refusal, not filtering). One false positive therefore drops every valid signal in that run, silently from the operator's point of view.
+
+Upstream measured wrong merges rising from 2.50% to 3.75% with a default-on hint. This vault has more short, polarity-sensitive rules than upstream's pages.
+
+**F2. Visibility leak that grows with the reach of the lookup.** The shipped `near-duplicate` lint already leaks: `lintWrittenPages` has no `readable` predicate (`page-lint.ts:583`, called at `notes-tools.ts:487`). It discloses withheld sibling paths and scores, and leaks through the `candidates_skipped` and `candidates_unreadable` counts. A shared lookup that goes vault-wide (FTS widening, embeddings, cross-topic retire siblings) turns a same-directory leak into a vault-wide one. The leak then appears on four surfaces at once: write receipts, `brain_review_candidates`, the extract envelope and the dream summary. Calling this "recall-quality regression" understates it. It is a reach-invariant break, and it is the most damaging outcome if it ships.
+
+**F3. Spend and latency leaks hidden by the instrumentation meant to measure them.** Two risks:
+- **Spend.** A semantic layer on the write path costs one embed per note write, multiplied by `brain_write_batch`. The tempting reuse, `detectSemanticDedup`, calls `provider.embed` directly with no gate (`dedup.ts:154`).
+- **Timing.** The stage card measures only MCP `tools/call`. Its natural stage set has no `similarity_lookup` stage, because none exists today. If the lookup lands without a stage name in the closed allowlist, the added cost is invisible, which defeats the wave's own spine claim. If stage names are built from input, or the `stages` array is emitted on every record including the gate-off path, records bloat and the content-free claim breaks.
+
+Secondary hazards:
+- **Windows CI.** Asserting positive or ordered sub-millisecond durations, or mtime-ordered candidate fixtures, will flake on Windows timer and filesystem resolution. Locale-formatted timestamps in the transcript line will differ by OS.
+- **Dry-run versus real-run drift.** Siblings of retires that are gated at apply time (`dream-apply.ts:298-311`) get reported as if they were retiring.
+
+## 2. Counter-variants
+
+### C1 "Fix the leak, measure, then defend" (minimal honest scope)
+Ship the reach fix for the existing lint, stage timings and the language-neutral extract rules. Ship the retire-sibling scan as a read-only, lexical, same-signal projection in `brain_review_candidates`. There is no shared vault-wide lookup, no embeddings and no write-side widening. The shared primitive is a small pure helper over `tokenise`/`jaccard`: `nearDuplicateCandidates(subject, pool, {threshold, minTokens, readable})`. It returns `{id, score, method: "lexical"}[]`, filters by the reach predicate before scoring and counting, and is called by the retire projection, the extract cross-item check and the lint.
+
+- **Pros**
+ - Zero new spend or egress.
+ - Closes a real shipped defect.
+ - Every new output is advisory or read-only.
+ - Each new behaviour can be traced through the timings.
+ - Roughly 30-38 files.
+- **Cons**
+ - The cross-directory and paraphrase cases from t_fda66477 stay unsolved.
+ - The card's headline ("hint an existing similar page") ships as "the hint is now safe", not "the hint is smarter".
+- **Cuts and defers**
+ - FTS-backed widening.
+ - The semantic layer and its labelled-pair evaluation.
+ - Decision-model pair verdicts on retire siblings.
+ - Rewording proposals, which are cut outright and not just deferred.
+ - Hook-capture timing.
+ - Assistant-turn acceptance.
+
+### C2 "Shared lookup, lexical lanes only, widening behind a flag"
+Same as C1, plus an index-backed lexical candidate pull (FTS/BM25 top-k, rescored with Jaccard) for the write lint. It is behind a config key that defaults to off, with a time and size bound and a disclosed skip when a bound trips.
+
+- **Pros**
+ - Catches cross-folder duplicates at zero embedding spend.
+ - Proves the shared lookup on a second candidate source.
+- **Cons**
+ - Adds a search dependency on the hot write path.
+ - Index staleness: the newly written page is not indexed yet, and recently written siblings may not be either.
+ - Needs its own cost test.
+ - Ships a default-off feature with no evaluation to justify ever turning it on.
+- **Cuts and defers**
+ - Embeddings.
+ - Pair verdicts.
+ - Rewording.
+ - The promotion evaluation.
+
+### C3 "Retire side only, write side is a bug fix" (narrowest)
+Ship only the retire-sibling projection and the reach fix. Defer t_b915b9cc and t_72e93e18 to the next wave.
+
+- **Pros**
+ - Smallest diff, about 20 files.
+ - A single new behaviour to review.
+- **Cons**
+ - Drops the spine's "measured cost" leg.
+ - The timing card is the cheapest and safest of the four, so cutting it removes risk nobody was carrying.
+ - The date-grounding part of t_b915b9cc is a free win: the downstream ISO extraction already exists.
+- **Cuts and defers**
+ - Two whole cards.
+
+**Attack on the ambitious first-consultant shape** (one embedding-backed lookup across all four paths): it fails all three tests in section 1 at once. It adds new egress and spend on the write path, a vault-wide reach surface and a whole-payload refusal triggered by semantic similarity. It also has no evaluation that can measure any of this, because `rerank-eval-gate.ts` measures recall hit@k and MRR, not duplicate-catch or wrong-hint rates. Reject any variant that turns on embeddings without a labelled-pair evaluation landing in the same PR.
+
+## 3. Minimal honest scope (my recommendation: C1)
+
+**Ship in one PR:**
+1. **t_fda66477, reach fix only.** Add a `readable` predicate to `lintWrittenPages` and `collectNearDuplicateCandidates`, threaded from `noteWriteResult` so it covers all four note-write tools. Withheld siblings are neither scored nor counted. Add a visibility-surface registry row for the note-write lint and a byte-identical clean-receipt test.
+2. **t_acab97de, a read-only `retire_siblings` field.** Compute it after `planAutoRetires` (`dream.ts:297`), and add it to `DreamRunSummary` and `buildReviewCandidates`.
+ - **Triggers:** superseded-by-context, rebutted, quarantine-violated, and user-rejected. Decay retires and merges are excluded.
+ - **Matching:** lexical, threshold ≥0.7, a minimum token count, same `signal` sign only, capped like `DEDUP_CANDIDATE_CAP`.
+ - **Reconciliation:** check against gated retires.
+ - **Acceptance:** only through existing verbs (`outdated` evidence, `reject`). No new executor. Nothing retires automatically.
+3. **t_b915b9cc.** Carry the turn timestamp into `MinedTurn`. Render it verbatim, and leave it out when empty, never substituting `now`. Add the ISO-bound instruction and the prompt-only hygiene text. The cross-item check refuses duplicate `topic` values only. Ship the Jaccard cross-item refusal only if fixtures prove zero false refusals on short same-signal principles; otherwise defer it.
+4. **t_72e93e18.** Use one shared ALS scope that carries both `decision_ms` and `stages`, so there is no second store. Stage names form a closed allowlist enforced in `emitMcpRouteLatency`, and it includes `near_duplicate_lookup` for the lint. The `stages` key appears only when at least one stage was noted, and only when the existing default-off gate is on. Durations use 0.1 ms rounding. Docs list which routes emit stages and state that hooks are out of scope.
+
+**Explicitly left undone, and stated in CHANGELOG and docs:**
+- Rewording proposals. A model-generated reword is a write of new content, so it is not "accept-gated surfacing".
+- Any embedding or semantic layer, and the labelled-pair evaluation it needs.
+- FTS cross-directory widening.
+- Decision-model verdicts on siblings.
+- Hook-capture stage timing.
+- Assistant-turn acceptance.
+- Any change to `computeDedupHash`.
+- The hygiene `forget` anchor, which stays untouched.
+
+## 4. Gates that must hard-fail the build
+
+| Gate | Must fail when |
+|---|---|
+| **Visibility-surface census** (`tests/core/architecture/visibility-surface-census.test.ts`) plus a reach test | Any surface returns a sibling id, path, score or count without a reach predicate. This is the gate that matters most for this wave. |
+| **Egress census** (`egress-census.test.ts`) | Any new network path appears. Under C1 the expected delta is zero, so any delta means the scope slipped. |
+| **Embedding spend counters / `planEmbeddingSpend` path** | Any `provider.embed` call is reachable from a dream, write or extract path without going through `prepareQueryEmbed`. Under C1 that call count must also be zero. Add an assertion that `detectSemanticDedup` still has no production caller. |
+| **Architecture census, specifically write-site and destructive-site** (`write-site-census`, `destructive-site-census`) | A new write or a new caller of `moveToRetired` appears. The retire projection must add neither, since it is accept-gated by construction. |
+| **Route-metrics payload safety** | A stage name outside the allowlist reaches disk. Test with a hostile name on a persisted record. |
+| **Response-check registry** | A second `registerResponseCheck` on the extract surface is registered (it already throws), or a violation code appears outside the closed set. |
+| **`sync-version --check`, `sync-plugin-mirrors:check`, Windows job** | These standing gates are already blocking. |
+
+**The rerank evaluation gate should not be a gate for this wave.** It measures something else, and citing it as cover would be a false assurance. If a later wave adds a semantic hint, its own duplicate-catch/wrong-hint evaluation must hard-fail before the config default can change.
+
+**Why C1:** the wave can close a real shipped reach leak, measure its own cost, and make retirements visible without adding any spend, egress or automatic action. Every ambitious variant depends on a precision claim (cross-bucket or semantic near-duplicate detection) that no gate in the repo can currently test. The near-duplicate helper stays small and reach-aware, so a later semantic lane plugs into it rather than forking a second threshold. Leaving rewording and embeddings undone is honest, since nothing here could catch it if they were wrong.
+
+## Final rationale (design writer)
+
+**Chosen: the primary consultant's Variant B, built in stages.** Three
+guards from the adversarial consultant's C1 are added on top:
+
+1. **The reach fix comes first and is unconditional.** The shipped
+ `near-duplicate` receipt lint (`page-lint.ts:583`, `notes-tools.ts:487`)
+ discloses withheld siblings today, and every wider lookup would make that
+ leak larger.
+2. **Every new behaviour is an advisory projection, off by default.** It
+ reads only, and an operator accepts it through existing verbs
+ (`brain_apply_evidence result: outdated`, `o2b brain reject`).
+3. **No embedding provider call anywhere in the wave.** A source-scan test
+ pins this.
+
+**Why B rather than C1.** The write-hint card's residual value is the
+extension beyond the hint that already ships: reach across directories,
+scoring by embedding, and comparing facts. C1 ships none of it. B's forget
+side delivers the semantic part at zero marginal spend, because both the
+retiring preference and its siblings are already indexed. The pattern is
+already proven in the review projection: `scoreSignalNovelty` in
+`src/core/brain/surprisal.ts` reads stored vectors through the
+`opts.searchConfig` path of `buildReviewCandidates`. The reader the first
+consultant feared might be missing exists as `storedEmbeddingsForDocument`
+(`src/core/search/store/vectors.ts:262`). On the write side, the extension
+beyond the directory is C2's BM25 pull from the keyword index, behind an
+off-by-default key. It adds no embeddings.
+
+**What the adversarial review changed:**
+
+- **Write-time embedding is cut**, not offered as an opt-in. The new body is
+ not indexed, so a semantic write hint is real spend. No gate in the repo
+ can measure how many duplicates it would catch against how many wrong
+ hints it would give, and `rerank-eval-gate.ts` measures recall, not
+ that.
+- **The Jaccard refusal across extract items is deferred.** Refusal rejects
+ the whole payload, so one false positive on two short, opposite rules
+ would drop every valid signal. Only the exact duplicate-topic rule ships.
+- **`readable` is a required argument of the shared kernel**, not an
+ option, and it is applied before scoring and before counting.
+- **One AsyncLocalStorage scope** carries both decision time and stages.
+ The allowlist of stage names is enforced where the record is emitted, and
+ `near_duplicate_lookup` is in the allowlist.
+- **The hard-fail gate list** (visibility, egress, spend, write-site,
+ destructive-site, route-metrics payload safety) became the integration
+ rule in `plan.md`. The rerank evaluation gate is not cited as cover.
+
+**One premise correction:** the adversarial consultant suggested comparing
+"same signal sign only" for retire siblings. Preferences carry no sign
+(`BrainPreference`, `src/core/brain/types.ts:875`), so that guard is
+replaced by a minimum of 4 tokens, the accept gate and the off-by-default
+key.
+
+**Rejected variants:**
+
+| Variant | Why rejected |
+|---|---|
+| C (out-of-band ledger) | The hint arrives late, and a new persisted store adds kernel state. |
+| D (model-assisted rewordings) | Breaks the budget, adds egress, and has the weakest correctness story. |
+| C3 (retire side only) | Drops two cards, including the cheapest and safest one. |
+| A on its own | Leaves the semantic half unbuilt, although stored vectors make it free on the forget side. |
diff --git a/docs/cli-reference.md b/docs/cli-reference.md
index eff65f88..d23edbe8 100644
--- a/docs/cli-reference.md
+++ b/docs/cli-reference.md
@@ -851,8 +851,9 @@ o2b brain orphan-repair [--apply --confirm "apply orphan repair"] - detach
o2b brain design-note <topic> [--payload <json> | --payload-file <path>] [--agent <name>] [--json] - the one-shot sibling of `o2b brain panel`. Without a payload it is read-only: it grounds the topic in the vault's tension records, decision records and truth projections and prints the single needs-llm-step envelope the calling agent answers, naming any store the vault holds nothing in (which is not the same as a store that matched nothing). With a payload it validates the written note and commits it as Brain/decisions/design-<date>-<topic>.md. The note must weigh named alternatives and mark EXACTLY ONE recommended: zero and two-plus are both refused, and the refusal states the count. A second note for the same topic on the same day is refused, never overwritten
o2b brain skill-proposals page-candidates [--json] - read-only: gate the vault's user pages on the page-meta trio (core tier, non-stale lifecycle, high confidence) and an observed-reuse floor, skip any page an installed skill already covers, and return one needs-llm-step envelope per admitted page plus every skip with its reason
o2b brain skill-proposals page-draft <page> (--payload <json> | --payload-file <path>) [--json] - validate a returned SKILL.md draft and stage it as a pending mature_page proposal INSIDE the vault; accept is what materializes the SKILL.md under the configured skills root, through the write-ahead journal
-o2b brain extract-signals <session-ref> [--payload <json> | --payload-file <path>] [--agent <name>] [--json] - mine durable taste signals from an already-imported session's user turns. Without a payload it is read-only: it prints the turns it would mine and the single needs-llm-step envelope the calling agent answers. With a payload it validates the answer and writes the accepted items into Brain/inbox/ as speculative source_type: auto_extract signals, subject to the durability denylist and to Brain/pending/ staging when write approval is on. A payload over the per-session cap, or an item below the confidence floor, refuses the whole payload by name and writes nothing; a session with no imported turns is refused, never reported as empty
+o2b brain extract-signals <session-ref> [--payload <json> | --payload-file <path>] [--agent <name>] [--json] - mine durable taste signals from an already-imported session's user turns. Without a payload it is read-only: it prints the turns it would mine and the single needs-llm-step envelope the calling agent answers. With a payload it validates the answer and writes the accepted items into Brain/inbox/ as speculative source_type: auto_extract signals, subject to the durability denylist and to Brain/pending/ staging when write approval is on. A payload over the per-session cap, an item below the confidence floor, or two items sharing a topic refuses the whole payload by name and writes nothing; a session with no imported turns is refused, never reported as empty
With the optional decision-model turn pre-filter (use extract_prefilter, docs/decision-models/extract-prefilter.md) the --json plan may add turns_dropped, skipped { reason: decision_model_prefilter, turns_dropped } with llm_step: null (nothing to mine), and decision_model { degraded }; none appears with the use off. Payload items accept an optional string source_turn
+ Since v1.74.0 every turns_mined entry carries timestamp, the turn's stored timestamp verbatim ("" when the source has none), and each prompt line reads [<turn-id> @ <timestamp>] <text>, or [<turn-id>] <text> for an undated turn - the run's clock is never substituted. The envelope instruction holds the items to five language-neutral rules: a time bound is written as an ISO 8601 date or interval resolved against the stating turn's timestamp (an end-only bound as an interval, because the dream pass reads a lone date as a start), conversational mechanics are skipped, one rule per item, a rule's conditions stay in its principle, and restatements are dropped. The dream pass turns the ISO bound into the preference's valid_from / valid_until through its existing temporal extraction; the extract lane writes neither field itself. Two items with the same topic refuse the payload under semantic_cross_item, the message naming both indices (items[i] and items[j] share topic "<topic>"), before any item is hashed or written
o2b brain deep-synthesis --json now also carries findings with causal_context, decomposed confidence (support, opposition, freshness, coverage), and the excluded_findings ledger with excluded_finding_count
```
diff --git a/docs/mcp.md b/docs/mcp.md
index 66f95b50..1ff5a18a 100644
--- a/docs/mcp.md
+++ b/docs/mcp.md
@@ -367,7 +367,7 @@ flags for a narrower per-process full server.
| `brain_context_pack` | Budgeted context slice; pass `lanes: true` to return directives, constraints, and consider lanes. Filtered items include `safety.reasons`. Each item carries a structural `epistemic` status (`observed` \| `derived` \| `hypothesis` \| `plan` \| `unknown`) plus `evidence_refs` derived from existing graph metadata; fields are absent when the status is `unknown`. Takes the same adequacy pair as `brain_recall_gate` under its own spelling, `recall_scores` AND `match_quality`: both or neither, an incomplete pair is refused, and the schema states it as `dependentRequired` keyed on this tool's own scores name. | `max_tokens` |
| `brain_context_receipts` | List or show opt-in prompt context receipt continuity records with budgets, hashes, source refs, safety/redaction metadata, and item IDs. | `operation` |
| `brain_recall_telemetry` | List or summarise opt-in recall telemetry records for search, context-pack, and pre-compress calls. | `operation` |
-| `brain_route_metrics` | List or summarise opt-in route-level MCP tool latency (`mcp_route_latency` records); `summary` rolls each tool up into count, error count, and min/avg/max + p50/p95/p99 latency, slowest-first. Emitted only when `mcp_route_metrics_enabled` is on; payload-safe (tool, scope, status, duration, arg key names). Read-only. | `operation` |
+| `brain_route_metrics` | List or summarise opt-in route-level MCP tool latency (`mcp_route_latency` records); `summary` rolls each tool up into count, error count, and min/avg/max + p50/p95/p99 latency, slowest-first, plus a per-stage count/avg/p95 for a route whose records carry write stages. Emitted only when `mcp_route_metrics_enabled` is on; payload-safe (tool, scope, status, duration, arg key names, stage names). Read-only. | `operation` |
| `brain_retrieval_plan` | Shadow-only retrieval advisor for one `question`: composes query intent/weights, the summary-surface route, the context-pack density allocation, the token-impact ledger, and observed route p95 latency into a strategy, token-budget allocation, graph-expansion advice, reliability, and a marginal-value stop. Optional `token_budget`. Read-only; exposes no mutating parameters and changes no ranking. | `question` |
| `brain_token_impact` | Durable value-of-memory ledger: `record` posts a context pack's tokenizer-exact prompt-token delta (`baseline` − `packed`, `method` exact/fallback) plus an optional modeled inference-avoidance estimate; `outcome` posts first-pass/repair/retry to calibrate the model; `summary` keeps EXACT prompt-token savings strictly separate from the MODELED (outcome-calibrated) figure; `list` reads raw samples. Writes gated on `token_impact_ledger_enabled` (default off); payload-safe (counts + opaque pack id only). Reads ignore the gate. | `operation` |
| `brain_context_pack_outcome`| Agent-operable outcome loop over the context-pack quality ledger: `post` records one compact outcome row for a carried context-pack quality-sample id — first-pass/repair/retry counters plus three STRICTLY SEPARATE token signals (`exact_prompt_token_savings`, `modeled_inference_avoidance`, `observed_provider_tokens`) — and composes the token-impact ledger by posting a matching first-pass/repair/retry calibration outcome; `list`/`summary` read the rows keeping the signals separate. Writes gated on `context_pack_outcome_enabled` (default off); payload-safe (counters + opaque sample id only), a field the caller omits is never invented. An optional `agent_id` names the ACTING agent and is recorded on every row the post lands; it is a tool ARGUMENT rather than the server's own config identity, so a config the server cannot read can never fail a telemetry post, and omitting it records no actor rather than a guessed one. Reads ignore the gate. | `operation` |
@@ -1137,6 +1137,18 @@ gated and fail-open, so it can never fail or slow-fail the call it
measures beyond one synchronous continuity append. `brain_route_metrics`
reads them back (`operation: "list"|"summary"`).
+Since v1.74.0 a record can also carry `stages`: the wall time the call
+spent in each write stage, as `{ name, ms }` entries. Names come from a
+closed nine-name allowlist (`validate`, `idempotency_lookup`,
+`near_duplicate_lookup`, `document_write`, `idempotency_remember`,
+`log_append`, `preference_write`, `write_receipt`, `lint`); any other
+name and any negative or non-finite value is dropped before the record is
+written, and the key is absent when no stage ran. Stage timings never
+appear in the response of the call being measured, only in its
+continuity record. `brain_route_metrics` `summary` adds a `stages`
+roll-up per route (count, average and p95 per stage name); a route
+without stages has no `stages` key.
+
The full observability contract behind these tools - event kinds,
always-on vs opt-in status, correlation IDs, payload safety, and the
continuity schema version - lives in `docs/observability.md`.
@@ -1310,6 +1322,25 @@ as authored, frontmatter is never compared or touched, an unreadable or
over-cap sibling is excluded as a candidate, and the finding never gates
the write - the receipt says what landed next to what.
+Since v1.74.0 the lint answers at the caller's reach. A sibling the caller
+may not read is never named, scored or counted on the receipt, and the
+broken-wikilink and merged-link checks treat a withheld page as absent: a
+wikilink to it reads as broken, and a merge chain ends at its first
+withheld hop.
+
+With the config key `near_duplicate_write_widening_enabled`
+(`OPEN_SECOND_BRAIN_NEAR_DUPLICATE_WRITE_WIDENING_ENABLED`, default off)
+set to `"true"`, the near-duplicate check also compares the written pages
+with candidates from other directories, pulled from the keyword index
+(the top 20 chunks per written page) and re-read from disk, under the
+same 0.8 bar, scope bucket and reach. The pull spends no embedding call.
+An index that cannot be opened, queried or closed, or any other failure
+while collecting the candidates, does not fail the write and does not
+disable the same-directory check: the report then carries
+`widening: "index_unavailable"` and a `widening_detail` code naming the
+failure, never a message or a path. With the key off nothing is opened
+and the receipt is unchanged.
+
The lint runs AFTER the commit and reads what is on disk. It never gates
the write, and it never throws. A call that authored no bytes - a
`brain_create_note` with `if_exists: "skip"` that skipped, a log-only
@@ -2426,3 +2457,39 @@ format characters), when it contains NUL, or when it exceeds the cap.
caller keyword-only (the composite deadline, an empty query vector, an
empty fit recorded as `semantic-query-empty-fit`), with a remedy naming `search_hybrid_deadline_ms` or the
input window. Each refusal names the remedy.
+- Since v1.74.0 a context-driven retire (`superseded-by-context`,
+ `rebutted`, `quarantine-violated`, `user-rejected`) can name the active
+ preferences that resemble it. With the config key
+ `near_duplicate_retire_siblings_enabled`
+ (`OPEN_SECOND_BRAIN_NEAR_DUPLICATE_RETIRE_SIBLINGS_ENABLED`, default
+ off) set to `"true"`, `brain_review_candidates` gains an additive
+ `retire_siblings` array of `{ retiring_id, sibling_id, score, method }`
+ pairs, the dream run summary - the CLI `o2b brain dream --json` and the
+ `brain_dream` tool response alike - carries the lexical pairs under the
+ same
+ key, and `o2b brain reject` lists the siblings of the rejected
+ preference with the command that rejects each (`--json` adds
+ `retire_siblings`). Decay retires and merges nominate nothing, a retire
+ the confirmed-evidence gate holds back nominates nothing, and nothing
+ is retired by the list: each pair is accepted through an existing verb
+ (`brain_apply_evidence` with `result: outdated`, or `o2b brain
+ reject`). Lexical pairs score at least 0.7 Jaccard on the principle.
+ With a search config, `brain_review_candidates` also compares the
+ stored vectors of the retiring and sibling preferences, at zero
+ embedding spend, and adds pairs at 0.92 cosine or above as
+ `method: "embedding"`; `retire_siblings_semantic` reports that tier's
+ outcome (`used`, `index_missing`, `vec_unavailable`, `not_embedded`,
+ `model_mismatch`, `index_unavailable` - the index exists but could not
+ be opened or read, in which case the lexical pairs still come back and
+ the search error code lands in `retire_siblings_semantic_detail`, a
+ field that appears only for that outcome and never carries a message
+ or a path). A pair is kept only when the caller may read both
+ preferences, and the stored-vector tier probes only the retires
+ `would_retire` keeps, so neither field reflects a withheld page.
+ `brain_extract_signals` now carries each mined turn's stored
+ `timestamp` on its `turns_mined` entries (`""` when the source has
+ none) and in the prompt lines; the envelope asks for a time bound as
+ an ISO 8601 date or interval resolved against the stating turn's
+ timestamp, which the dream pass turns into `valid_from` /
+ `valid_until`. A payload in which two items share a `topic` is refused
+ whole, naming both items. No new tool.
diff --git a/docs/observability.md b/docs/observability.md
index 1ad77b8a..12adceb4 100644
--- a/docs/observability.md
+++ b/docs/observability.md
@@ -114,7 +114,7 @@ Every continuity record shares one envelope, and exactly these eight fields in t
| `gate_telemetry` | config key `recall_gate_telemetry` (default off) | `brain_recall_gate` MCP handler; a call carrying `decision_answerable` while that decision-model use is on adds `adequacy_level` and `decision_answerable` (`probability`, `disagrees`); since v1.69.0 a call carrying `turn_id` records it under `turn_id`, which the read-model lifts like every other turn id |
| `negative_recall` | config key `recall_gate_telemetry` (default off - the SAME gate as `gate_telemetry` above, so one config decision governs both observations the recall gate makes) | `brain_recall_gate` MCP handler on the zero-usable-result path only (`emitNegativeRecallTelemetry` in `src/core/brain/gate-telemetry.ts`) - one record per gate call whose caller reported no usable result, carrying the typed corpus verdict (`state`, `complete`, the named `unknown_reason` when the verdict is `unknown`, `reason`) plus the reduced coverage receipt when one could be built (`coverage_digest`, `documents`, `chunks`, `schema_version`, `embedding_signature`, `last_indexed_at`, `scope`, `unindexed_roots`). Two receipt fields are not copied into the payload: `index_path` is deliberately dropped - a host filesystem path is not vault content and these records sync between devices, and the digest already binds it; the `embeddings` census the receipt carries is likewise absent from the payload and likewise bound by `coverage_digest`, so a reader comparing two records still sees a keyword-only index differ from a fully embedded one. Same prompt invariant as `gate_telemetry`: a SHA-256 prefix (`prompt_hash`) and the length (`prompt_chars`), never the prompt. A call with usable results writes NO record, so a corpus statement is never invented for an attempt that had one |
| `generation_report` | opt-in per call (`enable`) or config key `generation_trace_enabled` (default off) | inbound agent report after a generation handoff (`brain_generation_reports` action `record`, `o2b brain generation-reports record`) |
-| `mcp_route_latency` | config key `mcp_route_metrics_enabled` (default off) | the MCP server's tool-call seam (`MCPServer.invokeToolHandler` in `src/mcp/server.ts`) - one record per `tools/call` and per CLI-bridge `callTool`; payload carries the tool name, scope, status, duration, and argument KEY NAMES only, never argument values |
+| `mcp_route_latency` | config key `mcp_route_metrics_enabled` (default off) | the MCP server's tool-call seam (`MCPServer.invokeToolHandler` in `src/mcp/server.ts`) - one record per `tools/call` and per CLI-bridge `callTool`; payload carries the tool name, scope, status, duration, and argument KEY NAMES only, never argument values, plus the optional `decision_ms` and allowlisted `stages` timings |
| `decision_model_call` | the optional decision model is active (`decision_model_enabled: "true"` AND the named key variable set, which a self-hosted `laya` or `openjev` server on loopback does not need, and no vault opt-out); an install without the feature writes none | `runDecision` in `src/core/decision-model/run.ts` - one record per attempted decision request: `use`, `mode`, `provider`, `model` (the answering version), `calibrated`, `question_count`, `candidate_count`, `input_tokens`, `output_tokens`, `cost_usd`, `cost_source` (`reported`, `estimated` or `unknown`), `latency_ms`, `outcome` (`ok` or a degrade reason), `state_hash` (sha-256 of the redacted state), `origin` (`eval` or `ping`) when not an ordinary use call, plus per-use identifiers (a rerank adds `head_size`, `withheld_count`, `budget_dropped_count`, `frontmatter_only_count` when a frontmatter-only chunk was not sent, `injection_suspected` in `enforce`, `answerable_probability`, and in `shadow` `heuristic_order` and `decision_order` as paths, in `enforce` `decision_permutation` as head positions; every other use adds its own identifiers and counts, described on its page under `docs/decision-models/`). Never the state, a passage, a prompt or a question text. Fail-open: a failed write never fails the search. The daily cost gate and `o2b decision-model report` read these records |
| `decision_model_extract_commit` | the optional decision model is active and the `extract_prefilter` use is not `off` | commit phase of `brain_extract_signals` / `o2b brain extract-signals` (`src/core/brain/extract-signals-prefilter.ts`) - one record per commit that wrote items: `session_id`, `mode`, `written_count`, `source_turns` (the cited turn ids that name a real turn of the session) and `without_source_turn_count`; turn ids and counts only, never text. Read by `o2b decision-model report --use extract_prefilter` for its regret metric |
| `token_impact` | config key `token_impact_ledger_enabled` (default off) | `brain_token_impact` action `record` (`emitTokenImpact` in `src/core/brain/token-impact.ts`) - one sample per context pack; payload carries the tokenizer-exact prompt-token counts (`baseline_tokens`, `packed_tokens`, signed `delta_tokens`), the `method` label (`exact`/`fallback`), an optional opaque `pack_id`, an optional `source` attribution (`decision_model:<use>` for savings of a decision-model use), and the optional modeled inference-avoidance estimate - never raw prompts or recalled text |
@@ -161,6 +161,9 @@ Every continuity payload passes `safeContinuityPayload()` (`src/core/brain/conti
- `gate_telemetry` never stores the raw prompt by construction - only a SHA-256 prefix (`prompt_hash`) and the length (`prompt_chars`). The test suite asserts this against the persisted files, not just returned values. `negative_recall` shares that construction (same module, same hash helper) and adds one rule of its own: the coverage receipt it carries drops `index_path`, so no host filesystem path replicates with the vault.
- `generation_report` follows the same rule: the handoff prompt is hashed and counted but never persisted - only `prompt_hash` (full SHA-256 hex) and `prompt_chars` land on disk, plus token counts. The kernel never calls an LLM; the report is the agent's INBOUND record after it performed a generation for a brain handoff, so the local token estimate (`local_estimate.input_tokens`) is always present while the agent-reported `usage` block is present only when supplied - absent is reported as absent, never fabricated. A grep-guarded regression test pins that no `fetch`/provider HTTP call exists under `src/core` for this feature, and a persisted-file assertion confirms no raw prompt survives.
- `mcp_route_latency` carries an optional `decision_ms` (wall time spent in decision-model requests during that call) only when the call made at least one; records from an install without the feature are unchanged.
+- `mcp_route_latency` carries an optional `stages` array of `{ name, ms }` entries (v1.74.0): the wall time the call spent in each write stage, summed per name in first-seen order and rounded to 0.1 ms. Names come from the closed `ROUTE_STAGE` allowlist in `src/core/route-scope.ts` - `validate`, `idempotency_lookup`, `near_duplicate_lookup`, `document_write`, `idempotency_remember`, `log_append`, `preference_write`, `write_receipt`, `lint` - and `emitMcpRouteLatency` drops any other name, any negative value and any non-finite value before the record is written, so no path, topic or other free text can land on disk. The key is omitted when no stage ran, so a record from a route without stages is byte-identical to before. Stage timings never appear in the response of the call being measured; they live only in its continuity record, which `brain_route_metrics` reads back. Stages can nest, so they are not a partition of `duration_ms`: `lint` wraps the receipt lint, and the near-duplicate scoring inside that lint is also counted under `near_duplicate_lookup`. The same milliseconds then appear under both names, `lint` includes that scoring, and the stages of one record can add up to more than its duration.
+- Routes that emit stages, by the write helpers they reach: every route that writes a signal through `writeSignal` gets `validate` and `document_write`, plus `idempotency_lookup` and, when the signal is written, `idempotency_remember` only when the signal carries an idempotency key, which only `brain_feedback` accepts (its `idempotency_key` argument). That covers `brain_feedback` and `brain_extract_signals`; the latter writes one signal per accepted item, so its stage values are summed across items. `brain_feedback` adds `near_duplicate_lookup` (the write-conflict advisory), `log_append` (the feedback log line, and the second log line under `force_confirmed`; the advisory's own log line is not timed) and `preference_write` with `force_confirmed`; a retry that the idempotency key dedupes stops after `idempotency_lookup`. Every note write that ends in a write receipt - `brain_create_note`, `brain_update_note`, `brain_append_note` and `brain_write_batch` - gets `write_receipt` for each note it writes or rewrites, `document_write` for each note it creates, and, when at least one page was written, `lint` and `near_duplicate_lookup` from the receipt lint (the latter also covers the keyword-index widening when that is on). One route scope per tool call carries both `decision_ms` and `stages`; with `mcp_route_metrics_enabled` off no scope is opened and every stage helper reduces to a plain call.
+- Hook capture is not an MCP route: lifecycle hooks call `captureSessionLifecycleEvent` directly and hand the dedup walk to a detached worker, so they never pass through `invokeToolHandler` and carry no stage timings.
- `mcp_route_latency` records the route (tool name), MCP scope, status, duration, and the sorted set of argument KEY NAMES only - the tool's own JSON-Schema property names, never operator-supplied argument values - so no prompt, note body, or preference id can land on disk. A persisted-file test asserts key names survive while values never do.
- The masking policy lives in the read-model (`src/core/brain/continuity/read-model.ts`) and governs only the readers that go through it: records flagged `private` are dropped by default and kept only on explicit request; masked text is never un-masked. That is FOUR modules - the `o2b brain continuity` verbs (trajectory export and listing), the bench harness, `brain_event_trace`, and skill-usage aggregation. Twenty-two other modules call `listContinuityRecords` on the store directly and see raw records: no `private` drop and no schema-version dispatch. The policy is read-model-wide, NOT store-wide. Both lists are asserted by name in `tests/core/brain/continuity/reader-census.test.ts`, which fails when a module joins or leaves either set, and both COUNTS are pinned there as equalities rather than floors - the direct floor used to sit at twenty against a reality of twenty-one, which is how the number in this sentence came to be wrong. Auditing the direct readers is tracked separately.
- A hook's `tool_response` is READ and never written. Lifecycle capture consults it to decide whether the call it is reacting to actually succeeded - a failed `brain_feedback` must not be replayed, because replaying it mints the record the tool declined to create - and the decline is counted rather than swallowed. The response body itself is arbitrary tool output and never reaches a payload, so declaring it in the hook contract costs nothing on disk.
@@ -184,7 +187,7 @@ No-consumer regression tests cover each gated surface, so a future call site can
- `brain_context_receipts` and `o2b brain context-receipts` inspect served-context receipts.
- The SessionStart hook writes one `context_receipt` per injection (`host: "hook"`, `trigger: "session_inject"`) with one item per source it injected. On a measured run its `budget` block carries `inject_budget_chars` (the configured value), `budgeted_source_count` and, since v1.70.0, `scoped_rules_chars`: the length of the rendered scoped operator rules block, `0` when none rendered. The scoped block is a budgeted source named `scoped-rules`, but it is subtracted from the budget rather than charged against the full ceiling: the active body was budgeted to `inject_budget_chars` minus `scoped_rules_chars` (never below zero). The `budget` block is present whenever the scoped block rendered, also when there is no `active.md`. A run that fell back to the last good cache records no `budget` block, as before, but keeps the `standing-rules` and `scoped-rules` items, which are produced outside the fail-open boundary. See "Scoped operator rules" in [`how-it-works.md`](how-it-works.md).
- With `reground_parts_enabled` on, the same receipt's `extra.injection` gains the chunked re-grounding fields: `utf16_chars` (the joined payload length in UTF-16 code units), `part_ceiling_chars` (the per-part ceiling it was cut to, `null` on a runtime without the `reground-deliver` carrier or without a session id), `parts_total`, `parts_dropped` (parts past the 8-part maximum that were never queued; when it is above zero the last delivered part ends with `(context truncated: N further part(s) not delivered)`) and `over_budget` (`true` when parts were dropped, when a part is still longer than the ceiling, or when the queue could not be written and the whole payload went out as one part). That last case also sets `reground_fallback: "ledger_write_failed"`, which no other path writes, so a failed queue write is told apart from a plain over-budget payload. A part-ceiling key (`reground_part_chars`, `reground_part_chars_claudecode`, `reground_part_chars_codex`) whose value is out of range or not an integer falls through to the next level and is named in `config_invalid` (by its environment variable, such as `OPEN_SECOND_BRAIN_REGROUND_PART_CHARS`, when the rejected value came from the environment), present only then. With the flag off none of these fields appears and the receipt keeps its shape. Each part the `reground-deliver` hook hands out writes one hook-audit line (`actor: reground-deliver`, `action: reground_part_delivered`, `target` the host event `PostToolUse` or `UserPromptSubmit`) with `part` (the 1-based part number), `total`, `epoch` (the SessionStart that queued it, as `<source>:<epoch-ms>` with the source one of `startup`, `resume`, `clear` or `compact`, any other value recorded as `unknown`), `bytes` (UTF-8), `utf16_chars`, `part_ceiling_chars` and `over_budget` (the part alone is longer than the ceiling). A take that fails for a reason other than lock contention (an unreadable state directory, a full disk) writes `action: reground_take_failed` with `ok: false`, no part and `epoch` (the queue's SessionStart, `null` when unreadable), once per epoch on each machine: a state write that keeps failing leaves the queue undrained, and the repeats on later tool calls write no further line. A tool call inside a delegated sub-agent (the payload carries `agent_id`) takes no part and writes no line. A process-ceiling exit writes `action: hook_ceiling_exceeded`, `target: reground`.
-- `brain_route_metrics` (MCP, `operation: "list"|"summary"`) reads route-level MCP tool latency. `list` returns raw `mcp_route_latency` records (filter by `tool`/`status`/`since`/`until`/`limit`); `summary` rolls each tool up into count, error count, and min/avg/max plus p50/p95/p99 latency, ordered slowest-first so an operator can identify which endpoint blocks an agent turn by route rather than by aggregate benchmark. Records are emitted only when `mcp_route_metrics_enabled` is on (default off); read-only.
+- `brain_route_metrics` (MCP, `operation: "list"|"summary"`) reads route-level MCP tool latency. `list` returns raw `mcp_route_latency` records (filter by `tool`/`status`/`since`/`until`/`limit`); `summary` rolls each tool up into count, error count, and min/avg/max plus p50/p95/p99 latency, ordered slowest-first so an operator can identify which endpoint blocks an agent turn by route rather than by aggregate benchmark. A route whose records carry `stages` also gets a `stages` roll-up - per stage name, in `ROUTE_STAGE` order, the count, the average and the nearest-rank p95 in ms - and a route without stages has no `stages` key. Records are emitted only when `mcp_route_metrics_enabled` is on (default off); read-only.
- `brain_token_impact` (MCP, `operation: "record"|"outcome"|"list"|"summary"`) is the durable value-of-memory ledger. `record` posts one context pack's prompt-token delta (`baseline_tokens` − `packed_tokens`, labelled `method: tokenizer|heuristic` — how the CALLER produced the counts, never how accurate they are; the ledger counts nothing itself and verifies neither. Samples written under the retired `exact`/`fallback` labels are still read and classified) plus an optional modeled inference-avoidance estimate (`modeled_avoided_inferences` × `modeled_tokens_per_inference`); `outcome` posts a `first_pass`/`repair`/`retry` result. `summary` keeps the two ledgers STRICTLY SEPARATE: `prompt_token_delta` reports the net/saved/added tokens with a per-`method` split (so a tokenizer-produced count is never averaged in with a heuristic estimate), while `modeled_inference_avoidance` reports the raw modeled figure and a `calibrated_savings_tokens` dampened by the observed first-pass rate from posted outcomes (null until an outcome exists — uncalibrated, not zero). `list` returns raw samples (filter by `pack_id`/`method`/`host`/`since`/`until`/`limit`); `summary` accepts `max_samples` to bound aggregation. Writes are emitted only when `token_impact_ledger_enabled` is on (default off) and are payload-safe (counts + opaque `pack_id` only, never prompts or recalled text); `list`/`summary` read regardless of the gate. Aggregates are recomputed from the continuity store on every read, so they survive restarts.
- `brain_event_trace` (MCP) and `o2b brain event-trace` join the Brain log to the continuity store: given a logged event (by day, `--at` stamp, kind, or `--session-id`), they resolve the continuity records attached to it via the stable correlation ids above (`session_id`, `turn_id`, and artifact `sourceRefs`), so an operator can answer "which context was supplied to this action?" from one surface. Each attached record reports `joinedBy` (`session` / `turn` / `artifact`); open the full record through the per-kind readers (`context-receipts show`, `recall-telemetry`, `generation-reports show`). Read-only; `private` records are dropped unless explicitly kept.
- `brain_generation_reports` (MCP, actions `record`/`list`/`summary`) and `o2b brain generation-reports record|list|summary|show` post and read inbound LLM generation traces. `summary` rolls up call counts, the always-present local token estimate, agent-reported usage where present, and a per-path linkage map so a memory path resolves back to the generation reports that produced or consumed it.
diff --git a/openclaw.plugin.json b/openclaw.plugin.json
index 2e66af47..a87e3eeb 100644
--- a/openclaw.plugin.json
+++ b/openclaw.plugin.json
@@ -2,7 +2,7 @@
"id": "open-second-brain",
"name": "Open Second Brain",
"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",
- "version": "1.73.1",
+ "version": "1.74.0",
"activation": { "onStartup": true },
"skills": ["./skills"],
"contracts": {
diff --git a/package.json b/package.json
index e371fe66..89d92f17 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.73.1",
+ "version": "1.74.0",
"private": false,
"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.",
"keywords": [
@@ -63,8 +63,8 @@
"@types/bun": "^1.4.2",
"@types/node": "^22.20.4",
"@types/proper-lockfile": "^4.1.4",
- "oxfmt": "0.71.0",
- "oxlint": "1.86.0",
+ "oxfmt": "0.70.0",
+ "oxlint": "1.85.0",
"typescript": "^5.5.0"
},
"optionalDependencies": {
diff --git a/plugin.yaml b/plugin.yaml
index 6a5ced56..853282de 100644
--- a/plugin.yaml
+++ b/plugin.yaml
@@ -1,5 +1,5 @@
name: open-second-brain
-version: "1.73.1"
+version: "1.74.0"
description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."
author: "Open Second Brain contributors"
memory_provider: true
diff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.json
index 0d63043d..9a0f1ffd 100644
--- a/plugins/codex/.codex-plugin/plugin.json
+++ b/plugins/codex/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.73.1",
+ "version": "1.74.0",
"description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
"author": {
"name": "Open Second Brain contributors",
diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yaml
index 6a5ced56..853282de 100644
--- a/plugins/hermes/plugin.yaml
+++ b/plugins/hermes/plugin.yaml
@@ -1,5 +1,5 @@
name: open-second-brain
-version: "1.73.1"
+version: "1.74.0"
description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."
author: "Open Second Brain contributors"
memory_provider: true
diff --git a/pyproject.toml b/pyproject.toml
index 1c52a63e..0ce15915 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"
# CLI entry points (those moved to `package.json` `bin`).
[project]
name = "open-second-brain"
-version = "1.73.1"
+version = "1.74.0"
description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."
readme = "README.md"
requires-python = ">=3.11"
diff --git a/src/cli/brain/help-text.ts b/src/cli/brain/help-text.ts
index e5b55e90..b9c450ac 100644
--- a/src/cli/brain/help-text.ts
+++ b/src/cli/brain/help-text.ts
@@ -378,7 +378,10 @@ export const VERB_HELP: Record<string, string> = {
"Compare source-agent coverage using the same provenance foundation as agent-query.\n",
reject:
"usage: o2b brain reject --id <pref-id> --reason <text> [--yes] [--vault <path>] [--json]\n" +
- "Move a preference to retired/ with reason 'user-rejected'. --yes required when pinned.\n",
+ "Move a preference to retired/ with reason 'user-rejected'. --yes required when pinned.\n" +
+ "With near_duplicate_retire_siblings_enabled on, also lists the active preferences that resemble\n" +
+ "the rejected one (score, method) with the command that rejects each; --json adds retire_siblings.\n" +
+ "Nothing else is retired.\n",
freeze:
"usage: o2b brain freeze [--reason <text>] [--vault <path>] [--json]\n" +
"Write Brain/.state/frozen.json. While it exists every content write in this vault is refused - on this device and, once Syncthing has carried the marker, on every device that shares it. The Brain log keeps recording, so the freeze and the writes it refuses stay auditable. Idempotent: a second freeze keeps the first one's reason.\n",
diff --git a/src/cli/brain/verbs/extract-signals.ts b/src/cli/brain/verbs/extract-signals.ts
index d08eae44..30d3648a 100644
--- a/src/cli/brain/verbs/extract-signals.ts
+++ b/src/cli/brain/verbs/extract-signals.ts
@@ -83,7 +83,11 @@ export async function cmdBrainExtractSignals(argv: string[]): Promise<number> {
generated_at: plan.generatedAt,
boundary_decision: plan.boundaryDecision,
turns_scanned: plan.turnsScanned,
- turns_mined: plan.turnsMined.map((t) => ({ turn_id: t.turnId, text: t.text })),
+ turns_mined: plan.turnsMined.map((t) => ({
+ turn_id: t.turnId,
+ text: t.text,
+ timestamp: t.timestamp,
+ })),
cap: plan.cap,
confidence_floor: plan.confidenceFloor,
llm_step: plan.llmStep,
diff --git a/src/cli/brain/verbs/reject.ts b/src/cli/brain/verbs/reject.ts
index 7656d3d3..efa3956e 100644
--- a/src/cli/brain/verbs/reject.ts
+++ b/src/cli/brain/verbs/reject.ts
@@ -1,11 +1,21 @@
import { existsSync } from "node:fs";
-import { resolveAgentName } from "../../../core/config.ts";
+import {
+ resolveAgentName,
+ resolveNearDuplicateRetireSiblingsEnabled,
+} from "../../../core/config.ts";
+import { scanBrain } from "../../../core/brain/dream-scan.ts";
+import { READ_ALL_REFS } from "../../../core/brain/near-duplicate.ts";
import { moveToRetired, parsePreference } from "../../../core/brain/preference.ts";
+import {
+ planRetireSiblings,
+ retireSiblingPool,
+ type RetireSibling,
+} from "../../../core/brain/retire-siblings.ts";
import { preferencePath } from "../../../core/brain/paths.ts";
import { isoDate, isoSecond } from "../../../core/brain/time.ts";
import { renderPrefLink } from "../../../core/brain/wikilink.ts";
import { appendLogEvent } from "../../../core/brain/log.ts";
-import { BRAIN_LOG_EVENT_KIND } from "../../../core/brain/types.ts";
+import { BRAIN_LOG_EVENT_KIND, BRAIN_RETIRED_REASON } from "../../../core/brain/types.ts";
import { brainVerbContext, fail, ok, okJson, parse } from "../helpers.ts";
export async function cmdBrainReject(argv: string[]): Promise<number> {
@@ -77,10 +87,49 @@ export async function cmdBrainReject(argv: string[]): Promise<number> {
process.stderr.write(`warning: append reject log failed: ${(err as Error).message}\n`);
}
+ // Near-duplicate defense: list (never retire) the active preferences that
+ // resemble the one just rejected. The operator's own shell reads the whole
+ // vault, hence READ_ALL_REFS. Off, the output is unchanged. The retire
+ // has already happened, so a failing scan is a warning, never an error.
+ let siblings: ReadonlyArray<RetireSibling> = [];
+ if (resolveNearDuplicateRetireSiblingsEnabled(config)) {
+ try {
+ siblings = planRetireSiblings(
+ retireSiblingPool(scanBrain(vault).preferences.map((p) => p.pref)),
+ [
+ {
+ id: `pref-${slug}`,
+ principle: pref.principle,
+ reason: BRAIN_RETIRED_REASON.userRejected,
+ },
+ ],
+ { readable: READ_ALL_REFS, gated: new Set() },
+ );
+ } catch (err) {
+ process.stderr.write(`warning: retire siblings scan failed: ${(err as Error).message}\n`);
+ }
+ }
+
if (flags["json"]) {
- okJson({ id: `ret-${slug}`, reason: "user-rejected" });
+ okJson({
+ id: `ret-${slug}`,
+ reason: "user-rejected",
+ ...(siblings.length > 0 ? { retire_siblings: siblings } : {}),
+ });
} else {
ok(`retired: ret-${slug} (user-rejected)`);
+ if (siblings.length > 0) ok(formatRetireSiblings(siblings));
}
return 0;
}
+
+/** One line per sibling with its score, then the command that accepts it. */
+function formatRetireSiblings(siblings: ReadonlyArray<RetireSibling>): string {
+ return [
+ `retire siblings: ${siblings.length}`,
+ ...siblings.flatMap((s) => [
+ ` ${s.sibling_id} score=${s.score.toFixed(3)} method=${s.method}`,
+ ` accept: o2b brain reject --id ${s.sibling_id} --reason <text>`,
+ ]),
+ ].join("\n");
+}
diff --git a/src/core/brain/dream-types.ts b/src/core/brain/dream-types.ts
index 6e8f9ea4..3e71542d 100644
--- a/src/core/brain/dream-types.ts
+++ b/src/core/brain/dream-types.ts
@@ -14,6 +14,7 @@ import type { DreamPhaseSummary } from "./dream-phases.ts";
import type { DreamQuarantinedEntry } from "./dream-plan.ts";
import type { DreamOutcomeRegression } from "./dream-refresh.ts";
import type { BrainIntentReviewEntry } from "./intent-review.ts";
+import type { RetireSibling } from "./retire-siblings.ts";
import type { RollupLadderEntry } from "./rollup-ladder.ts";
import type { SalienceGateVerdict } from "./salience-gate.ts";
import type { BrainRetiredReason, DreamOpenQuestion } from "./types.ts";
@@ -127,6 +128,16 @@ export interface DreamRunSummary {
* when the config field is absent (the default).
*/
readonly gated_retires: ReadonlyArray<DreamGatedRetireEntry>;
+ /**
+ * Near-duplicate defense (t_acab97de): active preferences that resemble
+ * a context-driven retire of this run (`superseded-by-context`,
+ * `rebutted`, `quarantine-violated`, `user-rejected`), excluding
+ * siblings of a retire the confirmed-evidence gate holds back (on a dry
+ * run, one it would hold back). Advisory only: nothing here is retired. Absent
+ * when `near_duplicate_retire_siblings_enabled` is off or the list is
+ * empty, so an un-opted-in summary stays byte-identical.
+ */
+ readonly retire_siblings?: ReadonlyArray<RetireSibling>;
/**
* Outcome-regression findings (t_d478df53): confirmed preferences
* whose recent applied events co-occur with failure outcomes. The
@@ -258,4 +269,19 @@ export interface DreamOptions {
* `dryRun: true` throws `DreamPreviewReadableError`.
*/
readonly previewReadable?: (rel: string) => boolean;
+ /**
+ * Whether the summary carries `retire_siblings`. Omitted, it resolves
+ * `near_duplicate_retire_siblings_enabled` from the default config; a
+ * caller with its own config path (the MCP server) resolves it there.
+ */
+ readonly retireSiblingsEnabled?: boolean;
+ /**
+ * Receives every preference of the pass's full Brain scan, before any
+ * `previewReadable` filter, so a caller that needs the same records after
+ * the pass (the review-candidates projection) reuses this read instead of
+ * walking `Brain/` a second time.
+ */
+ readonly onScanPreferences?: (
+ preferences: ReadonlyArray<import("./dream-plan.ts").PreferenceRecord>,
+ ) => void;
}
diff --git a/src/core/brain/dream.ts b/src/core/brain/dream.ts
index 7c9a033b..d85a6db0 100644
--- a/src/core/brain/dream.ts
+++ b/src/core/brain/dream.ts
@@ -59,8 +59,10 @@ import {
gatedRetireSlugs,
plannedSignalArchiveIds,
plannedSignalMoveIds,
+ shouldGateRetireFromConfirmed,
type DreamApplyResult,
} from "./dream-apply.ts";
+import { resolveNearDuplicateRetireSiblingsEnabled } from "../config.ts";
import { planAutoRetires } from "./dream-plan-retires.ts";
import { planTopics, topicKeyContentionWarnings } from "./dream-plan-topics.ts";
import type { PlanState, ScanResult } from "./dream-plan.ts";
@@ -69,7 +71,12 @@ import { writeDreamLog } from "./dream-report.ts";
import { scanBrain } from "./dream-scan.ts";
import { planSignalArchive } from "./signal-archive.ts";
import { buildChangedSummary, buildNoOpSummary } from "./dream-summary.ts";
-import type { DreamOptions, DreamRunSummary, DreamWarning } from "./dream-types.ts";
+import type {
+ DreamGatedRetireEntry,
+ DreamOptions,
+ DreamRunSummary,
+ DreamWarning,
+} from "./dream-types.ts";
import { openWorkrun, WORKRUN_PHASE, type WorkrunHandle } from "./dream-workrun.ts";
import { buildIntentReview } from "./intent-review.ts";
import {
@@ -90,6 +97,8 @@ import {
resolveRollupThresholds,
type RollupLadderPlan,
} from "./rollup-ladder.ts";
+import { READ_ALL_REFS } from "./near-duplicate.ts";
+import { planRetireSiblings, retireSiblingPool } from "./retire-siblings.ts";
import { applySalienceGate } from "./salience-gate.ts";
import { withDestructiveSnapshot } from "./snapshot-gate.ts";
import { compactRunStamp, isoDate } from "./time.ts";
@@ -257,6 +266,7 @@ function dreamRun(
// checkpoints; the stream, by contrast, is this counter's, which
// already owns a `scan` stage.
const fullScan = scanBrain(vault, opts.safeguard ? { safeguard: opts.safeguard } : {});
+ opts.onScanPreferences?.(fullScan.preferences);
const scan =
opts.previewReadable === undefined
? fullScan
@@ -502,7 +512,7 @@ function dreamRun(
progress.finish();
noteProgressFaults(warnings, progressFaults);
- return buildChangedSummary({
+ const summary = buildChangedSummary({
runId,
// `brainDirsForWrite` asserts the vault is writable, so the log path
// is resolved only on the branch that actually wrote to it.
@@ -524,6 +534,60 @@ function dreamRun(
healEnriched: exec.healEnriched,
snapshotPath: snapshotPathStr,
});
+ return withRetireSiblings(summary, scan, plan, {
+ enabled: opts.retireSiblingsEnabled,
+ gatedRetires: exec.gatedRetires,
+ // A dry run applies nothing, so it gates nothing; the preview asks the
+ // same confirmed-evidence gate the apply step will.
+ previewGateThreshold: dryRun ? cfg.retire.confirmed_evidence_min_threshold : undefined,
+ });
+}
+
+/**
+ * Attach the retire-sibling projection (near-duplicate defense) to a
+ * changed run's summary. Computed from the completed retire plan, then
+ * reconciled against the retires the confirmed-evidence gate holds back:
+ * the ones the apply step gated, or on a dry run the ones it would gate,
+ * so the dry run and the real run agree. A retiring preference is never
+ * refreshed first, so its scanned frontmatter is what the apply step
+ * reads. The scan is the pass's own reach (a preview scan is already
+ * admitted), hence {@link READ_ALL_REFS}. Off (`enabled`, else the
+ * default config's key), or with nothing to report, the summary is
+ * returned untouched.
+ */
+function withRetireSiblings(
+ summary: DreamRunSummary,
+ scan: ScanResult,
+ plan: PlanState,
+ opts: {
+ readonly enabled: boolean | undefined;
+ readonly gatedRetires: ReadonlyArray<DreamGatedRetireEntry>;
+ readonly previewGateThreshold: number | undefined;
+ },
+): DreamRunSummary {
+ if (plan.retires.length === 0 || !(opts.enabled ?? resolveNearDuplicateRetireSiblingsEnabled())) {
+ return summary;
+ }
+ const gated = new Set(opts.gatedRetires.map((g) => g.pref_id));
+ if (opts.previewGateThreshold !== undefined) {
+ const byId = new Map(scan.preferences.map((p) => [p.pref.id, p.pref] as const));
+ for (const r of plan.retires) {
+ const existing = byId.get(`pref-${r.slug}`);
+ if (
+ existing !== undefined &&
+ shouldGateRetireFromConfirmed(existing, r.reason, opts.previewGateThreshold)
+ ) {
+ gated.add(existing.id);
+ }
+ }
+ }
+ const siblings = planRetireSiblings(
+ retireSiblingPool(scan.preferences.map((p) => p.pref)),
+ plan.retires.map((r) => ({ id: `pref-${r.slug}`, principle: r.principle, reason: r.reason })),
+ { readable: READ_ALL_REFS, gated },
+ );
+ if (siblings.length === 0) return summary;
+ return Object.freeze({ ...summary, retire_siblings: Object.freeze([...siblings]) });
}
/**
diff --git a/src/core/brain/extract-signals.ts b/src/core/brain/extract-signals.ts
index 84253cb9..134bb8e1 100644
--- a/src/core/brain/extract-signals.ts
+++ b/src/core/brain/extract-signals.ts
@@ -186,6 +186,13 @@ function describeDeadLetter(outcome: DeadLetterOutcome): string {
export interface MinedTurn {
readonly turnId: string;
readonly text: string;
+ /**
+ * The turn's stored timestamp, verbatim; "" when the source has none.
+ * Never a formatted `Date` and never the plan's clock: the caller grounds
+ * a relative time bound against it, so an invented value would ground it
+ * against the wrong day.
+ */
+ readonly timestamp: string;
}
/** Phase-one report: what will be mined, and the one envelope that mines it. */
@@ -290,10 +297,13 @@ export interface CommitExtractedSignalsOptions {
// ----- Semantic rules the descriptor language cannot express ---------------
/**
- * Cap and floor, registered once at module scope. Both read across the
- * payload rather than one value at a time - the cap counts the whole list,
- * and the floor compares a number to a limit the descriptor cannot name -
- * so they belong here rather than in {@link EXTRACTED_SIGNALS_SHAPE}.
+ * Cap, floor, topic uniqueness and interval sanity, registered once at
+ * module scope. Each reads across the payload or inside a value rather
+ * than one value at a time - the cap counts the whole list, the floor
+ * compares a number to a limit the descriptor cannot name, uniqueness
+ * compares items with each other, and an interval compares the two dates
+ * inside one principle - so they belong here rather than in
+ * {@link EXTRACTED_SIGNALS_SHAPE}.
*
* Registration is a module side effect on purpose: the registry is
* fail-closed, so a lane that forgot to register refuses its own writes
@@ -325,9 +335,87 @@ registerResponseCheck(EXTRACTED_SIGNALS_SURFACE, (payload) => {
);
}
});
+ violations.push(...duplicateTopicViolations(items));
+ violations.push(...intervalViolations(items));
return violations;
});
+/**
+ * An ISO interval inside a principle, the form the dream pass turns into
+ * a `[valid_from, valid_until)` window. Global so every interval in one
+ * principle is checked, not only the first.
+ */
+const PRINCIPLE_INTERVAL_RE = /\b(\d{4}-\d{2}-\d{2})\/(\d{4}-\d{2}-\d{2})\b/g;
+
+/**
+ * One violation per interval in a principle that names an impossible
+ * calendar date or ends on or before it starts. Either one yields a window
+ * that is never active, and nothing downstream would say why, so the
+ * payload is refused here where the caller can still correct it.
+ */
+function intervalViolations(items: ReadonlyArray<unknown>): SemanticViolation[] {
+ const violations: SemanticViolation[] = [];
+ items.forEach((item, index) => {
+ const principle = (item as { principle?: unknown }).principle;
+ if (typeof principle !== "string") return;
+ for (const [interval, start, end] of principle.matchAll(PRINCIPLE_INTERVAL_RE)) {
+ const impossible = [start!, end!].find((date) => !isCalendarDate(date));
+ const message =
+ impossible !== undefined
+ ? `interval ${interval} names ${impossible}, which is not a calendar date`
+ : end! <= start!
+ ? `interval ${interval} must end after it starts; its end date is exclusive, the day after the last day the rule holds`
+ : null;
+ if (message === null) continue;
+ violations.push(
+ semanticViolation(
+ SEMANTIC_VIOLATION_CODES.threshold,
+ `${SHAPE_ROOT_PATH}.items[${index}].principle`,
+ message,
+ ),
+ );
+ }
+ });
+ return violations;
+}
+
+/** Whether a `YYYY-MM-DD` string names a real day, so `2026-02-30` does not. */
+function isCalendarDate(date: string): boolean {
+ const parsed = new Date(`${date}T00:00:00Z`);
+ return !Number.isNaN(parsed.getTime()) && parsed.toISOString().slice(0, 10) === date;
+}
+
+/**
+ * One violation per item that repeats an earlier item's topic, naming both
+ * indices. The comparison is exact: two items under one topic are one rule
+ * stated twice or two rules filed under one name, and the lane cannot tell
+ * which - so the payload is refused whole rather than one of them kept by a
+ * guess. A fuzzy comparison of principles is deliberately absent: two short
+ * rules of opposite meaning read alike, and one false refusal here would
+ * drop every valid signal in the run.
+ */
+function duplicateTopicViolations(items: ReadonlyArray<unknown>): SemanticViolation[] {
+ const violations: SemanticViolation[] = [];
+ const firstIndex = new Map<string, number>();
+ items.forEach((item, index) => {
+ const topic = (item as { topic?: unknown }).topic;
+ if (typeof topic !== "string") return;
+ const first = firstIndex.get(topic);
+ if (first === undefined) {
+ firstIndex.set(topic, index);
+ return;
+ }
+ violations.push(
+ semanticViolation(
+ SEMANTIC_VIOLATION_CODES.crossItem,
+ `${SHAPE_ROOT_PATH}.items[${index}].topic`,
+ `items[${first}] and items[${index}] share topic ${JSON.stringify(topic)}`,
+ ),
+ );
+ });
+ return violations;
+}
+
/**
* The item list, or null when the payload is not shaped like one. The
* shape layer has already refused that case by the time the check runs;
@@ -382,7 +470,7 @@ export function planExtractSignals(
if (boundary.suppressMessage(turn.text)) continue;
const text = turn.text.trim();
if (text.length === 0) continue;
- mined.push(Object.freeze({ turnId: turn.turn_id, text }));
+ mined.push(Object.freeze({ turnId: turn.turn_id, text, timestamp: turn.timestamp }));
}
if (mined.length === 0) {
throw new ExtractSignalsError(
@@ -401,6 +489,33 @@ export function planExtractSignals(
});
}
+/**
+ * Hygiene rules the mined items are held to, in the instruction.
+ *
+ * Each names a CATEGORY and never an example word: the operator may write
+ * in any language, and a kernel that listed one language's greetings or
+ * date words would mine that language better than every other. Time is
+ * grounded the same way - the caller resolves a relative bound against the
+ * turn's timestamp and writes it as ISO, which the dream pass already turns
+ * into a validity window, so no date parser lives here.
+ */
+const EXTRACT_HYGIENE_RULES: ReadonlyArray<string> = Object.freeze([
+ "Write any time bound a rule carries as an ISO 8601 date or interval (YYYY-MM-DD, or " +
+ "YYYY-MM-DD/YYYY-MM-DD), resolving a relative bound against the timestamp of the turn " +
+ "that stated it; an interval's end date is exclusive, so write the day after the last " +
+ "day the rule holds, and an end that is not after the start refuses the whole payload; " +
+ "write a bound that only ends as an interval from that turn's date, " +
+ "because a lone date reads as the day the rule starts; a rule with no time bound stays " +
+ "undated.",
+ "Skip conversational mechanics - greetings, thanks, acknowledgements, requests to go on, " +
+ "and directions about the flow of this session - because they state no rule.",
+ "Return one rule per item: a turn that states two rules yields two items with distinct topics.",
+ "Keep every condition a rule was stated with - when, where, for which project or kind of " +
+ "work - inside its principle; never widen a conditional rule into an unconditional one.",
+ "Drop restatements: a rule the operator states more than once is returned once, under one " +
+ "topic, because two items sharing a topic refuse the whole payload.",
+]);
+
/**
* The one envelope. The turns ride inside the prompt rather than beside it
* because the receiving agent is answering a question about exactly these
@@ -413,7 +528,7 @@ export function buildMiningStep(
opts: { readonly sourceTurnHint?: boolean } = {},
): NeedsLlmStep {
const transcript = mined
- .map((turn) => `[${turn.turnId}] ${turn.text.slice(0, PROMPT_TURN_TEXT_MAX)}`)
+ .map((turn) => `${transcriptLabel(turn)} ${turn.text.slice(0, PROMPT_TURN_TEXT_MAX)}`)
.join("\n");
return buildNeedsLlmStep({
step: EXTRACT_SIGNALS_STEP,
@@ -423,13 +538,16 @@ export function buildMiningStep(
"correction, a prohibition - not a fact, a task, or a one-off instruction about this session. " +
`Return at most ${AUTO_EXTRACT_PER_SESSION_CAP} items, each with a confidence of at least ` +
`${AUTO_EXTRACT_CONFIDENCE_FLOOR}; omit anything you are less sure of rather than lowering the ` +
- "number, because an item below the floor refuses the whole payload.\n\n" +
+ "number, because an item below the floor refuses the whole payload. " +
+ EXTRACT_HYGIENE_RULES.join(" ") +
+ "\n\n" +
transcript,
schema_hints: [
'payload: { "items": [ { "topic", "signal", "principle", "confidence", "scope"? } ] }',
- "topic: stable kebab-slug naming the rule",
+ "topic: stable kebab-slug naming the rule, unique within the payload",
"signal: 'positive' when the principle is the rule to follow, 'negative' when it is what to avoid",
- "principle: one imperative line, in the language the operator used",
+ "principle: one imperative line, in the language the operator used, with any time bound " +
+ "written as an ISO 8601 date or interval",
`confidence: number in [${AUTO_EXTRACT_CONFIDENCE_FLOOR}, 1]`,
`items: at most ${AUTO_EXTRACT_PER_SESSION_CAP} entries`,
...(opts.sourceTurnHint === true
@@ -440,6 +558,11 @@ export function buildMiningStep(
});
}
+/** `[turnId @ timestamp]`, or `[turnId]` when the turn has no timestamp. */
+function transcriptLabel(turn: MinedTurn): string {
+ return turn.timestamp === "" ? `[${turn.turnId}]` : `[${turn.turnId} @ ${turn.timestamp}]`;
+}
+
// ----- Phase two: what the caller mined ------------------------------------
/**
@@ -589,9 +712,9 @@ export function commitExtractedSignals(
/**
* The key a missing item is NAMED by: its payload index and its topic.
- * The index carries the uniqueness (two items may share a topic) and the
- * topic carries the meaning, which is what an operator matches against
- * the payload they are about to re-run.
+ * The index carries the position and the topic carries the meaning,
+ * which is what an operator matches against the payload they are about to
+ * re-run.
*/
function unwrittenKey(index: number, topic: string): string {
return `items[${index}]:${topic}`;
diff --git a/src/core/brain/health/contradiction.ts b/src/core/brain/health/contradiction.ts
index 953ea1bd..c7b283f9 100644
--- a/src/core/brain/health/contradiction.ts
+++ b/src/core/brain/health/contradiction.ts
@@ -15,8 +15,13 @@
*/
import { dominantSignOf } from "../sign.ts";
-import { findSimilarPairs, jaccard, tokenise } from "../similarity.ts";
+import { findSimilarPairs, tokenise } from "../similarity.ts";
import { BRAIN_HEALTH_DEFAULTS } from "../policy.ts";
+import {
+ READ_ALL_REFS,
+ findNearDuplicates,
+ type NearDuplicatePoolEntry,
+} from "../near-duplicate.ts";
import {
BRAIN_PREFERENCE_STATUS,
type BrainPreferenceStatus,
@@ -273,9 +278,10 @@ export interface AdviseOnIncomingOptions {
/**
* Compute a write-time conflict advisory for a single incoming feedback
- * principle against already-confirmed preferences. Reuses the shared
- * similarity kernel (`tokenise` + `jaccard`) that `detectContradictions`
- * builds on, and mirrors its bucketing: only confirmed preferences in the
+ * principle against already-confirmed preferences. Scores through the
+ * shared near-duplicate kernel (`findNearDuplicates`, over the same
+ * `tokenise` + `jaccard` primitives `detectContradictions` builds on), and
+ * mirrors its bucketing: only confirmed preferences in the
* SAME scope bucket are compared (an unscoped incoming signal compares
* against the unscoped bucket). Every confirmed same-scope preference
* whose principle overlaps the incoming principle at or above the
@@ -289,6 +295,13 @@ export interface AdviseOnIncomingOptions {
* operator judges. Pure and deterministic; never throws on well-formed
* input.
*
+ * Two kernel rules apply: a principle (incoming or confirmed) under
+ * `NEAR_DUPLICATE_MIN_TOKENS` is not scored, because short principles
+ * share most of their tokens by accident, and each similarity is rounded
+ * to 3 decimals. Reach is not decided here: the caller hands in only the
+ * preferences it may read (`write-advisory.ts` applies the reach when it
+ * loads them), so the kernel's predicate is the explicit allow-all.
+ *
* Returns `null` when no confirmed same-scope preference clears the
* threshold - i.e. there is nothing to advise about.
*/
@@ -298,20 +311,25 @@ export function adviseOnIncoming(
confirmedPrefs: ReadonlyArray<PreferenceForContradiction>,
opts: AdviseOnIncomingOptions = {},
): IncomingConflictAdvisory | null {
- const threshold = opts.jaccard ?? BRAIN_HEALTH_DEFAULTS.contradiction_jaccard;
- const bucketKey = scope ?? "";
- const incomingTokens = tokenise(principle);
- const conflicts: IncomingConflictEvidence[] = [];
+ const pool: NearDuplicatePoolEntry[] = [];
for (const p of confirmedPrefs) {
if (p.status !== BRAIN_PREFERENCE_STATUS.confirmed) continue;
- if ((p.scope ?? "") !== bucketKey) continue;
- const sim = jaccard(incomingTokens, tokenise(p.principle));
- if (sim < threshold) continue;
- conflicts.push({ prefId: p.id, jaccard: sim });
+ pool.push({ ref: p.id, tokens: tokenise(p.principle), bucket: p.scope ?? "" });
}
- if (conflicts.length === 0) return null;
- conflicts.sort((a, b) => b.jaccard - a.jaccard || a.prefId.localeCompare(b.prefId));
- return { scope: scope ?? null, conflicts };
+ // The incoming signal has no id yet, so no preference id can equal the
+ // empty probe ref. Every confirmed preference is scored, as before: the
+ // pool is the caller's, so the kernel's cap is set to its size.
+ const { matches } = findNearDuplicates({ ref: "", tokens: tokenise(principle) }, pool, {
+ threshold: opts.jaccard ?? BRAIN_HEALTH_DEFAULTS.contradiction_jaccard,
+ readable: READ_ALL_REFS,
+ bucket: scope ?? "",
+ cap: pool.length,
+ });
+ if (matches.length === 0) return null;
+ return {
+ scope: scope ?? null,
+ conflicts: matches.map((m) => ({ prefId: m.ref, jaccard: m.score })),
+ };
}
/**
diff --git a/src/core/brain/mcp-route-metrics.ts b/src/core/brain/mcp-route-metrics.ts
index 3be4b331..fa79fec2 100644
--- a/src/core/brain/mcp-route-metrics.ts
+++ b/src/core/brain/mcp-route-metrics.ts
@@ -21,13 +21,21 @@
* duration, and the SORTED SET OF ARGUMENT KEY NAMES land on disk.
* Argument key names are the tool's own JSON-Schema property names
* (`query`, `max_tokens`, ...), never operator-supplied values - so no
- * prompt, note body, or preference id is ever recorded. The whole
- * payload still passes `safeContinuityPayload` redaction.
+ * prompt, note body, or preference id is ever recorded. Optional
+ * `decision_ms` and `stages` carry timings only; stage names come from the
+ * closed `ROUTE_STAGE_NAMES` allowlist. The whole payload still passes
+ * `safeContinuityPayload` redaction.
*/
import { emitGatedTelemetry } from "./continuity/emit.ts";
import { appendContinuityRecord, listContinuityRecords } from "./continuity/store.ts";
import type { ContinuityRecord } from "./continuity/types.ts";
+import {
+ isRouteStageName,
+ ROUTE_STAGE_NAMES,
+ type RouteStageName,
+ type RouteStageTiming,
+} from "../route-scope.ts";
export type McpRouteStatus = "ok" | "error";
@@ -51,6 +59,12 @@ export interface McpRouteLatencyInput {
* without the feature is unchanged.
*/
readonly decisionMs?: number;
+ /**
+ * Write-stage timings noted inside this call. Only names in
+ * `ROUTE_STAGE_NAMES` with a finite, non-negative value are persisted;
+ * absent or empty leaves the record unchanged.
+ */
+ readonly stages?: ReadonlyArray<RouteStageTiming>;
}
export interface McpRouteLatencyFilter {
@@ -72,6 +86,16 @@ export interface McpRouteStats {
readonly p50_ms: number;
readonly p95_ms: number;
readonly p99_ms: number;
+ /** Per-stage roll-up in `ROUTE_STAGE` order; absent when no record of the route carried stages. */
+ readonly stages?: ReadonlyArray<McpRouteStageStats>;
+}
+
+/** One write stage's roll-up within a route (nearest-rank p95). */
+export interface McpRouteStageStats {
+ readonly name: RouteStageName;
+ readonly count: number;
+ readonly avg_ms: number;
+ readonly p95_ms: number;
}
export interface McpRouteLatencySummary {
@@ -121,6 +145,8 @@ export function emitMcpRouteLatency<G>(
? { decision_ms: Math.max(0, Math.round(input.decisionMs)) }
: {}),
};
+ const stages = normalizeStages(input.stages);
+ if (stages.length > 0) payload["stages"] = stages;
return appendContinuityRecord(vault, {
kind: "mcp_route_latency",
createdAt,
@@ -139,6 +165,28 @@ function normalizeArgKeys(raw: ReadonlyArray<string> | undefined): string[] {
return [...seen].toSorted();
}
+/**
+ * Keep allowlisted stage names with finite, non-negative values, sum
+ * repeated names in first-seen order and round to 0.1 ms. A name outside
+ * `ROUTE_STAGE_NAMES` is dropped here, at the only write site, so free
+ * text never reaches the continuity record.
+ */
+function normalizeStages(
+ raw: ReadonlyArray<RouteStageTiming> | undefined,
+): Array<{ name: RouteStageName; ms: number }> {
+ if (!Array.isArray(raw)) return [];
+ const summed = new Map<RouteStageName, number>();
+ for (const stage of raw as ReadonlyArray<unknown>) {
+ if (typeof stage !== "object" || stage === null) continue;
+ const { name, ms } = stage as { name?: unknown; ms?: unknown };
+ if (!isRouteStageName(name) || typeof ms !== "number" || !Number.isFinite(ms) || ms < 0) {
+ continue;
+ }
+ summed.set(name, (summed.get(name) ?? 0) + ms);
+ }
+ return [...summed].map(([name, ms]) => ({ name, ms: Math.round(ms * 10) / 10 }));
+}
+
/** List `mcp_route_latency` records newest-first, after applying filters. */
export function listMcpRouteLatency(
vault: string,
@@ -166,6 +214,7 @@ export function summarizeMcpRouteLatency(
const byStatus: Partial<Record<McpRouteStatus, number>> = {};
const durationsByTool = new Map<string, number[]>();
const errorsByTool = new Map<string, number>();
+ const stagesByTool = new Map<string, Map<RouteStageName, number[]>>();
let errorCount = 0;
for (const record of records) {
@@ -185,12 +234,14 @@ export function summarizeMcpRouteLatency(
if (bucket) bucket.push(duration);
else durationsByTool.set(tool, [duration]);
}
+ collectStageTimes(payload["stages"], tool, stagesByTool);
}
const routes: McpRouteStats[] = [];
for (const [tool, durations] of durationsByTool) {
const sorted = durations.toSorted((a, b) => a - b);
const sum = sorted.reduce((acc, value) => acc + value, 0);
+ const stages = summarizeStages(stagesByTool.get(tool));
routes.push({
tool,
count: sorted.length,
@@ -201,6 +252,7 @@ export function summarizeMcpRouteLatency(
p50_ms: percentile(sorted, 50),
p95_ms: percentile(sorted, 95),
p99_ms: percentile(sorted, 99),
+ ...(stages.length > 0 ? { stages } : {}),
});
}
// Slowest surface first, tie-broken by name for a stable order.
@@ -214,6 +266,54 @@ export function summarizeMcpRouteLatency(
});
}
+/**
+ * Gather one persisted record's stage times under its tool. Re-checks the
+ * allowlist and the value guard on read, so a hand-edited record never
+ * puts a free-text name into the summary.
+ */
+function collectStageTimes(
+ raw: unknown,
+ tool: string,
+ stagesByTool: Map<string, Map<RouteStageName, number[]>>,
+): void {
+ if (!Array.isArray(raw)) return;
+ for (const stage of raw as ReadonlyArray<unknown>) {
+ if (typeof stage !== "object" || stage === null) continue;
+ const { name, ms } = stage as { name?: unknown; ms?: unknown };
+ if (!isRouteStageName(name) || typeof ms !== "number" || !Number.isFinite(ms) || ms < 0) {
+ continue;
+ }
+ let byStage = stagesByTool.get(tool);
+ if (byStage === undefined) {
+ byStage = new Map();
+ stagesByTool.set(tool, byStage);
+ }
+ const bucket = byStage.get(name);
+ if (bucket) bucket.push(ms);
+ else byStage.set(name, [ms]);
+ }
+}
+
+function summarizeStages(
+ byStage: ReadonlyMap<RouteStageName, ReadonlyArray<number>> | undefined,
+): McpRouteStageStats[] {
+ if (byStage === undefined) return [];
+ const stats: McpRouteStageStats[] = [];
+ for (const name of ROUTE_STAGE_NAMES) {
+ const times = byStage.get(name);
+ if (times === undefined || times.length === 0) continue;
+ const sorted = times.toSorted((a, b) => a - b);
+ const sum = sorted.reduce((acc, value) => acc + value, 0);
+ stats.push({
+ name,
+ count: sorted.length,
+ avg_ms: Math.round((sum / sorted.length) * 10) / 10,
+ p95_ms: percentile(sorted, 95),
+ });
+ }
+ return stats;
+}
+
/** Nearest-rank percentile over an ascending-sorted array; 0 for empty input. */
function percentile(sortedAsc: ReadonlyArray<number>, p: number): number {
if (sortedAsc.length === 0) return 0;
diff --git a/src/core/brain/near-duplicate-vectors.ts b/src/core/brain/near-duplicate-vectors.ts
new file mode 100644
index 00000000..ca9a87d4
--- /dev/null
+++ b/src/core/brain/near-duplicate-vectors.ts
@@ -0,0 +1,178 @@
+/**
+ * Stored-vector tier for retire siblings (near-duplicate defense,
+ * t_acab97de): cosine similarity between a probe page and candidate pages
+ * over the vectors the search index ALREADY holds. Zero provider calls -
+ * this module never imports from the embeddings layer, and a test pins
+ * that by scanning its source. Same precedent as `surprisal.ts`: open the
+ * store read-only and read what reindexing paid for.
+ *
+ * Only content chunks are compared. The indexer emits a page's frontmatter
+ * as chunks of their own, and two preference frontmatter blocks share
+ * their keys and most values, so a frontmatter pair would score near 1 for
+ * any two preferences.
+ *
+ * The outcome is a named status, never a silent degrade:
+ * - `index_missing`: no index has been built;
+ * - `vec_unavailable`: the sqlite-vec extension did not load;
+ * - `not_embedded`: the probe has no stored content vector yet, so
+ * nothing was compared;
+ * - `model_mismatch`: the probe's stored vectors were written under a
+ * model or dimension other than the configured one;
+ * - `used`: the probe was compared; a candidate without comparable
+ * stored vectors simply has no score.
+ * Any other store failure is a named `SearchError` and propagates.
+ */
+
+import { Store } from "../search/store.ts";
+import type { ChunkRow } from "../search/store.ts";
+import type { StoredChunkEmbedding } from "../search/store/vectors.ts";
+import { SearchError } from "../search/search-error.ts";
+import type { ResolvedSearchConfig } from "../search/types.ts";
+
+export type StoredVectorStatus =
+ | "used"
+ | "index_missing"
+ | "vec_unavailable"
+ | "not_embedded"
+ | "model_mismatch";
+
+export interface StoredVectorSimilarity {
+ readonly status: StoredVectorStatus;
+ /** Vault-relative candidate path -> max chunk-pair cosine. */
+ readonly scores: ReadonlyMap<string, number>;
+}
+
+export interface StoredVectorDeps {
+ /** Whether the store loads sqlite-vec; omitted loads it. */
+ readonly loadVec?: boolean;
+}
+
+const NO_SCORES: ReadonlyMap<string, number> = new Map();
+
+/**
+ * Score every candidate against the probe by the best cosine over their
+ * comparable chunk pairs (same model, same dimension, finite). The probe's
+ * rows must match the configured model and dimension when those are set.
+ */
+export async function storedVectorSimilarity(
+ config: ResolvedSearchConfig,
+ probePath: string,
+ candidatePaths: ReadonlyArray<string>,
+ deps: StoredVectorDeps = {},
+): Promise<StoredVectorSimilarity> {
+ return (await storedVectorSimilarities(config, [probePath], candidatePaths, deps))[0]!;
+}
+
+/**
+ * {@link storedVectorSimilarity} for several probes over one read store:
+ * one result per probe, in order, and each page's stored rows read once.
+ */
+export async function storedVectorSimilarities(
+ config: ResolvedSearchConfig,
+ probePaths: ReadonlyArray<string>,
+ candidatePaths: ReadonlyArray<string>,
+ deps: StoredVectorDeps = {},
+): Promise<ReadonlyArray<StoredVectorSimilarity>> {
+ const every = (status: StoredVectorStatus): ReadonlyArray<StoredVectorSimilarity> =>
+ probePaths.map(() => ({ status, scores: NO_SCORES }));
+ let store: Store;
+ try {
+ store = await Store.open(config, {
+ mode: "read",
+ ...(deps.loadVec !== undefined ? { loadVec: deps.loadVec } : {}),
+ });
+ } catch (e) {
+ if (e instanceof SearchError && e.code === "INDEX_MISSING") return every("index_missing");
+ throw e;
+ }
+ try {
+ if (!store.vecLoaded()) return every("vec_unavailable");
+ const rowsByPath = new Map<string, ReadonlyArray<StoredChunkEmbedding>>();
+ const stored = (path: string): ReadonlyArray<StoredChunkEmbedding> => {
+ const cached = rowsByPath.get(path);
+ if (cached !== undefined) return cached;
+ const docId = store.getDocumentIdByPath(path);
+ let rows: ReadonlyArray<StoredChunkEmbedding> = [];
+ if (docId !== null) {
+ const frontmatter = frontmatterChunkIds(store.getChunksByDocument(docId));
+ rows = store.storedEmbeddingsForDocument(docId).filter((r) => !frontmatter.has(r.chunkId));
+ }
+ rowsByPath.set(path, rows);
+ return rows;
+ };
+ const { model, dimension } = config.semantic;
+ return probePaths.map((probePath): StoredVectorSimilarity => {
+ const probeRows = stored(probePath);
+ if (probeRows.length === 0) return { status: "not_embedded", scores: NO_SCORES };
+ const usable = probeRows.filter(
+ (r) =>
+ (model === null || r.model === model) &&
+ (dimension === null || r.dimension === dimension),
+ );
+ if (usable.length === 0) return { status: "model_mismatch", scores: NO_SCORES };
+ const scores = new Map<string, number>();
+ for (const path of candidatePaths) {
+ if (path === probePath) continue;
+ const best = bestPairCosine(usable, stored(path));
+ if (best !== null) scores.set(path, best);
+ }
+ return { status: "used", scores };
+ });
+ } finally {
+ await store.close();
+ }
+}
+
+/**
+ * Ids of the leading chunks the chunker emitted for the frontmatter block:
+ * chunk 0 opens with the `---` fence, and the block runs through the chunk
+ * that ends on the closing fence. An oversize block spans several chunks,
+ * and no body text shares a chunk with it.
+ */
+function frontmatterChunkIds(chunks: ReadonlyArray<ChunkRow>): Set<number> {
+ const ids = new Set<number>();
+ const ordered = chunks.toSorted((a, b) => a.chunkIndex - b.chunkIndex);
+ const first = ordered[0];
+ if (first === undefined || first.content.split("\n")[0]!.trim() !== "---") return ids;
+ for (const [i, chunk] of ordered.entries()) {
+ ids.add(chunk.id);
+ const lines = chunk.content.trimEnd().split("\n");
+ // Chunk 0's first line is the opening fence, never the closing one.
+ const tail = i === 0 ? lines.slice(1) : lines;
+ if (tail.at(-1)?.trim() === "---") break;
+ }
+ return ids;
+}
+
+function bestPairCosine(
+ probe: ReadonlyArray<StoredChunkEmbedding>,
+ candidate: ReadonlyArray<StoredChunkEmbedding>,
+): number | null {
+ let best: number | null = null;
+ for (const a of probe) {
+ for (const b of candidate) {
+ if (a.model !== b.model || a.dimension !== b.dimension) continue;
+ if (a.vector.length !== a.dimension || b.vector.length !== b.dimension) continue;
+ const score = cosine(a.vector, b.vector);
+ // A zero or non-finite row from an index written before the store
+ // guard has no direction: unusable, never a NaN score.
+ if (!Number.isFinite(score)) continue;
+ if (best === null || score > best) best = score;
+ }
+ }
+ return best;
+}
+
+function cosine(a: Float32Array, b: Float32Array): number {
+ let dot = 0;
+ let normA = 0;
+ let normB = 0;
+ for (let i = 0; i < a.length; i++) {
+ const x = a[i]!;
+ const y = b[i]!;
+ dot += x * y;
+ normA += x * x;
+ normB += y * y;
+ }
+ return dot / (Math.sqrt(normA) * Math.sqrt(normB));
+}
diff --git a/src/core/brain/near-duplicate.ts b/src/core/brain/near-duplicate.ts
new file mode 100644
index 00000000..7ab13537
--- /dev/null
+++ b/src/core/brain/near-duplicate.ts
@@ -0,0 +1,141 @@
+/**
+ * Reach-aware near-duplicate kernel: one pure lookup shared by every
+ * point a fact enters or leaves the vault (write-receipt hint, feedback
+ * advisory, retire siblings).
+ *
+ * The caller projects its probe and pool into token sets with
+ * `tokenise` (see `similarity.ts`); the kernel never reads the vault,
+ * never sees a vector and never calls a model. The `readable`
+ * predicate is required and runs before anything else, so a withheld
+ * entry is neither scored nor counted: no match and no census number
+ * can disclose it. Operator-reach callers pass {@link READ_ALL_REFS}
+ * explicitly, so every unfiltered call site stays visible in the diff.
+ */
+
+import { jaccard } from "./similarity.ts";
+
+export type NearDuplicateMethod = "lexical" | "embedding";
+export type ReadableRef = (ref: string) => boolean;
+
+/** Explicit operator-reach predicate; every unfiltered call site names it. */
+export const READ_ALL_REFS: ReadableRef = () => true;
+
+/**
+ * The one named threshold table. `writeHint` is the shipped page-lint
+ * bar, `retireSiblingLexical` the doctor lint bar and
+ * `retireSiblingEmbedding` the entity semantic-dedup bar.
+ */
+export const NEAR_DUPLICATE_THRESHOLDS = Object.freeze({
+ writeHint: 0.8,
+ retireSiblingLexical: 0.7,
+ retireSiblingEmbedding: 0.92,
+} as const);
+
+/** Short texts share most of their tokens by accident; below this nothing is scored. */
+export const NEAR_DUPLICATE_MIN_TOKENS = 4;
+/** Upper bound on the entries one lookup scores. */
+export const NEAR_DUPLICATE_CANDIDATE_CAP = 200;
+/** Upper bound on keyword-index chunks a write-side widening pulls. */
+export const NEAR_DUPLICATE_WIDENING_TOP_K = 20;
+
+export interface NearDuplicateProbe {
+ /** Vault-relative path or preference id. */
+ readonly ref: string;
+ /** From `similarity.ts` `tokenise`. */
+ readonly tokens: ReadonlySet<string>;
+}
+
+export interface NearDuplicatePoolEntry {
+ readonly ref: string;
+ readonly tokens: ReadonlySet<string>;
+ /** Optional bucket; when the options carry `bucket`, entries must match it. */
+ readonly bucket?: string;
+}
+
+export interface NearDuplicateOptions {
+ readonly threshold: number;
+ /** Required; applied before scoring and counting. */
+ readonly readable: ReadableRef;
+ /** Default {@link NEAR_DUPLICATE_MIN_TOKENS}. */
+ readonly minTokens?: number;
+ /** Default {@link NEAR_DUPLICATE_CANDIDATE_CAP}. */
+ readonly cap?: number;
+ readonly bucket?: string;
+}
+
+export interface NearDuplicateMatch {
+ readonly ref: string;
+ /** Rounded to 3 decimals. */
+ readonly score: number;
+ readonly method: NearDuplicateMethod;
+}
+
+export interface NearDuplicateScan {
+ /** Readable entries actually scored. */
+ readonly compared: number;
+ /** Readable entries left out by the cap. */
+ readonly capped: number;
+ /** Readable entries skipped as too short. */
+ readonly below_min_tokens: number;
+}
+
+export interface NearDuplicateResult {
+ /** Sorted by score descending, then ref. */
+ readonly matches: ReadonlyArray<NearDuplicateMatch>;
+ readonly scan: NearDuplicateScan;
+}
+
+/**
+ * Score `probe` against `pool` by token Jaccard. Entries are filtered in
+ * this order: reach, the probe's own ref, the bucket, the minimum token
+ * count, then the cap (in pool order). A probe under the minimum token
+ * count scores nothing. Matches at or above `threshold` are returned.
+ */
+export function findNearDuplicates(
+ probe: NearDuplicateProbe,
+ pool: ReadonlyArray<NearDuplicatePoolEntry>,
+ opts: NearDuplicateOptions,
+): NearDuplicateResult {
+ const minTokens = opts.minTokens ?? NEAR_DUPLICATE_MIN_TOKENS;
+ const cap = opts.cap ?? NEAR_DUPLICATE_CANDIDATE_CAP;
+ if (probe.tokens.size < minTokens) {
+ return { matches: [], scan: { compared: 0, capped: 0, below_min_tokens: 0 } };
+ }
+ const matches: NearDuplicateMatch[] = [];
+ let compared = 0;
+ let capped = 0;
+ let belowMinTokens = 0;
+ for (const entry of pool) {
+ if (!opts.readable(entry.ref)) continue;
+ if (entry.ref === probe.ref) continue;
+ if (opts.bucket !== undefined && entry.bucket !== opts.bucket) continue;
+ if (entry.tokens.size < minTokens) {
+ belowMinTokens++;
+ continue;
+ }
+ if (compared >= cap) {
+ capped++;
+ continue;
+ }
+ compared++;
+ const sim = jaccard(probe.tokens, entry.tokens);
+ if (sim < opts.threshold) continue;
+ matches.push({ ref: entry.ref, score: roundScore(sim), method: "lexical" });
+ }
+ matches.sort(compareMatches);
+ return { matches, scan: { compared, capped, below_min_tokens: belowMinTokens } };
+}
+
+/** Round a similarity score to 3 decimals, the precision every surface reports. */
+export function roundScore(score: number): number {
+ return Math.round(score * 1000) / 1000;
+}
+
+/** Score descending, then ref by code unit, so the order never depends on locale. */
+export function compareMatches(
+ a: { readonly ref: string; readonly score: number },
+ b: { readonly ref: string; readonly score: number },
+): number {
+ if (a.score !== b.score) return b.score - a.score;
+ return a.ref < b.ref ? -1 : a.ref > b.ref ? 1 : 0;
+}
diff --git a/src/core/brain/notes/create-note.ts b/src/core/brain/notes/create-note.ts
index 4bd453aa..959ba4d7 100644
--- a/src/core/brain/notes/create-note.ts
+++ b/src/core/brain/notes/create-note.ts
@@ -75,6 +75,7 @@ import {
type NoteTemplateVariables,
} from "./note-template.ts";
import { NOTE_WRITE_OP, recordNoteWrite } from "./write-record.ts";
+import { ROUTE_STAGE, timeStageSync } from "../../route-scope.ts";
/** Machine-readable reason a {@link createNote} call was refused. */
export type CreateNoteErrorCode =
@@ -567,11 +568,13 @@ export function createNote(vault: string, input: CreateNoteInput): CreateNoteRes
mkdirSync(dirname(abs), { recursive: true });
try {
- writeFrontmatterAtomic(abs, frontmatter, body, {
- overwrite: false,
- existsErrorKind: "note",
- vaultForRelativePath: vault,
- });
+ timeStageSync(ROUTE_STAGE.documentWrite, () =>
+ writeFrontmatterAtomic(abs, frontmatter, body, {
+ overwrite: false,
+ existsErrorKind: "note",
+ vaultForRelativePath: vault,
+ }),
+ );
} catch (err) {
// One predicate for "the name was taken", shared with every other
// collision-aware call site. The hand-rolled pair it replaces read
@@ -591,12 +594,14 @@ export function createNote(vault: string, input: CreateNoteInput): CreateNoteRes
// throwing, and the reason rides out on the result rather than being
// swallowed here (who-wrote-what, Task A). A create replaces nothing,
// so there is no before-image to store and `hash_before` is `absent`.
- const receipt = recordNoteWrite(vault, {
- op: NOTE_WRITE_OP.create,
- target: relPath,
- before: null,
- after: { bytes: formatFrontmatter(frontmatter, body) },
- ...(input.configPath !== undefined ? { configPath: input.configPath } : {}),
- });
+ const receipt = timeStageSync(ROUTE_STAGE.writeReceipt, () =>
+ recordNoteWrite(vault, {
+ op: NOTE_WRITE_OP.create,
+ target: relPath,
+ before: null,
+ after: { bytes: formatFrontmatter(frontmatter, body) },
+ ...(input.configPath !== undefined ? { configPath: input.configPath } : {}),
+ }),
+ );
return { path: relPath, outcome: "created", created: true, ...receipt };
}
diff --git a/src/core/brain/page-lint-widening.ts b/src/core/brain/page-lint-widening.ts
new file mode 100644
index 00000000..42ff8f80
--- /dev/null
+++ b/src/core/brain/page-lint-widening.ts
@@ -0,0 +1,210 @@
+/**
+ * Keyword-index widening for the write-receipt near-duplicate hint.
+ *
+ * The receipt hint in `page-lint.ts` compares a written page only with the
+ * pages in its own directory, because that walk reads the disk and must
+ * stay bounded. A paraphrase usually lands somewhere else. This module
+ * widens the candidate set through the search index that already exists:
+ *
+ * 1. the written body's longest {@link WIDENING_QUERY_MAX_TERMS} distinct
+ * tokens become one FTS5 OR query through `buildFtsMatch`; the
+ * candidates are still scored on the full token set;
+ * 2. `keywordTopK` pulls the BM25 top {@link NEAR_DUPLICATE_WIDENING_TOP_K}
+ * chunks of other pages per written page (the page's own indexed
+ * chunks are excluded), so one call reads at most that many
+ * candidate pages per page it wrote;
+ * 3. each hit maps to its document path, the caller's `readable`
+ * predicate drops a withheld page before anything else happens, and
+ * the page is re-read from disk, so a page deleted since the last
+ * index pass, or one changed since, cannot produce a stale hint;
+ * 4. every page is projected into a {@link NearDuplicateCandidate} for
+ * the lint to score through the shared kernel.
+ *
+ * Zero embedding spend by construction: nothing here imports from
+ * `src/core/search/embeddings/`. The store is opened read-only and is
+ * never repaired from here (the FTS rebuild path takes the writer lock).
+ *
+ * Fail-open, never silent: an index that cannot be opened, queried or
+ * closed, and any other throw on the way to the candidates, yields the
+ * named status `index_unavailable` and no candidates, so the
+ * write and the same-directory hint are unaffected and the receipt says
+ * the widening did not run. The status carries the failure by name
+ * (`detail`), so a missing index, a locked or corrupt one and a defect in
+ * this module read differently on the receipt.
+ */
+
+import { readFileSync, statSync } from "node:fs";
+import { resolve } from "node:path";
+
+import { vaultRelative } from "../path-safety.ts";
+import { compositeScopeKey, scopeFromFrontmatter } from "../scope-key.ts";
+import { buildFtsMatch } from "../search/fts.ts";
+import { FTS_MATCH_MODE } from "../search/fts-match-mode.ts";
+import { Store } from "../search/store.ts";
+import type { ResolvedSearchConfig } from "../search/types.ts";
+import { parseFrontmatterText } from "../vault.ts";
+import { NEAR_DUPLICATE_WIDENING_TOP_K, type ReadableRef } from "./near-duplicate.ts";
+import {
+ failureCode,
+ type NearDuplicateCandidate,
+ type NearDuplicateWideningStatus,
+} from "./page-lint.ts";
+import { tokenise } from "./similarity.ts";
+import { ARTIFACT_MAX_BYTES } from "./write-session/validate.ts";
+
+export interface WideningResult {
+ readonly status: NearDuplicateWideningStatus;
+ readonly candidates: ReadonlyArray<NearDuplicateCandidate>;
+ /**
+ * Why the status is `index_unavailable`: the search error code, the
+ * errno code or the error name, never a message or a path. Absent when
+ * the index answered.
+ */
+ readonly detail?: string;
+}
+
+/** Extension every candidate page on disk carries. */
+const MARKDOWN_EXT = ".md";
+
+/** Prefix of a vault-relative spelling that leaves the vault. */
+const VAULT_ESCAPE_PREFIX = "../";
+
+/**
+ * Most distinct tokens one widening query OR-joins. FTS5 cost grows with
+ * the term count and a written page may be up to the artifact byte cap,
+ * so the query is bounded; the hint's quality does not depend on it,
+ * because the lint scores every candidate on the full token set.
+ */
+export const WIDENING_QUERY_MAX_TERMS = 64;
+
+/** The fail-open answer for a widening that could not run, naming why. */
+export function wideningUnavailable(err: unknown): WideningResult {
+ return Object.freeze({
+ status: "index_unavailable",
+ candidates: Object.freeze([]),
+ detail: failureCode(err),
+ });
+}
+
+/** The vault-relative spelling of a page, the one `page-lint.ts` reports. */
+function canonicalPage(vault: string, page: string): string {
+ return vaultRelative(resolve(vault, page), vault);
+}
+
+/**
+ * The page on disk as a candidate, or `null` when it provides no evidence:
+ * outside the vault, not Markdown, gone since the index saw it, over the
+ * artifact byte cap, or unreadable. The index is a hint about where to
+ * look, not evidence; the same-directory walk in `page-lint.ts` is the
+ * surface that names an unreadable sibling.
+ */
+function readCandidate(vault: string, page: string): NearDuplicateCandidate | null {
+ if (page.startsWith(VAULT_ESCAPE_PREFIX) || !page.endsWith(MARKDOWN_EXT)) return null;
+ const absolute = resolve(vault, page);
+ try {
+ if (statSync(absolute).size > ARTIFACT_MAX_BYTES) return null;
+ const [meta, body] = parseFrontmatterText(readFileSync(absolute, "utf8"));
+ return Object.freeze({
+ page,
+ scopeKey: compositeScopeKey(scopeFromFrontmatter(meta)),
+ tokens: tokenise(body),
+ });
+ } catch {
+ return null;
+ }
+}
+
+/**
+ * The FTS5 query for one written page: its distinct body tokens, OR-joined,
+ * capped at {@link WIDENING_QUERY_MAX_TERMS} by the longest-first rule.
+ * Longest first, ties in code-unit order: a long token is the rarer one
+ * in practice and needs no index read to rank, and the order is total, so
+ * the same body always yields the same query.
+ */
+function widenQuery(vault: string, page: string): string {
+ const candidate = readCandidate(vault, page);
+ if (candidate === null) return "";
+ const terms = [...candidate.tokens]
+ .toSorted((a, b) => b.length - a.length || (a < b ? -1 : a > b ? 1 : 0))
+ .slice(0, WIDENING_QUERY_MAX_TERMS);
+ return buildFtsMatch(terms.join(" "), { matchMode: FTS_MATCH_MODE.any });
+}
+
+/**
+ * Vault-relative paths of the documents behind the top keyword hits, in hit
+ * order. The written page's own indexed chunks are excluded by document id:
+ * its query is built from its own tokens, so on an update its older chunks
+ * would otherwise be the strongest matches and fill every slot. The pull
+ * grows by exactly that page's chunk count, so the top-k still counts
+ * other pages only.
+ */
+function keywordPaths(store: Store, query: string, page: string): ReadonlyArray<string> {
+ const ownId = store.getDocumentIdByPath(page);
+ const ownChunks = ownId === null ? 0 : store.chunksForDocument(ownId).length;
+ const hits = store
+ .keywordTopK(query, { limit: NEAR_DUPLICATE_WIDENING_TOP_K + ownChunks })
+ .filter((hit) => hit.documentId !== ownId)
+ .slice(0, NEAR_DUPLICATE_WIDENING_TOP_K);
+ const hydrated = store.hydrateChunks(hits.map((hit) => hit.chunkId));
+ const paths: string[] = [];
+ for (const hit of hits) {
+ const chunk = hydrated.get(hit.chunkId);
+ if (chunk !== undefined) paths.push(chunk.path);
+ }
+ return paths;
+}
+
+/**
+ * Near-duplicate candidates from the keyword index for the pages one write
+ * committed. Each candidate appears once, in first-hit order, whatever
+ * number of chunks or written pages pulled it in.
+ *
+ * Never throws: the write it hints on has already happened, so a failure
+ * anywhere - opening, querying or closing the store, the `readable`
+ * predicate, a candidate read - is the named `index_unavailable` status,
+ * never an error the caller would read as a failed write and retry.
+ */
+export async function collectWideningCandidates(
+ config: ResolvedSearchConfig,
+ vault: string,
+ pages: ReadonlyArray<string>,
+ readable: ReadableRef,
+): Promise<WideningResult> {
+ try {
+ return await widen(config, vault, pages, readable);
+ } catch (err) {
+ return wideningUnavailable(err);
+ }
+}
+
+async function widen(
+ config: ResolvedSearchConfig,
+ vault: string,
+ pages: ReadonlyArray<string>,
+ readable: ReadableRef,
+): Promise<WideningResult> {
+ const store = await Store.open(config, { mode: "read" });
+ const hitPaths: string[] = [];
+ try {
+ for (const page of pages) {
+ const written = canonicalPage(vault, page);
+ const query = widenQuery(vault, written);
+ if (query === "") continue;
+ hitPaths.push(...keywordPaths(store, query, written));
+ }
+ } finally {
+ await store.close();
+ }
+ const seen = new Set<string>();
+ const candidates: NearDuplicateCandidate[] = [];
+ for (const hitPath of hitPaths) {
+ const page = canonicalPage(vault, hitPath);
+ if (seen.has(page)) continue;
+ seen.add(page);
+ // Reach first: a withheld page is never read, so it cannot be returned.
+ if (!readable(page)) continue;
+ const candidate = readCandidate(vault, page);
+ if (candidate !== null) candidates.push(candidate);
+ }
+ return Object.freeze({ status: "used", candidates: Object.freeze(candidates) });
+}
diff --git a/src/core/brain/page-lint.ts b/src/core/brain/page-lint.ts
index f630ed37..dd585641 100644
--- a/src/core/brain/page-lint.ts
+++ b/src/core/brain/page-lint.ts
@@ -80,10 +80,11 @@
* absent `lint` key must mean clean, and only that.
*/
-import { readFileSync, readdirSync, statSync } from "node:fs";
-import { posix, resolve } from "node:path";
+import { lstatSync, readFileSync, readdirSync, statSync } from "node:fs";
+import { join, posix, resolve } from "node:path";
import { vaultRelative } from "../path-safety.ts";
+import { ROUTE_STAGE, timeStageSync } from "../route-scope.ts";
import { compositeScopeKey, scopeFromFrontmatter } from "../scope-key.ts";
import { parseFrontmatterText } from "../vault.ts";
import type { FrontmatterMap } from "../types.ts";
@@ -91,12 +92,21 @@ import { collectAllBasenames } from "./doctor/records.ts";
import {
LINT_CONSOLIDATE_KIND,
createMergedLinkResolver,
+ type MergedLinkResolution,
type MergedLinkResolver,
} from "./lint-consolidate.ts";
import { nextCommandField, type NextCommandField } from "./next-step.ts";
+import { pageIdToPath, reportMergeChain } from "./page-meta/page-id.ts";
+import { brainDirs } from "./paths.ts";
import { loadSchemaPack } from "./schema-pack.ts";
import type { BrainSchemaVocabulary } from "./schema-vocab.ts";
-import { jaccard, tokenise } from "./similarity.ts";
+import {
+ NEAR_DUPLICATE_THRESHOLDS,
+ findNearDuplicates,
+ type NearDuplicatePoolEntry,
+ type ReadableRef,
+} from "./near-duplicate.ts";
+import { tokenise } from "./similarity.ts";
import type { DoctorSeverity } from "./types.ts";
import { WIKILINK_TARGET_RE, isBrainArtifactId, normaliseWikilinkTarget } from "./wikilink.ts";
import { ARTIFACT_MAX_BYTES, validateArtifact } from "./write-session/validate.ts";
@@ -122,10 +132,10 @@ const BROKEN_WIKILINK_CODE = "broken-wikilink";
* Near-duplicate bar (t_d30c0548): the jaccard score at or above which the
* authored body counts as closely matching an existing page. Deliberately
* high - near-DUPLICATE claims near-identity, unlike the 0.5 contradiction
- * precedent that answers "same subject" - and a named constant rather than
- * config, so promoting it after real-world noise is a one-line change.
+ * precedent that answers "same subject". An alias of the kernel's one
+ * threshold table, so the write hint cannot drift from it.
*/
-export const NEAR_DUPLICATE_JACCARD = 0.8;
+export const NEAR_DUPLICATE_JACCARD = NEAR_DUPLICATE_THRESHOLDS.writeHint;
/** Code the near-duplicate detector reports under. */
export const NEAR_DUPLICATE_CODE = "near-duplicate";
@@ -198,7 +208,7 @@ const UNNAMED_FAILURE_CODE = "unknown";
* identifiers and integers cross this boundary, never a path and never an OS
* message.
*/
-function failureCode(err: unknown): string {
+export function failureCode(err: unknown): string {
const code = (err as NodeJS.ErrnoException | null)?.code;
if (typeof code === "string" && code.length > 0) return code;
const name = (err as Error | null)?.name;
@@ -271,6 +281,19 @@ export interface PageLintReport {
readonly candidates_skipped?: number;
/** Sibling pages the near-duplicate check could not read, each named. Present only when any. */
readonly candidates_unreadable?: ReadonlyArray<PageLintCandidateSkip>;
+ /**
+ * Whether the near-duplicate check looked beyond the written pages'
+ * directories. Present only when widening is on; `used` alone is not
+ * something to say, `index_unavailable` is.
+ */
+ readonly widening?: NearDuplicateWideningStatus;
+ /**
+ * Why widening reported `index_unavailable`: the search error code, the
+ * errno code or the error name ({@link failureCode}), so a missing index,
+ * a locked or corrupt one and a defect read differently. Present only
+ * with that status.
+ */
+ readonly widening_detail?: string;
}
/** A sibling page the near-duplicate check could not read, and the errno code why. */
@@ -279,10 +302,44 @@ export interface PageLintCandidateSkip {
readonly detail: string;
}
+/**
+ * Whether the near-duplicate hint looked beyond the written pages' own
+ * directories: `used` when the keyword index answered, `index_unavailable`
+ * when it could not be opened or queried. Absent from a report when
+ * widening is off.
+ */
+export type NearDuplicateWideningStatus = "used" | "index_unavailable";
+
+/** How one call to {@link lintWrittenPages} is bounded. */
+export interface LintWrittenPagesOptions {
+ /**
+ * The caller's reach. Required, never defaulted: a sibling page the
+ * caller may not read is dropped before it is scored and before the
+ * census counts it, and a wikilink target or merge-chain hop the caller
+ * may not read answers as absent, so neither a finding nor a census
+ * number can disclose it. An operator-reach caller passes `READ_ALL_REFS`.
+ */
+ readonly readable: ReadableRef;
+ /**
+ * Candidates from outside the written pages' directories, collected by
+ * `page-lint-widening.ts`. Scored under the same scope-bucket and reach
+ * rules; a page already in a written page's directory counts once.
+ */
+ readonly extraCandidates?: ReadonlyArray<NearDuplicateCandidate>;
+ /** Whether widening ran; absent when it is off. */
+ readonly widening?: NearDuplicateWideningStatus;
+ /** Why widening was unavailable; see {@link PageLintReport.widening_detail}. */
+ readonly wideningDetail?: string;
+}
+
/** What building the near-duplicate candidate index left out, by count and by name. */
export interface NearDuplicateCensus {
readonly candidatesSkipped: number;
readonly unreadable: ReadonlyArray<PageLintCandidateSkip>;
+ /** Whether widening ran; absent when it is off. */
+ readonly widening?: NearDuplicateWideningStatus;
+ /** Why widening was unavailable; see {@link PageLintReport.widening_detail}. */
+ readonly wideningDetail?: string;
}
/**
@@ -374,17 +431,42 @@ export interface LintContext {
* same scope bucket) needs nothing vault-wide.
*/
readonly nearDuplicateCandidates: ReadonlyMap<string, ReadonlyArray<NearDuplicateCandidate>>;
+ /**
+ * Candidates from outside the written pages' directories (widening).
+ * Every written page is compared with them as well as with its own
+ * directory's candidates.
+ */
+ readonly extraCandidates?: ReadonlyArray<NearDuplicateCandidate>;
+ /**
+ * The caller's reach, applied again by the kernel before any score, and
+ * by the wikilink and merged-link checks, so a withheld page answers as
+ * an absent one on every finding.
+ */
+ readonly readable: ReadableRef;
/** What the candidate index left out; absent means nothing was. */
readonly nearDuplicateCensus?: NearDuplicateCensus;
}
/**
* The near-duplicate findings for one written page: the authored body
- * against every candidate in the same directory and the same composite
- * scope bucket, at or above {@link NEAR_DUPLICATE_JACCARD}. The page
- * itself is never its own candidate - a batch can write two lookalikes,
- * but a single write cannot resemble itself - and the score rides the
- * message so an operator judgement call is informed.
+ * against every candidate in the same directory - plus every widened
+ * candidate - in the same composite scope bucket, scored through the
+ * shared kernel at {@link NEAR_DUPLICATE_JACCARD}. The page itself is
+ * never its own candidate - a batch can write two lookalikes, but a
+ * single write cannot resemble itself - and the score rides the message
+ * so an operator judgement call is informed.
+ *
+ * A same-directory finding keeps the message shipped since v1.64.0; a
+ * widened one appends `method=<method>`, so a reader can tell the hint
+ * came from the keyword index. One deliberate change from v1.73.0: the
+ * kernel scores nothing below `NEAR_DUPLICATE_MIN_TOKENS` body
+ * tokens, on either side, so two short stubs that share their few words
+ * by accident (formerly a jaccard=1.000 finding) are no longer reported.
+ * The kernel's own match order never reaches the receipt, which is ranked
+ * by {@link comparePageLintFindings}. The pool is bounded upstream (the
+ * directory cap is counted in `candidates_skipped`, the widening pull by
+ * its top-k), so the kernel's own cap is set to the pool size and never
+ * drops a candidate unreported.
*/
function nearDuplicateFindings(
ctx: LintContext,
@@ -392,28 +474,93 @@ function nearDuplicateFindings(
meta: FrontmatterMap,
body: string,
): PageLintFinding[] {
- const out: PageLintFinding[] = [];
- const scopeKey = compositeScopeKey(scopeFromFrontmatter(meta));
- const tokens = tokenise(body);
- for (const candidate of ctx.nearDuplicateCandidates.get(posix.dirname(page)) ?? NO_CANDIDATES) {
- if (candidate.page === page) continue;
- if (candidate.scopeKey !== scopeKey) continue;
- const sim = jaccard(tokens, candidate.tokens);
- if (sim < NEAR_DUPLICATE_JACCARD) continue;
- out.push(
- finding(
- "warning",
- NEAR_DUPLICATE_CODE,
- page,
- candidate.page,
- `body resembles [[${candidate.page}]] jaccard=${sim.toFixed(3)} (threshold ${NEAR_DUPLICATE_JACCARD})`,
- ),
- );
+ const local = ctx.nearDuplicateCandidates.get(posix.dirname(page)) ?? NO_CANDIDATES;
+ const widened = new Set<string>();
+ const pool: NearDuplicatePoolEntry[] = local.map(poolEntry);
+ const inPool = new Set(local.map((candidate) => candidate.page));
+ for (const candidate of ctx.extraCandidates ?? NO_CANDIDATES) {
+ if (inPool.has(candidate.page)) continue;
+ inPool.add(candidate.page);
+ widened.add(candidate.page);
+ pool.push(poolEntry(candidate));
}
- return out;
+ const { matches } = findNearDuplicates({ ref: page, tokens: tokenise(body) }, pool, {
+ threshold: NEAR_DUPLICATE_JACCARD,
+ readable: ctx.readable,
+ bucket: compositeScopeKey(scopeFromFrontmatter(meta)),
+ cap: pool.length,
+ });
+ return matches.map((match) => {
+ const evidence = `body resembles [[${match.ref}]] jaccard=${match.score.toFixed(3)} (threshold ${NEAR_DUPLICATE_JACCARD})`;
+ const message = widened.has(match.ref) ? `${evidence} method=${match.method}` : evidence;
+ return finding("warning", NEAR_DUPLICATE_CODE, page, match.ref, message);
+ });
+}
+
+function poolEntry(candidate: NearDuplicateCandidate): NearDuplicatePoolEntry {
+ return { ref: candidate.page, tokens: candidate.tokens, bucket: candidate.scopeKey };
+}
+
+const NOT_MERGED: MergedLinkResolution = Object.freeze({ canonical: null, unresolvable: null });
+
+/** Whether `id` names a Brain page on disk the caller may not read. */
+function isWithheldPageId(vault: string, ctx: LintContext, id: string): boolean {
+ const path = pageIdToPath(vault, id);
+ return path !== null && !ctx.readable(vaultRelative(path, vault));
}
-function lintOnePage(ctx: LintContext, page: string, raw: string): PageLintFinding[] {
+/**
+ * The merge-chain answer for one wikilink target at the caller's reach.
+ *
+ * The vault-wide resolver reads the `merged_into:` pointer of every page
+ * on the chain. A page the caller may not read answers as an absent one
+ * would: the walk ends at its id, the way it ends at a page with no file.
+ * So a withheld target is not merged at all, and a chain that reaches a
+ * withheld page names that page's id as the canonical end and nothing it
+ * points at. Only a link the resolver found merged pays the second walk.
+ */
+function mergedAtReach(vault: string, ctx: LintContext, target: string): MergedLinkResolution {
+ const merged = ctx.mergedLinks.resolve(target);
+ if (merged.canonical === null && merged.unresolvable === null) return merged;
+ const { visited } = reportMergeChain(vault, target);
+ const withheld = visited.findIndex((id) => isWithheldPageId(vault, ctx, id));
+ if (withheld === -1) return merged;
+ if (withheld === 0) return NOT_MERGED;
+ return Object.freeze({ canonical: visited[withheld]!, unresolvable: null });
+}
+
+/**
+ * Whether a Brain artifact named `target` exists where the caller may read
+ * it. `basenames` answers the vault-wide question once per call; a hit is
+ * asked again per directory `collectAllBasenames` indexes, so an artifact
+ * the caller may not read answers as an absent one. Only a linked name
+ * that exists pays the per-directory check.
+ */
+function hasReadableArtifact(vault: string, ctx: LintContext, target: string): boolean {
+ if (!ctx.basenames.has(target)) return false;
+ const dirs = brainDirs(vault);
+ for (const dir of [
+ dirs.brain,
+ dirs.inbox,
+ dirs.processed,
+ dirs.archived,
+ dirs.preferences,
+ dirs.retired,
+ dirs.log,
+ ]) {
+ const absolute = join(dir, `${target}.md`);
+ if (!ctx.readable(vaultRelative(absolute, vault))) continue;
+ if (lstatSync(absolute, { throwIfNoEntry: false })?.isFile() === true) return true;
+ }
+ return false;
+}
+
+function lintOnePage(
+ vault: string,
+ ctx: LintContext,
+ page: string,
+ raw: string,
+): PageLintFinding[] {
const out: PageLintFinding[] = [];
const [meta, body] = parseFrontmatterText(raw);
const schemaType = declaredSchemaType(meta);
@@ -425,7 +572,7 @@ function lintOnePage(ctx: LintContext, page: string, raw: string): PageLintFindi
// note, and is nobody's here to flag - the same boundary the doctor
// and the orphaned-reference fixer draw.
if (!isBrainArtifactId(target)) continue;
- const merged = ctx.mergedLinks.resolve(target);
+ const merged = mergedAtReach(vault, ctx, target);
if (merged.unresolvable !== null) {
out.push(
finding(
@@ -450,7 +597,7 @@ function lintOnePage(ctx: LintContext, page: string, raw: string): PageLintFindi
);
continue;
}
- if (!ctx.basenames.has(target)) {
+ if (!hasReadableArtifact(vault, ctx, target)) {
out.push(
finding(
"warning",
@@ -462,7 +609,11 @@ function lintOnePage(ctx: LintContext, page: string, raw: string): PageLintFindi
);
}
}
- out.push(...nearDuplicateFindings(ctx, page, meta, body));
+ out.push(
+ ...timeStageSync(ROUTE_STAGE.nearDuplicateLookup, () =>
+ nearDuplicateFindings(ctx, page, meta, body),
+ ),
+ );
return out;
}
@@ -510,10 +661,15 @@ function canonicalPage(vault: string, page: string): string {
* `unavailable` slot like every other context-building failure. A page
* spelled outside the vault ({@link VAULT_ESCAPE_PREFIX}) collects no
* candidates at all: the walk stops at the vault boundary.
+ *
+ * A sibling outside the caller's reach (`readable`) is dropped at the
+ * listing, before it is stat'ed: it is never scored, never named as
+ * unreadable and never counted against the cap.
*/
function collectNearDuplicateCandidates(
vault: string,
pages: ReadonlyArray<string>,
+ readable: ReadableRef,
): {
readonly index: ReadonlyMap<string, ReadonlyArray<NearDuplicateCandidate>>;
readonly census: NearDuplicateCensus;
@@ -529,15 +685,17 @@ function collectNearDuplicateCandidates(
let candidatesSkipped = 0;
for (const directory of directories) {
// Stat every sibling (cheap), then read only the newest few.
- const siblings: Array<{ absolute: string; size: number; mtimeMs: number }> = [];
+ const siblings: Array<{ page: string; absolute: string; size: number; mtimeMs: number }> = [];
for (const name of readdirSync(resolve(vault, directory))) {
if (!name.endsWith(MARKDOWN_EXT)) continue;
const absolute = resolve(vault, directory, name);
+ const page = canonicalPage(vault, absolute);
+ if (!readable(page)) continue;
try {
const stat = statSync(absolute);
- siblings.push({ absolute, size: stat.size, mtimeMs: stat.mtimeMs });
+ siblings.push({ page, absolute, size: stat.size, mtimeMs: stat.mtimeMs });
} catch (err) {
- unreadable.push({ page: canonicalPage(vault, absolute), detail: failureCode(err) });
+ unreadable.push({ page, detail: failureCode(err) });
}
}
const newest = siblings.toSorted((a, b) => b.mtimeMs - a.mtimeMs);
@@ -550,12 +708,12 @@ function collectNearDuplicateCandidates(
try {
const [meta, body] = parseFrontmatterText(readFileSync(sibling.absolute, "utf8"));
candidates.push({
- page: canonicalPage(vault, sibling.absolute),
+ page: sibling.page,
scopeKey: compositeScopeKey(scopeFromFrontmatter(meta)),
tokens: tokenise(body),
});
} catch (err) {
- unreadable.push({ page: canonicalPage(vault, sibling.absolute), detail: failureCode(err) });
+ unreadable.push({ page: sibling.page, detail: failureCode(err) });
}
}
index.set(directory, candidates);
@@ -579,18 +737,37 @@ function collectNearDuplicateCandidates(
* nothing here walks vault content beyond the written pages' own
* directories - the near-duplicate candidate index reads each of those
* once per call, sized against the same artifact cap as the pages.
+ *
+ * Every finding answers at the caller's reach (`opts.readable`): a
+ * near-duplicate candidate, a wikilink target and a merge-chain hop the
+ * caller may not read all read as absent. The written pages themselves
+ * are always linted.
*/
-export function lintWrittenPages(vault: string, pages: ReadonlyArray<string>): PageLintReport {
+export function lintWrittenPages(
+ vault: string,
+ pages: ReadonlyArray<string>,
+ opts: LintWrittenPagesOptions,
+): PageLintReport {
if (pages.length === 0) return emptyReport();
let ctx: LintContext;
try {
- const nearDuplicates = collectNearDuplicateCandidates(vault, pages);
+ // The candidate walk and the per-page scoring are the near-duplicate
+ // lookup; under an open route scope both add to that one stage.
+ const nearDuplicates = timeStageSync(ROUTE_STAGE.nearDuplicateLookup, () =>
+ collectNearDuplicateCandidates(vault, pages, opts.readable),
+ );
ctx = {
basenames: collectAllBasenames(vault),
vocabulary: loadSchemaPack(vault).vocabulary,
mergedLinks: createMergedLinkResolver(vault),
nearDuplicateCandidates: nearDuplicates.index,
- nearDuplicateCensus: nearDuplicates.census,
+ ...(opts.extraCandidates !== undefined ? { extraCandidates: opts.extraCandidates } : {}),
+ readable: opts.readable,
+ nearDuplicateCensus: {
+ ...nearDuplicates.census,
+ ...(opts.widening !== undefined ? { widening: opts.widening } : {}),
+ ...(opts.wideningDetail !== undefined ? { wideningDetail: opts.wideningDetail } : {}),
+ },
};
} catch (err) {
return emptyReport({
@@ -643,7 +820,7 @@ export function lintPagesWithContext(
continue;
}
try {
- detected.push(...lintOnePage(ctx, page, raw));
+ detected.push(...lintOnePage(vault, ctx, page, raw));
} catch (err) {
// One page's failure is one page's news. Reporting it as `unavailable`
// threw away the findings already collected for EARLIER pages and the
@@ -669,6 +846,8 @@ export function lintPagesWithContext(
...(census !== undefined && census.unreadable.length > 0
? { candidates_unreadable: Object.freeze([...census.unreadable]) }
: {}),
+ ...(census?.widening !== undefined ? { widening: census.widening } : {}),
+ ...(census?.wideningDetail !== undefined ? { widening_detail: census.wideningDetail } : {}),
});
}
@@ -684,7 +863,8 @@ function hasSomethingToSay(report: PageLintReport): boolean {
report.skipped.length > 0 ||
report.unavailable !== undefined ||
report.candidates_skipped !== undefined ||
- report.candidates_unreadable !== undefined
+ report.candidates_unreadable !== undefined ||
+ report.widening === "index_unavailable"
);
}
diff --git a/src/core/brain/retire-siblings.ts b/src/core/brain/retire-siblings.ts
new file mode 100644
index 00000000..920c3d36
--- /dev/null
+++ b/src/core/brain/retire-siblings.ts
@@ -0,0 +1,110 @@
+/**
+ * Retire-sibling planner (near-duplicate defense, t_acab97de).
+ *
+ * When a preference retires because context proved it wrong, an active
+ * paraphrase of it usually survives under another topic slug. This pure
+ * planner nominates those paraphrases as advisory pairs. It never
+ * retires anything: the operator accepts a pair through an existing verb
+ * (`brain_apply_evidence result: outdated` or `o2b brain reject`).
+ *
+ * Only context-driven retires nominate. Decay retires
+ * (`stale-no-evidence`, `expired-unconfirmed`) mean nobody used the rule,
+ * not that the rule is wrong, and `merged-into` keeps its sibling as the
+ * canonical page. Siblings are never cascaded: a sibling of a sibling is
+ * not computed.
+ */
+
+import { isMergeResolved, mergePointerLookup } from "./page-meta/page-id.ts";
+import {
+ compareMatches,
+ findNearDuplicates,
+ NEAR_DUPLICATE_THRESHOLDS,
+ type NearDuplicateMethod,
+ type NearDuplicatePoolEntry,
+ type ReadableRef,
+} from "./near-duplicate.ts";
+import { tokenise } from "./similarity.ts";
+import { BRAIN_RETIRED_REASON, type BrainRetiredReason } from "./types.ts";
+
+export const RETIRE_SIBLING_TRIGGER_REASONS: ReadonlySet<BrainRetiredReason> = new Set([
+ BRAIN_RETIRED_REASON.supersededByContext,
+ BRAIN_RETIRED_REASON.rebutted,
+ BRAIN_RETIRED_REASON.quarantineViolated,
+ BRAIN_RETIRED_REASON.userRejected,
+]);
+
+export interface RetireSibling {
+ /** `pref-*` id being retired. */
+ readonly retiring_id: string;
+ /** Active `pref-*` id that resembles it. */
+ readonly sibling_id: string;
+ readonly score: number;
+ readonly method: NearDuplicateMethod;
+}
+
+export interface RetireSiblingOptions {
+ readonly readable: ReadableRef;
+ /** Retiring ids the apply step held back; their siblings are dropped. */
+ readonly gated: ReadonlySet<string>;
+}
+
+/**
+ * Project active preferences into the sibling pool, leaving out pages a
+ * merge already resolved (the predicate every duplicate detector asks).
+ */
+export function retireSiblingPool(
+ prefs: ReadonlyArray<{
+ readonly id: string;
+ readonly principle: string;
+ readonly merged_into?: string;
+ }>,
+): ReadonlyArray<{ id: string; principle: string }> {
+ const pointerOf = mergePointerLookup(prefs.map((p) => [p.id, p.merged_into ?? null] as const));
+ return prefs
+ .filter((p) => !isMergeResolved(p.id, pointerOf))
+ .map((p) => ({ id: p.id, principle: p.principle }));
+}
+
+/**
+ * Nominate active siblings of every context-driven, ungated, readable
+ * retire. The pool crosses topic and scope buckets and excludes the whole
+ * retiring set. Output order is `(retiring_id, -score, sibling_id)`.
+ */
+export function planRetireSiblings(
+ active: ReadonlyArray<{ id: string; principle: string }>,
+ retiring: ReadonlyArray<{ id: string; principle: string; reason: BrainRetiredReason }>,
+ opts: RetireSiblingOptions,
+): ReadonlyArray<RetireSibling> {
+ const retiringIds = new Set(retiring.map((r) => r.id));
+ // Sorted by id so the output never depends on the order the caller
+ // scanned the vault in.
+ const pool: NearDuplicatePoolEntry[] = active
+ .filter((p) => !retiringIds.has(p.id))
+ .map((p) => ({ ref: p.id, tokens: tokenise(p.principle) }))
+ .toSorted((a, b) => (a.ref < b.ref ? -1 : a.ref > b.ref ? 1 : 0));
+ const out: RetireSibling[] = [];
+ for (const r of retiring) {
+ if (!RETIRE_SIBLING_TRIGGER_REASONS.has(r.reason)) continue;
+ if (opts.gated.has(r.id) || !opts.readable(r.id)) continue;
+ const { matches } = findNearDuplicates({ ref: r.id, tokens: tokenise(r.principle) }, pool, {
+ threshold: NEAR_DUPLICATE_THRESHOLDS.retireSiblingLexical,
+ readable: opts.readable,
+ // Lexical scoring over active preferences is cheap, and a capped
+ // scan would silently never score a sibling sorting past the cap.
+ cap: pool.length,
+ });
+ for (const m of matches) {
+ out.push({ retiring_id: r.id, sibling_id: m.ref, score: m.score, method: m.method });
+ }
+ }
+ return out.toSorted(compareRetireSiblings);
+}
+
+/** `(retiring_id, -score, sibling_id)`, by code unit so the order never depends on locale. */
+export function compareRetireSiblings(a: RetireSibling, b: RetireSibling): number {
+ if (a.retiring_id !== b.retiring_id) return a.retiring_id < b.retiring_id ? -1 : 1;
+ return compareMatches(
+ { ref: a.sibling_id, score: a.score },
+ { ref: b.sibling_id, score: b.score },
+ );
+}
diff --git a/src/core/brain/review-candidates.ts b/src/core/brain/review-candidates.ts
index a9bd0200..bee2a23f 100644
--- a/src/core/brain/review-candidates.ts
+++ b/src/core/brain/review-candidates.ts
@@ -16,15 +16,39 @@
import { existsSync, readdirSync } from "node:fs";
import { posix } from "node:path";
+import { resolveNearDuplicateRetireSiblingsEnabled } from "../config.ts";
import { vaultRelative } from "../path-safety.ts";
+import { SearchError } from "../search/search-error.ts";
import type { ResolvedSearchConfig } from "../search/types.ts";
-import { dream } from "./dream.ts";
-import type { DreamOptions } from "./dream-types.ts";
+import { dream, shouldGateRetireFromConfirmed } from "./dream.ts";
+import type { PreferenceRecord } from "./dream-plan.ts";
+import type { DreamOptions, DreamRunSummary } from "./dream-types.ts";
import type { BrainIntentReviewEntry } from "./intent-review.ts";
+import { NEAR_DUPLICATE_THRESHOLDS, READ_ALL_REFS, roundScore } from "./near-duplicate.ts";
+import {
+ storedVectorSimilarities,
+ type StoredVectorSimilarity,
+ type StoredVectorStatus,
+} from "./near-duplicate-vectors.ts";
+import { failureCode } from "./page-lint.ts";
import { brainDirs } from "./paths.ts";
+import { loadBrainConfig } from "./policy.ts";
+import {
+ compareRetireSiblings,
+ RETIRE_SIBLING_TRIGGER_REASONS,
+ retireSiblingPool,
+ type RetireSibling,
+} from "./retire-siblings.ts";
import { scoreSignalNovelty, sortByNovelty, type SignalNoveltyEntry } from "./surprisal.ts";
import { BRAIN_RETIRED_REASON, type BrainRetiredReason } from "./types.ts";
+/**
+ * Outcome of the retire-sibling stored-vector tier: a probe status, or
+ * `index_unavailable` when the store failed to open or read (busy,
+ * locked, schema). The preview degrades to the lexical pairs then.
+ */
+export type RetireSiblingSemanticStatus = StoredVectorStatus | "index_unavailable";
+
export interface ReviewCandidatesReport {
/** `pref-<slug>` ids that the dream pass would create new. */
readonly would_create: ReadonlyArray<string>;
@@ -73,6 +97,27 @@ export interface ReviewCandidatesReport {
* scored - vec-less vaults keep the report byte-identical.
*/
readonly signal_novelty?: ReadonlyArray<SignalNoveltyEntry>;
+ /**
+ * Near-duplicate defense (t_acab97de): active preferences resembling a
+ * context-driven retire this pass would make. Lexical pairs come from
+ * the dream summary; with a search config, stored-vector pairs at or
+ * above `retireSiblingEmbedding` merge in as `method: "embedding"`.
+ * Present only when `near_duplicate_retire_siblings_enabled` is on and
+ * the list is non-empty.
+ */
+ readonly retire_siblings?: ReadonlyArray<RetireSibling>;
+ /**
+ * Outcome of the stored-vector tier. Present only when retire siblings
+ * are computed with a search config and a context-driven retire passes
+ * `retiringVisible`.
+ */
+ readonly retire_siblings_semantic?: RetireSiblingSemanticStatus;
+ /**
+ * Why the tier reported `index_unavailable`: the search error code
+ * ({@link failureCode}), never a message or a path. Present only with
+ * that status.
+ */
+ readonly retire_siblings_semantic_detail?: string;
}
export interface BuildReviewCandidatesOptions {
@@ -84,6 +129,20 @@ export interface BuildReviewCandidatesOptions {
* so the clusters, counts and intent reviews fold no withheld record.
*/
readonly readable?: (rel: string) => boolean;
+ /**
+ * May the caller see this retiring `pref-*` id? A caller whose answer
+ * drops a retire row passes the same test here, so the stored-vector
+ * tier probes only the retires that answer keeps and
+ * `retire_siblings_semantic` folds no hidden probe. Omitted, every
+ * retire is probed.
+ */
+ readonly retiringVisible?: (prefId: string) => boolean;
+ /**
+ * Whether to project `retire_siblings`. Omitted, it resolves
+ * `near_duplicate_retire_siblings_enabled` from the default config; a
+ * caller with its own config path (the MCP server) resolves it there.
+ */
+ readonly retireSiblingsEnabled?: boolean;
/**
* When provided, annotate the report with surprisal novelty over
* the existing vec index (t_fddfe64a). Read-only; absent or
@@ -127,8 +186,14 @@ export async function buildReviewCandidates(
vault: string,
opts: BuildReviewCandidatesOptions = {},
): Promise<ReviewCandidatesReport> {
+ const siblingsEnabled = opts.retireSiblingsEnabled ?? resolveNearDuplicateRetireSiblingsEnabled();
+ let scannedPreferences: ReadonlyArray<PreferenceRecord> = [];
const summary = dream(vault, {
dryRun: true,
+ retireSiblingsEnabled: siblingsEnabled,
+ onScanPreferences: (preferences) => {
+ scannedPreferences = preferences;
+ },
...(opts.now ? { now: opts.now } : {}),
...(opts.safeguard !== undefined ? { safeguard: opts.safeguard } : {}),
...(opts.onProgress !== undefined ? { onProgress: opts.onProgress } : {}),
@@ -146,8 +211,17 @@ export async function buildReviewCandidates(
}
}
+ const siblings = siblingsEnabled
+ ? await projectRetireSiblings(vault, summary, scannedPreferences, opts)
+ : { siblings: [] };
+
return Object.freeze({
...(signalNovelty !== undefined ? { signal_novelty: signalNovelty } : {}),
+ ...(siblings.siblings.length > 0
+ ? { retire_siblings: Object.freeze(siblings.siblings.map((x) => Object.freeze({ ...x }))) }
+ : {}),
+ ...(siblings.semantic !== undefined ? { retire_siblings_semantic: siblings.semantic } : {}),
+ ...(siblings.detail !== undefined ? { retire_siblings_semantic_detail: siblings.detail } : {}),
would_create: Object.freeze([...summary.new_unconfirmed]),
would_promote: Object.freeze([...summary.confirmed]),
would_retire: Object.freeze(
@@ -186,3 +260,96 @@ export async function buildReviewCandidates(
),
} satisfies ReviewCandidatesReport);
}
+
+interface RetireSiblingProjection {
+ readonly siblings: ReadonlyArray<RetireSibling>;
+ readonly semantic?: RetireSiblingSemanticStatus;
+ readonly detail?: string;
+}
+
+/**
+ * The dream summary's lexical pairs, plus - when a search config is
+ * present - the stored-vector tier over the same pool: every readable
+ * active preference outside the retiring set that no merge resolved. A
+ * pair the lexical tier already found keeps its lexical entry. The pool
+ * comes from the dry run's own full scan (`preferences`), not a second walk.
+ * A store that fails to open or read is an advisory tier failing, not the
+ * preview: the lexical pairs stand and the status names the failure.
+ */
+async function projectRetireSiblings(
+ vault: string,
+ summary: DreamRunSummary,
+ preferences: ReadonlyArray<PreferenceRecord>,
+ opts: BuildReviewCandidatesOptions,
+): Promise<RetireSiblingProjection> {
+ const lexical = summary.retire_siblings ?? [];
+ const triggered = summary.retired.filter((r) => RETIRE_SIBLING_TRIGGER_REASONS.has(r.reason));
+ if (opts.searchConfig === undefined || triggered.length === 0) return { siblings: lexical };
+ const readable = opts.readable ?? READ_ALL_REFS;
+ const scanned = preferences
+ .filter((p) => readable(vaultRelative(p.path, vault)))
+ .map((p) => p.pref);
+ // The dry run gates nothing, so the retires the confirmed-evidence gate
+ // will hold back are dropped here, as the dream summary drops their
+ // lexical siblings.
+ const gateThreshold = loadBrainConfig(vault).retire.confirmed_evidence_min_threshold;
+ const byId = new Map(scanned.map((p) => [p.id, p] as const));
+ const retiringIds = triggered
+ .map((r) => ({ id: `pref-${r.id.replace(/^ret-/, "")}`, reason: r.reason }))
+ .filter(({ id, reason }) => {
+ const existing = byId.get(id);
+ return (
+ existing === undefined || !shouldGateRetireFromConfirmed(existing, reason, gateThreshold)
+ );
+ })
+ .map(({ id }) => id)
+ .filter((id) => opts.retiringVisible?.(id) ?? true);
+ if (retiringIds.length === 0) return { siblings: lexical };
+
+ const prefsRel = vaultRelative(brainDirs(vault).preferences, vault);
+ const pathOf = (id: string): string => posix.join(prefsRel, `${id}.md`);
+ const allRetiring = new Set(summary.retired.map((r) => `pref-${r.id.replace(/^ret-/, "")}`));
+ const pool = retireSiblingPool(scanned).filter((p) => !allRetiring.has(p.id));
+ const idByPath = new Map(pool.map((p) => [pathOf(p.id), p.id] as const));
+ const candidatePaths = [...idByPath.keys()].toSorted();
+
+ const seen = new Set(lexical.map((x) => `${x.retiring_id}\u0000${x.sibling_id}`));
+ const merged: RetireSibling[] = [...lexical];
+ let results: ReadonlyArray<StoredVectorSimilarity>;
+ try {
+ results = await storedVectorSimilarities(
+ opts.searchConfig,
+ retiringIds.map(pathOf),
+ candidatePaths,
+ );
+ } catch (e) {
+ if (!(e instanceof SearchError)) throw e;
+ return { siblings: lexical, semantic: "index_unavailable", detail: failureCode(e) };
+ }
+ results.forEach((result, i) => {
+ const retiringId = retiringIds[i]!;
+ for (const [path, score] of result.scores) {
+ const siblingId = idByPath.get(path);
+ if (siblingId === undefined) continue;
+ if (score < NEAR_DUPLICATE_THRESHOLDS.retireSiblingEmbedding) continue;
+ if (seen.has(`${retiringId}\u0000${siblingId}`)) continue;
+ merged.push({
+ retiring_id: retiringId,
+ sibling_id: siblingId,
+ score: roundScore(score),
+ method: "embedding",
+ });
+ }
+ });
+ const statuses = results.map((r) => r.status);
+ return { siblings: merged.toSorted(compareRetireSiblings), semantic: foldStatuses(statuses) };
+}
+
+/**
+ * One status for the whole tier: `used` when any probe was compared, else
+ * the first named reason it could not run (`not_embedded` when the probes
+ * have no stored content vector yet).
+ */
+function foldStatuses(statuses: ReadonlyArray<StoredVectorStatus>): StoredVectorStatus {
+ return statuses.includes("used") ? "used" : statuses[0]!;
+}
diff --git a/src/core/brain/signal.ts b/src/core/brain/signal.ts
index 6b97b809..e9230195 100644
--- a/src/core/brain/signal.ts
+++ b/src/core/brain/signal.ts
@@ -35,6 +35,7 @@ import {
rememberKey,
} from "./idempotency-ledger.ts";
import { sanitisePrinciple } from "./text/sanitize-principle.ts";
+import { ROUTE_STAGE, timeStageSync } from "../route-scope.ts";
import {
ORIGIN_CHANNEL_FIELD,
isOriginChannelStamp,
@@ -323,27 +324,9 @@ export function writeSignal(
// `topic`, `slug`, `agent`, `date`, `signal` are constrained by
// their own validators (slug rules / ISO date / enum) and are
// already rejected if they carry junk — don't double-process them.
- const sanitised = sanitiseSignalInput(resolvedInput);
-
- for (const field of REQUIRED_INPUT_FIELDS) {
- const value = sanitised[field];
- if (value === undefined || value === null || String(value).trim() === "") {
- throw new Error(`signal missing field: ${String(field)}`);
- }
- }
- if (
- sanitised.signal !== BRAIN_SIGNAL_SIGN.positive &&
- sanitised.signal !== BRAIN_SIGNAL_SIGN.negative
- ) {
- throw new Error(
- `signal field 'signal' must be 'positive' or 'negative'; got ${JSON.stringify(sanitised.signal)}`,
- );
- }
- if (sanitised.source_type !== undefined && !isBrainSignalSourceType(sanitised.source_type)) {
- throw new Error(
- `signal field 'source_type' must be one of ${renderSourceTypes()}; got ${JSON.stringify(sanitised.source_type)}`,
- );
- }
+ const sanitised = timeStageSync(ROUTE_STAGE.validate, () =>
+ validateSignalInput(sanitiseSignalInput(resolvedInput)),
+ );
// Idempotency consult (C1): when a client key is supplied, hash the
// semantic payload and check the ledger BEFORE allocating a slug or
@@ -354,7 +337,7 @@ export function writeSignal(
let contentHash: string | undefined;
if (idKey) {
contentHash = computePayloadHash(signalPayloadFields(sanitised));
- const existing = lookupKey(vault, idKey);
+ const existing = timeStageSync(ROUTE_STAGE.idempotencyLookup, () => lookupKey(vault, idKey));
if (existing) {
if (existing.contentHash === contentHash) {
const ref = (existing.ref ?? {}) as { id?: string; path?: string };
@@ -379,42 +362,47 @@ export function writeSignal(
// session import each write many signals per run on the same date and
// often the same topic stem - so losing the name to a concurrent
// writer has to cost this call the next candidate, not the event.
- const { allocation: allocated, value: id } = allocateAndCreate(
- {
- vault,
- targetDir: options.targetDir ?? dirs.inbox,
- prefix,
- slug: sanitised.slug,
- maxAttempts: options.maxSlugAttempts,
- },
- (allocation) => {
- // The on-disk id always equals the filename basename, including any
- // `-2`/`-3` collision suffix — which is why the document is rendered
- // per attempt rather than once: a retry that reused the first
- // attempt's bytes would ship a file whose `id` names a different
- // file. We surface it as the canonical id and duplicate it inside
- // the frontmatter so a manual `mv` keeps the link.
- const signalId = `${prefix}-${allocation.slug}`;
- const { metadata, body } = renderSignalDocument(sanitised, signalId);
- writeFrontmatterAtomic(allocation.path, metadata, body, {
- overwrite: false,
- existsErrorKind: "signal",
- vaultForRelativePath: vault,
- });
- return signalId;
- },
+ const { allocation: allocated, value: id } = timeStageSync(ROUTE_STAGE.documentWrite, () =>
+ allocateAndCreate(
+ {
+ vault,
+ targetDir: options.targetDir ?? dirs.inbox,
+ prefix,
+ slug: sanitised.slug,
+ maxAttempts: options.maxSlugAttempts,
+ },
+ (allocation) => {
+ // The on-disk id always equals the filename basename, including any
+ // `-2`/`-3` collision suffix — which is why the document is rendered
+ // per attempt rather than once: a retry that reused the first
+ // attempt's bytes would ship a file whose `id` names a different
+ // file. We surface it as the canonical id and duplicate it inside
+ // the frontmatter so a manual `mv` keeps the link.
+ const signalId = `${prefix}-${allocation.slug}`;
+ const { metadata, body } = renderSignalDocument(sanitised, signalId);
+ writeFrontmatterAtomic(allocation.path, metadata, body, {
+ overwrite: false,
+ existsErrorKind: "signal",
+ vaultForRelativePath: vault,
+ });
+ return signalId;
+ },
+ ),
);
// Record the key AFTER the file lands so a future retry dedupes. `ref`
// stores the vault-relative path + id so the dedupe branch above can
// return the original coordinates without re-deriving them.
if (idKey && contentHash !== undefined) {
- rememberKey(vault, {
- key: idKey,
- contentHash,
- createdAt: sanitised.created_at,
- ref: { id, path: relative(vault, allocated.path) },
- });
+ const hash = contentHash;
+ timeStageSync(ROUTE_STAGE.idempotencyRemember, () =>
+ rememberKey(vault, {
+ key: idKey,
+ contentHash: hash,
+ createdAt: sanitised.created_at,
+ ref: { id, path: relative(vault, allocated.path) },
+ }),
+ );
}
return { path: allocated.path, id };
@@ -545,6 +533,33 @@ function signalPayloadFields(input: WriteSignalInput): Record<string, unknown> {
// ----- Sanitisation --------------------------------------------------------
+/**
+ * Refuse a sanitised signal input that lacks a required field or carries
+ * an out-of-enum `signal` / `source_type`; returns the input unchanged.
+ */
+function validateSignalInput(sanitised: WriteSignalInput): WriteSignalInput {
+ for (const field of REQUIRED_INPUT_FIELDS) {
+ const value = sanitised[field];
+ if (value === undefined || value === null || String(value).trim() === "") {
+ throw new Error(`signal missing field: ${String(field)}`);
+ }
+ }
+ if (
+ sanitised.signal !== BRAIN_SIGNAL_SIGN.positive &&
+ sanitised.signal !== BRAIN_SIGNAL_SIGN.negative
+ ) {
+ throw new Error(
+ `signal field 'signal' must be 'positive' or 'negative'; got ${JSON.stringify(sanitised.signal)}`,
+ );
+ }
+ if (sanitised.source_type !== undefined && !isBrainSignalSourceType(sanitised.source_type)) {
+ throw new Error(
+ `signal field 'source_type' must be one of ${renderSourceTypes()}; got ${JSON.stringify(sanitised.source_type)}`,
+ );
+ }
+ return sanitised;
+}
+
/**
* Hard caps for free-form fields. `principle` is rendered as a
* single-line YAML scalar in frontmatter; `note`/`raw` shapes can
diff --git a/src/core/brain/write-advisory.ts b/src/core/brain/write-advisory.ts
index a1713a69..ab96f2ae 100644
--- a/src/core/brain/write-advisory.ts
+++ b/src/core/brain/write-advisory.ts
@@ -38,6 +38,7 @@ import { existsSync, readdirSync } from "node:fs";
import { join } from "node:path";
import { vaultRelative } from "../path-safety.ts";
+import { ROUTE_STAGE, timeStageSync } from "../route-scope.ts";
import { adviseOnIncoming, type PreferenceForContradiction } from "./health/contradiction.ts";
import { appendLogEvent } from "./log.ts";
@@ -165,8 +166,15 @@ export function adviseIncomingFeedback(
params: AdviseIncomingFeedbackParams,
): WriteConflictAdvisory | null {
try {
- const prefs = loadConfirmedScopePrefs(vault, params.scope, params.readable);
- const advisory = adviseOnIncoming(params.principle, params.scope, prefs);
+ // Loading the comparison pool and scoring it is the near-duplicate
+ // lookup of the feedback route; the log append below is its own stage.
+ const advisory = timeStageSync(ROUTE_STAGE.nearDuplicateLookup, () =>
+ adviseOnIncoming(
+ params.principle,
+ params.scope,
+ loadConfirmedScopePrefs(vault, params.scope, params.readable),
+ ),
+ );
if (advisory === null) return null;
const result: WriteConflictAdvisory = {
diff --git a/src/core/brain/write-batch.ts b/src/core/brain/write-batch.ts
index 97e2a1a4..61f75d2f 100644
--- a/src/core/brain/write-batch.ts
+++ b/src/core/brain/write-batch.ts
@@ -53,6 +53,7 @@ import { ORIGIN_CHANNEL_FIELD } from "../origin-channel.ts";
import { preferencePath, validateSlug } from "./paths.ts";
import { assertVaultIdentityForWrite } from "./vault-identity.ts";
import { BRAIN_APPLY_RESULT } from "./types.ts";
+import { ROUTE_STAGE, timeStageSync } from "../route-scope.ts";
import {
computePayloadHash,
IdempotencyPayloadMismatchError,
@@ -898,13 +899,15 @@ function commitNoteRewrite(
mkdirSync(dirname(target.abs), { recursive: true });
const wrote = atomicWriteFileSync(target.abs, contents, { skipIfUnchanged: true });
if (!wrote) return { wrote: false };
- const audit = recordNoteWrite(vault, {
- op,
- target: target.relPath,
- before: { bytes: before },
- after: { bytes: contents },
- ...(opts.configPath !== undefined ? { configPath: opts.configPath } : {}),
- });
+ const audit = timeStageSync(ROUTE_STAGE.writeReceipt, () =>
+ recordNoteWrite(vault, {
+ op,
+ target: target.relPath,
+ before: { bytes: before },
+ after: { bytes: contents },
+ ...(opts.configPath !== undefined ? { configPath: opts.configPath } : {}),
+ }),
+ );
return { wrote: true, ...audit };
}
diff --git a/src/core/config.ts b/src/core/config.ts
index 426ea95a..4baba118 100644
--- a/src/core/config.ts
+++ b/src/core/config.ts
@@ -786,6 +786,35 @@ export function resolveRecentTurnsResurface(configPath?: string): boolean {
);
}
+/**
+ * Retire-sibling projection gate (near-duplicate defense, t_acab97de).
+ * Default OFF: the dream summary, `brain_review_candidates` and
+ * `o2b brain reject` carry no `retire_siblings` unless
+ * `near_duplicate_retire_siblings_enabled: "true"`, so unchanged installs
+ * stay byte-identical.
+ */
+export function resolveNearDuplicateRetireSiblingsEnabled(configPath?: string): boolean {
+ return resolveConfigFlag(
+ "OPEN_SECOND_BRAIN_NEAR_DUPLICATE_RETIRE_SIBLINGS_ENABLED",
+ "near_duplicate_retire_siblings_enabled",
+ configPath,
+ );
+}
+
+/**
+ * Write-receipt near-duplicate widening gate (near-duplicate defense,
+ * t_fda66477). Default OFF: the receipt hint compares only pages in the
+ * written page's directory unless `near_duplicate_write_widening_enabled:
+ * "true"`, which adds keyword-index candidates from other directories.
+ */
+export function resolveNearDuplicateWriteWideningEnabled(configPath?: string): boolean {
+ return resolveConfigFlag(
+ "OPEN_SECOND_BRAIN_NEAR_DUPLICATE_WRITE_WIDENING_ENABLED",
+ "near_duplicate_write_widening_enabled",
+ configPath,
+ );
+}
+
/**
* SessionEnd handoff-note gate (Agent Surface Suite, t_28afa4d2).
* Default OFF: lifecycle capture writes no handoff note unless
diff --git a/src/core/decision-model/latency.ts b/src/core/decision-model/latency.ts
index 6c0d7888..543f5fd8 100644
--- a/src/core/decision-model/latency.ts
+++ b/src/core/decision-model/latency.ts
@@ -2,28 +2,14 @@
* Per-route decision latency, for the optional `decision_ms` component of
* `mcp_route_latency` records.
*
- * The MCP server opens a scope around a tool call only while route
- * metrics are enabled; `runDecision` adds each request's wall time to the
- * open scope, if any. With no scope open this is a no-op, so the feature
- * adds nothing to a call that is not being measured.
+ * The store lives in `src/core/route-scope.ts`, which also carries the
+ * write-stage timings; this module keeps the decision-only surface that
+ * `runDecision` and its tests use.
*/
-import { AsyncLocalStorage } from "node:async_hooks";
+import { createRouteScope } from "../route-scope.ts";
-interface LatencyScope {
- totalMs: number;
- calls: number;
-}
-
-const storage = new AsyncLocalStorage<LatencyScope>();
-
-/** Add one decision request's duration to the open scope, if any. */
-export function noteDecisionLatency(ms: number): void {
- const scope = storage.getStore();
- if (scope === undefined || !Number.isFinite(ms)) return;
- scope.totalMs += Math.max(0, ms);
- scope.calls += 1;
-}
+export { noteDecisionLatency } from "../route-scope.ts";
export interface DecisionLatencyScope {
/** Run `fn` with this scope open. */
@@ -33,9 +19,6 @@ export interface DecisionLatencyScope {
}
export function createDecisionLatencyScope(): DecisionLatencyScope {
- const scope: LatencyScope = { totalMs: 0, calls: 0 };
- return {
- run: (fn) => storage.run(scope, fn),
- decisionMs: () => (scope.calls > 0 ? Math.round(scope.totalMs) : undefined),
- };
+ const scope = createRouteScope();
+ return { run: (fn) => scope.run(fn), decisionMs: () => scope.decisionMs() };
}
diff --git a/src/core/route-scope.ts b/src/core/route-scope.ts
new file mode 100644
index 00000000..e688fd23
--- /dev/null
+++ b/src/core/route-scope.ts
@@ -0,0 +1,106 @@
+/**
+ * One per-call route scope for the optional `decision_ms` and `stages`
+ * components of `mcp_route_latency` records.
+ *
+ * The MCP server opens a scope around a tool call only while route metrics
+ * are enabled. `runDecision` adds each request's wall time to the open
+ * scope, and the write tools wrap their stages in `timeStageSync` or
+ * `timeStage`. With no scope open every helper is one `getStore()` that
+ * returns `undefined`, so the feature adds nothing to a call that is not
+ * being measured. Stage names are a closed allowlist: free text never
+ * reaches a continuity record.
+ */
+
+import { AsyncLocalStorage } from "node:async_hooks";
+import { performance } from "node:perf_hooks";
+
+export const ROUTE_STAGE = Object.freeze({
+ validate: "validate",
+ idempotencyLookup: "idempotency_lookup",
+ nearDuplicateLookup: "near_duplicate_lookup",
+ documentWrite: "document_write",
+ idempotencyRemember: "idempotency_remember",
+ logAppend: "log_append",
+ preferenceWrite: "preference_write",
+ writeReceipt: "write_receipt",
+ lint: "lint",
+} as const);
+
+export type RouteStageName = (typeof ROUTE_STAGE)[keyof typeof ROUTE_STAGE];
+
+export const ROUTE_STAGE_NAMES: ReadonlySet<RouteStageName> = new Set(Object.values(ROUTE_STAGE));
+
+export function isRouteStageName(value: unknown): value is RouteStageName {
+ return typeof value === "string" && ROUTE_STAGE_NAMES.has(value as RouteStageName);
+}
+
+export interface RouteStageTiming {
+ readonly name: RouteStageName;
+ readonly ms: number;
+}
+
+interface RouteScopeState {
+ decisionTotalMs: number;
+ decisionCalls: number;
+ readonly stageMs: Map<RouteStageName, number>;
+}
+
+const storage = new AsyncLocalStorage<RouteScopeState>();
+
+function noteStage(scope: RouteScopeState, name: RouteStageName, startedAt: number): void {
+ const elapsed = Math.max(0, performance.now() - startedAt);
+ scope.stageMs.set(name, (scope.stageMs.get(name) ?? 0) + elapsed);
+}
+
+/** Time a synchronous stage in the open scope; rethrows whatever `fn` throws. */
+export function timeStageSync<T>(name: RouteStageName, fn: () => T): T {
+ const scope = storage.getStore();
+ if (scope === undefined) return fn();
+ const startedAt = performance.now();
+ try {
+ return fn();
+ } finally {
+ noteStage(scope, name, startedAt);
+ }
+}
+
+/** Time an asynchronous stage in the open scope; rethrows whatever `fn` rejects with. */
+export async function timeStage<T>(name: RouteStageName, fn: () => Promise<T>): Promise<T> {
+ const scope = storage.getStore();
+ if (scope === undefined) return fn();
+ const startedAt = performance.now();
+ try {
+ return await fn();
+ } finally {
+ noteStage(scope, name, startedAt);
+ }
+}
+
+/** Add one decision request's duration to the open scope, if any. */
+export function noteDecisionLatency(ms: number): void {
+ const scope = storage.getStore();
+ if (scope === undefined || !Number.isFinite(ms)) return;
+ scope.decisionTotalMs += Math.max(0, ms);
+ scope.decisionCalls += 1;
+}
+
+export interface RouteScope {
+ /** Run `fn` with this scope open. */
+ run<T>(fn: () => Promise<T>): Promise<T>;
+ /** Total decision time in whole ms, or undefined when no decision request ran. */
+ decisionMs(): number | undefined;
+ /** Stage times summed per name in first-seen order at 0.1 ms, or undefined when none ran. */
+ stages(): ReadonlyArray<RouteStageTiming> | undefined;
+}
+
+export function createRouteScope(): RouteScope {
+ const scope: RouteScopeState = { decisionTotalMs: 0, decisionCalls: 0, stageMs: new Map() };
+ return {
+ run: (fn) => storage.run(scope, fn),
+ decisionMs: () => (scope.decisionCalls > 0 ? Math.round(scope.decisionTotalMs) : undefined),
+ stages: () =>
+ scope.stageMs.size === 0
+ ? undefined
+ : [...scope.stageMs].map(([name, ms]) => ({ name, ms: Math.round(ms * 10) / 10 })),
+ };
+}
diff --git a/src/core/search/visibility-surface-registry.ts b/src/core/search/visibility-surface-registry.ts
index 1ab11675..859b611c 100644
--- a/src/core/search/visibility-surface-registry.ts
+++ b/src/core/search/visibility-surface-registry.ts
@@ -816,7 +816,9 @@ export const VISIBILITY_SURFACE_REGISTRY: ReadonlyArray<VisibilitySurfaceEntry>
"withheld signal. would_create, would_promote, would_retire, would_supersede and " +
"gated_retires keep a row only when both the pref- and the ret- spelling of its id pass " +
"the gated owner view ANDed with reachView, and signal_novelty asks the same of each " +
- "signal's path and id.",
+ "signal's path and id. retire_siblings pairs two preference ids, and a pair is kept and " +
+ "counted only when both ids pass the same view, so a withheld retiring or sibling " +
+ "preference moves neither a row nor a count.",
},
{
surface: "brain_retention",
@@ -899,6 +901,70 @@ export const VISIBILITY_SURFACE_REGISTRY: ReadonlyArray<VisibilitySurfaceEntry>
"destination still refuses as occupied, the inherent create-collision residual, and an " +
"applied move still rewrites links inside withheld pages without naming them.",
},
+ {
+ surface: "brain_create_note",
+ kind: K.mcpTool,
+ category: C.covered,
+ reason:
+ "the write receipt carries the page lint (core/brain/page-lint.ts lintWrittenPages, through " +
+ "notes-tools.ts noteWriteResult), whose near-duplicate hint names sibling pages by path and " +
+ "counts the siblings it skipped or could not read; noteWriteResult binds the lint to " +
+ "readableAtContextReach(ctx), so a sibling the caller may not read is dropped at the " +
+ "directory listing, before it is scored and before candidates_skipped or " +
+ "candidates_unreadable count it; the wikilink and merged-link checks ask the same " +
+ "predicate, so a link to a withheld Brain page reads as broken and a merge chain ends at " +
+ "its first withheld hop, and the receipt answers as if the page were absent. " +
+ "A create onto an occupied withheld path still refuses as occupied, the inherent " +
+ "create-collision residual.",
+ },
+ {
+ surface: "brain_update_note",
+ kind: K.mcpTool,
+ category: C.covered,
+ reason:
+ "the write receipt carries the page lint (core/brain/page-lint.ts lintWrittenPages, through " +
+ "notes-tools.ts noteWriteResult), whose near-duplicate hint names sibling pages by path and " +
+ "counts the siblings it skipped or could not read; noteWriteResult binds the lint to " +
+ "readableAtContextReach(ctx), so a sibling the caller may not read is dropped at the " +
+ "directory listing, before it is scored and before candidates_skipped or " +
+ "candidates_unreadable count it; the wikilink and merged-link checks ask the same " +
+ "predicate, so a link to a withheld Brain page reads as broken and a merge chain ends at " +
+ "its first withheld hop, and the receipt answers as if the page were absent. " +
+ "The target itself resolves through the same predicate and a withheld one is refused " +
+ "with the error a missing note gets.",
+ },
+ {
+ surface: "brain_append_note",
+ kind: K.mcpTool,
+ category: C.covered,
+ reason:
+ "the write receipt carries the page lint (core/brain/page-lint.ts lintWrittenPages, through " +
+ "notes-tools.ts noteWriteResult), whose near-duplicate hint names sibling pages by path and " +
+ "counts the siblings it skipped or could not read; noteWriteResult binds the lint to " +
+ "readableAtContextReach(ctx), so a sibling the caller may not read is dropped at the " +
+ "directory listing, before it is scored and before candidates_skipped or " +
+ "candidates_unreadable count it; the wikilink and merged-link checks ask the same " +
+ "predicate, so a link to a withheld Brain page reads as broken and a merge chain ends at " +
+ "its first withheld hop, and the receipt answers as if the page were absent. " +
+ "The target itself resolves through the same predicate and a withheld one is refused " +
+ "with the error a missing note gets.",
+ },
+ {
+ surface: "brain_write_batch",
+ kind: K.mcpTool,
+ category: C.covered,
+ reason:
+ "the write receipt carries the page lint (core/brain/page-lint.ts lintWrittenPages, through " +
+ "notes-tools.ts noteWriteResult), whose near-duplicate hint names sibling pages by path and " +
+ "counts the siblings it skipped or could not read; noteWriteResult binds the lint to " +
+ "readableAtContextReach(ctx), so a sibling the caller may not read is dropped at the " +
+ "directory listing, before it is scored and before candidates_skipped or " +
+ "candidates_unreadable count it; the wikilink and merged-link checks ask the same " +
+ "predicate, so a link to a withheld Brain page reads as broken and a merge chain ends at " +
+ "its first withheld hop, and the receipt answers as if the page were absent. " +
+ "Every page the batch committed is linted under the one predicate, and the update and " +
+ "append operations refuse a withheld target with the error a missing note gets.",
+ },
{
surface: "brain_expire",
kind: K.mcpTool,
diff --git a/src/mcp/brain/extract-tools.ts b/src/mcp/brain/extract-tools.ts
index 78e1f08f..5d2a537d 100644
--- a/src/mcp/brain/extract-tools.ts
+++ b/src/mcp/brain/extract-tools.ts
@@ -58,7 +58,11 @@ async function toolBrainExtractSignals(
generated_at: plan.generatedAt,
boundary_decision: plan.boundaryDecision,
turns_scanned: plan.turnsScanned,
- turns_mined: plan.turnsMined.map((t) => ({ turn_id: t.turnId, text: t.text })),
+ turns_mined: plan.turnsMined.map((t) => ({
+ turn_id: t.turnId,
+ text: t.text,
+ timestamp: t.timestamp,
+ })),
cap: plan.cap,
confidence_floor: plan.confidenceFloor,
llm_step: plan.llmStep,
@@ -110,7 +114,7 @@ export const EXTRACT_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([
{
name: TOOL,
description:
- "Mine taste signals from an imported session's USER turns; runs no model. Without `items` returns the turns plus one needs-llm-step envelope. With `items` writes them to Brain/inbox/ as speculative `source_type: auto_extract` signals. Over the cap or under the floor refuses the payload.",
+ "Mine taste signals from an imported session's USER turns; runs no model. Without `items` returns the turns plus one needs-llm-step envelope. With `items` writes them to Brain/inbox/ as speculative `source_type: auto_extract` signals. Refuses over-cap, under-floor or repeated-topic payloads.",
inputSchema: {
type: "object",
properties: {
@@ -124,7 +128,10 @@ export const EXTRACT_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([
items: {
type: "object",
properties: {
- topic: { type: "string", description: "Stable kebab-slug naming the rule." },
+ topic: {
+ type: "string",
+ description: "Stable kebab-slug naming the rule, unique within the payload.",
+ },
signal: {
type: "string",
enum: ["positive", "negative"],
diff --git a/src/mcp/brain/feedback-tools.ts b/src/mcp/brain/feedback-tools.ts
index efafac92..cefdd690 100644
--- a/src/mcp/brain/feedback-tools.ts
+++ b/src/mcp/brain/feedback-tools.ts
@@ -7,7 +7,7 @@
*/
import { resolve } from "node:path";
-import { resolveAgentName } from "../../core/config.ts";
+import { resolveAgentName, resolveNearDuplicateRetireSiblingsEnabled } from "../../core/config.ts";
import {
appendApplyEvidence,
BrainPreferenceNotFoundError,
@@ -37,6 +37,7 @@ import {
} from "../../core/brain/dream-stage.ts";
import { BRAIN_ROLES } from "../../core/brain/trust/role.ts";
import { resolveEffectiveScope, writeSignal } from "../../core/brain/signal.ts";
+import { ROUTE_STAGE, timeStageSync } from "../../core/route-scope.ts";
import {
adviseIncomingFeedback,
adviseUnroutableCapture,
@@ -221,16 +222,18 @@ async function toolBrainFeedback(
: mirrorSignal(sharedNamespace, ctx.vault, signalInput, writeOpts);
try {
- appendLogEvent(ctx.vault, {
- timestamp: createdAt,
- eventType: BRAIN_LOG_EVENT_KIND.feedback,
- body: {
- signal: `[[${sigResult.id}]]`,
- topic: topic.trim(),
- sign: signalRaw,
- agent,
- },
- });
+ timeStageSync(ROUTE_STAGE.logAppend, () =>
+ appendLogEvent(ctx.vault, {
+ timestamp: createdAt,
+ eventType: BRAIN_LOG_EVENT_KIND.feedback,
+ body: {
+ signal: `[[${sigResult.id}]]`,
+ topic: topic.trim(),
+ sign: signalRaw,
+ agent,
+ },
+ }),
+ );
} catch (err) {
process.stderr.write(`warning: append feedback log failed: ${(err as Error).message}\n`);
}
@@ -267,50 +270,55 @@ async function toolBrainFeedback(
// `confirmed_at` is now; `unconfirmed_until` is also now so the trial
// window collapses on inspection. The just-written signal is recorded
// as the rule's origin under `evidenced_by`.
- prefResult = writePreference(
- ctx.vault,
- {
- slug,
- topic: topic.trim(),
- principle: principle.trim(),
- created_at: createdAt,
- unconfirmed_until: createdAt,
- status: BRAIN_PREFERENCE_STATUS.confirmed,
- evidenced_by: [`[[${sigResult.id}]]`],
- confirmed_at: createdAt,
- // Issue #149: an explicit zero, not an absent value. The
- // on-disk encoding of "absent" is the literal `null`, and two
- // ranking surfaces (`pre-compress-pack`, `morning-brief`) map
- // `null` to negative infinity before sorting - so a rule
- // force-confirmed a second ago would sort BELOW every rule
- // that has a number, including one measured at zero. Zero is
- // also the true Wilson lower bound on no evidence, which is
- // why the dream pass already pre-seeds it for new
- // preferences. This writer now matches it.
- confidence_value: 0,
- ...(effectiveScope !== undefined ? { scope: effectiveScope } : {}),
- // The same lifetime the signal carries: a rule confirmed from an
- // observation that expires on a date does not outlive it.
- ...(expires !== undefined ? { expiration_date: expires } : {}),
- },
- // Ownership is resolved by the writer, never echoed from `agent`:
- // that argument is caller-supplied, and a caller must not be able to
- // name whose memory this becomes. The server's config path is handed
- // over rather than a resolved name for the same reason.
- ctx.configPath !== null ? { configPath: ctx.configPath } : {},
+ prefResult = timeStageSync(ROUTE_STAGE.preferenceWrite, () =>
+ writePreference(
+ ctx.vault,
+ {
+ slug,
+ topic: topic.trim(),
+ principle: principle.trim(),
+ created_at: createdAt,
+ unconfirmed_until: createdAt,
+ status: BRAIN_PREFERENCE_STATUS.confirmed,
+ evidenced_by: [`[[${sigResult.id}]]`],
+ confirmed_at: createdAt,
+ // Issue #149: an explicit zero, not an absent value. The
+ // on-disk encoding of "absent" is the literal `null`, and two
+ // ranking surfaces (`pre-compress-pack`, `morning-brief`) map
+ // `null` to negative infinity before sorting - so a rule
+ // force-confirmed a second ago would sort BELOW every rule
+ // that has a number, including one measured at zero. Zero is
+ // also the true Wilson lower bound on no evidence, which is
+ // why the dream pass already pre-seeds it for new
+ // preferences. This writer now matches it.
+ confidence_value: 0,
+ ...(effectiveScope !== undefined ? { scope: effectiveScope } : {}),
+ // The same lifetime the signal carries: a rule confirmed from an
+ // observation that expires on a date does not outlive it.
+ ...(expires !== undefined ? { expiration_date: expires } : {}),
+ },
+ // Ownership is resolved by the writer, never echoed from `agent`:
+ // that argument is caller-supplied, and a caller must not be able to
+ // name whose memory this becomes. The server's config path is handed
+ // over rather than a resolved name for the same reason.
+ ctx.configPath !== null ? { configPath: ctx.configPath } : {},
+ ),
);
try {
// Offset by 1s so the force-confirmed event sorts after the feedback
// event on the same UTC second (parseLogDay is stable on ties, but a
// visible chronology reads cleaner).
- appendLogEvent(ctx.vault, {
- timestamp: isoSecond(new Date(now.getTime() + 1000)),
- eventType: BRAIN_LOG_EVENT_KIND.forceConfirmed,
- body: {
- preference: `[[${prefResult.id}]]`,
- agent,
- },
- });
+ const preferenceId = prefResult.id;
+ timeStageSync(ROUTE_STAGE.logAppend, () =>
+ appendLogEvent(ctx.vault, {
+ timestamp: isoSecond(new Date(now.getTime() + 1000)),
+ eventType: BRAIN_LOG_EVENT_KIND.forceConfirmed,
+ body: {
+ preference: `[[${preferenceId}]]`,
+ agent,
+ },
+ }),
+ );
} catch (err) {
process.stderr.write(
`warning: append force-confirmed log failed: ${(err as Error).message}\n`,
@@ -653,6 +661,11 @@ async function toolBrainDream(
// spends the run's budget - which is the honest reading of a budget the
// operator set for this operation.
const safeguard = toolSafeguard(ctx, OPERATION.dream);
+ // The retire-siblings key honours the server's own config file, as
+ // `brain_review_candidates` does.
+ const retireSiblingsEnabled = resolveNearDuplicateRetireSiblingsEnabled(
+ ctx.configPath ?? undefined,
+ );
// Only pay for a preview pass when a guard is actually requested; otherwise
// the run is byte-identical to before.
if (expect !== null || strict) {
@@ -660,6 +673,7 @@ async function toolBrainDream(
dryRun: true,
safeguard,
readable,
+ retireSiblingsEnabled,
...previewScope,
...(nowDate ? { now: nowDate } : {}),
...(agent ? { agentName: agent } : {}),
@@ -685,6 +699,7 @@ async function toolBrainDream(
dryRun,
safeguard,
readable,
+ retireSiblingsEnabled,
...previewScope,
...(nowDate ? { now: nowDate } : {}),
...(agent ? { agentName: agent } : {}),
@@ -692,6 +707,17 @@ async function toolBrainDream(
...(onProgress ? { onProgress } : {}),
});
const changeList = dreamChangeList(summary);
+ // A retire-sibling pair names two preferences; it is kept only when the
+ // caller may see both, each in both spellings (`ret-<slug>` after the
+ // pass, `pref-<slug>` before it).
+ const bothSpellings = (id: string): ReadonlyArray<string> => {
+ const slug = brainArtifactSlug(id);
+ return [`pref-${slug}`, `ret-${slug}`];
+ };
+ const retireSiblings = dreamView.keep(summary.retire_siblings ?? [], (p) => [
+ ...bothSpellings(p.retiring_id),
+ ...bothSpellings(p.sibling_id),
+ ]);
// The summary is already a Plain Old Frozen Object — JSON-serialise
// verbatim. We surface `snapshot_path` / `log_path` as vault-relative
@@ -705,6 +731,16 @@ async function toolBrainDream(
changed_count: dryRun ? 0 : changeList.length,
dry_run: dryRun,
...scopedDreamRows(dreamView, summary),
+ ...(retireSiblings.length > 0
+ ? {
+ retire_siblings: retireSiblings.map((p) => ({
+ retiring_id: p.retiring_id,
+ sibling_id: p.sibling_id,
+ score: p.score,
+ method: p.method,
+ })),
+ }
+ : {}),
// Inbox archive (issue #195): a count, not the ids. A first pass over a
// long-lived vault archives thousands of signals; the run's log event
// names every one of them.
diff --git a/src/mcp/brain/notes-tools.ts b/src/mcp/brain/notes-tools.ts
index 207f1454..9348968e 100644
--- a/src/mcp/brain/notes-tools.ts
+++ b/src/mcp/brain/notes-tools.ts
@@ -40,8 +40,23 @@ import {
type NoteWriteAudit,
type WriteBatchOpResult,
} from "../../core/brain/write-batch.ts";
+import type { ReadableRef } from "../../core/brain/near-duplicate.ts";
import { nextCommandField } from "../../core/brain/next-step.ts";
-import { lintWrittenPages, pageLintField, type PageLintField } from "../../core/brain/page-lint.ts";
+import {
+ lintWrittenPages,
+ pageLintField,
+ type LintWrittenPagesOptions,
+ type PageLintField,
+} from "../../core/brain/page-lint.ts";
+import {
+ collectWideningCandidates,
+ wideningUnavailable,
+ type WideningResult,
+} from "../../core/brain/page-lint-widening.ts";
+import { resolveNearDuplicateWriteWideningEnabled } from "../../core/config.ts";
+import { ROUTE_STAGE, timeStage, timeStageSync } from "../../core/route-scope.ts";
+import { resolveSearchConfig } from "../../core/search/index.ts";
+import type { ResolvedSearchConfig } from "../../core/search/types.ts";
import {
WRITE_PATH_ADVISORY_KEY,
type WritePathAdvisoryField,
@@ -272,7 +287,7 @@ async function toolBrainCreateNote(
// has always returned and stays in lockstep with it, so a skip can
// never be read as a create by either field. A skip authored no
// bytes, so it names no page for the lint.
- return noteWriteResult(ctx, res.created ? [res.path] : [], {
+ return await noteWriteResult(ctx, res.created ? [res.path] : [], {
created: res.created,
outcome: res.outcome,
path: res.path,
@@ -477,14 +492,66 @@ function runSingleWrite<K extends SingleNoteOperation["kind"]>(
* to say - so a receipt for a clean write is byte-identical to the one
* that shipped before. `pages` empty (a skipped create, a log-only batch)
* means no bytes were authored, so there is nothing to lint.
+ *
+ * The lint answers at the caller's reach: a near-duplicate sibling the
+ * caller may not read is never named, scored or counted on the receipt, a
+ * wikilink to a withheld Brain page reads as broken, and a merge chain
+ * ends at its first withheld hop, as they would if the page were absent.
+ *
+ * With `near_duplicate_write_widening_enabled` on, the near-duplicate hint
+ * also compares the pages with keyword-index candidates from other
+ * directories ({@link collectWideningCandidates}), and the report states
+ * whether that ran. With the key off nothing is opened and the receipt is
+ * exactly the one that shipped before.
+ *
+ * Under an open route scope the lint is timed as the `lint` stage and the
+ * widening as `near_duplicate_lookup`, the stage the lint's own
+ * near-duplicate scoring also adds to.
*/
-export function noteWriteResult<T extends Record<string, unknown>>(
+export async function noteWriteResult<T extends Record<string, unknown>>(
ctx: ServerContext,
pages: ReadonlyArray<string>,
receipt: T,
-): T & PageLintField {
+): Promise<T & PageLintField> {
if (pages.length === 0) return receipt;
- return { ...receipt, ...pageLintField(lintWrittenPages(ctx.vault, pages)) };
+ const readable = readableAtContextReach(ctx);
+ const widening = await timeStage(ROUTE_STAGE.nearDuplicateLookup, () =>
+ widenNearDuplicates(ctx, pages, readable),
+ );
+ const opts: LintWrittenPagesOptions =
+ widening === undefined
+ ? { readable }
+ : {
+ readable,
+ extraCandidates: widening.candidates,
+ widening: widening.status,
+ ...(widening.detail !== undefined ? { wideningDetail: widening.detail } : {}),
+ };
+ const report = timeStageSync(ROUTE_STAGE.lint, () => lintWrittenPages(ctx.vault, pages, opts));
+ return { ...receipt, ...pageLintField(report) };
+}
+
+/**
+ * Keyword-index candidates for the near-duplicate hint, or `undefined`
+ * when widening is off. A search config that does not resolve leaves no
+ * index to read, which is the same named `index_unavailable` an index
+ * that will not open reports, with the failure named in its detail: the
+ * write is never failed by its hint.
+ */
+async function widenNearDuplicates(
+ ctx: ServerContext,
+ pages: ReadonlyArray<string>,
+ readable: ReadableRef,
+): Promise<WideningResult | undefined> {
+ const configPath = ctx.configPath ?? undefined;
+ if (!resolveNearDuplicateWriteWideningEnabled(configPath)) return undefined;
+ let config: ResolvedSearchConfig;
+ try {
+ config = resolveSearchConfig({ vault: ctx.vault, configPath });
+ } catch (err) {
+ return wideningUnavailable(err);
+ }
+ return collectWideningCandidates(config, ctx.vault, pages, readable);
}
export const NOTES_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([
diff --git a/src/mcp/brain/recall-tools.ts b/src/mcp/brain/recall-tools.ts
index 6147e491..23df6e6d 100644
--- a/src/mcp/brain/recall-tools.ts
+++ b/src/mcp/brain/recall-tools.ts
@@ -1204,7 +1204,7 @@ export const RECALL_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([
name: "brain_route_metrics",
previewBudget: MCP_PREVIEW_BUDGET,
description:
- "Route-level MCP latency: list mcp_route_latency records or summarize per-tool latency (count, errors, min/avg/max, p50/p95/p99) slowest-first to find slow surfaces by endpoint. Emitted only when mcp_route_metrics_enabled is on; payload-safe (tool, scope, status, duration, arg keys). Read-only.",
+ "Route-level MCP latency: list mcp_route_latency records or summarize per-tool latency (count, errors, min/avg/max, p50/p95/p99, per-stage count/avg/p95) slowest-first. Emitted only when mcp_route_metrics_enabled is on; payload-safe (tool, scope, status, duration, arg keys, stage names). Read-only.",
inputSchema: {
type: "object",
properties: {
diff --git a/src/mcp/brain/review-tools.ts b/src/mcp/brain/review-tools.ts
index b4ea289e..fb8733af 100644
--- a/src/mcp/brain/review-tools.ts
+++ b/src/mcp/brain/review-tools.ts
@@ -6,6 +6,7 @@
* BRAIN_TOOLS surface.
*/
+import { resolveNearDuplicateRetireSiblingsEnabled } from "../../core/config.ts";
import { resolveSearchConfig } from "../../core/search/index.ts";
import { buildTimelineIndex } from "../../core/brain/temporal/build-index.ts";
import { findStaleEntries } from "../../core/brain/temporal/stale-watch.ts";
@@ -115,8 +116,22 @@ async function toolBrainReviewCandidates(
// over inbox signal clusters keyed by topic - count no withheld signal,
// and no withheld preference routes a cluster.
const readable = readableAtContextReachOrUndefined(ctx);
+ // The projected rows name preferences that EXIST by the id they would
+ // get after the pass, so `ret-<slug>` is resolved back through the
+ // shared slug fold and both spellings are asked about
+ // (a-label-is-not-a-boundary, U3).
+ const view = reviewView(ctx);
+ const bothSpellings = (id: string): ReadonlyArray<string> => {
+ const slug = brainArtifactSlug(id);
+ return [`pref-${slug}`, `ret-${slug}`];
+ };
const report = await buildReviewCandidates(ctx.vault, {
...(readable !== undefined ? { readable } : {}),
+ // The stored-vector tier probes only the retires `would_retire` keeps,
+ // so `retire_siblings_semantic` reflects no hidden retire.
+ retiringVisible: (id) => view.row(...bothSpellings(id)),
+ // The gate honours the server's own config file, as the write side does.
+ retireSiblingsEnabled: resolveNearDuplicateRetireSiblingsEnabled(ctx.configPath ?? undefined),
// The projection is read-only, but it runs a full dry-run
// consolidation pass to produce it - the same pass, and so the same
// budget, as `brain_dream`.
@@ -125,20 +140,37 @@ async function toolBrainReviewCandidates(
...(nowDate ? { now: nowDate } : {}),
...(searchConfig !== undefined ? { searchConfig } : {}),
});
- // The projected rows name preferences that EXIST by the id they would
- // get after the pass, so `ret-<slug>` is resolved back through the
- // shared slug fold and both spellings are asked about
- // (a-label-is-not-a-boundary, U3).
- const view = reviewView(ctx);
- const bothSpellings = (id: string): ReadonlyArray<string> => {
- const slug = brainArtifactSlug(id);
- return [`pref-${slug}`, `ret-${slug}`];
- };
+ // A retire-sibling pair names two preferences; it is kept only when the
+ // caller may see both, in both spellings, and the list is filtered
+ // before anything is counted, so a withheld page moves no number.
+ const retireSiblings =
+ report.retire_siblings !== undefined
+ ? view.keep(report.retire_siblings, (p) => [
+ ...bothSpellings(p.retiring_id),
+ ...bothSpellings(p.sibling_id),
+ ])
+ : [];
return {
// Signal rows name an inbox signal by id AND by vault-relative path.
...(report.signal_novelty !== undefined
? { signal_novelty: view.keep(report.signal_novelty, (s) => [s.path, s.id]) }
: {}),
+ ...(retireSiblings.length > 0
+ ? {
+ retire_siblings: retireSiblings.map((p) => ({
+ retiring_id: p.retiring_id,
+ sibling_id: p.sibling_id,
+ score: p.score,
+ method: p.method,
+ })),
+ }
+ : {}),
+ ...(report.retire_siblings_semantic !== undefined
+ ? { retire_siblings_semantic: report.retire_siblings_semantic }
+ : {}),
+ ...(report.retire_siblings_semantic_detail !== undefined
+ ? { retire_siblings_semantic_detail: report.retire_siblings_semantic_detail }
+ : {}),
// `would_create` names ids the pass has not written yet, so most of
// them resolve to nothing and pass; asking anyway is what keeps a
// projection over an id that DOES already exist from crossing.
@@ -253,7 +285,7 @@ export const REVIEW_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([
{
name: "brain_review_candidates",
description:
- "Read-only preview of the next `brain_dream` pass: would_create / would_promote / would_retire / would_supersede, clusters below threshold, gated retires, and intent reviews. Mutates nothing.",
+ "Read-only preview of the next `brain_dream` pass: would_create / would_promote / would_retire / would_supersede, clusters below threshold, gated retires, intent reviews, and opt-in retire_siblings (lexical or stored-vector pairs; tier status in retire_siblings_semantic). Mutates nothing.",
inputSchema: {
type: "object",
properties: {
diff --git a/src/mcp/server.ts b/src/mcp/server.ts
index 6a59a90a..5a9b748b 100644
--- a/src/mcp/server.ts
+++ b/src/mcp/server.ts
@@ -10,7 +10,7 @@ import {
resolveMcpRouteMetricsEnabled,
} from "../core/config.ts";
import { emitMcpRouteLatency, type McpRouteStatus } from "../core/brain/mcp-route-metrics.ts";
-import { createDecisionLatencyScope } from "../core/decision-model/latency.ts";
+import { createRouteScope } from "../core/route-scope.ts";
import { assertKnownArguments } from "./argument-guard.ts";
import { VaultFrozenError } from "../core/brain/freeze-marker.ts";
import { UNRESOLVED_AGENT, vaultFrozenRefusal } from "./frozen-refusal.ts";
@@ -263,8 +263,10 @@ export class MCPServer {
* metrics off it is a transparent pass-through; with them on it times
* the handler and emits one payload-safe `mcp_route_latency` record
* (status `error` on throw), then re-raises so error handling upstream
- * is unchanged. The emit is gated and fail-open, so it can never fail
- * or slow-fail the call beyond one synchronous continuity append.
+ * is unchanged. One route scope per call carries the decision time and
+ * the write stages the handler notes; neither ever reaches the response.
+ * The emit is gated and fail-open, so it can never fail or slow-fail the
+ * call beyond one synchronous continuity append.
*
* The unknown-argument gate runs FIRST, before the timer and before the
* handler, because a refused call is not a route to measure. Placing it
@@ -297,9 +299,9 @@ export class MCPServer {
}
const start = performance.now();
let status: McpRouteStatus = "ok";
- const decisionScope = createDecisionLatencyScope();
+ const routeScope = createRouteScope();
try {
- return await decisionScope.run(async () => tool.handler(this.context, args, onProgress));
+ return await routeScope.run(async () => tool.handler(this.context, args, onProgress));
} catch (exc) {
status = "error";
throw this.mapFrozen(tool, exc);
@@ -312,9 +314,8 @@ export class MCPServer {
status,
durationMs: performance.now() - start,
argKeys: Object.keys(args),
- ...(decisionScope.decisionMs() !== undefined
- ? { decisionMs: decisionScope.decisionMs() }
- : {}),
+ ...(routeScope.decisionMs() !== undefined ? { decisionMs: routeScope.decisionMs() } : {}),
+ ...(routeScope.stages() !== undefined ? { stages: routeScope.stages() } : {}),
},
this.routeMetricsEnabled,
);
diff --git a/tests/cli/brain-reject-siblings.test.ts b/tests/cli/brain-reject-siblings.test.ts
new file mode 100644
index 00000000..2f6f57d5
--- /dev/null
+++ b/tests/cli/brain-reject-siblings.test.ts
@@ -0,0 +1,127 @@
+/**
+ * `o2b brain reject` lists retire siblings (near-duplicate defense, A6).
+ *
+ * With `near_duplicate_retire_siblings_enabled` on, the verb prints every
+ * active preference that resembles the one it just retired, with its score
+ * and the command that would accept the pair. It retires nothing else. With
+ * the key off its output is exactly what it was before.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { mkdtempSync, readdirSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { bootstrapBrain } from "../../src/core/brain/init.ts";
+import { writePreference } from "../../src/core/brain/preference.ts";
+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";
+import { runCli } from "../helpers/run-cli.ts";
+
+const FLAG_ENV = "OPEN_SECOND_BRAIN_NEAR_DUPLICATE_RETIRE_SIBLINGS_ENABLED";
+const RULE = "always run the formatter before every commit in this repository";
+const PARAPHRASE = "always run the formatter before each commit in this repository";
+const UNRELATED = "prefer tabs over spaces inside every makefile you touch";
+
+let tmp: string;
+let vault: string;
+let config: string;
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "o2b-reject-siblings-"));
+ vault = join(tmp, "vault");
+ config = join(tmp, "config.yaml");
+ atomicWriteFileSync(config, `vault: ${vault}\n`);
+ bootstrapBrain(vault, { configPath: config });
+ seed("old", "formatting", RULE);
+ seed("paraphrase", "commit-hygiene", PARAPHRASE);
+ seed("unrelated", "makefiles", UNRELATED);
+});
+
+afterEach(() => {
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+function seed(slug: string, topic: string, principle: string): void {
+ writePreference(vault, {
+ slug,
+ topic,
+ principle,
+ created_at: "2026-06-01T00:00:00Z",
+ confirmed_at: "2026-06-02T00:00:00Z",
+ unconfirmed_until: "2026-06-15T00:00:00Z",
+ status: "confirmed",
+ evidenced_by: ["[[sig-1]]"],
+ applied_count: 1,
+ violated_count: 0,
+ last_evidence_at: "2026-06-02T00:00:00Z",
+ confidence: "low",
+ });
+}
+
+function reject(flagOn: boolean, extra: string[] = []) {
+ const env: Record<string, string> = { OPEN_SECOND_BRAIN_CONFIG: config };
+ if (flagOn) env[FLAG_ENV] = "1";
+ return runCli(
+ ["brain", "reject", "--vault", vault, "--id", "pref-old", "--reason", "obsolete", ...extra],
+ { env },
+ );
+}
+
+function names(dir: string): string[] {
+ return readdirSync(join(vault, "Brain", dir))
+ .filter((n) => n.endsWith(".md"))
+ .toSorted();
+}
+
+describe("brain reject retire siblings", () => {
+ test("with the key on, each sibling is printed with its score and the accept command", async () => {
+ const r = await reject(true);
+ expect(r.returncode).toBe(0);
+ expect(r.stdout).toBe(
+ "retired: ret-old (user-rejected)\n" +
+ "retire siblings: 1\n" +
+ " pref-paraphrase score=0.818 method=lexical\n" +
+ " accept: o2b brain reject --id pref-paraphrase --reason <text>\n",
+ );
+ });
+
+ test("with the key on, --json carries retire_siblings", async () => {
+ const r = await reject(true, ["--json"]);
+ expect(r.returncode).toBe(0);
+ expect(JSON.parse(r.stdout)).toEqual({
+ ok: true,
+ id: "ret-old",
+ reason: "user-rejected",
+ retire_siblings: [
+ { retiring_id: "pref-old", sibling_id: "pref-paraphrase", score: 0.818, method: "lexical" },
+ ],
+ });
+ });
+
+ test("with the key off, the output is exactly the pre-existing line", async () => {
+ const text = await reject(false);
+ expect(text.stdout).toBe("retired: ret-old (user-rejected)\n");
+ seed("old", "formatting", RULE);
+ rmSync(join(vault, "Brain", "retired", "ret-old.md"));
+ const json = await reject(false, ["--json"]);
+ expect(JSON.parse(json.stdout)).toEqual({ ok: true, id: "ret-old", reason: "user-rejected" });
+ });
+
+ test("a sibling scan that fails after the retire is a warning, not a failed reject", async () => {
+ // A file where the archived-signal directory belongs makes the scan throw.
+ const archived = join(vault, "Brain", "inbox", "archived");
+ rmSync(archived, { recursive: true, force: true });
+ writeFileSync(archived, "not a directory");
+ const r = await reject(true);
+ expect(r.returncode).toBe(0);
+ expect(r.stdout).toBe("retired: ret-old (user-rejected)\n");
+ expect(r.stderr).toContain("warning: retire siblings scan failed");
+ expect(names("retired")).toEqual(["ret-old.md"]);
+ });
+
+ test("nothing besides the rejected preference is retired", async () => {
+ await reject(true);
+ expect(names("preferences")).toEqual(["pref-paraphrase.md", "pref-unrelated.md"]);
+ expect(names("retired")).toEqual(["ret-old.md"]);
+ });
+});
diff --git a/tests/core/architecture/visibility-surface-census.test.ts b/tests/core/architecture/visibility-surface-census.test.ts
index e5604a6a..a15578b7 100644
--- a/tests/core/architecture/visibility-surface-census.test.ts
+++ b/tests/core/architecture/visibility-surface-census.test.ts
@@ -343,6 +343,11 @@ const NOTE_CONTENT_PRODUCERS: ReadonlyArray<ProducerRule> = Object.freeze([
},
{ specifierIncludes: "/brain/diarization.ts", identifiers: ["diarize"] },
{ specifierIncludes: "/brain/recompile.ts", identifiers: ["planRecompile"] },
+ // The write-receipt lint names a near-duplicate sibling page by path and
+ // counts the siblings it skipped, so every note-write tool that returns
+ // through the shared receipt envelope hands paths to its caller.
+ { specifierIncludes: "/brain/page-lint.ts", identifiers: ["lintWrittenPages"] },
+ { specifierIncludes: "/notes-tools.ts", identifiers: ["noteWriteResult"] },
]);
/**
@@ -574,8 +579,13 @@ function reasonProblems(entries: ReadonlyArray<VisibilitySurfaceEntry>): {
* `brain_derive_fact` and `brain_scaffold_stub`, which accepted a
* withheld premise or source, and `brain_note_lifecycle` on the
* file-level rule.
+ *
+ * 76 before the vocabulary gained the write-receipt lint and the shared
+ * receipt envelope. The four new names are `brain_create_note`,
+ * `brain_update_note`, `brain_append_note` and `brain_write_batch`, whose
+ * near-duplicate hint named sibling pages the caller could not read.
*/
-const MCP_TOOL_POPULATION_SIZE = 76;
+const MCP_TOOL_POPULATION_SIZE = 80;
/** Measured: MCP resources + templates. */
const MCP_RESOURCE_POPULATION_SIZE = 8;
/** Measured: hand-enumerated CLI verb mirrors. */
@@ -635,12 +645,14 @@ describe("visibility surface census", () => {
expect(covered).toEqual([
"brain_agent_diff",
"brain_agent_query",
+ "brain_append_note",
"brain_apply_evidence",
"brain_backlinks",
"brain_bridges",
"brain_claims",
"brain_clusters",
"brain_context",
+ "brain_create_note",
"brain_dead_ends",
"brain_decision",
"brain_deep_synthesis",
@@ -678,6 +690,8 @@ describe("visibility surface census", () => {
"brain_tiers",
"brain_trigger",
"brain_unlinked_mentions",
+ "brain_update_note",
+ "brain_write_batch",
"brain_writes",
"schema_inspect",
"second_brain_query",
diff --git a/tests/core/brain/extract-signals-prefilter.test.ts b/tests/core/brain/extract-signals-prefilter.test.ts
index a125afbc..bf2d52ac 100644
--- a/tests/core/brain/extract-signals-prefilter.test.ts
+++ b/tests/core/brain/extract-signals-prefilter.test.ts
@@ -225,9 +225,11 @@ describe("enforce", () => {
expect(plan.skipped).toBeUndefined();
const prompt = plan.llmStep!.prompt;
expect(prompt).toContain("the 2 user turn(s)");
- expect(prompt.indexOf("[t1]")).toBeLessThan(prompt.indexOf("[t5]"));
- expect(prompt).not.toContain("[t3]");
- expect(prompt).not.toContain("[t7]");
+ // Every turn line carries its timestamp: `[turnId @ <timestamp>] text`.
+ expect(prompt).toContain("[t1 @ ");
+ expect(prompt.indexOf("[t1 @ ")).toBeLessThan(prompt.indexOf("[t5 @ "));
+ expect(prompt).not.toContain("[t3 @ ");
+ expect(prompt).not.toContain("[t7 @ ");
});
test("an invalid item keeps its turn", async () => {
diff --git a/tests/core/brain/extract-signals-temporal.test.ts b/tests/core/brain/extract-signals-temporal.test.ts
new file mode 100644
index 00000000..2bf95189
--- /dev/null
+++ b/tests/core/brain/extract-signals-temporal.test.ts
@@ -0,0 +1,152 @@
+/**
+ * Extract date grounding reaches the dream temporal window
+ * (near-duplicate-defense, C5). Claims pinned here:
+ *
+ * 1. An extract payload whose principle carries an ISO interval lands a
+ * signal whose principle keeps the interval verbatim; the extract lane
+ * writes no `valid_from` / `valid_until` of its own.
+ * 2. A dream pass over those signals derives `valid_from` and
+ * `valid_until` through the existing temporal extraction - no parser
+ * lives in the extract lane.
+ * 3. The envelope tells the caller to write an end-only bound as an
+ * interval, because the dream pass reads a lone ISO date as the day a
+ * rule STARTS: an end bound written as a lone date would invert it.
+ * 4. The envelope says an interval's end date is exclusive, because the
+ * window is `[valid_from, valid_until)`: "to the 20th" is written as
+ * the 21st, or the rule stops applying for the whole of the 20th.
+ * 5. An interval with an impossible date, or one that does not end after
+ * it starts, is refused whole: it would yield a window that is never
+ * active, and nothing downstream would say why.
+ */
+
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import { existsSync, mkdtempSync, readdirSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { dream } from "../../../src/core/brain/dream.ts";
+import {
+ commitExtractedSignals,
+ planExtractSignals,
+} from "../../../src/core/brain/extract-signals.ts";
+import { bootstrapBrain } from "../../../src/core/brain/init.ts";
+import { preferencePath } from "../../../src/core/brain/paths.ts";
+import { parsePreference } from "../../../src/core/brain/preference.ts";
+import {
+ ResponseCheckError,
+ SEMANTIC_VIOLATION_CODES,
+} from "../../../src/core/brain/response-checks.ts";
+import { importSessionRecall } from "../../../src/core/brain/session-recall.ts";
+import { parseSignal } from "../../../src/core/brain/signal.ts";
+import { atomicWriteFileSync } from "../../../src/core/fs-atomic.ts";
+
+const TOPIC = "release-freeze";
+// "From the 10th to the 20th": the end date is exclusive, so it is the 21st.
+const INTERVAL = "2026-10-10/2026-10-21";
+const NOW = new Date("2026-10-07T12:00:00Z");
+
+/**
+ * Three sessions, one rule each: the dream pass promotes a topic once
+ * three signals agree, and one payload may not repeat a topic. The
+ * principles differ in wording so the dedup index keeps all three.
+ */
+const SESSIONS: ReadonlyArray<{ id: string; principle: string }> = [
+ { id: "sess-a", principle: `Freeze merges to the release branch during ${INTERVAL}.` },
+ { id: "sess-b", principle: `Hold every release-branch merge during ${INTERVAL}.` },
+ { id: "sess-c", principle: `Merge nothing into the release branch during ${INTERVAL}.` },
+];
+
+let vault: string;
+let configHome: string;
+
+beforeEach(() => {
+ vault = mkdtempSync(join(tmpdir(), "o2b-extract-temporal-"));
+ configHome = mkdtempSync(join(tmpdir(), "o2b-extract-temporal-cfg-"));
+ const configPath = join(configHome, "config.yaml");
+ atomicWriteFileSync(configPath, `vault: ${vault}\n`);
+ bootstrapBrain(vault, { configPath });
+ SESSIONS.forEach((session, index) => {
+ importSessionRecall(vault, {
+ sessionId: session.id,
+ turns: [
+ {
+ turnId: "t1",
+ timestamp: `2026-10-0${index + 1}T09:00:00Z`,
+ role: "user",
+ text: "No release-branch merges from the 10th to the 20th.",
+ },
+ ],
+ createdAt: NOW.toISOString(),
+ });
+ });
+});
+
+afterEach(() => {
+ rmSync(vault, { recursive: true, force: true });
+ rmSync(configHome, { recursive: true, force: true });
+});
+
+function commit(session: { id: string; principle: string }) {
+ return commitExtractedSignals(
+ vault,
+ session.id,
+ {
+ items: [{ topic: TOPIC, signal: "negative", principle: session.principle, confidence: 0.9 }],
+ },
+ { agent: "tester", now: NOW, writeApprovalEnabled: false },
+ );
+}
+
+test("an ISO interval in an extracted principle becomes the dream validity window", () => {
+ for (const session of SESSIONS) {
+ const res = commit(session);
+ expect(res.written).toHaveLength(1);
+ const signal = parseSignal(res.written[0]!.path);
+ expect(signal.principle).toContain(INTERVAL);
+ // The extract lane grounds the date in the text and nothing else.
+ expect(signal.valid_from).toBeUndefined();
+ expect(signal.valid_until).toBeUndefined();
+ }
+
+ const summary = dream(vault, { now: NOW });
+ expect(summary.new_unconfirmed).toContain(`pref-${TOPIC}`);
+ const pref = parsePreference(preferencePath(vault, TOPIC));
+ expect(pref.valid_from).toBe("2026-10-10T00:00:00Z");
+ expect(pref.valid_until).toBe("2026-10-21T00:00:00Z");
+});
+
+test("the envelope asks for an end-only bound as an interval, never a lone date", () => {
+ const plan = planExtractSignals(vault, SESSIONS[0]!.id, { now: NOW });
+ const head = plan.llmStep.prompt.slice(0, plan.llmStep.prompt.indexOf("\n\n"));
+ expect(head).toContain("YYYY-MM-DD/YYYY-MM-DD");
+ expect(head).toContain("a lone date reads as the day the rule starts");
+});
+
+test("the envelope says an interval's end date is exclusive", () => {
+ const plan = planExtractSignals(vault, SESSIONS[0]!.id, { now: NOW });
+ const head = plan.llmStep.prompt.slice(0, plan.llmStep.prompt.indexOf("\n\n"));
+ expect(head).toContain("an interval's end date is exclusive");
+ expect(head).toContain("the day after the last day the rule holds");
+});
+
+test.each([
+ ["a reversed interval", "2026-10-21/2026-10-10", "must end after it starts"],
+ ["an empty interval", "2026-10-10/2026-10-10", "must end after it starts"],
+ ["an impossible end date", "2026-02-01/2026-02-30", "2026-02-30, which is not a calendar date"],
+ ["an impossible start date", "2026-13-01/2026-12-20", "2026-13-01, which is not a calendar date"],
+])("%s is refused whole, naming the principle", (_label, interval, reason) => {
+ let caught: unknown;
+ try {
+ commit({ id: SESSIONS[0]!.id, principle: `Freeze release-branch merges during ${interval}.` });
+ } catch (err) {
+ caught = err;
+ }
+ expect(caught).toBeInstanceOf(ResponseCheckError);
+ const err = caught as ResponseCheckError;
+ expect(err.code).toBe(SEMANTIC_VIOLATION_CODES.threshold);
+ expect(err.message).toContain("items[0].principle");
+ expect(err.message).toContain(`interval ${interval}`);
+ expect(err.message).toContain(reason);
+ const inbox = join(vault, "Brain", "inbox");
+ expect(existsSync(inbox) ? readdirSync(inbox).filter((f) => f.endsWith(".md")) : []).toEqual([]);
+});
diff --git a/tests/core/brain/extract-signals.test.ts b/tests/core/brain/extract-signals.test.ts
index c8cbdc62..c5898fec 100644
--- a/tests/core/brain/extract-signals.test.ts
+++ b/tests/core/brain/extract-signals.test.ts
@@ -39,12 +39,16 @@ import {
} from "../../../src/core/brain/extract-signals.ts";
import { listDeadLetters } from "../../../src/core/brain/dead-letter.ts";
import { NEEDS_LLM_STEP } from "../../../src/core/brain/llm-step.ts";
-import { ResponseCheckError } from "../../../src/core/brain/response-checks.ts";
+import {
+ ResponseCheckError,
+ SEMANTIC_VIOLATION_CODES,
+} from "../../../src/core/brain/response-checks.ts";
import { ResponseShapeError } from "../../../src/core/brain/response-shape.ts";
import { bootstrapBrain } from "../../../src/core/brain/init.ts";
import { brainConfigPath } from "../../../src/core/brain/paths.ts";
import { atomicWriteFileSync } from "../../../src/core/fs-atomic.ts";
import { importSessionRecall } from "../../../src/core/brain/session-recall.ts";
+import type { DedupIndexEntry } from "../../../src/core/brain/dedup-hash.ts";
import { parseSignal } from "../../../src/core/brain/signal.ts";
import { BRAIN_SIGNAL_SOURCE_TYPE } from "../../../src/core/brain/types.ts";
import type { SessionTurn } from "../../../src/core/brain/sessions/types.ts";
@@ -396,3 +400,125 @@ test("a partial write leaves a durable dead letter naming the unwritten items",
expect(err.deadLetter.recorded).toBe(true);
if (err.deadLetter.recorded) expect(err.message).toContain(err.deadLetter.id);
});
+
+// ----- Turn timestamps in the envelope (near-duplicate-defense, C1) --------
+//
+// 15. A mined turn carries the stored turn timestamp verbatim, and the
+// transcript line shows it as `[turnId @ <timestamp>] text`.
+// 16. A turn stored without a timestamp renders `[turnId] text`; the plan's
+// clock is never substituted for the missing value.
+
+test("a mined turn carries the stored timestamp verbatim into its transcript line", () => {
+ const plan = planExtractSignals(vault, SESSION, { now: NOW });
+ expect(plan.turnsMined.map((t) => t.timestamp)).toEqual([
+ "2026-08-22T09:01:00Z",
+ "2026-08-22T09:03:00Z",
+ ]);
+ expect(plan.llmStep.prompt).toContain(
+ "[t1 @ 2026-08-22T09:01:00Z] Always name the release theme in the heading.",
+ );
+ expect(plan.llmStep.prompt).toContain(
+ "[t3 @ 2026-08-22T09:03:00Z] And never abbreviate the module names.",
+ );
+ // Assistant turns stay out, timestamp or not.
+ expect(plan.llmStep.prompt).not.toContain("[t2");
+});
+
+test("a turn without a stored timestamp renders bare and never borrows the clock", () => {
+ importTurns("sess-undated", [
+ { turnId: "u1", timestamp: "", role: "user", text: "Keep the changelog terse." },
+ ]);
+ const plan = planExtractSignals(vault, "sess-undated", { now: NOW });
+ expect(plan.turnsMined.map((t) => t.timestamp)).toEqual([""]);
+ expect(plan.llmStep.prompt).toContain("[u1] Keep the changelog terse.");
+ expect(plan.llmStep.prompt).not.toContain("[u1 @");
+ expect(plan.llmStep.prompt).not.toContain(NOW.toISOString());
+});
+
+// ----- Hygiene rules in the envelope (near-duplicate-defense, C2) ----------
+//
+// 17. The instruction carries five language-neutral rules: time bounds as
+// ISO dates or intervals, conversational mechanics skipped, one rule
+// per item, conditions kept, restatements dropped.
+// 18. The rules name categories, never example words: the instruction and
+// the schema hints quote no phrase and stay ASCII, so no language's
+// vocabulary is baked into the kernel.
+
+/** The instruction half of the prompt, everything before the transcript. */
+function instruction(prompt: string): string {
+ return prompt.slice(0, prompt.indexOf("\n\n"));
+}
+
+test("the envelope instruction carries the five hygiene rules", () => {
+ const plan = planExtractSignals(vault, SESSION, { now: NOW });
+ const head = instruction(plan.llmStep.prompt);
+ expect(head).toContain("ISO 8601 date or interval");
+ expect(head).toContain("timestamp of the turn");
+ expect(head).toContain("conversational mechanics");
+ expect(head).toContain("one rule per item");
+ expect(head).toContain("condition");
+ expect(head).toContain("restatement");
+ expect(plan.llmStep.schema_hints.join("\n")).toContain("ISO 8601");
+});
+
+test("the hygiene rules name categories and quote no example words", () => {
+ const plan = planExtractSignals(vault, SESSION, { now: NOW });
+ const head = instruction(plan.llmStep.prompt);
+ const hints = plan.llmStep.schema_hints.filter((hint) => !hint.startsWith("payload:"));
+ for (const text of [head, ...hints]) {
+ expect(text).not.toMatch(/"/);
+ expect(text).toMatch(/^[\x20-\x7e]*$/);
+ }
+});
+
+// ----- Duplicate-topic refusal (near-duplicate-defense, C3) -----------------
+//
+// 19. Two items sharing a `topic` refuse the payload whole under the
+// cross-item code, and the message names both indices and the topic.
+// 20. The refusal lands before any write and before any item is hashed for
+// dedup, so a refused payload leaves the vault and the index untouched.
+
+/** A dedup index that counts every keyed read; one read follows every hash. */
+class CountingDedup extends Map<string, DedupIndexEntry> {
+ lookups = 0;
+ override has(key: string): boolean {
+ this.lookups += 1;
+ return super.has(key);
+ }
+ override get(key: string): DedupIndexEntry | undefined {
+ this.lookups += 1;
+ return super.get(key);
+ }
+}
+
+test("a payload that repeats a topic is refused whole, naming both indices", () => {
+ const dedup = new CountingDedup();
+ const items = [
+ item(),
+ item({ topic: "module-names", principle: "Never abbreviate module names." }),
+ item({ principle: "Put the release theme in the heading." }),
+ ];
+ let caught: unknown;
+ try {
+ commitExtractedSignals(vault, SESSION, { items }, { agent: "tester", now: NOW, dedup });
+ } catch (err) {
+ caught = err;
+ }
+ expect(caught).toBeInstanceOf(ResponseCheckError);
+ const err = caught as ResponseCheckError;
+ expect(err.code).toBe(SEMANTIC_VIOLATION_CODES.crossItem);
+ expect(err.message).toContain('items[0] and items[2] share topic "release-notes-style"');
+ expect(dedup.lookups).toBe(0);
+ expect(dedup.size).toBe(0);
+ expect(inboxFiles()).toEqual([]);
+});
+
+test("distinct topics pass the duplicate-topic rule", () => {
+ const res = commitExtractedSignals(
+ vault,
+ SESSION,
+ { items: [item(), item({ topic: "module-names", principle: "Never abbreviate modules." })] },
+ { agent: "tester", now: NOW },
+ );
+ expect(res.written.map((w) => w.topic)).toEqual(["release-notes-style", "module-names"]);
+});
diff --git a/tests/core/brain/mcp-route-metrics.test.ts b/tests/core/brain/mcp-route-metrics.test.ts
index 83fe5310..63fd2c22 100644
--- a/tests/core/brain/mcp-route-metrics.test.ts
+++ b/tests/core/brain/mcp-route-metrics.test.ts
@@ -8,6 +8,7 @@ import {
listMcpRouteLatency,
summarizeMcpRouteLatency,
} from "../../../src/core/brain/mcp-route-metrics.ts";
+import type { RouteStageTiming } from "../../../src/core/route-scope.ts";
let tmp: string;
let vault: string;
@@ -171,3 +172,125 @@ describe("summarizeMcpRouteLatency", () => {
expect(summary.routes[0]!.count).toBe(3);
});
});
+
+describe("mcp route latency stages", () => {
+ const base = {
+ createdAt: "2026-06-01T00:00:00.000Z",
+ tool: "brain_feedback",
+ status: "ok" as const,
+ durationMs: 4,
+ };
+
+ test("stages are written only when non-empty", () => {
+ const withStages = emitMcpRouteLatency(
+ vault,
+ {
+ ...base,
+ stages: [
+ { name: "validate", ms: 0.4 },
+ { name: "log_append", ms: 1.26 },
+ ],
+ },
+ true,
+ );
+ expect(withStages!.payload["stages"]).toEqual([
+ { name: "validate", ms: 0.4 },
+ { name: "log_append", ms: 1.3 },
+ ]);
+ const empty = emitMcpRouteLatency(vault, { ...base, stages: [] }, true);
+ expect(Object.hasOwn(empty!.payload, "stages")).toBe(false);
+ });
+
+ test("unknown names, negative and non-finite values are dropped", () => {
+ const hostile = [
+ { name: "validate", ms: -1 },
+ { name: "document_write", ms: Number.NaN },
+ { name: "lint", ms: Number.POSITIVE_INFINITY },
+ { name: "not_a_stage", ms: 2 },
+ { name: "write_receipt", ms: 2 },
+ ] as unknown as ReadonlyArray<RouteStageTiming>;
+ const record = emitMcpRouteLatency(vault, { ...base, stages: hostile }, true);
+ expect(record!.payload["stages"]).toEqual([{ name: "write_receipt", ms: 2 }]);
+
+ const allBad = [{ name: "validate", ms: -3 }] as unknown as ReadonlyArray<RouteStageTiming>;
+ const none = emitMcpRouteLatency(vault, { ...base, stages: allBad }, true);
+ expect(Object.hasOwn(none!.payload, "stages")).toBe(false);
+ });
+
+ test("repeated names are summed in first-seen order", () => {
+ const record = emitMcpRouteLatency(
+ vault,
+ {
+ ...base,
+ stages: [
+ { name: "write_receipt", ms: 1 },
+ { name: "lint", ms: 0.5 },
+ { name: "write_receipt", ms: 0.25 },
+ ],
+ },
+ true,
+ );
+ expect(record!.payload["stages"]).toEqual([
+ { name: "write_receipt", ms: 1.3 },
+ { name: "lint", ms: 0.5 },
+ ]);
+ });
+
+ test("a hostile stage name never reaches the persisted continuity file", () => {
+ const PRIVATE_PATH = "Notes/private-topic-marker.md";
+ const PRIVATE_TOPIC = "private-topic-slug-marker";
+ const hostile = [
+ { name: PRIVATE_PATH, ms: 1 },
+ { name: PRIVATE_TOPIC, ms: 1 },
+ { name: "validate", ms: 1 },
+ ] as unknown as ReadonlyArray<RouteStageTiming>;
+ emitMcpRouteLatency(vault, { ...base, stages: hostile }, true);
+ const raw = readFileSync(join(vault, "Brain", "log", "continuity", "2026-06.jsonl"), "utf8");
+ expect(raw).not.toContain(PRIVATE_PATH);
+ expect(raw).not.toContain(PRIVATE_TOPIC);
+ expect(raw).toContain('"validate"');
+ });
+
+ test("records without stages keep the pre-stage payload byte-for-byte", () => {
+ const record = emitMcpRouteLatency(
+ vault,
+ { ...base, scope: "full", argKeys: ["topic"], decisionMs: 2.4 },
+ true,
+ );
+ expect(JSON.stringify(record!.payload)).toBe(
+ '{"tool":"brain_feedback","scope":"full","status":"ok","duration_ms":4,"arg_keys":["topic"],"decision_ms":2}',
+ );
+ });
+});
+
+function stagesFor(ms: number): ReadonlyArray<RouteStageTiming> {
+ return [
+ { name: "validate", ms },
+ { name: "log_append", ms: ms * 2 },
+ ];
+}
+
+describe("summarizeMcpRouteLatency stages", () => {
+ test("each route gets a per-stage count, average and p95", () => {
+ for (const ms of [1, 2, 3, 4]) {
+ emitMcpRouteLatency(
+ vault,
+ { tool: "brain_feedback", status: "ok", durationMs: 10, stages: stagesFor(ms) },
+ true,
+ );
+ }
+ emitMcpRouteLatency(vault, { tool: "brain_feedback", status: "ok", durationMs: 10 }, true);
+
+ const route = summarizeMcpRouteLatency(vault).routes.find((r) => r.tool === "brain_feedback")!;
+ expect(route.stages).toEqual([
+ { name: "validate", count: 4, avg_ms: 2.5, p95_ms: 4 },
+ { name: "log_append", count: 4, avg_ms: 5, p95_ms: 8 },
+ ]);
+ });
+
+ test("a route without stages has no stages key", () => {
+ emitMcpRouteLatency(vault, { tool: "second_brain_status", status: "ok", durationMs: 2 }, true);
+ const route = summarizeMcpRouteLatency(vault).routes[0]!;
+ expect(Object.hasOwn(route, "stages")).toBe(false);
+ });
+});
diff --git a/tests/core/brain/near-duplicate-vectors.test.ts b/tests/core/brain/near-duplicate-vectors.test.ts
new file mode 100644
index 00000000..f21adbf0
--- /dev/null
+++ b/tests/core/brain/near-duplicate-vectors.test.ts
@@ -0,0 +1,284 @@
+/**
+ * Stored-vector tier for retire siblings (near-duplicate defense, A4).
+ *
+ * Each status comes from a real fixture index (or its absence): the tier
+ * reads vectors the indexer already paid for and never embeds anything. A
+ * source scan pins that by construction: neither this module nor the
+ * write-side widening collector imports from the embeddings layer, and
+ * `detectSemanticDedup` still has no production caller.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { existsSync, readdirSync, readFileSync } from "node:fs";
+import { dirname, join, relative, resolve, sep } from "node:path";
+
+import {
+ storedVectorSimilarities,
+ storedVectorSimilarity,
+} from "../../../src/core/brain/near-duplicate-vectors.ts";
+import { indexVault } from "../../../src/core/search/indexer.ts";
+import { Store } from "../../../src/core/search/store.ts";
+import type {
+ ResolvedEmbeddingConfig,
+ ResolvedSearchConfig,
+} from "../../../src/core/search/types.ts";
+import { sqliteVecLoadable } from "../../helpers/sqlite-vec.ts";
+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";
+
+const MODEL = "nd-model";
+const OTHER_MODEL = "nd-older-model";
+const DIMENSION = 4;
+const VEC_LOADABLE = sqliteVecLoadable();
+
+const PROBE = "Brain/preferences/pref-old.md";
+const NEAR = "Brain/preferences/pref-near.md";
+const FAR = "Brain/preferences/pref-far.md";
+const UNINDEXED = "Brain/preferences/pref-unindexed.md";
+
+let vault: string;
+let dbPath: string;
+let cleanup: () => void;
+
+beforeEach(() => {
+ ({ vault, dbPath, cleanup } = createTempVault("nd-vectors"));
+});
+
+afterEach(() => cleanup());
+
+function unit(values: number[]): number[] {
+ const norm = Math.hypot(...values);
+ return values.map((v) => v / norm);
+}
+
+function vecConfig(semantic: Partial<ResolvedEmbeddingConfig> = {}): ResolvedSearchConfig {
+ return makeConfig({
+ vault,
+ dbPath,
+ semantic: {
+ enabled: true,
+ baseUrl: "https://x/v1",
+ model: MODEL,
+ apiKey: "k",
+ dimension: DIMENSION,
+ ...semantic,
+ },
+ });
+}
+
+function writePref(relPath: string, principle: string): void {
+ writeMd(vault, relPath, `---\ntopic: t\nprinciple: ${principle}\n---\n\n${principle}\n`);
+}
+
+async function plant(
+ config: ResolvedSearchConfig,
+ byPath: Record<string, { vector: number[]; model?: string; frontmatter?: number[] }>,
+): Promise<void> {
+ const store = await Store.open(config, { mode: "write" });
+ try {
+ for (const [path, planted] of Object.entries(byPath)) {
+ const docId = store.getDocumentIdByPath(path);
+ expect(docId).not.toBeNull();
+ for (const chunk of store.chunksForDocument(docId!)) {
+ // Chunk 0 is the page's frontmatter block (see `writePref`).
+ const vector =
+ chunk.chunkIndex === 0 ? (planted.frontmatter ?? planted.vector) : planted.vector;
+ store.vecUpsert(
+ chunk.id,
+ unit(vector),
+ planted.model ?? MODEL,
+ DIMENSION,
+ `eh-${path}-${chunk.id}`,
+ );
+ }
+ }
+ } finally {
+ await store.close();
+ }
+}
+
+async function indexedFixture(config: ResolvedSearchConfig): Promise<void> {
+ writePref(PROBE, "Run the formatter before every commit");
+ writePref(NEAR, "Format the code before each commit");
+ writePref(FAR, "Use tabs inside makefiles");
+ await indexVault(config);
+}
+
+describe("storedVectorSimilarity", () => {
+ test("a missing index reports index_missing with no scores", async () => {
+ const result = await storedVectorSimilarity(vecConfig(), PROBE, [NEAR]);
+ expect(existsSync(dbPath)).toBe(false);
+ expect(result.status).toBe("index_missing");
+ expect(result.scores.size).toBe(0);
+ });
+
+ test("a store without sqlite-vec reports vec_unavailable", async () => {
+ const config = vecConfig();
+ await indexedFixture(config);
+ const result = await storedVectorSimilarity(config, PROBE, [NEAR], { loadVec: false });
+ expect(result.status).toBe("vec_unavailable");
+ expect(result.scores.size).toBe(0);
+ });
+
+ test.skipIf(!VEC_LOADABLE)(
+ "an indexed probe with no stored vector reports not_embedded, not used",
+ async () => {
+ const config = vecConfig();
+ await indexedFixture(config);
+ await plant(config, { [NEAR]: { vector: [1, 0, 0, 0] } });
+ const result = await storedVectorSimilarity(config, PROBE, [NEAR]);
+ expect(result.status).toBe("not_embedded");
+ expect(result.scores.size).toBe(0);
+ },
+ );
+
+ test.skipIf(!VEC_LOADABLE)(
+ "frontmatter chunks are never compared: matching frontmatter alone scores nothing",
+ async () => {
+ const config = vecConfig();
+ await indexedFixture(config);
+ // Identical frontmatter vectors, orthogonal principle vectors.
+ await plant(config, {
+ [PROBE]: { frontmatter: [1, 0, 0, 0], vector: [0, 1, 0, 0] },
+ [NEAR]: { frontmatter: [1, 0, 0, 0], vector: [0, 0, 1, 0] },
+ });
+ const result = await storedVectorSimilarity(config, PROBE, [NEAR]);
+ expect(result.status).toBe("used");
+ expect(result.scores.get(NEAR)).toBeCloseTo(0, 5);
+ },
+ );
+
+ test.skipIf(!VEC_LOADABLE)(
+ "a probe whose only stored vector is its frontmatter reports not_embedded",
+ async () => {
+ const config = vecConfig();
+ await indexedFixture(config);
+ const store = await Store.open(config, { mode: "write" });
+ try {
+ const docId = store.getDocumentIdByPath(PROBE)!;
+ const fm = store.chunksForDocument(docId).find((c) => c.chunkIndex === 0)!;
+ store.vecUpsert(fm.id, unit([1, 0, 0, 0]), MODEL, DIMENSION, "eh-probe-fm");
+ } finally {
+ await store.close();
+ }
+ await plant(config, { [NEAR]: { vector: [1, 0, 0, 0] } });
+ const result = await storedVectorSimilarity(config, PROBE, [NEAR]);
+ expect(result.status).toBe("not_embedded");
+ },
+ );
+
+ test.skipIf(!VEC_LOADABLE)(
+ "a probe stored under another model reports model_mismatch",
+ async () => {
+ const config = vecConfig();
+ await indexedFixture(config);
+ await plant(config, {
+ [PROBE]: { vector: [1, 0, 0, 0], model: OTHER_MODEL },
+ [NEAR]: { vector: [1, 0, 0, 0], model: OTHER_MODEL },
+ });
+ const result = await storedVectorSimilarity(config, PROBE, [NEAR]);
+ expect(result.status).toBe("model_mismatch");
+ expect(result.scores.size).toBe(0);
+ },
+ );
+
+ test.skipIf(!VEC_LOADABLE)(
+ "a configured dimension that differs from the stored one reports model_mismatch",
+ async () => {
+ await indexedFixture(vecConfig());
+ await plant(vecConfig(), {
+ [PROBE]: { vector: [1, 0, 0, 0] },
+ [NEAR]: { vector: [1, 0, 0, 0] },
+ });
+ const result = await storedVectorSimilarity(vecConfig({ dimension: 8 }), PROBE, [NEAR]);
+ expect(result.status).toBe("model_mismatch");
+ },
+ );
+
+ test.skipIf(!VEC_LOADABLE)(
+ "used: each candidate scores its max chunk-pair cosine; incomparable rows are unscored",
+ async () => {
+ const config = vecConfig();
+ await indexedFixture(config);
+ await plant(config, {
+ [PROBE]: { vector: [1, 0, 0, 0] },
+ [NEAR]: { vector: [1, 0.05, 0, 0] },
+ [FAR]: { vector: [0, 0, 0, 1], model: OTHER_MODEL },
+ });
+ const result = await storedVectorSimilarity(config, PROBE, [NEAR, FAR, UNINDEXED]);
+ expect(result.status).toBe("used");
+ expect([...result.scores.keys()]).toEqual([NEAR]);
+ expect(result.scores.get(NEAR)).toBeCloseTo(1 / Math.hypot(1, 0.05), 5);
+ },
+ );
+
+ test.skipIf(!VEC_LOADABLE)(
+ "several probes over one store: one result per probe, in order",
+ async () => {
+ const config = vecConfig();
+ await indexedFixture(config);
+ await plant(config, {
+ [PROBE]: { vector: [1, 0, 0, 0] },
+ [NEAR]: { vector: [1, 0.05, 0, 0] },
+ });
+ const results = await storedVectorSimilarities(config, [FAR, PROBE], [PROBE, NEAR]);
+ expect(results.map((r) => r.status)).toEqual(["not_embedded", "used"]);
+ expect([...results[1]!.scores.keys()]).toEqual([NEAR]);
+ const mismatched = await storedVectorSimilarities(
+ vecConfig({ dimension: 8 }),
+ [PROBE, NEAR],
+ [],
+ );
+ expect(mismatched.map((r) => r.status)).toEqual(["model_mismatch", "model_mismatch"]);
+ },
+ );
+});
+
+const REPO = resolve(import.meta.dir, "../../..");
+const EMBEDDINGS_DIR = join(REPO, "src", "core", "search", "embeddings") + sep;
+const SPEND_FREE_MODULES = [
+ "src/core/brain/near-duplicate-vectors.ts",
+ "src/core/brain/page-lint-widening.ts",
+];
+
+function importSpecifiers(source: string): string[] {
+ return [...source.matchAll(/\bfrom\s+["']([^"']+)["']|\bimport\s*\(\s*["']([^"']+)["']/g)].map(
+ (m) => (m[1] ?? m[2])!,
+ );
+}
+
+function sourceFiles(dir: string): string[] {
+ return readdirSync(dir, { recursive: true, withFileTypes: true })
+ .filter((e) => e.isFile() && e.name.endsWith(".ts"))
+ .map((e) => join(e.parentPath, e.name));
+}
+
+describe("spend gates by construction", () => {
+ test("neither spend-free module imports from the embeddings layer", () => {
+ for (const rel of SPEND_FREE_MODULES) {
+ const abs = join(REPO, rel);
+ // The widening collector belongs to another lane and may not exist yet.
+ if (!existsSync(abs)) continue;
+ const offending = importSpecifiers(readFileSync(abs, "utf8"))
+ .filter((spec) => spec.startsWith("."))
+ .map((spec) => resolve(dirname(abs), spec))
+ .filter((target) => target.startsWith(EMBEDDINGS_DIR));
+ expect({ module: rel, offending }).toEqual({ module: rel, offending: [] });
+ }
+ });
+
+ test("detectSemanticDedup has no caller outside tests", () => {
+ const callers = sourceFiles(join(REPO, "src"))
+ .filter((abs) => /\bdetectSemanticDedup\b/.test(stripComments(readFileSync(abs, "utf8"))))
+ .map((abs) => relative(REPO, abs).split(sep).join("/"));
+ expect(callers).toEqual(["src/core/brain/hygiene/detectors/dedup.ts"]);
+ const own = stripComments(
+ readFileSync(join(REPO, "src/core/brain/hygiene/detectors/dedup.ts"), "utf8"),
+ );
+ expect(own.match(/\bdetectSemanticDedup\b/g)).toEqual(["detectSemanticDedup"]);
+ });
+});
+
+/** Drop line and block comments so a doc mention is not counted as a caller. */
+function stripComments(source: string): string {
+ return source.replace(/\/\*[\s\S]*?\*\//g, "").replace(/(^|[^:])\/\/.*$/gm, "$1");
+}
diff --git a/tests/core/brain/near-duplicate.test.ts b/tests/core/brain/near-duplicate.test.ts
new file mode 100644
index 00000000..bc72dbe7
--- /dev/null
+++ b/tests/core/brain/near-duplicate.test.ts
@@ -0,0 +1,119 @@
+import { describe, expect, test } from "bun:test";
+
+import {
+ findNearDuplicates,
+ NEAR_DUPLICATE_CANDIDATE_CAP,
+ NEAR_DUPLICATE_MIN_TOKENS,
+ NEAR_DUPLICATE_THRESHOLDS,
+ READ_ALL_REFS,
+ type NearDuplicatePoolEntry,
+} from "../../../src/core/brain/near-duplicate.ts";
+import { NEAR_DUPLICATE_JACCARD } from "../../../src/core/brain/page-lint.ts";
+import { tokenise } from "../../../src/core/brain/similarity.ts";
+
+const BASE = "always run the formatter before every commit in this repository";
+
+function entry(ref: string, text: string, bucket?: string): NearDuplicatePoolEntry {
+ return bucket === undefined
+ ? { ref, tokens: tokenise(text) }
+ : { ref, tokens: tokenise(text), bucket };
+}
+
+const probe = { ref: "pref-probe", tokens: tokenise(BASE) };
+const opts = { threshold: 0.5, readable: READ_ALL_REFS };
+const notHidden = (ref: string) => !ref.startsWith("pref-hidden");
+
+describe("findNearDuplicates", () => {
+ test("an unreadable entry never appears in matches or in any scan counter", () => {
+ const pool = [
+ entry("pref-hidden-match", BASE),
+ entry("pref-hidden-short", "tiny"),
+ entry("pref-visible", BASE),
+ ];
+ const all = findNearDuplicates(probe, pool, { ...opts, readable: notHidden, cap: 1 });
+ expect(all.matches.map((m) => m.ref)).toEqual(["pref-visible"]);
+ expect(all.scan).toEqual({ compared: 1, capped: 0, below_min_tokens: 0 });
+ });
+
+ test("an entry under minTokens counts in below_min_tokens and is not scored", () => {
+ const pool = [entry("pref-short", "run formatter"), entry("pref-long", BASE)];
+ const result = findNearDuplicates(probe, pool, opts);
+ expect(result.matches.map((m) => m.ref)).toEqual(["pref-long"]);
+ expect(result.scan).toEqual({ compared: 1, capped: 0, below_min_tokens: 1 });
+ });
+
+ test("a probe under minTokens scores nothing", () => {
+ const short = { ref: "pref-short-probe", tokens: tokenise("run formatter") };
+ const result = findNearDuplicates(short, [entry("pref-long", BASE)], opts);
+ expect(result.matches).toEqual([]);
+ expect(result.scan.compared).toBe(0);
+ });
+
+ test("an explicit minTokens overrides the default", () => {
+ const pool = [entry("pref-short", "run formatter")];
+ const short = { ref: "pref-p", tokens: tokenise("run formatter") };
+ const result = findNearDuplicates(short, pool, { ...opts, minTokens: 2 });
+ expect(result.matches).toEqual([{ ref: "pref-short", score: 1, method: "lexical" }]);
+ });
+
+ test("the cap counts as capped", () => {
+ const pool = ["AA-1", "AA-2", "AA-3", "AA-4"].map((ref) => entry(ref, BASE));
+ const result = findNearDuplicates(probe, pool, { ...opts, cap: 3 });
+ expect(result.matches.map((m) => m.ref)).toEqual(["AA-1", "AA-2", "AA-3"]);
+ expect(result.scan).toEqual({ compared: 3, capped: 1, below_min_tokens: 0 });
+ });
+
+ test("the probe's own ref is excluded", () => {
+ const result = findNearDuplicates(probe, [entry("pref-probe", BASE)], opts);
+ expect(result.matches).toEqual([]);
+ expect(result.scan).toEqual({ compared: 0, capped: 0, below_min_tokens: 0 });
+ });
+
+ test("a bucket option keeps only entries of the same bucket", () => {
+ const pool = [entry("pref-a", BASE, "x"), entry("pref-b", BASE, "y"), entry("pref-c", BASE)];
+ const result = findNearDuplicates(probe, pool, { ...opts, bucket: "x" });
+ expect(result.matches.map((m) => m.ref)).toEqual(["pref-a"]);
+ expect(result.scan.compared).toBe(1);
+ });
+
+ test("matches below the threshold are compared but not returned", () => {
+ const pool = [entry("pref-far", "prefer tabs over spaces in makefiles always")];
+ const result = findNearDuplicates(probe, pool, opts);
+ expect(result.matches).toEqual([]);
+ expect(result.scan.compared).toBe(1);
+ });
+
+ test("sorting is deterministic: score descending, then ref", () => {
+ const near = `${BASE} today`;
+ const pool = [entry("pref-c", near), entry("pref-b", BASE), entry("pref-a", BASE)];
+ const forward = findNearDuplicates(probe, pool, opts);
+ const reversed = findNearDuplicates(probe, pool.toReversed(), opts);
+ expect(forward.matches.map((m) => m.ref)).toEqual(["pref-a", "pref-b", "pref-c"]);
+ expect(reversed).toEqual(forward);
+ expect(forward.matches[2]!.score).toBe(0.909);
+ expect(forward.matches.every((m) => m.method === "lexical")).toBe(true);
+ });
+
+ test("multilingual tokens score through the shared tokenise", () => {
+ const ru = "всегда запускай форматтер перед каждым коммитом";
+ const zh = "提交 之前 总是 运行 格式化 工具";
+ const ruProbe = { ref: "pref-ru", tokens: tokenise(ru) };
+ const zhProbe = { ref: "pref-zh", tokens: tokenise(zh) };
+ const pool = [entry("pref-ru-2", `${ru} проекта`), entry("pref-zh-2", `${zh} 仓库`)];
+ expect(findNearDuplicates(ruProbe, pool, opts).matches.map((m) => m.ref)).toEqual([
+ "pref-ru-2",
+ ]);
+ expect(findNearDuplicates(zhProbe, pool, opts).matches.map((m) => m.ref)).toEqual([
+ "pref-zh-2",
+ ]);
+ });
+
+ test("the threshold table and the shared constants hold their pinned values", () => {
+ expect(NEAR_DUPLICATE_THRESHOLDS.writeHint).toBe(NEAR_DUPLICATE_JACCARD);
+ expect(NEAR_DUPLICATE_THRESHOLDS.retireSiblingLexical).toBe(0.7);
+ expect(NEAR_DUPLICATE_THRESHOLDS.retireSiblingEmbedding).toBe(0.92);
+ expect(NEAR_DUPLICATE_MIN_TOKENS).toBe(4);
+ expect(NEAR_DUPLICATE_CANDIDATE_CAP).toBe(200);
+ expect(READ_ALL_REFS("anything")).toBe(true);
+ });
+});
diff --git a/tests/core/brain/page-lint-widening.test.ts b/tests/core/brain/page-lint-widening.test.ts
new file mode 100644
index 00000000..326aecee
--- /dev/null
+++ b/tests/core/brain/page-lint-widening.test.ts
@@ -0,0 +1,209 @@
+/**
+ * The keyword-index widening collector for the write-receipt
+ * near-duplicate hint. It pulls BM25 candidates from the existing search
+ * index so a write can be compared with pages outside its own directory,
+ * re-reads every candidate from disk, and answers at the caller's reach.
+ * An index that cannot be opened is a named status, never a failed write.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { mkdirSync, mkdtempSync, readFileSync, rmSync, unlinkSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { bootstrapBrain } from "../../../src/core/brain/init.ts";
+import {
+ NEAR_DUPLICATE_WIDENING_TOP_K,
+ READ_ALL_REFS,
+} from "../../../src/core/brain/near-duplicate.ts";
+import {
+ collectWideningCandidates,
+ WIDENING_QUERY_MAX_TERMS,
+} from "../../../src/core/brain/page-lint-widening.ts";
+import { tokenise } from "../../../src/core/brain/similarity.ts";
+import { atomicWriteFileSync } from "../../../src/core/fs-atomic.ts";
+import { resolveSearchConfig } from "../../../src/core/search/index.ts";
+import { indexVault } from "../../../src/core/search/indexer.ts";
+import { Store } from "../../../src/core/search/store.ts";
+
+const BODY = "orchard lantern copper meadow violet harbor";
+
+let vault: string;
+let configHome: string;
+
+beforeEach(() => {
+ vault = mkdtempSync(join(tmpdir(), "o2b-widening-vault-"));
+ configHome = mkdtempSync(join(tmpdir(), "o2b-widening-cfg-"));
+ const configPath = join(configHome, "config.yaml");
+ atomicWriteFileSync(configPath, `vault: ${vault}\n`);
+ bootstrapBrain(vault, { configPath });
+});
+
+afterEach(() => {
+ rmSync(vault, { recursive: true, force: true });
+ rmSync(configHome, { recursive: true, force: true });
+});
+
+function writeNote(rel: string, body: string, frontmatter = "title: T"): string {
+ const abs = join(vault, rel);
+ mkdirSync(join(abs, ".."), { recursive: true });
+ writeFileSync(abs, `---\n${frontmatter}\n---\n\n${body}\n`, "utf8");
+ return rel;
+}
+
+async function indexed() {
+ const config = resolveSearchConfig({ vault });
+ await indexVault(config, { force: true });
+ return config;
+}
+
+describe("collectWideningCandidates", () => {
+ test("a near-identical page in another directory is returned, re-read from disk", async () => {
+ const earlier = writeNote("Projects/Earlier.md", BODY, "title: Earlier\nsession: atlas");
+ const config = await indexed();
+ const written = writeNote("Notes/Later.md", BODY);
+ const result = await collectWideningCandidates(config, vault, [written], READ_ALL_REFS);
+ expect(result.status).toBe("used");
+ const found = result.candidates.find((c) => c.page === earlier);
+ expect(found).toBeDefined();
+ expect([...found!.tokens].toSorted()).toEqual([...tokenise(BODY)].toSorted());
+ // The scope bucket comes from the page's own frontmatter.
+ expect(found!.scopeKey).toContain("atlas");
+ });
+
+ test("a withheld page is dropped before it is returned", async () => {
+ writeNote("Projects/Hidden.md", BODY);
+ const visible = writeNote("Projects/Visible.md", BODY);
+ const config = await indexed();
+ const written = writeNote("Notes/Later.md", BODY);
+ const result = await collectWideningCandidates(
+ config,
+ vault,
+ [written],
+ (rel) => rel !== "Projects/Hidden.md",
+ );
+ const pages = result.candidates.map((c) => c.page);
+ expect(pages).toContain(visible);
+ expect(pages).not.toContain("Projects/Hidden.md");
+ });
+
+ test("a page deleted from disk but still in the index is dropped", async () => {
+ const gone = writeNote("Projects/Gone.md", BODY);
+ const config = await indexed();
+ unlinkSync(join(vault, gone));
+ const written = writeNote("Notes/Later.md", BODY);
+ const result = await collectWideningCandidates(config, vault, [written], READ_ALL_REFS);
+ expect(result.status).toBe("used");
+ expect(result.candidates.map((c) => c.page)).not.toContain(gone);
+ });
+
+ test("a missing index is a named status with no candidates", async () => {
+ const config = resolveSearchConfig({ vault });
+ const written = writeNote("Notes/Later.md", BODY);
+ const result = await collectWideningCandidates(config, vault, [written], READ_ALL_REFS);
+ expect(result).toEqual({
+ status: "index_unavailable",
+ candidates: [],
+ detail: "INDEX_MISSING",
+ });
+ });
+
+ test("an index that is there but will not open names a different failure", async () => {
+ const config = resolveSearchConfig({ vault });
+ mkdirSync(join(config.dbPath, ".."), { recursive: true });
+ writeFileSync(config.dbPath, "not a database");
+ const written = writeNote("Notes/Later.md", BODY);
+ const result = await collectWideningCandidates(config, vault, [written], READ_ALL_REFS);
+ expect(result).toEqual({
+ status: "index_unavailable",
+ candidates: [],
+ detail: "INDEX_UNREADABLE",
+ });
+ });
+
+ test("a reach predicate that throws is a named status, never a thrown error", async () => {
+ writeNote("Projects/Earlier.md", BODY);
+ const config = await indexed();
+ const written = writeNote("Notes/Later.md", BODY);
+ const result = await collectWideningCandidates(config, vault, [written], () => {
+ throw new RangeError("reach lookup failed");
+ });
+ expect(result).toEqual({
+ status: "index_unavailable",
+ candidates: [],
+ detail: "RangeError",
+ });
+ });
+
+ test("a long body queries only its longest tokens, and a shared one still finds the lookalike", async () => {
+ // Every BODY token is longer than every filler token, so the capped
+ // query keeps all of BODY and drops the shortest fillers.
+ const fillers = Array.from({ length: WIDENING_QUERY_MAX_TERMS * 30 }, (_, i) => `q${i}`);
+ expect(fillers.every((t) => t.length < 6)).toBe(true);
+ const shortOnly = writeNote("Projects/ShortOnly.md", fillers.slice(0, 100).join(" "));
+ const lookalike = writeNote("Projects/Lookalike.md", BODY);
+ const config = await indexed();
+ const written = writeNote("Notes/Long.md", `${BODY} ${fillers.join(" ")}`);
+ const result = await collectWideningCandidates(config, vault, [written], READ_ALL_REFS);
+ expect(result.status).toBe("used");
+ const pages = result.candidates.map((c) => c.page);
+ expect(pages).toContain(lookalike);
+ // Its two- and three-character tokens are past the cap, so no query term reaches it.
+ expect(pages).not.toContain(shortOnly);
+ });
+
+ test("the keyword pull is bounded by the widening top-k", async () => {
+ for (let i = 0; i < NEAR_DUPLICATE_WIDENING_TOP_K + 10; i++) {
+ writeNote(`Projects/AA-Copy-${String(i).padStart(2, "0")}.md`, BODY);
+ }
+ const config = await indexed();
+ const written = writeNote("Notes/Later.md", BODY);
+ const result = await collectWideningCandidates(config, vault, [written], READ_ALL_REFS);
+ expect(result.candidates.length).toBeGreaterThan(0);
+ expect(result.candidates.length).toBeLessThanOrEqual(NEAR_DUPLICATE_WIDENING_TOP_K);
+ });
+
+ test("a rewritten long page's own indexed chunks do not crowd out a lookalike", async () => {
+ // Every section repeats the probe tokens, so each of the page's own
+ // chunks outranks the lookalike under BM25. The filler tokens are
+ // shorter than every probe token, so the capped query keeps the probe.
+ const filler = Array.from({ length: 320 }, (_, i) => `f${i}`).join(" ");
+ const sections = Array.from(
+ { length: NEAR_DUPLICATE_WIDENING_TOP_K + 10 },
+ (_, i) => `## Section ${i}\n\n${BODY} ${BODY} ${BODY}\n\n${filler}`,
+ );
+ const long = writeNote("Notes/Long.md", sections.join("\n\n"));
+ const lookalike = writeNote("Projects/Lookalike.md", BODY);
+ const config = await indexed();
+ const store = await Store.open(config, { mode: "read" });
+ try {
+ const ownId = store.getDocumentIdByPath(long);
+ expect(ownId).not.toBeNull();
+ expect(store.chunksForDocument(ownId!).length).toBeGreaterThan(NEAR_DUPLICATE_WIDENING_TOP_K);
+ } finally {
+ await store.close();
+ }
+ writeNote(long, `${sections.join("\n\n")}\n\nOne more line.`);
+ const result = await collectWideningCandidates(config, vault, [long], READ_ALL_REFS);
+ const pages = result.candidates.map((c) => c.page);
+ expect(pages).toContain(lookalike);
+ expect(pages).not.toContain(long);
+ });
+
+ test("a candidate two written pages both pull in is listed once", async () => {
+ const earlier = writeNote("Projects/Earlier.md", BODY);
+ const config = await indexed();
+ const first = writeNote("Notes/Later.md", BODY);
+ const second = writeNote("Notes/Later-Again.md", BODY);
+ const result = await collectWideningCandidates(config, vault, [first, second], READ_ALL_REFS);
+ expect(result.candidates.filter((c) => c.page === earlier)).toHaveLength(1);
+ });
+
+ test("no embedding provider is imported", () => {
+ const source = readFileSync(
+ join(import.meta.dir, "..", "..", "..", "src", "core", "brain", "page-lint-widening.ts"),
+ "utf8",
+ );
+ expect(source).not.toMatch(/from\s+"[^"]*search\/embeddings\//);
+ });
+});
diff --git a/tests/core/brain/page-lint.test.ts b/tests/core/brain/page-lint.test.ts
index bfefd3d5..a5422509 100644
--- a/tests/core/brain/page-lint.test.ts
+++ b/tests/core/brain/page-lint.test.ts
@@ -11,7 +11,7 @@
*/
import { afterEach, beforeEach, describe, expect, test } from "bun:test";
-import { mkdirSync, mkdtempSync, rmSync, utimesSync, writeFileSync } from "node:fs";
+import { mkdirSync, mkdtempSync, rmSync, unlinkSync, utimesSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
@@ -31,11 +31,20 @@ import {
type NearDuplicateCandidate,
} from "../../../src/core/brain/page-lint.ts";
import { LINT_CONSOLIDATE_KIND } from "../../../src/core/brain/lint-consolidate.ts";
+import {
+ NEAR_DUPLICATE_MIN_TOKENS,
+ NEAR_DUPLICATE_THRESHOLDS,
+ READ_ALL_REFS,
+} from "../../../src/core/brain/near-duplicate.ts";
import { loadSchemaPack } from "../../../src/core/brain/schema-pack.ts";
+import { tokenise } from "../../../src/core/brain/similarity.ts";
import { ARTIFACT_MAX_BYTES } from "../../../src/core/brain/write-session/validate.ts";
let vault: string;
+/** Operator reach: every page readable. */
+const READ_ALL = { readable: READ_ALL_REFS } as const;
+
beforeEach(() => {
vault = mkdtempSync(join(tmpdir(), "o2b-page-lint-"));
for (const dir of ["preferences", "retired", "log", "inbox"]) {
@@ -70,7 +79,7 @@ function writePref(slug: string, fields: Record<string, string> = {}): void {
describe("lintWrittenPages - the clean path", () => {
test("a valid page with no broken links yields nothing to say", () => {
const rel = writeNote("Notes/Clean.md", "---\ntitle: Clean\n---\n\nplain prose\n");
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(report.findings).toEqual([]);
expect(report.total).toBe(0);
expect(report.returned).toBe(0);
@@ -83,14 +92,14 @@ describe("lintWrittenPages - the clean path", () => {
test("a wikilink to a non-Brain note is nobody's business here", () => {
writeNote("Notes/Other.md", "---\ntitle: Other\n---\n\nx\n");
const rel = writeNote("Notes/Links.md", "---\ntitle: Links\n---\n\nsee [[Other]]\n");
- expect(lintWrittenPages(vault, [rel]).findings).toEqual([]);
+ expect(lintWrittenPages(vault, [rel], READ_ALL).findings).toEqual([]);
});
});
describe("lintWrittenPages - findings", () => {
test("a broken Brain wikilink is one warning carrying a next command", () => {
const rel = writeNote("Notes/Broken.md", "---\ntitle: B\n---\n\nsee [[pref-ghost]]\n");
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(report.findings.length).toBe(1);
expect(report.findings[0]).toMatchObject({
severity: "warning",
@@ -109,7 +118,7 @@ describe("lintWrittenPages - findings", () => {
writePref("mid", { merged_into: "pref-canon" });
writePref("dup", { merged_into: "pref-mid" });
const rel = writeNote("Notes/Merged.md", "---\ntitle: M\n---\n\nsee [[pref-dup]]\n");
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(report.findings.length).toBe(1);
expect(report.findings[0]).toMatchObject({
severity: "warning",
@@ -123,7 +132,7 @@ describe("lintWrittenPages - findings", () => {
test("an invalid document reports error findings, ranked ahead of warnings", () => {
const rel = writeNote("Notes/Invalid.md", "no frontmatter at all, see [[pref-ghost]]\n");
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(report.findings.length).toBeGreaterThanOrEqual(2);
expect(report.findings[0]!.severity).toBe("error");
expect(report.findings[0]!.code).toBe("frontmatter-missing");
@@ -133,7 +142,7 @@ describe("lintWrittenPages - findings", () => {
test("a declared type outside the schema pack is an error finding", () => {
const rel = writeNote("Notes/Typed.md", "---\ntitle: T\ntype: not-a-declared-type\n---\n\nx\n");
- const codes = lintWrittenPages(vault, [rel]).findings.map((f) => f.code);
+ const codes = lintWrittenPages(vault, [rel], READ_ALL).findings.map((f) => f.code);
expect(codes).toContain("schema-type-unknown");
});
});
@@ -142,7 +151,7 @@ describe("lintWrittenPages - bounds", () => {
test("a page over the artifact byte cap is skipped with a reason, never dropped", () => {
const filler = "x".repeat(ARTIFACT_MAX_BYTES + 1024);
const rel = writeNote("Notes/Huge.md", `---\ntitle: H\n---\n\n${filler}\n`);
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(report.findings).toEqual([]);
expect(report.skipped).toEqual([
{ page: rel, reason: PAGE_LINT_SKIP_REASON.overByteCap, detail: expect.any(String) },
@@ -156,7 +165,7 @@ describe("lintWrittenPages - bounds", () => {
(_, i) => `see [[pref-ghost-${i}]]`,
).join("\n");
const rel = writeNote("Notes/Many.md", `---\ntitle: M\n---\n\n${links}\n`);
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(report.total).toBe(PAGE_LINT_MAX_FINDINGS + 5);
expect(report.returned).toBe(PAGE_LINT_MAX_FINDINGS);
expect(report.findings.length).toBe(PAGE_LINT_MAX_FINDINGS);
@@ -164,7 +173,7 @@ describe("lintWrittenPages - bounds", () => {
});
test("an unreadable page is skipped with its own reason", () => {
- const report = lintWrittenPages(vault, ["Notes/NeverWritten.md"]);
+ const report = lintWrittenPages(vault, ["Notes/NeverWritten.md"], READ_ALL);
expect(report.findings).toEqual([]);
expect(report.skipped.map((s) => s.reason)).toEqual([PAGE_LINT_SKIP_REASON.unreadable]);
});
@@ -179,7 +188,7 @@ describe("lintWrittenPages - bounds", () => {
*/
describe("lintWrittenPages - nothing of the operator's filesystem crosses the wire", () => {
test("an unreadable page carries the errno CODE, not the kernel's sentence", () => {
- const report = lintWrittenPages(vault, ["Notes/NeverWritten.md"]);
+ const report = lintWrittenPages(vault, ["Notes/NeverWritten.md"], READ_ALL);
expect(report.skipped).toEqual([
{
page: "Notes/NeverWritten.md",
@@ -193,14 +202,14 @@ describe("lintWrittenPages - nothing of the operator's filesystem crosses the wi
test("a lint that cannot start names its errno code and no path", () => {
mkdirSync(join(vault, "Brain", "_brain.yaml"), { recursive: true });
const rel = writeNote("Notes/Any.md", "---\ntitle: A\n---\n\nx\n");
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(report.unavailable!.message).not.toContain(vault);
expect(report.unavailable!.message).toMatch(/could not start: [A-Za-z][A-Za-z0-9_]*$/);
});
test("an over-cap skip is stated in bytes, which name nothing on disk", () => {
const rel = writeNote("Notes/Huge.md", "x".repeat(ARTIFACT_MAX_BYTES + 8));
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(report.skipped[0]!.reason).toBe(PAGE_LINT_SKIP_REASON.overByteCap);
expect(JSON.stringify(report)).not.toContain(vault);
});
@@ -222,6 +231,7 @@ describe("lintPagesWithContext - one page's failure is not the report's", () =>
basenames: new Set<string>(),
vocabulary: loadSchemaPack(vault).vocabulary,
nearDuplicateCandidates: new Map<string, ReadonlyArray<NearDuplicateCandidate>>(),
+ readable: READ_ALL_REFS,
mergedLinks: {
resolve() {
throw reason;
@@ -266,7 +276,7 @@ describe("lintWrittenPages - failure is named, never absent", () => {
// EISDIR, which is a failure of the LINT, not of the page.
mkdirSync(join(vault, "Brain", "_brain.yaml"), { recursive: true });
const rel = writeNote("Notes/Any.md", "---\ntitle: A\n---\n\nx\n");
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(report.unavailable).toMatchObject({ code: PAGE_LINT_UNAVAILABLE_CODE });
expect(report.unavailable!.message.length).toBeGreaterThan(0);
expect(report.total).toBe(0);
@@ -327,12 +337,37 @@ describe("lintWrittenPages - near-duplicate findings", () => {
test("NEAR_DUPLICATE_JACCARD is the designed threshold, a named constant", () => {
expect(NEAR_DUPLICATE_JACCARD).toBe(0.8);
+ expect(NEAR_DUPLICATE_JACCARD).toBe(NEAR_DUPLICATE_THRESHOLDS.writeHint);
+ });
+
+ test("a same-directory finding is byte-identical to the shipped receipt", () => {
+ const first = writeNote("Notes/Alpha.md", alphaNote("Alpha"));
+ const rel = writeNote("Notes/Beta.md", alphaNote("Beta"));
+ expect(nearDuplicateFindings(lintWrittenPages(vault, [rel], READ_ALL))).toEqual([
+ {
+ severity: "warning",
+ code: NEAR_DUPLICATE_CODE,
+ page: rel,
+ path: first,
+ message: `body resembles [[${first}]] jaccard=1.000 (threshold 0.8)`,
+ },
+ ]);
+ });
+
+ test("same-directory stubs below the kernel's token floor are not reported", () => {
+ // Three tokens each: identical bodies, a jaccard=1.000 finding before the
+ // shared kernel's minimum token count.
+ const stub = "alpha beta gamma";
+ expect(tokenise(stub).size).toBe(NEAR_DUPLICATE_MIN_TOKENS - 1);
+ writeNote("Notes/Alpha.md", `---\ntitle: Alpha\n---\n\n${stub}\n`);
+ const rel = writeNote("Notes/Beta.md", `---\ntitle: Beta\n---\n\n${stub}\n`);
+ expect(nearDuplicateFindings(lintWrittenPages(vault, [rel], READ_ALL))).toEqual([]);
});
test("a write closely matching a same-directory same-scope page yields the finding", () => {
const first = writeNote("Notes/Alpha.md", alphaNote("Alpha"));
const rel = writeNote("Notes/Beta.md", alphaNote("Beta"));
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
const findings = nearDuplicateFindings(report);
expect(findings).toHaveLength(1);
expect(findings[0]).toMatchObject({
@@ -351,7 +386,7 @@ describe("lintWrittenPages - near-duplicate findings", () => {
// 4 shared tokens over a 5-token union is exactly 4/5 = 0.8.
writeNote("Notes/Alpha.md", `---\ntitle: Alpha\n---\n\n${ALPHA_BODY}\n`);
const rel = writeNote("Notes/Beta.md", "---\ntitle: Beta\n---\n\nalpha beta gamma delta\n");
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
const findings = nearDuplicateFindings(report);
expect(findings).toHaveLength(1);
expect(findings[0]!.message).toContain("jaccard=0.800");
@@ -360,7 +395,7 @@ describe("lintWrittenPages - near-duplicate findings", () => {
test("a below-threshold resemblance yields nothing", () => {
writeNote("Notes/Alpha.md", alphaNote("Alpha"));
const rel = writeNote("Notes/Beta.md", "---\ntitle: Beta\n---\n\nalpha zeta eta theta iota\n");
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(report.findings).toEqual([]);
expect(pageLintField(report)).toEqual({});
});
@@ -368,21 +403,21 @@ describe("lintWrittenPages - near-duplicate findings", () => {
test("a same-body page in a different directory is not a candidate", () => {
writeNote("Elsewhere/Alpha.md", alphaNote("Alpha"));
const rel = writeNote("Notes/Beta.md", alphaNote("Beta"));
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(report.findings).toEqual([]);
});
test("a same-body page in a different scope bucket is not a candidate", () => {
writeNote("Notes/Alpha.md", alphaNote("Alpha", ["session: atlas"]));
const rel = writeNote("Notes/Beta.md", alphaNote("Beta"));
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(report.findings).toEqual([]);
});
test("a same-body page in the SAME scope bucket is a candidate", () => {
writeNote("Notes/Alpha.md", alphaNote("Alpha", ["session: atlas"]));
const rel = writeNote("Notes/Beta.md", alphaNote("Beta", ["session: atlas"]));
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(nearDuplicateFindings(report)).toHaveLength(1);
});
@@ -391,7 +426,7 @@ describe("lintWrittenPages - near-duplicate findings", () => {
// be a REAL difference - it is a scope axis - so it must not appear here.)
writeNote("Notes/Alpha.md", `---\ntitle: Alpha\ntype: note\n---\n\n${ALPHA_BODY}\n`);
const rel = writeNote("Notes/Beta.md", alphaNote("Beta"));
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(nearDuplicateFindings(report)).toHaveLength(1);
});
@@ -401,7 +436,7 @@ describe("lintWrittenPages - near-duplicate findings", () => {
"Notes/Beta.md",
"---\ntitle: Same\ntype: note\n---\n\nanother kind entirely\n",
);
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(report.findings).toEqual([]);
});
@@ -414,7 +449,7 @@ describe("lintWrittenPages - near-duplicate findings", () => {
utimesSync(join(vault, rel), at, at);
}
const rel = writeNote("Notes/Beta.md", alphaNote("Beta"));
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
// The written page is the newest sibling, so the cap leaves out the oldest fillers.
expect(report.candidates_skipped).toBe(extra + 1);
expect(pageLintField(report)).toHaveProperty("lint");
@@ -422,14 +457,14 @@ describe("lintWrittenPages - near-duplicate findings", () => {
test("the written page never resembles itself", () => {
const rel = writeNote("Notes/Alpha.md", alphaNote("Alpha"));
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(report.findings).toEqual([]);
});
test("two near-identical pages written in one call report each other", () => {
const first = writeNote("Notes/Alpha.md", alphaNote("Alpha"));
const second = writeNote("Notes/Beta.md", alphaNote("Beta"));
- const report = lintWrittenPages(vault, [first, second]);
+ const report = lintWrittenPages(vault, [first, second], READ_ALL);
const pairs = nearDuplicateFindings(report)
.map((f) => `${f.page} -> ${f.path}`)
.toSorted();
@@ -442,7 +477,7 @@ describe("lintWrittenPages - near-duplicate findings", () => {
// write-conflict advisory takes toward corrupt preference files.
mkdirSync(join(vault, "Notes", "Stuck.md"), { recursive: true });
const rel = writeNote("Notes/Beta.md", alphaNote("Beta"));
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(report.unavailable).toBeUndefined();
expect(report.findings).toEqual([]);
// Excluded by name, with the errno code, never silently.
@@ -453,7 +488,7 @@ describe("lintWrittenPages - near-duplicate findings", () => {
const filler = `${ALPHA_BODY} `.repeat(ARTIFACT_MAX_BYTES / ALPHA_BODY.length + 1);
writeNote("Notes/Alpha.md", `---\ntitle: Alpha\n---\n\n${filler}\n`);
const rel = writeNote("Notes/Beta.md", alphaNote("Beta"));
- const report = lintWrittenPages(vault, [rel]);
+ const report = lintWrittenPages(vault, [rel], READ_ALL);
expect(report.unavailable).toBeUndefined();
expect(report.findings).toEqual([]);
// The skip list stays reserved for WRITTEN pages.
@@ -471,7 +506,7 @@ describe("lintWrittenPages - near-duplicate findings", () => {
writeFileSync(join(outside, "Escaped.md"), alphaNoteText, "utf8");
// vault/../Escaped.md IS the file written above - the same bytes as
// its neighbor, yet no near-duplicate may be reported about them.
- const report = lintWrittenPages(vault, ["../Escaped.md"]);
+ const report = lintWrittenPages(vault, ["../Escaped.md"], READ_ALL);
expect(report.unavailable).toBeUndefined();
expect(report.findings.filter((f) => f.code === NEAR_DUPLICATE_CODE)).toEqual([]);
} finally {
@@ -480,6 +515,216 @@ describe("lintWrittenPages - near-duplicate findings", () => {
});
});
+/**
+ * The receipt answers at the caller's reach. A sibling the caller may not
+ * read is dropped before it is scored and before any census counts it, so
+ * neither a finding nor `candidates_skipped` nor `candidates_unreadable`
+ * can disclose that it exists.
+ */
+describe("lintWrittenPages - near-duplicate candidates at the caller's reach", () => {
+ const BODY = "alpha beta gamma delta epsilon";
+ const HIDDEN_PREFIX = "Notes/Hidden-";
+ const withholdHidden = { readable: (rel: string) => !rel.startsWith(HIDDEN_PREFIX) } as const;
+
+ function note(title: string): string {
+ return `---\ntitle: ${title}\n---\n\n${BODY}\n`;
+ }
+
+ test("a withheld near-identical sibling is neither scored nor named", () => {
+ writeNote("Notes/Hidden-Alpha.md", note("Alpha"));
+ const rel = writeNote("Notes/Beta.md", note("Beta"));
+ const report = lintWrittenPages(vault, [rel], withholdHidden);
+ expect(report.findings).toEqual([]);
+ expect(JSON.stringify(report)).not.toContain("Hidden-Alpha");
+ // A clean write still contributes no key at all.
+ expect(pageLintField(report)).toEqual({});
+ });
+
+ test("the same sibling at full reach is reported, so the filter is what hid it", () => {
+ const sibling = writeNote("Notes/Hidden-Alpha.md", note("Alpha"));
+ const rel = writeNote("Notes/Beta.md", note("Beta"));
+ const findings = nearDuplicateFindings(lintWrittenPages(vault, [rel], READ_ALL));
+ expect(findings.map((f) => f.path)).toEqual([sibling]);
+ });
+
+ test("a withheld sibling that cannot be read is not counted as unreadable", () => {
+ mkdirSync(join(vault, "Notes", "Hidden-Stuck.md"), { recursive: true });
+ const rel = writeNote("Notes/Beta.md", note("Beta"));
+ const report = lintWrittenPages(vault, [rel], withholdHidden);
+ expect(report.candidates_unreadable).toBeUndefined();
+ expect(pageLintField(report)).toEqual({});
+ });
+
+ test("withheld siblings never count toward the candidate cap", () => {
+ const oldest = Date.parse("2026-01-01T00:00:00Z");
+ for (let i = 0; i < NEAR_DUPLICATE_MAX_CANDIDATES + 3; i++) {
+ const rel = writeNote(`${HIDDEN_PREFIX}${i}.md`, `---\ntitle: H${i}\n---\n\nfiller ${i}\n`);
+ const at = new Date(oldest + i * 1000);
+ utimesSync(join(vault, rel), at, at);
+ }
+ const rel = writeNote("Notes/Beta.md", note("Beta"));
+ const report = lintWrittenPages(vault, [rel], withholdHidden);
+ expect(report.candidates_skipped).toBeUndefined();
+ expect(pageLintField(report)).toEqual({});
+ });
+
+ test("the predicate is asked about vault-relative paths", () => {
+ writeNote("Notes/Alpha.md", note("Alpha"));
+ const rel = writeNote("Notes/Beta.md", note("Beta"));
+ const asked: string[] = [];
+ lintWrittenPages(vault, [rel], {
+ readable: (path) => {
+ asked.push(path);
+ return true;
+ },
+ });
+ expect([...new Set(asked)].toSorted()).toEqual(["Notes/Alpha.md", "Notes/Beta.md"]);
+ });
+});
+
+/**
+ * The wikilink and merged-link findings answer at the caller's reach too. A
+ * Brain page the caller may not read must produce the receipt a vault
+ * without that page produces, byte for byte, whichever finding it would
+ * otherwise have shaped.
+ */
+describe("lintWrittenPages - link findings at the caller's reach", () => {
+ const prefPath = (slug: string) => `Brain/preferences/pref-${slug}.md`;
+ const withhold = (slug: string) => ({ readable: (rel: string) => rel !== prefPath(slug) });
+
+ /** The report with `slug` withheld, then the report with `slug` deleted. */
+ function withheldThenAbsent(slug: string, rel: string): [string, string] {
+ const withheld = JSON.stringify(lintWrittenPages(vault, [rel], withhold(slug)));
+ unlinkSync(join(vault, prefPath(slug)));
+ return [withheld, JSON.stringify(lintWrittenPages(vault, [rel], READ_ALL))];
+ }
+
+ test("a link to a withheld page is reported as broken, as an absent one is", () => {
+ writePref("hidden");
+ const rel = writeNote("Notes/Links.md", "---\ntitle: L\n---\n\nsee [[pref-hidden]]\n");
+ expect(lintWrittenPages(vault, [rel], READ_ALL).findings).toEqual([]);
+ const [withheld, absent] = withheldThenAbsent("hidden", rel);
+ expect(withheld).toBe(absent);
+ expect(withheld).toContain("broken-wikilink");
+ });
+
+ test("a withheld merged-away target is not followed into its canonical", () => {
+ writePref("canon");
+ writePref("dup", { merged_into: "pref-canon" });
+ const rel = writeNote("Notes/Merged.md", "---\ntitle: M\n---\n\nsee [[pref-dup]]\n");
+ const [withheld, absent] = withheldThenAbsent("dup", rel);
+ expect(withheld).toBe(absent);
+ expect(withheld).not.toContain("pref-canon");
+ });
+
+ test("a merge chain ends at a withheld hop and names nothing past it", () => {
+ writePref("canon");
+ writePref("mid", { merged_into: "pref-canon" });
+ writePref("dup", { merged_into: "pref-mid" });
+ const rel = writeNote("Notes/Merged.md", "---\ntitle: M\n---\n\nsee [[pref-dup]]\n");
+ const [withheld, absent] = withheldThenAbsent("mid", rel);
+ expect(withheld).toBe(absent);
+ expect(withheld).toContain("merged into pref-mid");
+ expect(withheld).not.toContain("pref-canon");
+ });
+});
+
+/**
+ * Widening: candidates from outside the written page's directory, handed in
+ * by the keyword-index collector. They are scored through the same kernel
+ * under the same scope-bucket rule; a cross-directory finding keeps the
+ * `near-duplicate` code and names its method, and the report states whether
+ * widening ran only when the caller says it was asked to.
+ */
+describe("lintWrittenPages - widened near-duplicate candidates", () => {
+ const BODY = "alpha beta gamma delta epsilon";
+
+ function note(title: string): string {
+ return `---\ntitle: ${title}\n---\n\n${BODY}\n`;
+ }
+
+ function candidate(page: string, scopeKey = ""): NearDuplicateCandidate {
+ return { page, scopeKey, tokens: new Set(BODY.split(" ")) };
+ }
+
+ test("a cross-directory candidate yields the same code with the method in the message", () => {
+ const rel = writeNote("Notes/Beta.md", note("Beta"));
+ const report = lintWrittenPages(vault, [rel], {
+ ...READ_ALL,
+ extraCandidates: [candidate("Projects/Alpha.md")],
+ widening: "used",
+ });
+ expect(nearDuplicateFindings(report)).toEqual([
+ {
+ severity: "warning",
+ code: NEAR_DUPLICATE_CODE,
+ page: rel,
+ path: "Projects/Alpha.md",
+ message:
+ "body resembles [[Projects/Alpha.md]] jaccard=1.000 (threshold 0.8) method=lexical",
+ },
+ ]);
+ expect(report.widening).toBe("used");
+ });
+
+ test("a widened candidate already in the directory is reported once, as shipped", () => {
+ const first = writeNote("Notes/Alpha.md", note("Alpha"));
+ const rel = writeNote("Notes/Beta.md", note("Beta"));
+ const findings = nearDuplicateFindings(
+ lintWrittenPages(vault, [rel], {
+ ...READ_ALL,
+ extraCandidates: [candidate(first)],
+ widening: "used",
+ }),
+ );
+ expect(findings.map((f) => f.message)).toEqual([
+ `body resembles [[${first}]] jaccard=1.000 (threshold 0.8)`,
+ ]);
+ });
+
+ test("a widened candidate in another scope bucket is not reported", () => {
+ const rel = writeNote("Notes/Beta.md", note("Beta"));
+ const report = lintWrittenPages(vault, [rel], {
+ ...READ_ALL,
+ extraCandidates: [candidate("Projects/Alpha.md", "session=atlas")],
+ widening: "used",
+ });
+ expect(report.findings).toEqual([]);
+ });
+
+ test("a widened candidate outside the caller's reach is not reported", () => {
+ const rel = writeNote("Notes/Beta.md", note("Beta"));
+ const report = lintWrittenPages(vault, [rel], {
+ readable: (page) => page !== "Projects/Alpha.md",
+ extraCandidates: [candidate("Projects/Alpha.md")],
+ widening: "used",
+ });
+ expect(report.findings).toEqual([]);
+ });
+
+ test("the widening status is absent unless passed, and a clean used widening says nothing", () => {
+ const rel = writeNote("Notes/Beta.md", note("Beta"));
+ const off = lintWrittenPages(vault, [rel], READ_ALL);
+ expect("widening" in off).toBe(false);
+ const used = lintWrittenPages(vault, [rel], { ...READ_ALL, widening: "used" });
+ expect(used.widening).toBe("used");
+ expect(pageLintField(used)).toEqual({});
+ });
+
+ test("an unavailable index is stated on the receipt", () => {
+ const rel = writeNote("Notes/Beta.md", note("Beta"));
+ const report = lintWrittenPages(vault, [rel], {
+ ...READ_ALL,
+ extraCandidates: [],
+ widening: "index_unavailable",
+ wideningDetail: "INDEX_LOCKED",
+ });
+ expect(pageLintField(report)).toMatchObject({
+ [PAGE_LINT_KEY]: { widening: "index_unavailable", widening_detail: "INDEX_LOCKED" },
+ });
+ });
+});
+
/**
* A caller already holding the indexes can lint with them - including the
* near-duplicate candidate index, so the seam stays honest about what is
@@ -502,6 +747,7 @@ describe("lintPagesWithContext - caller-supplied near-duplicate candidates", ()
basenames: new Set<string>(),
vocabulary: loadSchemaPack(vault).vocabulary,
nearDuplicateCandidates: new Map([["Notes", candidates]]),
+ readable: READ_ALL_REFS,
mergedLinks: { resolve: () => ({ canonical: null, unresolvable: null }) },
};
const report = lintPagesWithContext(vault, ctx, [rel]);
@@ -526,6 +772,7 @@ describe("lintPagesWithContext - caller-supplied near-duplicate candidates", ()
basenames: new Set<string>(),
vocabulary: loadSchemaPack(vault).vocabulary,
nearDuplicateCandidates: new Map([["Notes", candidates]]),
+ readable: READ_ALL_REFS,
mergedLinks: { resolve: () => ({ canonical: null, unresolvable: null }) },
};
expect(lintPagesWithContext(vault, ctx, [rel]).findings).toEqual([]);
diff --git a/tests/core/brain/retire-siblings.test.ts b/tests/core/brain/retire-siblings.test.ts
new file mode 100644
index 00000000..bb5a4777
--- /dev/null
+++ b/tests/core/brain/retire-siblings.test.ts
@@ -0,0 +1,249 @@
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { appendApplyEvidence } from "../../../src/core/brain/apply-evidence.ts";
+import { dream } from "../../../src/core/brain/dream.ts";
+import { bootstrapBrain } from "../../../src/core/brain/init.ts";
+
+import {
+ NEAR_DUPLICATE_CANDIDATE_CAP,
+ READ_ALL_REFS,
+} from "../../../src/core/brain/near-duplicate.ts";
+import {
+ planRetireSiblings,
+ RETIRE_SIBLING_TRIGGER_REASONS,
+ retireSiblingPool,
+ type RetireSibling,
+} from "../../../src/core/brain/retire-siblings.ts";
+import { writePreference } from "../../../src/core/brain/preference.ts";
+import { BRAIN_RETIRED_REASON, type BrainRetiredReason } from "../../../src/core/brain/types.ts";
+import { atomicWriteFileSync } from "../../../src/core/fs-atomic.ts";
+import { digestVaultTree } from "../../helpers/vault-digest.ts";
+
+const RULE = "always run the formatter before every commit in this repository";
+const PARAPHRASE = "always run the formatter before each commit in this repository";
+const UNRELATED = "prefer tabs over spaces inside every makefile you touch";
+
+const hideSibling = (ref: string) => ref !== "pref-paraphrase";
+const hideRetiring = (ref: string) => ref !== "pref-old";
+const NO_GATE = { readable: READ_ALL_REFS, gated: new Set<string>() };
+
+function retiring(reason: BrainRetiredReason, id = "pref-old") {
+ return [{ id, principle: RULE, reason }];
+}
+
+const ACTIVE = [
+ { id: "pref-paraphrase", principle: PARAPHRASE },
+ { id: "pref-unrelated", principle: UNRELATED },
+];
+
+describe("planRetireSiblings", () => {
+ test("only the four context-driven reasons nominate", () => {
+ expect([...RETIRE_SIBLING_TRIGGER_REASONS].toSorted()).toEqual([
+ "quarantine-violated",
+ "rebutted",
+ "superseded-by-context",
+ "user-rejected",
+ ]);
+ for (const reason of RETIRE_SIBLING_TRIGGER_REASONS) {
+ const out = planRetireSiblings(ACTIVE, retiring(reason), NO_GATE);
+ expect(out).toEqual([
+ { retiring_id: "pref-old", sibling_id: "pref-paraphrase", score: 0.818, method: "lexical" },
+ ]);
+ }
+ });
+
+ test("decay reasons and merged-into never nominate", () => {
+ for (const reason of [
+ BRAIN_RETIRED_REASON.staleNoEvidence,
+ BRAIN_RETIRED_REASON.expiredUnconfirmed,
+ BRAIN_RETIRED_REASON.mergedInto,
+ ]) {
+ expect(planRetireSiblings(ACTIVE, retiring(reason), NO_GATE)).toEqual([]);
+ }
+ });
+
+ test("siblings of a gated id are dropped", () => {
+ const out = planRetireSiblings(ACTIVE, retiring(BRAIN_RETIRED_REASON.rebutted), {
+ readable: READ_ALL_REFS,
+ gated: new Set(["pref-old"]),
+ });
+ expect(out).toEqual([]);
+ });
+
+ test("an unreadable sibling or retiring id is never listed", () => {
+ const reason = BRAIN_RETIRED_REASON.rebutted;
+ const gated = new Set<string>();
+ expect(planRetireSiblings(ACTIVE, retiring(reason), { readable: hideSibling, gated })).toEqual(
+ [],
+ );
+ expect(planRetireSiblings(ACTIVE, retiring(reason), { readable: hideRetiring, gated })).toEqual(
+ [],
+ );
+ });
+
+ test("merge-resolved pages are excluded from the pool", () => {
+ const pool = retireSiblingPool([
+ { id: "pref-paraphrase", principle: PARAPHRASE, merged_into: "pref-canonical" },
+ { id: "pref-live", principle: PARAPHRASE },
+ ]);
+ expect(pool).toEqual([{ id: "pref-live", principle: PARAPHRASE }]);
+ const out = planRetireSiblings(pool, retiring(BRAIN_RETIRED_REASON.rebutted), NO_GATE);
+ expect(out.map((s) => s.sibling_id)).toEqual(["pref-live"]);
+ });
+
+ test("preferences from other topics and scopes are compared", () => {
+ // The planner takes no topic or scope: the whole active set is one pool.
+ const active = [{ id: "pref-other-topic-slug", principle: PARAPHRASE }];
+ const out = planRetireSiblings(active, retiring(BRAIN_RETIRED_REASON.rebutted), NO_GATE);
+ expect(out.map((s) => s.sibling_id)).toEqual(["pref-other-topic-slug"]);
+ });
+
+ test("the retiring set is never its own sibling and siblings are not cascaded", () => {
+ const secondHop = `${PARAPHRASE} today`;
+ const active = [
+ { id: "pref-old", principle: RULE },
+ { id: "pref-paraphrase", principle: PARAPHRASE },
+ { id: "pref-second-hop", principle: `each ${secondHop} quickly` },
+ ];
+ const out = planRetireSiblings(active, retiring(BRAIN_RETIRED_REASON.rebutted), NO_GATE);
+ expect(out.map((s) => s.sibling_id)).toEqual(["pref-paraphrase"]);
+ });
+
+ test("another retiring preference is not offered as a sibling", () => {
+ const both = [
+ { id: "pref-old", principle: RULE, reason: BRAIN_RETIRED_REASON.rebutted },
+ {
+ id: "pref-paraphrase",
+ principle: PARAPHRASE,
+ reason: BRAIN_RETIRED_REASON.staleNoEvidence,
+ },
+ ];
+ expect(planRetireSiblings(ACTIVE, both, NO_GATE)).toEqual([]);
+ });
+
+ test("a sibling sorting past the near-duplicate candidate cap is still scored", () => {
+ const fillers = Array.from({ length: NEAR_DUPLICATE_CANDIDATE_CAP + 50 }, (_, i) => ({
+ id: `pref-filler-${String(i).padStart(4, "0")}`,
+ principle: UNRELATED,
+ }));
+ const active = [...fillers, { id: "pref-zzz-dup", principle: RULE }];
+ const out = planRetireSiblings(active, retiring(BRAIN_RETIRED_REASON.rebutted), NO_GATE);
+ expect(out.map((s) => s.sibling_id)).toEqual(["pref-zzz-dup"]);
+ });
+
+ test("the output is stable across runs and input orders", () => {
+ const active = [
+ { id: "pref-b", principle: PARAPHRASE },
+ { id: "pref-a", principle: PARAPHRASE },
+ { id: "pref-c", principle: RULE },
+ ];
+ const retires = [
+ { id: "pref-y", principle: RULE, reason: BRAIN_RETIRED_REASON.rebutted },
+ { id: "pref-x", principle: RULE, reason: BRAIN_RETIRED_REASON.userRejected },
+ ];
+ const first = planRetireSiblings(active, retires, NO_GATE);
+ const again = planRetireSiblings(active.toReversed(), retires.toReversed(), NO_GATE);
+ expect(again).toEqual(first);
+ expect(first.map((s) => `${s.retiring_id}>${s.sibling_id}`)).toEqual([
+ "pref-x>pref-c",
+ "pref-x>pref-a",
+ "pref-x>pref-b",
+ "pref-y>pref-c",
+ "pref-y>pref-a",
+ "pref-y>pref-b",
+ ]);
+ });
+});
+
+const FLAG_ENV = "OPEN_SECOND_BRAIN_NEAR_DUPLICATE_RETIRE_SIBLINGS_ENABLED";
+const NOW = new Date("2026-06-10T12:00:00Z");
+
+describe("dream retire_siblings", () => {
+ let vault: string;
+ let configHome: string;
+ let savedFlag: string | undefined;
+
+ beforeEach(() => {
+ savedFlag = process.env[FLAG_ENV];
+ delete process.env[FLAG_ENV];
+ vault = mkdtempSync(join(tmpdir(), "o2b-retire-siblings-vault-"));
+ configHome = mkdtempSync(join(tmpdir(), "o2b-retire-siblings-cfg-"));
+ const configPath = join(configHome, "config.yaml");
+ atomicWriteFileSync(configPath, `vault: ${vault}\n`);
+ bootstrapBrain(vault, { configPath });
+ seedPreference("old", "formatting", RULE);
+ seedPreference("paraphrase", "commit-hygiene", PARAPHRASE);
+ seedPreference("unrelated", "makefiles", UNRELATED);
+ appendApplyEvidence(
+ vault,
+ { pref_id: "pref-old", artifact: "[[AA-artifact]]", result: "outdated", agent: "test-agent" },
+ { now: new Date("2026-06-09T12:00:00Z") },
+ );
+ });
+
+ afterEach(() => {
+ if (savedFlag === undefined) delete process.env[FLAG_ENV];
+ else process.env[FLAG_ENV] = savedFlag;
+ rmSync(vault, { recursive: true, force: true });
+ rmSync(configHome, { recursive: true, force: true });
+ });
+
+ function seedPreference(slug: string, topic: string, principle: string): void {
+ writePreference(vault, {
+ slug,
+ topic,
+ principle,
+ created_at: "2026-06-01T00:00:00Z",
+ confirmed_at: "2026-06-02T00:00:00Z",
+ unconfirmed_until: "2026-06-15T00:00:00Z",
+ status: "confirmed",
+ evidenced_by: ["[[sig-1]]"],
+ applied_count: 1,
+ violated_count: 0,
+ last_evidence_at: "2026-06-02T00:00:00Z",
+ confidence: "low",
+ });
+ }
+
+ const EXPECTED: RetireSibling[] = [
+ { retiring_id: "pref-old", sibling_id: "pref-paraphrase", score: 0.818, method: "lexical" },
+ ];
+
+ test("with the key on, the dry run and the real run report the same siblings", () => {
+ process.env[FLAG_ENV] = "1";
+ const before = digestVaultTree(vault);
+ const preview = dream(vault, { dryRun: true, now: NOW });
+ expect(digestVaultTree(vault)).toBe(before);
+ expect(preview.retired).toEqual([{ id: "ret-old", reason: "superseded-by-context" }]);
+ expect(preview.retire_siblings).toEqual(EXPECTED);
+ const real = dream(vault, { now: NOW });
+ expect(real.retire_siblings).toEqual(EXPECTED);
+ });
+
+ test("with the key off, the field is absent and the summary is otherwise unchanged", () => {
+ const off = dream(vault, { dryRun: true, now: NOW });
+ expect("retire_siblings" in off).toBe(false);
+ process.env[FLAG_ENV] = "1";
+ const { retire_siblings: _siblings, ...on } = dream(vault, { dryRun: true, now: NOW });
+ expect(JSON.stringify(on)).toBe(JSON.stringify(off));
+ });
+
+ test("a sibling of a gated retire does not appear, in the dry run or the real run", () => {
+ process.env[FLAG_ENV] = "1";
+ const yamlPath = join(vault, "Brain", "_brain.yaml");
+ const yaml = readFileSync(yamlPath, "utf8").replace(
+ "# confirmed_evidence_min_threshold: 3",
+ "confirmed_evidence_min_threshold: 50",
+ );
+ writeFileSync(yamlPath, yaml);
+ const preview = dream(vault, { dryRun: true, now: NOW });
+ expect(preview.retired).toEqual([{ id: "ret-old", reason: "superseded-by-context" }]);
+ expect("retire_siblings" in preview).toBe(false);
+ const real = dream(vault, { now: NOW });
+ expect(real.gated_retires.map((g) => g.pref_id)).toEqual(["pref-old"]);
+ expect("retire_siblings" in real).toBe(false);
+ });
+});
diff --git a/tests/core/brain/review-candidates.test.ts b/tests/core/brain/review-candidates.test.ts
index 5730dda5..3831506c 100644
--- a/tests/core/brain/review-candidates.test.ts
+++ b/tests/core/brain/review-candidates.test.ts
@@ -5,15 +5,31 @@
*/
import { afterEach, beforeEach, describe, expect, test } from "bun:test";
-import { existsSync, mkdtempSync, readdirSync, rmSync } from "node:fs";
+import {
+ existsSync,
+ mkdirSync,
+ mkdtempSync,
+ readdirSync,
+ readFileSync,
+ rmSync,
+ writeFileSync,
+} from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
+import { appendApplyEvidence } from "../../../src/core/brain/apply-evidence.ts";
import { dream, DreamPreviewReadableError } from "../../../src/core/brain/dream.ts";
import { bootstrapBrain } from "../../../src/core/brain/init.ts";
import { dreamRunsDir } from "../../../src/core/brain/paths.ts";
+import { writePreference } from "../../../src/core/brain/preference.ts";
import { buildReviewCandidates } from "../../../src/core/brain/review-candidates.ts";
+import type { RetireSibling } from "../../../src/core/brain/retire-siblings.ts";
import { atomicWriteFileSync } from "../../../src/core/fs-atomic.ts";
+import { indexVault } from "../../../src/core/search/indexer.ts";
+import { Store } from "../../../src/core/search/store.ts";
+import type { ResolvedSearchConfig } from "../../../src/core/search/types.ts";
+import { makeConfig } from "../../helpers/search-fixtures.ts";
+import { sqliteVecLoadable } from "../../helpers/sqlite-vec.ts";
let vault: string;
let configHome: string;
@@ -71,3 +87,189 @@ describe("buildReviewCandidates", () => {
expect(() => dream(vault, { previewReadable: () => true })).toThrow(DreamPreviewReadableError);
});
});
+
+const FLAG_ENV = "OPEN_SECOND_BRAIN_NEAR_DUPLICATE_RETIRE_SIBLINGS_ENABLED";
+const NOW = new Date("2026-06-10T12:00:00Z");
+const MODEL = "rc-model";
+const DIMENSION = 4;
+const RULE = "always run the formatter before every commit in this repository";
+const PARAPHRASE = "always run the formatter before each commit in this repository";
+const SEMANTIC = "keep source tidy with the code styler ahead of saving changes";
+const UNRELATED = "prefer tabs over spaces inside every makefile you touch";
+
+function seedPreference(slug: string, topic: string, principle: string): void {
+ writePreference(vault, {
+ slug,
+ topic,
+ principle,
+ created_at: "2026-06-01T00:00:00Z",
+ confirmed_at: "2026-06-02T00:00:00Z",
+ unconfirmed_until: "2026-06-15T00:00:00Z",
+ status: "confirmed",
+ evidenced_by: ["[[sig-1]]"],
+ applied_count: 1,
+ violated_count: 0,
+ last_evidence_at: "2026-06-02T00:00:00Z",
+ confidence: "low",
+ });
+}
+
+function searchConfig(): ResolvedSearchConfig {
+ return makeConfig({
+ vault,
+ dbPath: join(vault, ".open-second-brain", "brain.sqlite"),
+ semantic: {
+ enabled: true,
+ baseUrl: "https://x/v1",
+ model: MODEL,
+ apiKey: "k",
+ dimension: DIMENSION,
+ },
+ });
+}
+
+async function plant(config: ResolvedSearchConfig, bySlug: Record<string, number[]>) {
+ const store = await Store.open(config, { mode: "write" });
+ try {
+ for (const [slug, values] of Object.entries(bySlug)) {
+ const docId = store.getDocumentIdByPath(`Brain/preferences/pref-${slug}.md`);
+ expect(docId).not.toBeNull();
+ const norm = Math.hypot(...values);
+ for (const chunk of store.chunksForDocument(docId!)) {
+ const vector = values.map((v) => v / norm);
+ store.vecUpsert(chunk.id, vector, MODEL, DIMENSION, `eh-${slug}-${chunk.id}`);
+ }
+ }
+ } finally {
+ await store.close();
+ }
+}
+
+describe("buildReviewCandidates retire_siblings", () => {
+ let savedFlag: string | undefined;
+
+ beforeEach(() => {
+ savedFlag = process.env[FLAG_ENV];
+ delete process.env[FLAG_ENV];
+ seedPreference("old", "formatting", RULE);
+ seedPreference("paraphrase", "commit-hygiene", PARAPHRASE);
+ seedPreference("semantic", "tidiness", SEMANTIC);
+ seedPreference("unrelated", "makefiles", UNRELATED);
+ appendApplyEvidence(
+ vault,
+ { pref_id: "pref-old", artifact: "[[AA-artifact]]", result: "outdated", agent: "test-agent" },
+ { now: new Date("2026-06-09T12:00:00Z") },
+ );
+ });
+
+ afterEach(() => {
+ if (savedFlag === undefined) delete process.env[FLAG_ENV];
+ else process.env[FLAG_ENV] = savedFlag;
+ });
+
+ const LEXICAL: RetireSibling = {
+ retiring_id: "pref-old",
+ sibling_id: "pref-paraphrase",
+ score: 0.818,
+ method: "lexical",
+ };
+
+ test("with the key on, the lexical siblings are projected", async () => {
+ process.env[FLAG_ENV] = "1";
+ const r = await buildReviewCandidates(vault, { now: NOW });
+ expect(r.retire_siblings).toEqual([LEXICAL]);
+ expect("retire_siblings_semantic" in r).toBe(false);
+ });
+
+ test.skipIf(!sqliteVecLoadable())(
+ "stored-vector matches merge in with method embedding at or above the bar",
+ async () => {
+ process.env[FLAG_ENV] = "1";
+ const config = searchConfig();
+ await indexVault(config);
+ await plant(config, {
+ old: [1, 0, 0, 0],
+ paraphrase: [1, 0.05, 0, 0],
+ semantic: [1, 0.3, 0, 0],
+ unrelated: [1, 1, 0, 0],
+ });
+ const r = await buildReviewCandidates(vault, { now: NOW, searchConfig: config });
+ expect(r.retire_siblings_semantic).toBe("used");
+ // `paraphrase` stays lexical; `semantic` (cosine 0.958) joins by
+ // embedding; `unrelated` (0.707) is below the 0.92 bar.
+ expect(r.retire_siblings).toEqual([
+ { retiring_id: "pref-old", sibling_id: "pref-semantic", score: 0.958, method: "embedding" },
+ LEXICAL,
+ ]);
+ },
+ );
+
+ test.skipIf(!sqliteVecLoadable())(
+ "a retire the confirmed-evidence gate will hold back gets no embedding sibling",
+ async () => {
+ process.env[FLAG_ENV] = "1";
+ const yamlPath = join(vault, "Brain", "_brain.yaml");
+ writeFileSync(
+ yamlPath,
+ readFileSync(yamlPath, "utf8").replace(
+ "# confirmed_evidence_min_threshold: 3",
+ "confirmed_evidence_min_threshold: 50",
+ ),
+ );
+ const config = searchConfig();
+ await indexVault(config);
+ await plant(config, { old: [1, 0, 0, 0], semantic: [1, 0.3, 0, 0] });
+ const r = await buildReviewCandidates(vault, { now: NOW, searchConfig: config });
+ expect(r.would_retire).toEqual([{ id: "ret-old", reason: "superseded-by-context" }]);
+ expect("retire_siblings" in r).toBe(false);
+ expect("retire_siblings_semantic" in r).toBe(false);
+ },
+ );
+
+ test("a missing index is reported by name and leaves the lexical list", async () => {
+ process.env[FLAG_ENV] = "1";
+ const r = await buildReviewCandidates(vault, { now: NOW, searchConfig: searchConfig() });
+ expect(r.retire_siblings_semantic).toBe("index_missing");
+ expect(r.retire_siblings).toEqual([LEXICAL]);
+ expect("retire_siblings_semantic_detail" in r).toBe(false);
+ });
+
+ test("an index that will not open degrades to the lexical list and names the failure", async () => {
+ process.env[FLAG_ENV] = "1";
+ const config = searchConfig();
+ mkdirSync(join(config.dbPath, ".."), { recursive: true });
+ writeFileSync(config.dbPath, "not a database");
+ const r = await buildReviewCandidates(vault, { now: NOW, searchConfig: config });
+ expect(r.retire_siblings_semantic).toBe("index_unavailable");
+ expect(r.retire_siblings_semantic_detail).toBe("INDEX_UNREADABLE");
+ expect(r.retire_siblings).toEqual([LEXICAL]);
+ expect(r.would_retire).toEqual([{ id: "ret-old", reason: "superseded-by-context" }]);
+ });
+
+ test.skipIf(!sqliteVecLoadable())(
+ "an index that has not embedded the preferences reports not_embedded, not used",
+ async () => {
+ process.env[FLAG_ENV] = "1";
+ const config = searchConfig();
+ await indexVault(config);
+ const r = await buildReviewCandidates(vault, { now: NOW, searchConfig: config });
+ expect(r.retire_siblings_semantic).toBe("not_embedded");
+ expect(r.retire_siblings).toEqual([LEXICAL]);
+ },
+ );
+
+ test("an explicit retireSiblingsEnabled decides over the default config", async () => {
+ const on = await buildReviewCandidates(vault, { now: NOW, retireSiblingsEnabled: true });
+ expect(on.retire_siblings).toEqual([LEXICAL]);
+ process.env[FLAG_ENV] = "1";
+ const off = await buildReviewCandidates(vault, { now: NOW, retireSiblingsEnabled: false });
+ expect("retire_siblings" in off).toBe(false);
+ });
+
+ test("with the key off, both fields are absent", async () => {
+ const r = await buildReviewCandidates(vault, { now: NOW, searchConfig: searchConfig() });
+ expect("retire_siblings" in r).toBe(false);
+ expect("retire_siblings_semantic" in r).toBe(false);
+ expect(r.would_retire).toEqual([{ id: "ret-old", reason: "superseded-by-context" }]);
+ });
+});
diff --git a/tests/core/brain/write-advisory.test.ts b/tests/core/brain/write-advisory.test.ts
index 79407245..c9eba76f 100644
--- a/tests/core/brain/write-advisory.test.ts
+++ b/tests/core/brain/write-advisory.test.ts
@@ -21,6 +21,13 @@ import { routeExtractedFacts } from "../../../src/core/brain/fact-extract.ts";
import type { DedupIndexEntry } from "../../../src/core/brain/dedup-hash.ts";
import { BRAIN_LOG_EVENT_KIND, BRAIN_PREFERENCE_STATUS } from "../../../src/core/brain/types.ts";
import { adviseIncomingFeedback } from "../../../src/core/brain/write-advisory.ts";
+import {
+ adviseOnIncoming,
+ type PreferenceForContradiction,
+} from "../../../src/core/brain/health/contradiction.ts";
+import { NEAR_DUPLICATE_MIN_TOKENS } from "../../../src/core/brain/near-duplicate.ts";
+import { BRAIN_HEALTH_DEFAULTS } from "../../../src/core/brain/policy.ts";
+import { createRouteScope, ROUTE_STAGE } from "../../../src/core/route-scope.ts";
let tmp: string;
let vault: string;
@@ -155,3 +162,119 @@ describe("adviseIncomingFeedback", () => {
expect(advisoryEvents.length).toBe(0);
});
});
+
+/** A confirmed `coding` preference as `adviseOnIncoming` reads it. */
+function codingPref(id: string, principle: string): PreferenceForContradiction {
+ return {
+ id,
+ principle,
+ status: BRAIN_PREFERENCE_STATUS.confirmed,
+ scope: "coding",
+ evidenced_by: [],
+ };
+}
+
+/**
+ * The advisory scores through the shared near-duplicate kernel. Its output
+ * and its log line stay what they were, the threshold still comes from the
+ * operator-configurable health default, and the one allowed change is that
+ * a principle under the kernel's minimum token count no longer advises.
+ */
+describe("adviseIncomingFeedback - scored through the shared kernel", () => {
+ const PRINCIPLE = "always indent source with tabs not spaces";
+
+ test("the advisory and its log line are byte-identical to the shipped shape", () => {
+ confirmPref("tabs", PRINCIPLE, "coding");
+ const advisory = adviseIncomingFeedback(vault, {
+ principle: PRINCIPLE,
+ scope: "coding",
+ agent: "test-agent",
+ now: NOW,
+ });
+ expect(advisory).toEqual({
+ scope: "coding",
+ conflicts: [{ pref_id: "pref-tabs", jaccard: 1 }],
+ });
+ const events = readAllLogEntries(vault).filter(
+ (e) => e.eventType === BRAIN_LOG_EVENT_KIND.writeConflictAdvisory,
+ );
+ expect(events.map((e) => e.body)).toEqual([
+ {
+ scope: "coding",
+ conflicts: ["[[pref-tabs]] jaccard=1.000"],
+ agent: "test-agent",
+ origin_channel: "unset",
+ },
+ ]);
+ });
+
+ test("an unscoped principle compares against the unscoped bucket only", () => {
+ confirmPref("tabs", PRINCIPLE);
+ confirmPref("scoped", PRINCIPLE, "coding");
+ const advisory = adviseIncomingFeedback(vault, { principle: PRINCIPLE, agent: "a", now: NOW });
+ expect(advisory).toEqual({ scope: null, conflicts: [{ pref_id: "pref-tabs", jaccard: 1 }] });
+ });
+
+ test("the threshold is the health default, inclusive", () => {
+ const threshold = BRAIN_HEALTH_DEFAULTS.contradiction_jaccard;
+ expect(threshold).toBe(0.5);
+ // 4 shared tokens over an 8-token union is exactly 0.5.
+ const at = adviseOnIncoming("alpha bravo charlie delta echo foxtrot", "coding", [
+ codingPref("pref-at", "alpha bravo charlie delta golf hotel"),
+ // 3 shared over 9 is below the bar.
+ codingPref("pref-below", "alpha bravo charlie golf hotel india"),
+ ]);
+ expect(at).toEqual({ scope: "coding", conflicts: [{ prefId: "pref-at", jaccard: 0.5 }] });
+ });
+
+ test("a principle under the kernel's minimum token count no longer advises", () => {
+ const short = "use tabs always";
+ expect(short.split(" ").length).toBeLessThan(NEAR_DUPLICATE_MIN_TOKENS);
+ confirmPref("short", short, "coding");
+ const advisory = adviseIncomingFeedback(vault, {
+ principle: short,
+ scope: "coding",
+ agent: "test-agent",
+ now: NOW,
+ });
+ expect(advisory).toBeNull();
+ expect(
+ readAllLogEntries(vault).filter(
+ (e) => e.eventType === BRAIN_LOG_EVENT_KIND.writeConflictAdvisory,
+ ),
+ ).toEqual([]);
+ });
+});
+
+describe("adviseIncomingFeedback - stage timing", () => {
+ test("a feedback advisory records near_duplicate_lookup in the open route scope", async () => {
+ confirmPref("tabs", "always indent source with tabs not spaces", "coding");
+ const scope = createRouteScope();
+ await scope.run(async () =>
+ adviseIncomingFeedback(vault, {
+ principle: "always indent source with tabs not spaces",
+ scope: "coding",
+ agent: "test-agent",
+ now: NOW,
+ }),
+ );
+ const stage = scope.stages()?.find((s) => s.name === ROUTE_STAGE.nearDuplicateLookup);
+ expect(stage).toBeDefined();
+ expect(Number.isFinite(stage!.ms)).toBe(true);
+ expect(stage!.ms).toBeGreaterThanOrEqual(0);
+ });
+
+ test("with no scope open the advisory is unchanged", () => {
+ confirmPref("tabs", "always indent source with tabs not spaces", "coding");
+ const advisory = adviseIncomingFeedback(vault, {
+ principle: "always indent source with tabs not spaces",
+ scope: "coding",
+ agent: "test-agent",
+ now: NOW,
+ });
+ expect(advisory).toEqual({
+ scope: "coding",
+ conflicts: [{ pref_id: "pref-tabs", jaccard: 1 }],
+ });
+ });
+});
diff --git a/tests/core/config-near-duplicate.test.ts b/tests/core/config-near-duplicate.test.ts
new file mode 100644
index 00000000..da3574a8
--- /dev/null
+++ b/tests/core/config-near-duplicate.test.ts
@@ -0,0 +1,61 @@
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import {
+ resolveNearDuplicateRetireSiblingsEnabled,
+ resolveNearDuplicateWriteWideningEnabled,
+} from "../../src/core/config.ts";
+
+const CASES = [
+ {
+ resolve: resolveNearDuplicateRetireSiblingsEnabled,
+ key: "near_duplicate_retire_siblings_enabled",
+ env: "OPEN_SECOND_BRAIN_NEAR_DUPLICATE_RETIRE_SIBLINGS_ENABLED",
+ },
+ {
+ resolve: resolveNearDuplicateWriteWideningEnabled,
+ key: "near_duplicate_write_widening_enabled",
+ env: "OPEN_SECOND_BRAIN_NEAR_DUPLICATE_WRITE_WIDENING_ENABLED",
+ },
+] as const;
+
+let tmp: string;
+let configPath: string;
+const savedEnv: Record<string, string | undefined> = {};
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "o2b-config-near-dup-"));
+ configPath = join(tmp, "config.yaml");
+ for (const { env } of CASES) {
+ savedEnv[env] = process.env[env];
+ delete process.env[env];
+ }
+});
+
+afterEach(() => {
+ for (const { env } of CASES) {
+ if (savedEnv[env] === undefined) delete process.env[env];
+ else process.env[env] = savedEnv[env];
+ }
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+for (const { resolve, key, env } of CASES) {
+ describe(key, () => {
+ test("defaults to off", () => {
+ expect(resolve(configPath)).toBe(false);
+ });
+
+ test("the config key turns it on", () => {
+ writeFileSync(configPath, `${key}: "true"\n`);
+ expect(resolve(configPath)).toBe(true);
+ });
+
+ test("the env var turns it on", () => {
+ process.env[env] = "1";
+ expect(resolve(configPath)).toBe(true);
+ });
+ });
+}
diff --git a/tests/core/route-scope.test.ts b/tests/core/route-scope.test.ts
new file mode 100644
index 00000000..3871d55a
--- /dev/null
+++ b/tests/core/route-scope.test.ts
@@ -0,0 +1,110 @@
+import { describe, expect, test } from "bun:test";
+import { performance } from "node:perf_hooks";
+
+import {
+ createRouteScope,
+ isRouteStageName,
+ noteDecisionLatency,
+ ROUTE_STAGE,
+ ROUTE_STAGE_NAMES,
+ timeStage,
+ timeStageSync,
+} from "../../src/core/route-scope.ts";
+
+describe("route scope", () => {
+ test("with no scope open, both helpers return the value of fn", async () => {
+ expect(timeStageSync(ROUTE_STAGE.validate, () => 7)).toBe(7);
+ expect(await timeStage(ROUTE_STAGE.documentWrite, async () => "ok")).toBe("ok");
+ });
+
+ test("a scope with no stage noted reports undefined", async () => {
+ const scope = createRouteScope();
+ await scope.run(async () => undefined);
+ expect(scope.stages()).toBeUndefined();
+ expect(scope.decisionMs()).toBeUndefined();
+ });
+
+ test("inside run, sync and async stages are recorded", async () => {
+ const scope = createRouteScope();
+ await scope.run(async () => {
+ timeStageSync(ROUTE_STAGE.validate, () => 1);
+ await timeStage(ROUTE_STAGE.logAppend, async () => 2);
+ });
+ const stages = scope.stages() ?? [];
+ expect(stages.map((s) => s.name)).toEqual(["validate", "log_append"]);
+ for (const stage of stages) {
+ expect(Number.isFinite(stage.ms)).toBe(true);
+ expect(stage.ms).toBeGreaterThanOrEqual(0);
+ }
+ });
+
+ test("repeated names are summed in first-seen order", async () => {
+ const scope = createRouteScope();
+ await scope.run(async () => {
+ // The first write_receipt spins on the clock the scope reads, so its
+ // share is at least 20 ms on any timer; the second one is near zero.
+ // A scope that kept only the last value would report under 20 ms.
+ timeStageSync(ROUTE_STAGE.writeReceipt, () => {
+ const until = performance.now() + 20;
+ while (performance.now() < until) {
+ // busy wait
+ }
+ });
+ timeStageSync(ROUTE_STAGE.lint, () => undefined);
+ timeStageSync(ROUTE_STAGE.writeReceipt, () => undefined);
+ });
+ const stages = scope.stages() ?? [];
+ expect(stages.map((s) => s.name)).toEqual(["write_receipt", "lint"]);
+ expect(stages[0]!.ms).toBeGreaterThanOrEqual(20);
+ });
+
+ test("values are rounded to 0.1 ms", async () => {
+ const scope = createRouteScope();
+ await scope.run(async () => {
+ await timeStage(ROUTE_STAGE.documentWrite, () => new Promise((r) => setTimeout(r, 3)));
+ });
+ const [stage] = scope.stages() ?? [];
+ expect(stage).toBeDefined();
+ expect(Math.round(stage!.ms * 10) / 10).toBe(stage!.ms);
+ });
+
+ test("an error inside a stage still records and rethrows", async () => {
+ const scope = createRouteScope();
+ await scope.run(async () => {
+ expect(() =>
+ timeStageSync(ROUTE_STAGE.validate, () => {
+ throw new Error("sync-boom");
+ }),
+ ).toThrow("sync-boom");
+ await expect(
+ timeStage(ROUTE_STAGE.idempotencyLookup, async () => {
+ throw new Error("async-boom");
+ }),
+ ).rejects.toThrow("async-boom");
+ });
+ expect((scope.stages() ?? []).map((s) => s.name)).toEqual(["validate", "idempotency_lookup"]);
+ });
+
+ test("decisionMs sums noted decision latency and rounds to whole ms", async () => {
+ const scope = createRouteScope();
+ await scope.run(async () => {
+ noteDecisionLatency(10.4);
+ noteDecisionLatency(5.3);
+ noteDecisionLatency(Number.NaN);
+ noteDecisionLatency(-4);
+ });
+ expect(scope.decisionMs()).toBe(16);
+ expect(createRouteScope().decisionMs()).toBeUndefined();
+ // Outside every scope a note is dropped, never added to a closed scope.
+ expect(() => noteDecisionLatency(3)).not.toThrow();
+ expect(scope.decisionMs()).toBe(16);
+ });
+
+ test("isRouteStageName accepts exactly the nine allowlisted names", () => {
+ expect(ROUTE_STAGE_NAMES.size).toBe(9);
+ for (const name of Object.values(ROUTE_STAGE)) expect(isRouteStageName(name)).toBe(true);
+ for (const bad of ["Notes/private.md", "topic-x", "", "VALIDATE", 3, undefined, null]) {
+ expect(isRouteStageName(bad)).toBe(false);
+ }
+ });
+});
diff --git a/tests/mcp/brain-create-note.test.ts b/tests/mcp/brain-create-note.test.ts
index 77c68f45..0a2239cb 100644
--- a/tests/mcp/brain-create-note.test.ts
+++ b/tests/mcp/brain-create-note.test.ts
@@ -7,7 +7,7 @@
*/
import { afterEach, beforeEach, describe, expect, test } from "bun:test";
-import { existsSync, mkdtempSync, readFileSync, rmSync } from "node:fs";
+import { appendFileSync, existsSync, mkdtempSync, readFileSync, rmSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
@@ -17,8 +17,13 @@ import { NOTES_TOOLS } from "../../src/mcp/brain/notes-tools.ts";
import { WRITE_BATCH_TOOLS } from "../../src/mcp/brain/write-batch-tools.ts";
import { resolveNextStep } from "../../src/core/brain/next-step.ts";
import { PAGE_LINT_KEY } from "../../src/core/brain/page-lint.ts";
+import { resolveSearchConfig } from "../../src/core/search/index.ts";
+import { indexVault } from "../../src/core/search/indexer.ts";
import { WRITE_BINDING_REFUSED_CODE } from "../../src/core/write-binding/index.ts";
+import { listMcpRouteLatency } from "../../src/core/brain/mcp-route-metrics.ts";
+import { isRouteStageName } from "../../src/core/route-scope.ts";
import { MCPError } from "../../src/mcp/protocol.ts";
+import { MCPServer } from "../../src/mcp/server.ts";
import type { ServerContext } from "../../src/mcp/tool-contract.ts";
/**
@@ -305,3 +310,123 @@ describe("brain_create_note - the lint attached to the receipt", () => {
expect(PAGE_LINT_KEY in res).toBe(false);
});
});
+
+function enableWidening(): void {
+ appendFileSync(ctx.configPath!, "near_duplicate_write_widening_enabled: true\n");
+}
+
+/**
+ * Opt-in widening of the near-duplicate hint beyond the written page's
+ * directory, through the keyword index. With the key off the receipt is
+ * exactly what shipped; with it on, a create in another folder reports the
+ * earlier page and names the method.
+ */
+describe("brain_create_note - widened near-duplicate hint", () => {
+ const BODY = "orchard lantern copper meadow violet harbor";
+
+ async function seedIndexedEarlierPage(): Promise<void> {
+ await handler(ctx, { path: "Projects/Earlier.md", frontmatter: { title: "E" }, content: BODY });
+ await indexVault(resolveSearchConfig({ vault, configPath: ctx.configPath! }), { force: true });
+ }
+
+ test("with the key on, a create in another folder reports the earlier page", async () => {
+ enableWidening();
+ await seedIndexedEarlierPage();
+ const res = (await handler(ctx, {
+ path: "Notes/Later.md",
+ frontmatter: { title: "L" },
+ content: BODY,
+ })) as Record<string, unknown>;
+ const lint = res[PAGE_LINT_KEY] as {
+ findings: ReadonlyArray<Record<string, unknown>>;
+ widening: string;
+ };
+ expect(lint.widening).toBe("used");
+ expect(lint.findings).toEqual([
+ expect.objectContaining({
+ code: "near-duplicate",
+ page: "Notes/Later.md",
+ path: "Projects/Earlier.md",
+ message: expect.stringContaining("method=lexical"),
+ }),
+ ]);
+ });
+
+ test("with the key off, the receipt is unchanged", async () => {
+ await seedIndexedEarlierPage();
+ const res = (await handler(ctx, {
+ path: "Notes/Later.md",
+ frontmatter: { title: "L" },
+ content: BODY,
+ })) as Record<string, unknown>;
+ expect(Object.keys(res)).toEqual(["created", "outcome", "path", "write_id"]);
+ });
+
+ test("with the key on and no index, the receipt says widening did not run", async () => {
+ enableWidening();
+ const res = (await handler(ctx, {
+ path: "Notes/Later.md",
+ frontmatter: { title: "L" },
+ content: BODY,
+ })) as Record<string, unknown>;
+ expect(res[PAGE_LINT_KEY]).toMatchObject({
+ total: 0,
+ widening: "index_unavailable",
+ widening_detail: "INDEX_MISSING",
+ });
+ });
+});
+
+async function createdStages(): Promise<ReadonlyArray<{ name: unknown; ms: unknown }>> {
+ const server = new MCPServer({ vault, configPath: ctx.configPath! });
+ await server.callTool("brain_create_note", {
+ path: "Notes/AA-stage-probe.md",
+ content: "orchard lantern copper meadow violet harbor",
+ });
+ const [record] = listMcpRouteLatency(vault, { tool: "brain_create_note" });
+ const stages = record!.payload["stages"];
+ expect(Array.isArray(stages)).toBe(true);
+ return stages as ReadonlyArray<{ name: unknown; ms: unknown }>;
+}
+
+function expectTimed(stages: ReadonlyArray<{ name: unknown; ms: unknown }>, name: string): void {
+ const stage = stages.find((s) => s.name === name);
+ expect(stage).toBeDefined();
+ expect(isRouteStageName(stage!.name)).toBe(true);
+ expect(Number.isFinite(stage!.ms)).toBe(true);
+ expect(stage!.ms as number).toBeGreaterThanOrEqual(0);
+}
+
+/**
+ * With route metrics on, a create's continuity record names the time the
+ * receipt lint and the near-duplicate lookup took. Only presence and a
+ * finite non-negative value are asserted: never an order, never a size.
+ */
+describe("brain_create_note - receipt lint stage timings", () => {
+ const CONFIG_ENV = "OPEN_SECOND_BRAIN_CONFIG";
+ let savedConfigEnv: string | undefined;
+
+ beforeEach(() => {
+ savedConfigEnv = process.env[CONFIG_ENV];
+ process.env[CONFIG_ENV] = ctx.configPath!;
+ appendFileSync(ctx.configPath!, 'mcp_route_metrics_enabled: "true"\n');
+ });
+
+ afterEach(() => {
+ if (savedConfigEnv === undefined) delete process.env[CONFIG_ENV];
+ else process.env[CONFIG_ENV] = savedConfigEnv;
+ });
+
+ test("the record carries lint and near_duplicate_lookup", async () => {
+ const stages = await createdStages();
+ expectTimed(stages, "lint");
+ expectTimed(stages, "near_duplicate_lookup");
+ });
+
+ test("with widening on, the lookup is still one named stage", async () => {
+ enableWidening();
+ const stages = await createdStages();
+ expectTimed(stages, "near_duplicate_lookup");
+ expect(stages.filter((s) => s.name === "near_duplicate_lookup")).toHaveLength(1);
+ });
+});
diff --git a/tests/mcp/extract-signals-tool.test.ts b/tests/mcp/extract-signals-tool.test.ts
index 3cca71ce..45e031e8 100644
--- a/tests/mcp/extract-signals-tool.test.ts
+++ b/tests/mcp/extract-signals-tool.test.ts
@@ -11,6 +11,9 @@
* 4. The tool is full-tier, absent from the writer surface.
* 5. A committed signal's path is vault-relative: no MCP response of
* this tool carries the absolute host path.
+ * 6. Every `turns_mined` entry carries the turn's stored timestamp
+ * verbatim, and the CLI `--json` plan over the same fixture carries
+ * the identical projection.
*/
import { afterEach, beforeEach, expect, test } from "bun:test";
@@ -22,6 +25,7 @@ import { AUTO_EXTRACT_PER_SESSION_CAP } from "../../src/core/brain/extract-signa
import { importSessionRecall } from "../../src/core/brain/session-recall.ts";
import { buildToolTable, findTool } from "../../src/mcp/tools.ts";
import type { ServerContext } from "../../src/mcp/tool-contract.ts";
+import { runCli } from "../helpers/run-cli.ts";
const TOOL = "brain_extract_signals";
const SESSION = "sess-mcp";
@@ -117,3 +121,24 @@ test("is a full-tier tool, absent from the writer surface", () => {
expect(buildToolTable("full").find((t) => t.name === TOOL)).toBeDefined();
expect(buildToolTable("writer").find((t) => t.name === TOOL)).toBeUndefined();
});
+
+test("turns_mined carries the stored turn timestamp, on the tool and the CLI alike", async () => {
+ const expected = [
+ {
+ turn_id: "t1",
+ text: "Always name the release theme in the heading.",
+ timestamp: "2026-08-22T09:01:00Z",
+ },
+ ];
+ const res = (await tool().handler(ctx, { session: SESSION })) as {
+ turns_mined: ReadonlyArray<Record<string, unknown>>;
+ };
+ expect(res.turns_mined).toEqual(expected);
+
+ const cli = await runCli(["brain", "extract-signals", SESSION, "--json", "--vault", vault], {
+ env: { OPEN_SECOND_BRAIN_CONFIG: ctx.configPath! },
+ });
+ expect(cli.returncode).toBe(0);
+ const payload = JSON.parse(cli.stdout) as { turns_mined: unknown };
+ expect(payload.turns_mined).toEqual(expected);
+});
diff --git a/tests/mcp/feedback-advisory-reach.test.ts b/tests/mcp/feedback-advisory-reach.test.ts
index b493556c..feaa3170 100644
--- a/tests/mcp/feedback-advisory-reach.test.ts
+++ b/tests/mcp/feedback-advisory-reach.test.ts
@@ -23,6 +23,7 @@ import { REMOTE_DENY_VISIBILITY_TOKEN } from "../../src/core/graph/visibility.ts
import {
buildReachLogFixture,
maskVolatile,
+ PRIVATE_PATH,
PRIVATE_SLUG,
reachServer,
type ReachLogFixture as Fixture,
@@ -63,10 +64,25 @@ function reserveFile(abs: string): void {
);
}
+/**
+ * The shared fixture's `Rule <slug>.` principle is two tokens, under the
+ * near-duplicate kernel's minimum, so the withheld preference gets a
+ * principle long enough for the conflict advisory to score it.
+ */
+const PRIVATE_PRINCIPLE = `Rule ${PRIVATE_SLUG} governs every release checklist.`;
+
+function lengthenPrivatePrinciple(vault: string): void {
+ const abs = join(vault, PRIVATE_PATH);
+ const text = readFileSync(abs, "utf8");
+ expect(text).toContain(`Rule ${PRIVATE_SLUG}.`);
+ writeFileSync(abs, text.replaceAll(`Rule ${PRIVATE_SLUG}.`, PRIVATE_PRINCIPLE));
+}
+
function fixture(withPrivate: boolean): Fixture {
const base = mkdtempSync(join(tmpdir(), "o2b-feedback-reach-"));
bases.push(base);
const f = buildReachLogFixture(base, withPrivate);
+ if (withPrivate) lengthenPrivatePrinciple(f.vault);
seedSignal(f.vault, "public-capture", PUBLIC_SCOPE);
if (withPrivate) reserveFile(seedSignal(f.vault, "withheld-capture", WITHHELD_SCOPE));
return f;
@@ -82,7 +98,7 @@ async function answer(
}
/** A scope-less capture whose principle repeats the withheld rule word for word. */
-const PROBE = { topic: "probe", signal: "positive", principle: `Rule ${PRIVATE_SLUG}.` };
+const PROBE = { topic: "probe", signal: "positive", principle: PRIVATE_PRINCIPLE };
describe("brain_feedback advisories answer at the caller's reach", () => {
test("remote reach: neither advisory names or counts a withheld record", async () => {
diff --git a/tests/mcp/note-write-reach.test.ts b/tests/mcp/note-write-reach.test.ts
index c97d539c..5dacb1e2 100644
--- a/tests/mcp/note-write-reach.test.ts
+++ b/tests/mcp/note-write-reach.test.ts
@@ -67,6 +67,11 @@ async function answer(f: Fixture, tool: string, args: Record<string, unknown>):
return raw.split(f.vault).join("<vault>");
}
+/** The answer with the per-write receipt id normalised. */
+async function receipt(f: Fixture, tool: string, args: Record<string, unknown>): Promise<string> {
+ return (await answer(f, tool, args)).replace(/nw_\d{14}_[0-9a-f]{16}/g, "<nw>");
+}
+
const CALLS: ReadonlyArray<readonly [string, Record<string, unknown>]> = [
["brain_note_lifecycle", { action: "delete", path: PRIVATE_PATH }],
["brain_note_lifecycle", { action: "delete", path: PRIVATE_PATH, apply: true, confirm: true }],
@@ -117,3 +122,93 @@ describe("at local reach the page is planned and written", () => {
expect(readFileSync(join(local.vault, PRIVATE_PATH), "utf8")).toContain("appended");
});
});
+
+/**
+ * The write receipt's near-duplicate hint answers at the caller's reach.
+ * A body that copies the private page would be named as its near
+ * duplicate at local reach; at a reach that withholds the page, every
+ * note-write tool answers exactly as if the page never existed.
+ */
+describe("the near-duplicate receipt hint never names a withheld sibling", () => {
+ const COPIED_BODY = "# Secret plan\nThe PIN is 4711.\n";
+ const DRAFT_PATH = "Notes/draft.md";
+
+ /** A fixture with an empty-bodied draft to append to. */
+ function writeFixture(withSecret: boolean, reach?: typeof TRANSPORT_REACH.local): Fixture {
+ const f = fixture(withSecret, reach);
+ writeFileSync(join(f.vault, DRAFT_PATH), "---\ntitle: Draft\n---\n");
+ return f;
+ }
+
+ const WRITES: ReadonlyArray<readonly [string, Record<string, unknown>]> = [
+ ["brain_create_note", { path: "Notes/mine.md", content: COPIED_BODY }],
+ ["brain_update_note", { path: "Notes/open.md", content: COPIED_BODY }],
+ ["brain_append_note", { path: DRAFT_PATH, content: COPIED_BODY }],
+ [
+ "brain_write_batch",
+ { operations: [{ op: "create_note", path: "Notes/mine.md", content: COPIED_BODY }] },
+ ],
+ ];
+
+ for (const [tool, args] of WRITES) {
+ test(`${tool} answers as if the page were absent`, async () => {
+ const hidden = await receipt(writeFixture(true), tool, args);
+ expect(hidden).not.toContain(PRIVATE_PATH);
+ expect(hidden).toBe(await receipt(writeFixture(false), tool, args));
+ });
+
+ test(`${tool} names the page at local reach`, async () => {
+ const local = await receipt(writeFixture(true, TRANSPORT_REACH.local), tool, args);
+ expect(local).toContain("near-duplicate");
+ expect(local).toContain(PRIVATE_PATH);
+ });
+ }
+});
+
+/**
+ * The receipt's wikilink and merged-link findings answer at the caller's
+ * reach as well. A preference that reserved itself is reported the way a
+ * preference that never existed is: a link to it reads as broken, and a
+ * merge chain through it ends at its id without naming where it points.
+ */
+describe("the receipt's link findings never reach past a withheld page", () => {
+ const PREFS = join("Brain", "preferences");
+
+ function writePref(f: Fixture, slug: string, fields: string): void {
+ writeFileSync(
+ join(f.vault, PREFS, `pref-${slug}.md`),
+ `---\nid: pref-${slug}\ntopic: x\nprinciple: y\n${fields}---\n`,
+ );
+ }
+
+ /** A vault with a public pref merged into `pref-held`, private or absent. */
+ function linkFixture(withHeld: boolean, reach?: typeof TRANSPORT_REACH.local): Fixture {
+ const f = fixture(false, reach);
+ writePref(f, "canon", "");
+ writePref(f, "dup", "merged_into: pref-held\n");
+ if (withHeld) writePref(f, "held", "visibility: private\nmerged_into: pref-canon\n");
+ return f;
+ }
+
+ const LINKS = ["[[pref-held]]", "[[pref-dup]]"];
+
+ for (const link of LINKS) {
+ const args = { path: "Notes/mine.md", content: `see ${link}\n` };
+
+ test(`a link ${link} answers as if the page were absent`, async () => {
+ const hidden = await receipt(linkFixture(true), "brain_create_note", args);
+ expect(hidden).not.toContain("pref-canon");
+ expect(hidden).toBe(await receipt(linkFixture(false), "brain_create_note", args));
+ });
+ }
+
+ test("at local reach the chain is followed to its canonical end", async () => {
+ const args = { path: "Notes/mine.md", content: "see [[pref-dup]]\n" };
+ const local = await receipt(
+ linkFixture(true, TRANSPORT_REACH.local),
+ "brain_create_note",
+ args,
+ );
+ expect(local).toContain("merged into pref-canon");
+ });
+});
diff --git a/tests/mcp/review-tools-reach.test.ts b/tests/mcp/review-tools-reach.test.ts
index be9078a5..018fb63a 100644
--- a/tests/mcp/review-tools-reach.test.ts
+++ b/tests/mcp/review-tools-reach.test.ts
@@ -21,14 +21,16 @@
*/
import { afterEach, describe, expect, test } from "bun:test";
-import { mkdtempSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
+import { mkdirSync, mkdtempSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
-import { join } from "node:path";
+import { dirname, join } from "node:path";
+import { appendApplyEvidence } from "../../src/core/brain/apply-evidence.ts";
import { processedSignalPath, signalPath } from "../../src/core/brain/paths.ts";
import { writePreference } from "../../src/core/brain/preference.ts";
import { writeSignal } from "../../src/core/brain/signal.ts";
import { BRAIN_CONFIDENCE, BRAIN_PREFERENCE_STATUS } from "../../src/core/brain/types.ts";
+import { resolveSearchConfig } from "../../src/core/search/index.ts";
import { TRANSPORT_REACH, type TransportReach } from "../../src/core/graph/transport-reach.ts";
import { REMOTE_DENY_VISIBILITY_TOKEN } from "../../src/core/graph/visibility.ts";
import {
@@ -187,3 +189,250 @@ describe("the lifecycle review readers answer at the caller's reach", () => {
expect(await answer(fixture(true), "brain_trigger", undefined, scan)).toContain(READABLE_SLUG);
});
});
+
+const SIBLINGS_FLAG = "OPEN_SECOND_BRAIN_NEAR_DUPLICATE_RETIRE_SIBLINGS_ENABLED";
+const SIBLING_RULE = "always run the formatter before every commit in this repository";
+const SIBLING_PARAPHRASE = "always run the formatter before each commit in this repository";
+
+function rule(vault: string, slug: string, principle: string, reserved: boolean): void {
+ writePreference(vault, {
+ slug,
+ topic: slug,
+ principle,
+ created_at: "2026-05-01T00:00:00Z",
+ unconfirmed_until: "2026-05-08T00:00:00Z",
+ status: BRAIN_PREFERENCE_STATUS.confirmed,
+ evidenced_by: [],
+ confirmed_at: "2026-05-02T00:00:00Z",
+ applied_count: 1,
+ violated_count: 0,
+ last_evidence_at: new Date().toISOString().slice(0, 10) + "T00:00:00Z",
+ confidence: BRAIN_CONFIDENCE.high,
+ confidence_value: 0.8,
+ });
+ if (reserved) reserve(join(vault, "Brain", "preferences", `pref-${slug}.md`));
+}
+
+function evidence(vault: string, slug: string, result: "applied" | "outdated"): void {
+ appendApplyEvidence(vault, {
+ pref_id: `pref-${slug}`,
+ artifact: "[[AA-artifact]]",
+ result,
+ agent: "test",
+ });
+}
+
+function siblingsOf(answerText: string): unknown {
+ return (JSON.parse(answerText) as { retire_siblings?: unknown }).retire_siblings;
+}
+
+/**
+ * Two context-driven retires, each with one paraphrase sibling. In the
+ * first pair the sibling is withheld, in the second the retiring
+ * preference is; a readable pair is in both vaults.
+ */
+function siblingFixture(withPrivate: boolean): Fixture {
+ const base = mkdtempSync(join(tmpdir(), "o2b-review-siblings-reach-"));
+ bases.push(base);
+ const f = buildReachLogFixture(base, withPrivate);
+ rule(f.vault, "public-old", `${SIBLING_RULE} alpha`, false);
+ rule(f.vault, "public-twin", `${SIBLING_PARAPHRASE} alpha`, false);
+ evidence(f.vault, "public-twin", "applied");
+ evidence(f.vault, "public-old", "outdated");
+ if (withPrivate) {
+ rule(f.vault, "public-retiring", `${SIBLING_RULE} beta`, false);
+ rule(f.vault, `${WITHHELD_PREFIX}-twin`, `${SIBLING_PARAPHRASE} beta`, true);
+ evidence(f.vault, `${WITHHELD_PREFIX}-twin`, "applied");
+ evidence(f.vault, "public-retiring", "outdated");
+ rule(f.vault, `${WITHHELD_PREFIX}-old`, `${SIBLING_RULE} gamma`, true);
+ rule(f.vault, "public-gamma-twin", `${SIBLING_PARAPHRASE} gamma`, false);
+ evidence(f.vault, "public-gamma-twin", "applied");
+ evidence(f.vault, `${WITHHELD_PREFIX}-old`, "outdated");
+ }
+ return f;
+}
+
+describe("brain_review_candidates retire_siblings answer at the caller's reach", () => {
+ const savedFlag = process.env[SIBLINGS_FLAG];
+
+ afterEach(() => {
+ if (savedFlag === undefined) delete process.env[SIBLINGS_FLAG];
+ else process.env[SIBLINGS_FLAG] = savedFlag;
+ });
+
+ test("remote reach: a pair with a withheld retiring or sibling id is neither listed nor counted", async () => {
+ process.env[SIBLINGS_FLAG] = "1";
+ const withheld = await answer(siblingFixture(true), "brain_review_candidates");
+ const absent = await answer(siblingFixture(false), "brain_review_candidates");
+ expect(withheld).not.toContain(WITHHELD_PREFIX);
+ expect(siblingsOf(withheld)).toEqual(siblingsOf(absent));
+ expect(siblingsOf(withheld)).toEqual([
+ {
+ retiring_id: "pref-public-old",
+ sibling_id: "pref-public-twin",
+ score: 0.833,
+ method: "lexical",
+ },
+ ]);
+ });
+
+ test("remote reach: a sibling whose ret- spelling is withheld is not listed", async () => {
+ // The dry run admits the readable `pref-bygone`, but the vault also holds
+ // a withheld `ret-bygone`: the view asks about both spellings.
+ process.env[SIBLINGS_FLAG] = "1";
+ const f = siblingFixture(true);
+ rule(f.vault, RETIRED_SLUG, `${SIBLING_PARAPHRASE} beta`, false);
+ evidence(f.vault, RETIRED_SLUG, "applied");
+ const remote = await answer(f, "brain_review_candidates");
+ expect(remote).not.toContain(RETIRED_SLUG);
+ const local = await answer(f, "brain_review_candidates", TRANSPORT_REACH.local);
+ expect(local).toContain(`"sibling_id":"pref-${RETIRED_SLUG}"`);
+ });
+
+ test("remote reach: a retire hidden by its withheld ret- spelling moves no semantic status", async () => {
+ // The only context-driven retire is the readable `pref-bygone`, whose
+ // withheld `ret-bygone` hides it; the absent vault has neither.
+ process.env[SIBLINGS_FLAG] = "1";
+ const withheldBase = mkdtempSync(join(tmpdir(), "o2b-review-semantic-reach-"));
+ const absentBase = mkdtempSync(join(tmpdir(), "o2b-review-semantic-reach-"));
+ bases.push(withheldBase, absentBase);
+ const withheld = buildReachLogFixture(withheldBase, true);
+ rule(withheld.vault, RETIRED_SLUG, SIBLING_RULE, false);
+ evidence(withheld.vault, RETIRED_SLUG, "outdated");
+ const absent = buildReachLogFixture(absentBase, false);
+ const local = await answer(withheld, "brain_review_candidates", TRANSPORT_REACH.local);
+ expect(local).toContain('"retire_siblings_semantic"');
+ const remote = await answer(withheld, "brain_review_candidates");
+ expect(remote).not.toContain("retire_siblings_semantic");
+ expect(remote).toBe(await answer(absent, "brain_review_candidates"));
+ });
+
+ test("the key is read from the server's own config file, not the default one", async () => {
+ delete process.env[SIBLINGS_FLAG];
+ const base = mkdtempSync(join(tmpdir(), "o2b-review-siblings-config-"));
+ bases.push(base);
+ const f = buildReachLogFixture(base, false, ['near_duplicate_retire_siblings_enabled: "true"']);
+ rule(f.vault, "public-old", `${SIBLING_RULE} alpha`, false);
+ rule(f.vault, "public-twin", `${SIBLING_PARAPHRASE} alpha`, false);
+ evidence(f.vault, "public-twin", "applied");
+ evidence(f.vault, "public-old", "outdated");
+ const server = reachServer(f, TRANSPORT_REACH.local);
+ // The default config is now a file without the key.
+ const otherConfig = join(base, "other-config.yaml");
+ writeFileSync(otherConfig, `vault: ${f.vault}\n`);
+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = otherConfig;
+ const result = await server.callTool("brain_review_candidates", {});
+ expect(siblingsOf(maskVolatile(f, result["structuredContent"] ?? result))).toEqual([
+ {
+ retiring_id: "pref-public-old",
+ sibling_id: "pref-public-twin",
+ score: 0.833,
+ method: "lexical",
+ },
+ ]);
+ });
+
+ test("an index that will not open keeps the lexical pairs and names the failure", async () => {
+ process.env[SIBLINGS_FLAG] = "1";
+ const f = siblingFixture(false);
+ const { dbPath } = resolveSearchConfig({ vault: f.vault, configPath: f.configPath });
+ mkdirSync(dirname(dbPath), { recursive: true });
+ writeFileSync(dbPath, "not a database");
+ const local = await answer(f, "brain_review_candidates", TRANSPORT_REACH.local);
+ expect(local).toContain('"retire_siblings_semantic":"index_unavailable"');
+ expect(local).toContain('"retire_siblings_semantic_detail":"INDEX_UNREADABLE"');
+ expect(siblingsOf(local)).toEqual([
+ {
+ retiring_id: "pref-public-old",
+ sibling_id: "pref-public-twin",
+ score: 0.833,
+ method: "lexical",
+ },
+ ]);
+ });
+
+ test("local control: the operator's own shell lists every pair", async () => {
+ process.env[SIBLINGS_FLAG] = "1";
+ const local = await answer(
+ siblingFixture(true),
+ "brain_review_candidates",
+ TRANSPORT_REACH.local,
+ );
+ const pairs = (siblingsOf(local) as Array<{ retiring_id: string; sibling_id: string }>).map(
+ (p) => `${p.retiring_id}>${p.sibling_id}`,
+ );
+ expect(pairs).toEqual([
+ "pref-public-old>pref-public-twin",
+ "pref-public-retiring>pref-zzwithheld-stale-twin",
+ "pref-zzwithheld-stale-old>pref-public-gamma-twin",
+ ]);
+ });
+});
+
+describe("brain_dream retire_siblings answer at the caller's reach", () => {
+ const savedFlag = process.env[SIBLINGS_FLAG];
+ const DRY_RUN = { dry_run: true };
+ const PUBLIC_PAIR = {
+ retiring_id: "pref-public-old",
+ sibling_id: "pref-public-twin",
+ score: 0.833,
+ method: "lexical",
+ };
+
+ afterEach(() => {
+ if (savedFlag === undefined) delete process.env[SIBLINGS_FLAG];
+ else process.env[SIBLINGS_FLAG] = savedFlag;
+ });
+
+ test("remote reach: a pair with a withheld retiring or sibling id is neither listed nor counted", async () => {
+ process.env[SIBLINGS_FLAG] = "1";
+ const withheld = await answer(siblingFixture(true), "brain_dream", undefined, DRY_RUN);
+ const absent = await answer(siblingFixture(false), "brain_dream", undefined, DRY_RUN);
+ expect(siblingsOf(withheld)).toEqual(siblingsOf(absent));
+ expect(siblingsOf(withheld)).toEqual([PUBLIC_PAIR]);
+ });
+
+ test("local control: the operator's own shell lists every pair", async () => {
+ process.env[SIBLINGS_FLAG] = "1";
+ const local = await answer(siblingFixture(true), "brain_dream", TRANSPORT_REACH.local, DRY_RUN);
+ const pairs = (siblingsOf(local) as Array<{ retiring_id: string; sibling_id: string }>).map(
+ (p) => `${p.retiring_id}>${p.sibling_id}`,
+ );
+ expect(pairs).toEqual([
+ "pref-public-old>pref-public-twin",
+ "pref-public-retiring>pref-zzwithheld-stale-twin",
+ "pref-zzwithheld-stale-old>pref-public-gamma-twin",
+ ]);
+ });
+
+ test("with the key off, the field is absent", async () => {
+ delete process.env[SIBLINGS_FLAG];
+ const local = await answer(
+ siblingFixture(false),
+ "brain_dream",
+ TRANSPORT_REACH.local,
+ DRY_RUN,
+ );
+ expect(local).not.toContain("retire_siblings");
+ });
+
+ test("the key is read from the server's own config file, not the default one", async () => {
+ delete process.env[SIBLINGS_FLAG];
+ const base = mkdtempSync(join(tmpdir(), "o2b-dream-siblings-config-"));
+ bases.push(base);
+ const f = buildReachLogFixture(base, false, ['near_duplicate_retire_siblings_enabled: "true"']);
+ rule(f.vault, "public-old", `${SIBLING_RULE} alpha`, false);
+ rule(f.vault, "public-twin", `${SIBLING_PARAPHRASE} alpha`, false);
+ evidence(f.vault, "public-twin", "applied");
+ evidence(f.vault, "public-old", "outdated");
+ const server = reachServer(f, TRANSPORT_REACH.local);
+ // The default config is now a file without the key.
+ const otherConfig = join(base, "other-config.yaml");
+ writeFileSync(otherConfig, `vault: ${f.vault}\n`);
+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = otherConfig;
+ const result = await server.callTool("brain_dream", DRY_RUN);
+ expect(siblingsOf(maskVolatile(f, result["structuredContent"] ?? result))).toEqual([
+ PUBLIC_PAIR,
+ ]);
+ });
+});
diff --git a/tests/mcp/route-metrics-tool.test.ts b/tests/mcp/route-metrics-tool.test.ts
index a76844d1..b9dd4afe 100644
--- a/tests/mcp/route-metrics-tool.test.ts
+++ b/tests/mcp/route-metrics-tool.test.ts
@@ -5,6 +5,7 @@ import { join } from "node:path";
import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";
import { listMcpRouteLatency } from "../../src/core/brain/mcp-route-metrics.ts";
+import { isRouteStageName } from "../../src/core/route-scope.ts";
import { JSONRPC_VERSION, MCPServer, PROTOCOL_VERSION } from "../../src/mcp/index.ts";
import { buildToolTable } from "../../src/mcp/tools.ts";
@@ -93,6 +94,8 @@ describe("route latency capture through tools/call", () => {
const server = new MCPServer({ vault, configPath });
await initialize(server);
await callTool(server, "second_brain_status", {});
+ // A route that times write stages records nothing either.
+ await callTool(server, "brain_feedback", FEEDBACK_ARGS);
expect(listMcpRouteLatency(vault)).toHaveLength(0);
});
@@ -178,3 +181,113 @@ describe("route latency capture through tools/call", () => {
expect(response.error ?? response.result?.isError).toBeTruthy();
});
});
+
+const FEEDBACK_ARGS = {
+ topic: "route-stage-probe",
+ signal: "positive",
+ principle: "Keep route stage timings out of tool responses entirely.",
+};
+
+describe("route scope stages through callTool", () => {
+ test("gate on: a brain_feedback call yields one record and no decision_ms", async () => {
+ writeConfig(true);
+ const server = new MCPServer({ vault, configPath });
+ const result = await server.callTool("brain_feedback", FEEDBACK_ARGS);
+ expect(JSON.stringify(result)).not.toContain("stages");
+
+ const records = listMcpRouteLatency(vault, { tool: "brain_feedback" });
+ expect(records).toHaveLength(1);
+ expect(records[0]!.payload["status"]).toBe("ok");
+ expect(Object.hasOwn(records[0]!.payload, "decision_ms")).toBe(false);
+ });
+});
+
+function stageNames(payload: Record<string, unknown>): string[] {
+ const stages = payload["stages"];
+ if (!Array.isArray(stages)) return [];
+ for (const stage of stages as Array<{ name: unknown; ms: unknown }>) {
+ expect(isRouteStageName(stage.name)).toBe(true);
+ expect(typeof stage.ms).toBe("number");
+ expect(Number.isFinite(stage.ms as number)).toBe(true);
+ expect(stage.ms as number).toBeGreaterThanOrEqual(0);
+ }
+ return (stages as Array<{ name: string }>).map((stage) => stage.name);
+}
+
+describe("signal write stages", () => {
+ test("a brain_feedback record carries the signal write stages", async () => {
+ writeConfig(true);
+ const server = new MCPServer({ vault, configPath });
+ await server.callTool("brain_feedback", { ...FEEDBACK_ARGS, idempotency_key: "stage-key-1" });
+
+ const [record] = listMcpRouteLatency(vault, { tool: "brain_feedback" });
+ const names = stageNames(record!.payload);
+ for (const name of [
+ "validate",
+ "idempotency_lookup",
+ "document_write",
+ "idempotency_remember",
+ "log_append",
+ ]) {
+ expect(names).toContain(name);
+ }
+ expect(names).not.toContain("preference_write");
+ });
+
+ test("force_confirmed adds preference_write", async () => {
+ writeConfig(true);
+ const server = new MCPServer({ vault, configPath });
+ await server.callTool("brain_feedback", { ...FEEDBACK_ARGS, force_confirmed: true });
+
+ const [record] = listMcpRouteLatency(vault, { tool: "brain_feedback" });
+ expect(stageNames(record!.payload)).toContain("preference_write");
+ });
+
+ test("an idempotency mismatch still propagates and records an error", async () => {
+ writeConfig(true);
+ const server = new MCPServer({ vault, configPath });
+ await server.callTool("brain_feedback", { ...FEEDBACK_ARGS, idempotency_key: "stage-key-2" });
+ await expect(
+ server.callTool("brain_feedback", {
+ ...FEEDBACK_ARGS,
+ principle: "A different principle under the same idempotency key.",
+ idempotency_key: "stage-key-2",
+ }),
+ ).rejects.toThrow();
+
+ const errors = listMcpRouteLatency(vault, { tool: "brain_feedback", status: "error" });
+ expect(errors).toHaveLength(1);
+ expect(stageNames(errors[0]!.payload)).toContain("idempotency_lookup");
+ });
+});
+
+describe("note write stages", () => {
+ test("a brain_create_note record carries document_write and write_receipt", async () => {
+ writeConfig(true);
+ const server = new MCPServer({ vault, configPath });
+ await server.callTool("brain_create_note", {
+ path: "Notes/AA-stage-probe.md",
+ content: "stage probe body",
+ });
+
+ const [record] = listMcpRouteLatency(vault, { tool: "brain_create_note" });
+ const names = stageNames(record!.payload);
+ expect(names).toContain("document_write");
+ expect(names).toContain("write_receipt");
+ });
+
+ test("a brain_write_batch rewrite record carries write_receipt", async () => {
+ writeConfig(true);
+ const server = new MCPServer({ vault, configPath });
+ await server.callTool("brain_create_note", {
+ path: "Notes/AA-batch-target.md",
+ content: "first body",
+ });
+ await server.callTool("brain_write_batch", {
+ operations: [{ op: "update_note", path: "Notes/AA-batch-target.md", content: "second" }],
+ });
+
+ const [record] = listMcpRouteLatency(vault, { tool: "brain_write_batch" });
+ expect(stageNames(record!.payload)).toContain("write_receipt");
+ });
+});
diff --git a/uv.lock b/uv.lock
index 9409c171..a0a02f99 100644
--- a/uv.lock
+++ b/uv.lock
@@ -4,5 +4,5 @@ requires-python = ">=3.11"
[[package]]
name = "open-second-brain"
-version = "1.73.1"
+version = "1.74.0"
source = { editable = "." }