Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
refactor: make search and classification language-agnostic (#84)
Sol Aitken9886d9abdeee
diff --git a/docs/plans/2026-05-16-brain-search-design.md b/docs/plans/2026-05-16-brain-search-design.mdindex f8564cbe..46fad5c8 100644--- a/docs/plans/2026-05-16-brain-search-design.md+++ b/docs/plans/2026-05-16-brain-search-design.md@@ -407,7 +407,7 @@ Human format:```[1] Brain/preferences/pref-russian-replies.md • 0.81line 12-38 • hybrid • mtime 2026-05-14- Правила общения: всегда отвечать по-русски, кроме когда речь о…+ Communication rules: always reply in the user's language, except when…```JSON format: objectdiff --git a/docs/plans/2026-05-18-agent-discipline-tail-design.md b/docs/plans/2026-05-18-agent-discipline-tail-design.mdindex 92520f4c..ec6f3938 100644--- a/docs/plans/2026-05-18-agent-discipline-tail-design.md+++ b/docs/plans/2026-05-18-agent-discipline-tail-design.md@@ -625,8 +625,8 @@ for §D`). Each line names the trigger that would bring it back toactive planning.`Plan/3. Agent logging discipline.md` status: `shipped`. §B / §D /-§E sections move from `open:` to `shipped:`. The "Триггер на-возврат" section is dropped — §30 is closed.+§E sections move from `open:` to `shipped:`. The "return+trigger" section is dropped — §30 is closed.`Plan/2. Improvement.md` and `Plan/4. v0.10.5 review followups.md`are untouched.diff --git a/docs/plans/2026-05-18-agent-discipline-tail-impl.md b/docs/plans/2026-05-18-agent-discipline-tail-impl.mdindex 7c00306b..3f16e629 100644--- a/docs/plans/2026-05-18-agent-discipline-tail-impl.md+++ b/docs/plans/2026-05-18-agent-discipline-tail-impl.md@@ -3136,7 +3136,7 @@ existing `js-yaml` already on the dep tree.- Modify: `/root/vault/Projects/OpenSecondBrain/Features/_summary.md`.- [ ] **Step 1: Update §30 entry** — set to `✅ shipped fully in- v0.10.7`. The current entry says `реализовано частично в+ v0.10.7`. The current entry says `partially implemented inv0.10.6 (§A+§C)`. Replace the closing parenthetical withshipped-in-full and link to `[[Plan/3. Agent logging discipline]]`.@@ -3152,7 +3152,7 @@ existing `js-yaml` already on the dep tree.false-positive/false-negative pairs.- [ ] **Step 3: Update the Agent-logging-discipline marker** —- the line that previously said `реализовано частично в v0.10.6`+ the line that previously said `partially implemented in v0.10.6`becomes `shipped fully in v0.10.7 — §B (writer MCP split), §D(daily discipline cron), §E (claude-memory bridge)`.@@ -3167,7 +3167,7 @@ existing `js-yaml` already on the dep tree.three open items move into `shipped:` (or rewrite the front-matterto remove the `open:` field entirely).-- [ ] **Step 2: Drop the "Триггер на возврат" section** — §30 is+- [ ] **Step 2: Drop the "return trigger" section** — §30 isclosed; the trigger is no longer relevant.- [ ] **Step 3: Add a tail note** — "v0.10.7 closed §B / §D / §E.diff --git a/docs/plans/2026-05-18-brain-maturity-impl.md b/docs/plans/2026-05-18-brain-maturity-impl.mdindex 21157d71..37bd9368 100644--- a/docs/plans/2026-05-18-brain-maturity-impl.md+++ b/docs/plans/2026-05-18-brain-maturity-impl.md@@ -2039,8 +2039,8 @@ Expected: exit 0.**Step 1: Replace the two deferred entries that this PR retires**Find the deferred-work section, locate the two entries:-- `§4 — per-runtime steering text для Claude Code и Codex.`-- `§15 «good vs bad» SKILL-секция.`+- `§4 — per-runtime steering text for Claude Code and Codex.`+- `§15 «good vs bad» SKILL section.`Replace them with one consolidated line:diff --git a/docs/plans/2026-05-18-v0.10.6-design.md b/docs/plans/2026-05-18-v0.10.6-design.mdindex 693e1888..b3db0c71 100644--- a/docs/plans/2026-05-18-v0.10.6-design.md+++ b/docs/plans/2026-05-18-v0.10.6-design.md@@ -806,18 +806,18 @@ no separate `§31.3` entry needed.- §5 — mark as fully shipped (manifest + abort drift detectiondelivered in v0.10.6). Update tail text.-- §14 — append "v0.10.6: keyboard-доступная инспекция узлов и- layout-state persistence через localStorage". Remove the two+- §14 — append "v0.10.6: keyboard-accessible node inspection and+ layout-state persistence via localStorage". Remove the twomatching items from `## Deferred work` and from the §14 inlineparenthetical.-- §22 — replace "6" header with "✅ … *(реализовано в v0.10.6)*".-- §28 — same: "✅ … *(реализовано в v0.10.6)*".-- §30 — "✅ Agent logging discipline — §A + §C *(реализовано в- v0.10.6: stop-guardrail расширен на brain-events, SKILL trigger- переформулирован; §B / §D / §E остаются открытыми)*".-- §31 — "✅ v0.10.5 review followups *(закрыто в v0.10.6: MD040 в- трёх planning-доках, доп. тест на пустой DYLD_LIBRARY_PATH;- keyboard a11y делён §14 polish)*".+- §22 — replace "6" header with "✅ … *(implemented in v0.10.6)*".+- §28 — same: "✅ … *(implemented in v0.10.6)*".+- §30 — "✅ Agent logging discipline — §A + §C *(implemented in+ v0.10.6: stop-guardrail extended to brain-events, SKILL trigger+ rephrased; §B / §D / §E remain open)*".+- §31 — "✅ v0.10.5 review followups *(closed in v0.10.6: MD040 in+ the three planning docs, extra test for an empty DYLD_LIBRARY_PATH;+ keyboard a11y split into §14 polish)*".`Projects/OpenSecondBrain/Plan/3. Agent logging discipline.md`:diff --git a/docs/plans/2026-05-18-v0.10.6-impl.md b/docs/plans/2026-05-18-v0.10.6-impl.mdindex 55844851..ee21be6d 100644--- a/docs/plans/2026-05-18-v0.10.6-impl.md+++ b/docs/plans/2026-05-18-v0.10.6-impl.md@@ -1346,7 +1346,7 @@ section:- §5: mark fully shipped (manifest + abort drift detection).- §14: append v0.10.6 keyboard + layout-persistence note. Removethe two matching items from `## Deferred work`.-- §22, §28: prepend `✅ … *(реализовано в v0.10.6)*`.+- §22, §28: prepend `✅ … *(implemented in v0.10.6)*`.- §30: mark §A + §C shipped, leave §B / §D / §E open.- §31: mark §31.1 + §31.2 closed, §31.3 subsumed by §14 polish.diff --git a/docs/plans/2026-05-19-v0.10.8-impl.md b/docs/plans/2026-05-19-v0.10.8-impl.mdindex f2ba7fb4..e666356e 100644--- a/docs/plans/2026-05-19-v0.10.8-impl.md+++ b/docs/plans/2026-05-19-v0.10.8-impl.md@@ -1429,33 +1429,33 @@ In `/root/vault/Projects/OpenSecondBrain/Features/_summary.md`, change the headito:```-### ✅ 32. Retire `event_log_append` for agent-facing surface, single source = `Brain/log/` — 8 *(реализовано в v0.10.8: §32A облегчённый verification gate; §32B brain_note tool в always-loaded writer scope; §32C-D drop event_log_append из stop-guardrail / detector / identity-reminder; §32E hooks/README.md переписан; §32F cmdAppendEvent резолвит identity через resolveAgentName; §32G полный дроп handler из всех рантаймов (Claude/Codex/Hermes/OpenClaw); §32H drift-check операторский, без auto-watcher)*+### ✅ 32. Retire `event_log_append` for agent-facing surface, single source = `Brain/log/` — 8 *(implemented in v0.10.8: §32A lightweight verification gate; §32B brain_note tool in the always-loaded writer scope; §32C-D drop event_log_append from stop-guardrail / detector / identity-reminder; §32E hooks/README.md rewritten; §32F cmdAppendEvent resolves identity via resolveAgentName; §32G full handler drop from all runtimes (Claude/Codex/Hermes/OpenClaw); §32H operator-driven drift-check, no auto-watcher)*```- [ ] **Step 14.2: Mark §23 shipped**In the same file, change §23 (line 224) from:-```-### 23. Structured JSONL log параллельно с Markdown — 6+```markdown+### 23. Structured JSONL log alongside Markdown — 6```to:-```-### ✅ 23. Structured JSONL log параллельно с Markdown — 6 *(реализовано в v0.10.8: Brain/log/<date>.jsonl рядом с .md, atomic dual-write через proper-lockfile lock; src/core/brain/log-jsonl.ts:readLogDay - JSONL-preferred reader с markdown fallback; discipline-report переключен; no backfill - исторические дни обслуживает fallback)*+```markdown+### ✅ 23. Structured JSONL log alongside Markdown — 6 *(implemented in v0.10.8: Brain/log/<date>.jsonl next to the .md, atomic dual-write via a proper-lockfile lock; src/core/brain/log-jsonl.ts:readLogDay - JSONL-preferred reader with a markdown fallback; discipline-report switched over; no backfill - the fallback serves historical days)*```- [ ] **Step 14.3: Append Deferred-work entries**-In `## Deferred work`, add (in the same Russian style as existing entries):+In `## Deferred work`, add (in the same style as existing entries):```markdown-- **§32B CLI verb `o2b brain note`.** MCP tool отгружен в v0.10.8, но CLI verb для cron-job/shell-script use-case'а отложен. Триггер на возврат: первый случай когда заметку нужно записать вне MCP (cron, ad-hoc script). Bash-needle для detector добавляется одновременно с CLI verb.+- **§32B CLI verb `o2b brain note`.** The MCP tool shipped in v0.10.8, but the CLI verb for the cron-job / shell-script use-case is deferred. Return trigger: the first time a note must be written outside MCP (cron, ad-hoc script). The bash needle for the detector is added together with the CLI verb.-- **§32H auto-watcher Daily/-записей от @agent-identity.** Операторская проверка (`o2b discipline report` + грeп Daily/) сейчас достаточна. Триггер: drift-check находит повторные случаи которые операторский проход пропустил.+- **§32H auto-watcher for Daily/ entries from @agent-identity.** The operator check (`o2b discipline report` + grep over Daily/) is sufficient for now. Trigger: drift-check finds recurring cases the operator pass missed.-- **§23 one-shot JSONL backfill исторических markdown-дней.** Лeniвый fallback в readLogDay обслуживает все pre-v0.10.8 дни без сидекара. Триггер: машинный consumer материально выигрывает от JSONL для дней старше v0.10.8 (ни один не назван сейчас).+- **§23 one-shot JSONL backfill of historical markdown days.** The lazy fallback in readLogDay serves all pre-v0.10.8 days without a sidecar. Trigger: a machine consumer materially benefits from JSONL for days older than v0.10.8 (none named yet).```- [ ] **Step 14.4: Update `CHANGELOG.md`**@@ -1626,7 +1626,7 @@ language convention. Markdown posts, chat replies, descriptions, andin-line comments in non-em-dash languages get ` - `. Technicaldocumentation and code stay as-is.-Связано: [[feedback_always_russian]] (dialog language is the+Related: [[feedback_always_russian]] (dialog language is thedefault selector). Brain preference candidate: `pref-no-em-dashes`.```diff --git a/docs/plans/2026-05-20-v0.10.10-design.md b/docs/plans/2026-05-20-v0.10.10-design.mdindex f15bb401..4fce7e3d 100644--- a/docs/plans/2026-05-20-v0.10.10-design.md+++ b/docs/plans/2026-05-20-v0.10.10-design.md@@ -464,7 +464,7 @@ After implementation lands, the vault-sidesmall places (operator-confirmed edit, no automated cron writes):- §1 shipped-in header gains "+v0.10.10 Most-applied (30d) section".-- "Спорные / отложенные" — the `brain_context` bullet is removed+- "Disputed / deferred" — the `brain_context` bullet is removed(no longer disputed).- "Deferred work" — the §32B CLI bullet moves to a shipped referencepointing at the v0.10.10 release notes.diff --git a/docs/plans/2026-05-20-v0.10.10-impl.md b/docs/plans/2026-05-20-v0.10.10-impl.mdindex 14aabafe..7bc043d4 100644--- a/docs/plans/2026-05-20-v0.10.10-impl.md+++ b/docs/plans/2026-05-20-v0.10.10-impl.md@@ -2270,9 +2270,9 @@ Wait for a yes.If yes:1. In the `## Tier S` section, find the `### ✅ 1.` entry. Append to- its `*(реализовано в v0.9.1)*` parenthetical:+ its `*(implemented in v0.9.1)*` parenthetical:`; + v0.10.10 Most-applied (30d) section`.-2. In the `## Спорные / отложенные` section, remove the bullet+2. In the `## Disputed / deferred` section, remove the bulletstarting with **Pull-bootstrap MCP-tool `brain_context`/`brain_principles`**.3. In the `## Deferred work` section, find the bullet that startswith **§32B CLI verb `o2b brain note`** and replace it with adiff --git a/src/core/brain/context-lanes.ts b/src/core/brain/context-lanes.tsindex 4ada194e..a6087f65 100644--- a/src/core/brain/context-lanes.ts+++ b/src/core/brain/context-lanes.ts@@ -30,9 +30,6 @@ export interface ContextLaneInput {readonly manualLane?: ContextLaneName | null;}-const CONSTRAINT_RE =- /\b(?:avoid|do not|don't|must not|never|no longer|forbid|forbidden)\b|\b(?:не делай|никогда|запрещено|избегай)\b/iu;-export function normalizeContextLane(raw: unknown): ContextLaneName | null {if (typeof raw !== "string") return null;const value = raw.trim().toLowerCase();@@ -40,10 +37,18 @@ export function normalizeContextLane(raw: unknown): ContextLaneName | null {return null;}+/**+ * Classify a source into a context lane.+ *+ * Language-agnostic by construction: the lane is never inferred from+ * the words of the principle/body (an English/Russian "never/forbidden"+ * prose scan would silently misclassify every other language). The+ * `constraints` lane is opt-in via the explicit `context_lane:`+ * frontmatter field (surfaced here as `manualLane`); without it, the+ * lane follows only the structural page tier.+ */export function classifyContextLane(input: ContextLaneInput): ContextLaneName {if (input.manualLane) return input.manualLane;- const text = `${input.principle}\n${input.body}`;- if (CONSTRAINT_RE.test(text)) return "constraints";if (input.tier === "peripheral") return "consider";return "directives";}diff --git a/src/core/brain/git/decisions.ts b/src/core/brain/git/decisions.tsindex c0b02e74..4f323b6c 100644--- a/src/core/brain/git/decisions.ts+++ b/src/core/brain/git/decisions.ts@@ -2,9 +2,10 @@* Commit-decision miner (Project History Suite, t_93d299bb).** Deterministic heuristics over INGESTED commit records (never live- * git): conventional breaking markers, BREAKING CHANGE footers, a- * word-boundary decision keyword set, and revert shape. Same store,- * same candidates, same order - no LLM classification anywhere.+ * git): conventional breaking markers, BREAKING CHANGE footers, and+ * revert shape - all fixed commit-protocol tokens, language-agnostic.+ * Same store, same candidates, same order - no LLM classification+ * anywhere.** Each decision-shaped commit becomes one draft ADR candidate note at* `Brain/decisions/candidates/adr-<shortsha>-<slug>.md` with the@@ -21,20 +22,6 @@ import { atomicWriteFileSync } from "../../fs-atomic.ts";import { listGitCommits } from "./store.ts";import type { GitCommitRecord } from "./store.ts";-/** Phrases that mark a commit message as decision-shaped. */-const DECISION_KEYWORDS = [- "decide",- "decided",- "decision",- "adopt",- "adopted",- "switch to",- "migrate to",- "instead of",- "adr",- "rationale",-] as const;-const CONVENTIONAL_BREAKING_RE = /^[a-z]+(\([^)]*\))?!:/;const REVERT_RE = /^revert\b/i;const SLUG_MAX = 40;@@ -48,15 +35,15 @@ const CANDIDATE_SHA_LEN = 12;* Empty array = not decision-shaped.*/export function detectDecisionSignals(subject: string, body: string): ReadonlyArray<string> {+ // Language-agnostic by construction: only structural commit-protocol+ // markers count as decision signals. The old English keyword set+ // ("decide", "adopt", ...) never fired on a commit history written in+ // any other language; conventional-commit and git-native markers+ // (`type!:`, the `BREAKING CHANGE:` footer, the `Revert` prefix) are+ // fixed protocol tokens git/tooling emit, not free prose.const signals: string[] = [];if (CONVENTIONAL_BREAKING_RE.test(subject)) signals.push("conventional_breaking");if (/^BREAKING CHANGE\b/m.test(body)) signals.push("breaking_change_footer");- const haystack = `${subject}\n${body}`.toLowerCase();- for (const keyword of DECISION_KEYWORDS) {- // Word-boundary match: 'adopt' must not fire inside 'adoption'.- const re = new RegExp(`\\b${keyword.replace(/ /g, "\\s+")}\\b`);- if (re.test(haystack)) signals.push(`decision_keyword:${keyword}`);- }if (REVERT_RE.test(subject)) signals.push("revert");return Object.freeze(signals);}diff --git a/src/core/search/coverage.ts b/src/core/search/coverage.tsindex be42e555..8102fa7b 100644--- a/src/core/search/coverage.ts+++ b/src/core/search/coverage.ts@@ -14,34 +14,24 @@* in. Deterministic, no LLM, no clock.*/-const STOPWORDS = new Set([- "about",- "and",- "are",- "for",- "from",- "how",- "the",- "this",- "that",- "what",- "when",- "where",- "with",-]);-/** Share of the corpus a term may appear in and still count as rare. */export const RARE_TERM_CORPUS_SHARE = 0.02;/**- * Significant query terms: length >= 3, not a stopword, deduplicated,- * in query order. The same extraction the evidence pack has used since- * PR #54 — now owned here so every consumer agrees.+ * Significant query terms: length >= 3, deduplicated, in query order.+ *+ * Language-agnostic by construction: there is deliberately NO stopword+ * list. A per-language stopword set (the old English-only one) would+ * under-filter every other language while pretending to help. Instead,+ * corpus-common terms are handled downstream by the IDF weighting in+ * {@link buildCoverageReport} — a term that appears in most documents+ * earns near-zero IDF and contributes almost nothing to the weighted+ * coverage, in any language, without a vocabulary list.*/export function significantTerms(query: string): string[] {const terms = new Set<string>();for (const token of query.toLocaleLowerCase().split(/[^\p{L}\p{N}_-]+/u)) {- if (token.length >= 3 && !STOPWORDS.has(token)) terms.add(token);+ if (token.length >= 3) terms.add(token);}return [...terms];}diff --git a/src/core/search/evidence-pack.ts b/src/core/search/evidence-pack.tsindex c271be53..53e48e04 100644--- a/src/core/search/evidence-pack.ts+++ b/src/core/search/evidence-pack.ts@@ -72,8 +72,29 @@ export interface EvidencePack {readonly completeness?: CompletenessReport;}-const TERMINAL_STATE_RE =- /\b(?:archived|closed|deprecated|done|resolved|retired|superseded|terminal)\b/iu;+/**+ * Controlled terminal-status vocabulary. These are accepted values of+ * the frontmatter `status:` field - a controlled enum the system owns -+ * NOT words scanned inside a note's prose. A note is terminal when its+ * DECLARED status is one of these, so the language the note is written+ * in is irrelevant (the old regex scanned path/title/content for these+ * English words and false-fired on any note that merely mentioned them).+ */+const TERMINAL_STATUS_VALUES: ReadonlySet<string> = new Set([+ "archived",+ "closed",+ "deprecated",+ "done",+ "resolved",+ "retired",+ "superseded",+ "terminal",+]);++/** True when a frontmatter `status` value denotes a terminal state. */+export function isTerminalStatus(status: unknown): boolean {+ return typeof status === "string" && TERMINAL_STATUS_VALUES.has(status.trim().toLowerCase());+}function includesTerm(result: BrainSearchResult, term: string): boolean {return termIncludedIn(`${result.path}\n${result.title ?? ""}\n${result.content}`, term);@@ -87,12 +108,25 @@ function supportCoverage(return matched.length / significant.length;}-export function evidenceTerminalState(result: BrainSearchResult): boolean {- return TERMINAL_STATE_RE.test(`${result.path}\n${result.title ?? ""}\n${result.content}`);+/**+ * Whether a result is in a terminal state for the current query.+ * Terminality is decided structurally by the caller (reading the+ * frontmatter `status:` field via {@link isTerminalStatus}) and handed+ * in as the set of terminal paths - this function never inspects the+ * note's text.+ */+export function evidenceTerminalState(+ result: BrainSearchResult,+ terminalPaths: ReadonlySet<string>,+): boolean {+ return terminalPaths.has(result.path);}-function withTerminalReason(result: BrainSearchResult): BrainSearchResult {- if (!evidenceTerminalState(result)) return result;+function withTerminalReason(+ result: BrainSearchResult,+ terminalPaths: ReadonlySet<string>,+): BrainSearchResult {+ if (!evidenceTerminalState(result, terminalPaths)) return result;if (result.reasons.some((reason) => reason.startsWith("evidence_terminal_downrank:"))) {return result;}@@ -104,14 +138,17 @@ function withTerminalReason(result: BrainSearchResult): BrainSearchResult {export function downrankTerminalEvidenceResults(results: ReadonlyArray<BrainSearchResult>,+ terminalPaths: ReadonlySet<string>,): ReadonlyArray<BrainSearchResult> {- return results.map(withTerminalReason).toSorted((left, right) => {- const leftTerminal = evidenceTerminalState(left);- const rightTerminal = evidenceTerminalState(right);- if (leftTerminal !== rightTerminal) return leftTerminal ? 1 : -1;- if (right.score !== left.score) return right.score - left.score;- return left.chunkId - right.chunkId;- });+ return results+ .map((result) => withTerminalReason(result, terminalPaths))+ .toSorted((left, right) => {+ const leftTerminal = evidenceTerminalState(left, terminalPaths);+ const rightTerminal = evidenceTerminalState(right, terminalPaths);+ if (leftTerminal !== rightTerminal) return leftTerminal ? 1 : -1;+ if (right.score !== left.score) return right.score - left.score;+ return left.chunkId - right.chunkId;+ });}function abstentionMessage(@@ -131,6 +168,7 @@ export function buildEvidencePack(query: string,results: ReadonlyArray<BrainSearchResult>,verification?: EvidenceVerification,+ terminalPaths: ReadonlySet<string> = new Set(),): EvidencePack {const significant = Object.freeze(significantTerms(query));const matchedSet = new Set<string>();@@ -138,7 +176,7 @@ export function buildEvidencePack(const matched = significant.filter((term) => includesTerm(result, term));for (const term of matched) matchedSet.add(term);const missing = significant.filter((term) => !matched.includes(term));- const terminalState = evidenceTerminalState(result);+ const terminalState = evidenceTerminalState(result, terminalPaths);const terminalDownranked = result.reasons.some((reason) =>reason.startsWith("evidence_terminal_downrank:"),);diff --git a/src/core/search/query-expansion.ts b/src/core/search/query-expansion.tsindex b8c998aa..c4d6451f 100644--- a/src/core/search/query-expansion.ts+++ b/src/core/search/query-expansion.ts@@ -8,9 +8,10 @@* string work plus one read of the vault's entity registry - no local* model, no paid call, identical output for identical input.*- * - lex: query tokens minus stopwords (the FTS lane is implicit AND,- * so one stopword absent from the target note kills the match);- * falls back to the raw tokens when everything is a stopword.+ * - lex: the query tokens (deduped, capped), minus any the caller+ * flags as corpus-common via `commonTokens` (high document frequency,+ * not a hardcoded stopword list - the FTS lane is implicit AND, so a+ * ubiquitous token would otherwise kill the match). Language-agnostic.* - vec: the raw query, plus one entity-context line when registry* entities match a query token - anchors the semantic lane to the* vault's own vocabulary.@@ -31,53 +32,16 @@ export const EXPANSION_MAX_LEX_TERMS = 8;/** Default cap on matched registry entities woven into vec/hyde. */export const EXPANSION_MAX_ENTITIES = 3;-/**- * Small closed-class English stopword set. Deliberately conservative:- * a missed stopword only leaves one extra AND term, while an- * over-eager list would silently drop signal.- */-const STOPWORDS: ReadonlySet<string> = new Set([- "a",- "an",- "and",- "are",- "as",- "at",- "be",- "but",- "by",- "for",- "from",- "how",- "in",- "is",- "it",- "me",- "my",- "of",- "on",- "or",- "our",- "show",- "that",- "the",- "their",- "this",- "to",- "was",- "we",- "what",- "when",- "where",- "which",- "who",- "why",- "with",-]);-export interface ExpandQueryOptions {readonly maxLexTerms?: number;readonly maxEntities?: number;+ /**+ * Corpus-common tokens to drop from the implicit-AND lex lane. The+ * caller derives these from document frequency (a token present in+ * most documents carries little signal, in ANY language), never from a+ * hardcoded stopword list. Defaults to none.+ */+ readonly commonTokens?: ReadonlySet<string>;}/**@@ -93,11 +57,16 @@ export function expandQuery(const maxEntities = Math.max(0, opts.maxEntities ?? EXPANSION_MAX_ENTITIES);const trimmed = query.trim();- const tokens = [...new Set(tokenizeForExpansion(trimmed))];- const meaningful = tokens.filter((t) => !STOPWORDS.has(t));- // Fall back to the raw tokens when the whole query is stopwords -- // an empty lex lane would turn the FTS lane off entirely.- const baseTokens = meaningful.length > 0 ? meaningful : tokens;+ // Language-agnostic by construction: no stopword list. Corpus-common+ // tokens (high document frequency, supplied by the caller) are dropped+ // from the implicit-AND lex lane so one ubiquitous word cannot kill the+ // match - in ANY language, driven by data rather than an English word+ // list. Fall back to all tokens when every token is common, so the lex+ // lane never goes empty.+ const commonTokens = opts.commonTokens ?? new Set<string>();+ const allTokens = [...new Set(tokenizeForExpansion(trimmed))];+ const meaningful = allTokens.filter((t) => !commonTokens.has(t));+ const baseTokens = meaningful.length > 0 ? meaningful : allTokens;const lexTerms = baseTokens.slice(0, maxLexTerms);const entityNames = matchEntities(vault, baseTokens).slice(0, maxEntities);diff --git a/src/core/search/search.ts b/src/core/search/search.tsindex ab1d7087..7931abe4 100644--- a/src/core/search/search.ts+++ b/src/core/search/search.ts@@ -35,7 +35,11 @@ import {import { extractEntities } from "./entities.ts";import { expandQueryEntities } from "./entity-alias.ts";import { buildCoverageReport, significantTerms, termIncludedIn } from "./coverage.ts";-import { buildEvidencePack, downrankTerminalEvidenceResults } from "./evidence-pack.ts";+import {+ buildEvidencePack,+ downrankTerminalEvidenceResults,+ isTerminalStatus,+} from "./evidence-pack.ts";import type { EvidenceUnionRecord, EvidenceVerification } from "./evidence-pack.ts";import { runFtsQueryDetailed } from "./fts.ts";import { mmrRerank } from "./mmr.ts";@@ -215,16 +219,10 @@ export async function search(const pathPrefix = assertSafePathPrefix(opts.pathPrefix);const policy = resolveSemanticPolicy(config, opts);const warnings: string[] = [];- // Opt-in local expansion (t_2fa95db1): an explicit structured- // document always wins; expansion only fills the gap.- const structured =- opts.structuredQuery ?? (expandActive === true ? expandQuery(config.vault, query) : undefined);const sessionFocus =opts.sessionFocus === undefined? readActiveSessionFocus(config, opts.focusSession, Date.now()): opts.sessionFocus;- const keywordQuery = structuredKeywordQuery(query, structured);- const semanticLaneQuery = structuredSemanticQuery(structured);// Time-aware recall (recall-trust-suite): resolve since/until up front// so invalid input fails fast, before any store I/O.const timeRange =@@ -232,11 +230,6 @@ export async function search(? resolveTimeRange({ since: opts.since, until: opts.until }, Date.now()): null;- // Query plan (v0.20.0): one structural pass yields the intent weight- // profile and the cache key. Pure; no I/O. Expanded terms (if any) are- // folded in once they have been derived from the store below.- const basePlan = buildQueryPlan(keywordQuery, [], structured?.intent);-// Read-only origins (cross-vault search) disable self-healing: a// rebuild would write an index INTO the external vault. Default// (selfHeal absent) keeps the legacy heal-and-retry behaviour.@@ -245,6 +238,25 @@ export async function search(? await Store.open(config, { mode: "read" }): await openReadOrSelfHeal(config);try {+ // Opt-in local expansion (t_2fa95db1): an explicit structured+ // document always wins; expansion only fills the gap. The lex lane's+ // corpus-common tokens are derived from document frequency here+ // (language-agnostic, no stopword list) so an implicit-AND query is+ // not killed by a word that is ubiquitous in this vault.+ const commonTokens =+ opts.structuredQuery === undefined && expandActive === true+ ? highFrequencyTokens(store, query)+ : new Set<string>();+ const structured =+ opts.structuredQuery ??+ (expandActive === true ? expandQuery(config.vault, query, { commonTokens }) : undefined);+ const keywordQuery = structuredKeywordQuery(query, structured);+ const semanticLaneQuery = structuredSemanticQuery(structured);+ // Query plan (v0.20.0): one structural pass yields the intent weight+ // profile and the cache key. Expanded terms (if any) are folded in+ // once they have been derived from the store below.+ const basePlan = buildQueryPlan(keywordQuery, [], structured?.intent);+// Persistent query cache (v0.20.0): opt-in. Keyed by the request +// base plan hash + a config fingerprint, gated by the corpus// generation and a TTL. A hit returns the previously computed@@ -785,9 +797,19 @@ export async function search(}),): withCanonicalReasons;+ // Terminal-state downrank (recall-trust-suite) is now structural and+ // language-agnostic: a result is terminal when its frontmatter+ // `status:` field declares a terminal value (controlled vocabulary),+ // never because the note's prose happens to contain an English word+ // like "done". One cached frontmatter read per candidate path, only+ // in evidence-pack mode.+ const terminalPaths =+ opts.evidencePack === true+ ? buildTerminalPaths(config.vault, withSecondPassReasons)+ : new Set<string>();const finalResults =opts.evidencePack === true- ? downrankTerminalEvidenceResults(withSecondPassReasons)+ ? downrankTerminalEvidenceResults(withSecondPassReasons, terminalPaths): withSecondPassReasons;const evidencePack =opts.evidencePack === true@@ -795,6 +817,7 @@ export async function search(query,finalResults,buildEvidenceVerification(store, query, finalResults, pathPrefix),+ terminalPaths,): undefined;@@ -894,6 +917,71 @@ function applyTraversal();}+/** A token in at least this share of the corpus is treated as common. */+const COMMON_TOKEN_CORPUS_SHARE = 0.5;+/**+ * Floor on document frequency before a token can be "common". Below this+ * a corpus is too small to tell a stopword-like word from a rare one, so+ * nothing is flagged (a 2-document vault must not call a word that appears+ * once "ubiquitous").+ */+const MIN_COMMON_DOCUMENT_FREQUENCY = 2;++/**+ * Corpus-common query tokens, derived from document frequency. A token+ * present in at least {@link COMMON_TOKEN_CORPUS_SHARE} of the indexed+ * documents (and in at least {@link MIN_COMMON_DOCUMENT_FREQUENCY} of+ * them) carries little discriminating signal - in ANY language - so the+ * lex expansion lane drops it rather than letting one ubiquitous word+ * kill an implicit-AND match. Language-agnostic: no stopword list.+ *+ * Note: document frequency is measured through the FTS index, whose+ * tokenization can differ slightly from `tokenizeForExpansion`. A miss+ * only fails to flag a token as common (it stays in the lex lane), so the+ * fallback is always the safe, non-lossy direction.+ */+function highFrequencyTokens(store: Store, query: string): ReadonlySet<string> {+ const tokens = [...new Set(tokenizeForExpansion(query))];+ if (tokens.length === 0) return new Set();+ const documentCount = store.counts().documents;+ if (documentCount === 0) return new Set();+ const threshold = COMMON_TOKEN_CORPUS_SHARE * documentCount;+ const df = store.documentFrequencies(tokens);+ const common = new Set<string>();+ for (const token of tokens) {+ const freq = df.get(token) ?? 0;+ if (freq >= MIN_COMMON_DOCUMENT_FREQUENCY && freq >= threshold) common.add(token);+ }+ return common;+}++/**+ * Build the set of terminal-state paths for evidence-pack downranking.+ * Reads each unique candidate path's frontmatter `status:` field once+ * and includes the path when the declared status is terminal (controlled+ * vocabulary). A missing or unreadable status is non-terminal. This is+ * the language-agnostic replacement for scanning note prose for English+ * status words.+ */+function buildTerminalPaths(+ vault: string,+ results: ReadonlyArray<BrainSearchResult>,+): ReadonlySet<string> {+ const terminal = new Set<string>();+ const seen = new Set<string>();+ for (const r of results) {+ if (seen.has(r.path)) continue;+ seen.add(r.path);+ try {+ const [meta] = parseFrontmatter(join(vault, r.path));+ if (isTerminalStatus((meta as Record<string, unknown>)["status"])) terminal.add(r.path);+ } catch {+ // Unreadable frontmatter is non-terminal.+ }+ }+ return terminal;+}+/** Cap on extra records fetched per uncovered term (Feature C union). */const UNION_RECORDS_PER_TERM = 2;/** Cap on the total recall-union fetch per query. */diff --git a/src/core/search/surfacing-gate.ts b/src/core/search/surfacing-gate.tsindex 74c16996..77b65ca7 100644--- a/src/core/search/surfacing-gate.ts+++ b/src/core/search/surfacing-gate.ts@@ -1,13 +1,10 @@export type SurfacingGateReason =| "explicit"- | "memory_question"- | "uncertain_question"| "duplicate"| "empty"- | "greeting"| "slash_command"| "shell_command"- | "no_recall_intent";+ | "default_retrieve";export interface SurfacingGateInput {readonly prompt: string;@@ -20,16 +17,6 @@ export interface SurfacingGateDecision {readonly reason: SurfacingGateReason;}-const GREETINGS = new Set([- "hello",- "hi",- "hey",- "привет",- "здравствуй",- "здравствуйте",- "добрый день",-]);-const SHELL_COMMANDS = new Set(["awk","bun",@@ -54,21 +41,10 @@ const SHELL_COMMANDS = new Set(["yarn",]);-const MEMORY_PATTERNS = [- /\b(remember|recall|memory|context|notes?|decid(?:e|ed)|decision|search|find)\b/iu,- /\b(what did we|where did we|when did we|have we|did we)\b/iu,- /\b(помнишь|вспомни|найди|контекст|решили|обсуждали|заметки)\b/iu,-];-function normalizePrompt(prompt: string): string {return prompt.trim().replace(/\s+/gu, " ").toLocaleLowerCase();}-function isGreeting(normalized: string): boolean {- const stripped = normalized.replace(/[!.?]+$/u, "");- return GREETINGS.has(stripped);-}-function isSlashCommand(normalized: string): boolean {return normalized.startsWith("/") && !normalized.includes(" ");}@@ -79,10 +55,18 @@ function isShellOnlyPrompt(normalized: string): boolean {return SHELL_COMMANDS.has(firstToken);}-function hasMemoryIntent(normalized: string): boolean {- return MEMORY_PATTERNS.some((pattern) => pattern.test(normalized));-}-+/**+ * Decide whether a prompt should trigger memory retrieval.+ *+ * Language-agnostic by construction: the gate never inspects prompt+ * words against any natural-language vocabulary, so a prompt in any+ * language is treated identically. Only structural signals suppress+ * retrieval — an empty prompt, a verbatim repeat of the previous one, a+ * slash command, or a single shell command (command names, not human+ * language). Everything else FAILS OPEN: we retrieve and let ranking+ * decide relevance, because a missed recall is worse than a cheap+ * no-result search.+ */export function evaluateSurfacingGate(input: SurfacingGateInput): SurfacingGateDecision {if (input.explicit === true) return Object.freeze({ retrieve: true, reason: "explicit" });@@ -93,17 +77,10 @@ export function evaluateSurfacingGate(input: SurfacingGateInput): SurfacingGateDif (previous !== null && previous === normalized) {return Object.freeze({ retrieve: false, reason: "duplicate" });}- if (isGreeting(normalized)) return Object.freeze({ retrieve: false, reason: "greeting" });if (isSlashCommand(normalized))return Object.freeze({ retrieve: false, reason: "slash_command" });if (isShellOnlyPrompt(normalized)) {return Object.freeze({ retrieve: false, reason: "shell_command" });}- if (hasMemoryIntent(normalized)) {- return Object.freeze({ retrieve: true, reason: "memory_question" });- }- if (normalized.endsWith("?") && normalized.length >= 60) {- return Object.freeze({ retrieve: true, reason: "uncertain_question" });- }- return Object.freeze({ retrieve: false, reason: "no_recall_intent" });+ return Object.freeze({ retrieve: true, reason: "default_retrieve" });}diff --git a/tests/cli/brain-benchmark.test.ts b/tests/cli/brain-benchmark.test.tsindex 54d8b82f..d0203bf4 100644--- a/tests/cli/brain-benchmark.test.ts+++ b/tests/cli/brain-benchmark.test.ts@@ -27,7 +27,11 @@ beforeEach(async () => {join(vault, "canary.md"),"# Canary rollout\n\nShip one instance first, observe, expand gradually.\n",);- writeFileSync(join(vault, "other.md"), "# Other\n\nNothing relevant whatsoever.\n");+ // "the" is seeded across >= 2 of the other notes (majority of the+ // corpus) but NOT canary.md, so the DF-driven, language-agnostic+ // common-token filter drops it from the implicit-AND lex lane.+ writeFileSync(join(vault, "other.md"), "# Other\n\nNothing in the list is relevant here.\n");+ writeFileSync(join(vault, "notes.md"), "# Notes\n\nThe team reviewed the plan in the meeting.\n");datasetPath = join(tmp, "dataset.json");writeFileSync(datasetPath,diff --git a/tests/cli/brain-context-pack.test.ts b/tests/cli/brain-context-pack.test.tsindex 995788b5..4b7e86bc 100644--- a/tests/cli/brain-context-pack.test.ts+++ b/tests/cli/brain-context-pack.test.ts@@ -27,7 +27,9 @@ function writePref(slug: string, principle: string, extra = ""): void {test("brain context-pack --lanes --json returns polarity lanes", async () => {writePref("directive", "Prefer concise answers", "tier: core");- writePref("constraint", "Never expose tokens", "tier: core");+ // The constraints lane is opt-in via the explicit context_lane field,+ // not inferred from prose words like "Never" (language-agnostic).+ writePref("constraint", "Never expose tokens", "tier: core\ncontext_lane: constraints");const out = await runCli(["brain", "context-pack", "--vault", vault, "--max-tokens", "10000", "--lanes", "--json"],diff --git a/tests/core/brain/context-pack.test.ts b/tests/core/brain/context-pack.test.tsindex 2deaf7d4..e984d52c 100644--- a/tests/core/brain/context-pack.test.ts+++ b/tests/core/brain/context-pack.test.ts@@ -257,6 +257,10 @@ describe("packContext", () => {topic: "security",principle: "Never expose secret tokens",tier: "core",+ // The constraints lane is opt-in via the explicit context_lane+ // field - it is no longer inferred from prose words like "Never"+ // (that only ever worked for English/Russian).+ context_lane: "constraints",});writePref("consider", {topic: "taste",diff --git a/tests/core/brain/git/decisions.test.ts b/tests/core/brain/git/decisions.test.tsindex c5a9d909..80d54cf1 100644--- a/tests/core/brain/git/decisions.test.ts+++ b/tests/core/brain/git/decisions.test.ts@@ -46,7 +46,7 @@ afterEach(() => {// ── signal detection ────────────────────────────────────────────────────────-test("detectDecisionSignals matches deterministic decision shapes", () => {+test("detectDecisionSignals matches only structural commit-protocol shapes", () => {expect(detectDecisionSignals("feat!: drop legacy index format", "")).toContain("conventional_breaking",);@@ -56,17 +56,16 @@ test("detectDecisionSignals matches deterministic decision shapes", () => {expect(detectDecisionSignals("feat: new api", "BREAKING CHANGE: payload renamed")).toContain("breaking_change_footer",);- expect(detectDecisionSignals("chore: migrate to bun test runner", "")).toContain(- "decision_keyword:migrate to",- );- expect(- detectDecisionSignals("docs: explain choice", "We decided to adopt JSONL instead of SQLite."),- ).toEqual(expect.arrayContaining(["decision_keyword:decided", "decision_keyword:adopt"]));expect(detectDecisionSignals('Revert "feat: bad idea"', "")).toContain("revert");// Plain work is NOT decision-shaped.expect(detectDecisionSignals("fix: off-by-one in pager", "small fix")).toEqual([]);- // Substrings inside words do not match (no false 'adopt' in 'adoption-rate.ts').- expect(detectDecisionSignals("chore: update adoption-rate metrics", "")).toEqual([]);+ // Language-agnostic by construction: prose words ("decided", "adopt",+ // "migrate to") are NOT decision signals - they only ever fired on+ // English commit histories. Only structural protocol markers count.+ expect(detectDecisionSignals("chore: migrate to bun test runner", "")).toEqual([]);+ expect(+ detectDecisionSignals("docs: explain choice", "We decided to adopt JSONL instead of SQLite."),+ ).toEqual([]);});// ── mining ──────────────────────────────────────────────────────────────────@@ -75,7 +74,7 @@ test("mineCommitDecisions writes candidates for decision commits only, with provappendGitRecords(vault, KEY, [commit("a".repeat(40), "feat!: drop legacy index format", "BREAKING CHANGE: rebuild needed"),commit("b".repeat(40), "fix: typo"),- commit("c".repeat(40), "chore: migrate to bun test runner"),+ commit("c".repeat(40), 'Revert "feat: half-baked migration"'),]);const res = mineCommitDecisions(vault, KEY);expect(res.scanned).toBe(3);diff --git a/tests/core/search/coverage.test.ts b/tests/core/search/coverage.test.tsindex fd3bdf2d..45a073a8 100644--- a/tests/core/search/coverage.test.ts+++ b/tests/core/search/coverage.test.ts@@ -23,8 +23,16 @@ import { search } from "../../../src/core/search/search.ts";import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";describe("coverage engine (pure)", () => {- test("significantTerms drops stopwords and short tokens", () => {- expect(significantTerms("what is the alpha zephyr")).toEqual(["alpha", "zephyr"]);+ test("significantTerms drops short tokens but keeps every word - no stopword list", () => {+ // Language-agnostic: there is no stopword list. Tokens under length 3+ // are dropped structurally ("is"); common words like "the"/"what" are+ // kept here and deprioritised downstream by IDF, in any language.+ expect(significantTerms("what is the alpha zephyr")).toEqual([+ "what",+ "the",+ "alpha",+ "zephyr",+ ]);});test("idf is monotonically decreasing in document frequency", () => {diff --git a/tests/core/search/evidence-pack.test.ts b/tests/core/search/evidence-pack.test.tsindex 8296a761..f9d20658 100644--- a/tests/core/search/evidence-pack.test.ts+++ b/tests/core/search/evidence-pack.test.ts@@ -36,20 +36,36 @@ test("buildEvidencePack reports matched and missing significant terms", () => {expect(pack.records[0]?.whyRetrieved).toEqual(["fts5_bm25: 1.000"]);});-test("buildEvidencePack marks terminal-state records", () => {- const pack = buildEvidencePack("alpha", [result("done.md", "alpha superseded by another note")]);+test("buildEvidencePack marks terminal-state records from the terminal path set", () => {+ // Terminality is structural (declared frontmatter `status:`), handed in+ // as the terminal-path set - never inferred from the note's prose.+ const terminalPaths = new Set(["done.md"]);+ const pack = buildEvidencePack(+ "alpha",+ [result("done.md", "alpha and more")],+ undefined,+ terminalPaths,+ );expect(pack.records[0]?.terminalState).toBe(true);expect(pack.records[0]?.droppedCandidateReasons).toEqual([]);});+test("buildEvidencePack does not infer terminal state from prose", () => {+ // "superseded" / "done" in the body must NOT make a record terminal+ // when no terminal path set is supplied (the old regex false-fired here).+ const pack = buildEvidencePack("alpha", [result("notes.md", "alpha superseded by another note")]);++ expect(pack.records[0]?.terminalState).toBe(false);+});+test("buildEvidencePack reports terminal downrank only when applied", () => {- const pack = buildEvidencePack("alpha", [- result("done.md", "alpha superseded by another note", [- "fts5_bm25: 1.000",- "evidence_terminal_downrank: true",- ]),- ]);+ const pack = buildEvidencePack(+ "alpha",+ [result("done.md", "alpha and more", ["fts5_bm25: 1.000", "evidence_terminal_downrank: true"])],+ undefined,+ new Set(["done.md"]),+ );expect(pack.records[0]?.droppedCandidateReasons).toContain("terminal_state_downranked");});diff --git a/tests/core/search/query-expansion.test.ts b/tests/core/search/query-expansion.test.tsindex 7af6ac65..d62b7bda 100644--- a/tests/core/search/query-expansion.test.ts+++ b/tests/core/search/query-expansion.test.ts@@ -2,9 +2,10 @@* Deterministic query expansion producer (link-recall-intelligence,* Task 5 / t_2fa95db1): a bare query becomes a structured lex/vec/hyde* document for the EXISTING structured-query consumer - no model, no- * paid call. Lex strips stopwords for the implicit-AND FTS lane, vec- * adds an entity-context line when registry entities match, hyde is- * one template passage. Opt-in via `search(config, {expand: true})`.+ * paid call. Lex keeps every query token for the implicit-AND FTS lane+ * (no stopword list - language-agnostic), vec adds an entity-context+ * line when registry entities match, hyde is one template passage.+ * Opt-in via `search(config, {expand: true})`.*/import { test, expect, beforeEach, afterEach, describe } from "bun:test";@@ -39,7 +40,9 @@ describe("expandQuery", () => {const first = expandQuery(vault, "the canary rollout plan");const second = expandQuery(vault, "the canary rollout plan");expect(first).toEqual(second);- expect(first.lex.include).toEqual(["canary", "rollout", "plan"]);+ // No stopword list: every token is kept (deduped, capped), in any+ // language. Common words are deprioritised by ranking, not dropped here.+ expect(first.lex.include).toEqual(["the", "canary", "rollout", "plan"]);expect(first.lex.exclude).toEqual([]);expect(first.vec[0]).toBe("the canary rollout plan");expect(first.hyde).toHaveLength(1);@@ -47,7 +50,7 @@ describe("expandQuery", () => {expect(first.intent).toBeNull();});- test("keeps the raw tokens when every token is a stopword", () => {+ test("keeps every token - there is no stopword list to strip", () => {const doc = expandQuery(vault, "the and of");expect(doc.lex.include).toEqual(["the", "and", "of"]);});@@ -81,7 +84,10 @@ describe("expandQuery", () => {});describe("search --expand", () => {- test("stopword stripping recovers a hit the implicit-AND lane misses", async () => {+ test("--expand annotates the lex lane and preserves real hits", async () => {+ // Language-agnostic: expansion no longer strips a hardcoded English+ // stopword list (that recovery only ever worked for English). A query+ // whose own terms match is still found, with the lex lane annotated.writeFileSync(join(vault, "canary.md"),"# Canary rollout\n\nShip one instance first, observe, expand gradually.\n",@@ -89,10 +95,7 @@ describe("search --expand", () => {writeFileSync(join(vault, "other.md"), "# Other\n\nNothing relevant here at all.\n");await indexVault(config);- const plain = await search(config, { query: "the canary rollout" });- expect(plain.results).toHaveLength(0);-- const expanded = await search(config, { query: "the canary rollout", expand: true });+ const expanded = await search(config, { query: "canary rollout", expand: true });expect(expanded.results.some((r) => r.path === "canary.md")).toBe(true);expect(expanded.resultsdiff --git a/tests/core/search/search.test.ts b/tests/core/search/search.test.tsindex ff56b4d3..aaac402e 100644--- a/tests/core/search/search.test.ts+++ b/tests/core/search/search.test.ts@@ -216,7 +216,13 @@ test("search degrades semantic structured lanes when semantic search is disabled});test("search can return an evidence pack and downrank terminal-state support", async () => {- writeMd(vault, "Notes/terminal.md", "# Terminal\n\nalpha beta superseded by active note.");+ // Terminal state is declared via the controlled frontmatter `status:`+ // field, not inferred from prose - language-agnostic downranking.+ writeMd(+ vault,+ "Notes/terminal.md",+ "---\nstatus: superseded\n---\n\n# Terminal\n\nalpha beta old support.",+ );writeMd(vault, "Notes/active.md", "# Active\n\nalpha beta current support.");const cfg = makeConfig({ vault, dbPath });await indexVault(cfg);diff --git a/tests/core/search/surfacing-gate.test.ts b/tests/core/search/surfacing-gate.test.tsindex 444cee36..3b030ebc 100644--- a/tests/core/search/surfacing-gate.test.ts+++ b/tests/core/search/surfacing-gate.test.ts@@ -2,8 +2,8 @@ import { expect, test } from "bun:test";import { evaluateSurfacingGate } from "../../../src/core/search/surfacing-gate.ts";-test("evaluateSurfacingGate skips greetings, slash commands, shell-only prompts, and duplicates", () => {- expect(evaluateSurfacingGate({ prompt: "hello" }).reason).toBe("greeting");+test("evaluateSurfacingGate suppresses only structural non-recall prompts", () => {+ expect(evaluateSurfacingGate({ prompt: "" }).reason).toBe("empty");expect(evaluateSurfacingGate({ prompt: "/help" }).reason).toBe("slash_command");expect(evaluateSurfacingGate({ prompt: "git status" }).reason).toBe("shell_command");expect(@@ -14,26 +14,28 @@ test("evaluateSurfacingGate skips greetings, slash commands, shell-only prompts,).toBe("duplicate");});-test("evaluateSurfacingGate retrieves real memory questions and explicit requests", () => {- expect(- evaluateSurfacingGate({- prompt: "what did we decide about recall diagnostics?",- }),- ).toEqual({- retrieve: true,- reason: "memory_question",- });+test("evaluateSurfacingGate fails open for any non-suppressed prompt, in any language", () => {+ // No natural-language keyword list: an English memory question, a+ // greeting, a long uncertain question, and a non-Latin prompt all+ // fall through to the same language-agnostic retrieve decision.+ for (const prompt of [+ "what did we decide about recall diagnostics?",+ "hello",+ "Can you compare the options we discussed and tell me which path is less risky?",+ "что мы решили про диагностику recall?",+ "私たちは何を決めましたか",+ "هل ناقشنا هذا من قبل",+ ]) {+ expect(evaluateSurfacingGate({ prompt })).toEqual({+ retrieve: true,+ reason: "default_retrieve",+ });+ }+});++test("evaluateSurfacingGate honours an explicit recall request", () => {expect(evaluateSurfacingGate({ prompt: "hello", explicit: true })).toEqual({retrieve: true,reason: "explicit",});});--test("evaluateSurfacingGate fails open for uncertain long questions", () => {- expect(- evaluateSurfacingGate({- prompt:- "Can you compare the options we discussed and tell me which implementation path is less risky?",- }),- ).toEqual({ retrieve: true, reason: "uncertain_question" });-});diff --git a/tests/core/search/tuning.test.ts b/tests/core/search/tuning.test.tsindex 424bd582..576f99e1 100644--- a/tests/core/search/tuning.test.ts+++ b/tests/core/search/tuning.test.ts@@ -48,16 +48,24 @@ const DATASET = parseRecallBenchmarkDataset({],});+// "the" is deliberately seeded into 2 of the 3 notes (canary, other) but+// NOT into backup.md. It is therefore corpus-common (>= 50% document+// frequency) and dropped from the implicit-AND lex lane by the+// language-agnostic, DF-driven common-token filter - which is what lets+// the "the nightly backup" query reach backup.md only WITH expansion.function writeNotes(): void {writeFileSync(join(vault, "canary.md"),- "# Canary rollout\n\nShip one instance first, observe, expand gradually.\n",+ "# Canary rollout\n\nShip the first instance, observe, then expand the rollout gradually.\n",);writeFileSync(join(vault, "backup.md"),"# Nightly backup\n\nSnapshot databases every night, verify checksums offsite.\n",);- writeFileSync(join(vault, "other.md"), "# Other\n\nNothing relevant whatsoever.\n");+ writeFileSync(+ join(vault, "other.md"),+ "# Other\n\nNothing in the list here is relevant to the task.\n",+ );}describe("grid + apply", () => {diff --git a/tests/e2e/link-recall-intelligence.integration.test.ts b/tests/e2e/link-recall-intelligence.integration.test.tsindex b0975dd4..a93f1531 100644--- a/tests/e2e/link-recall-intelligence.integration.test.ts+++ b/tests/e2e/link-recall-intelligence.integration.test.ts@@ -59,9 +59,15 @@ afterEach(() => {test("the suite composes end to end on one vault", async () => {// -- 1. Vault: an alias owner, twin unlinked notes, one community ------+ // "the" is seeded into the majority of the prose notes below (but NOT+ // weekly-review.md) so that the DF-driven, language-agnostic common-token+ // filter treats it as corpus-common and drops it from the implicit-AND+ // lex lane - the language-neutral replacement for the old English+ // stopword list. weekly-review.md is the recovery target, so it must+ // omit "the" for the bare implicit-AND query to miss it.writeFileSync(join(vault, "project-alpha.md"),- '---\ntitle: Project Alpha\naliases: ["PA"]\n---\n\n# Project Alpha\n\nUmbrella initiative: importer, dashboard, billing pipeline.\n',+ '---\ntitle: Project Alpha\naliases: ["PA"]\n---\n\n# Project Alpha\n\nThe umbrella initiative covers the importer, the dashboard, and the billing pipeline.\n',);writeFileSync(join(vault, "weekly-review.md"),@@ -81,7 +87,7 @@ test("the suite composes end to end on one vault", async () => {.filter((t) => t !== name).map((t) => `[[${t}]]`).join(" ");- writeFileSync(join(vault, `${name}.md`), `# ${name}\n\nSee ${others}.\n`);+ writeFileSync(join(vault, `${name}.md`), `# ${name}\n\nSee the related teams ${others}.\n`);}// -- 2. Alias resolution materializes at index time --------------------diff --git a/tests/e2e/project-history.integration.test.ts b/tests/e2e/project-history.integration.test.tsindex 6c34db88..c0ea436f 100644--- a/tests/e2e/project-history.integration.test.ts+++ b/tests/e2e/project-history.integration.test.ts@@ -93,7 +93,7 @@ test("ingest -> find -> mine -> architect -> re-run: one coherent project memoryexpect(mined.created).toBe(1);const candidate = readFileSync(mined.notes[0]!, "utf8");expect(candidate).toContain("conventional_breaking");- expect(candidate).toContain("decision_keyword:adopt");+ expect(candidate).toContain("breaking_change_footer");expect(candidate).toContain("migrate to jsonl store");// ── Architect: structural notes for the same project. ──diff --git a/tests/mcp/context-pack-tool.test.ts b/tests/mcp/context-pack-tool.test.tsindex eb2aa368..7c8a4e5b 100644--- a/tests/mcp/context-pack-tool.test.ts+++ b/tests/mcp/context-pack-tool.test.ts@@ -112,7 +112,9 @@ describe("brain_context_pack tool — round trip", () => {);writeFileSync(join(vault, "Brain", "preferences", "pref-constraint.md"),- "---\nid: pref-constraint\ntopic: t\nprinciple: Never expose tokens\ntier: core\n---\n",+ // Constraints lane is opt-in via the explicit context_lane field,+ // not inferred from the prose word "Never" (language-agnostic).+ "---\nid: pref-constraint\ntopic: t\nprinciple: Never expose tokens\ntier: core\ncontext_lane: constraints\n---\n",);const server = new MCPServer({ vault, configPath });await initialize(server);diff --git a/tests/mcp/recall-gate-telemetry.test.ts b/tests/mcp/recall-gate-telemetry.test.tsindex 96cf7238..59ebe92a 100644--- a/tests/mcp/recall-gate-telemetry.test.ts+++ b/tests/mcp/recall-gate-telemetry.test.ts@@ -36,7 +36,7 @@ function tool(name: string) {}test("default off: the gate emits no telemetry", async () => {- const decision = (await tool("brain_recall_gate").handler(ctx(), { prompt: "hello" })) as {+ const decision = (await tool("brain_recall_gate").handler(ctx(), { prompt: "/help" })) as {retrieve: boolean;};expect(decision.retrieve).toBe(false);@@ -50,7 +50,7 @@ test("with recall_gate_telemetry on, decisions land as continuity records", asyntelemetry_host: "hermes",session_id: "sess-1",});- await tool("brain_recall_gate").handler(ctx(), { prompt: "hi" });+ await tool("brain_recall_gate").handler(ctx(), { prompt: "/help" });const records = listGateTelemetry(vault);expect(records).toHaveLength(2);expect(records.some((r) => r.payload["decision"] === "retrieve")).toBe(true);