Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript

refactor: make search and classification language-agnostic (#84)

Sol Aitken9886d9abdeee
diff --git a/docs/plans/2026-05-16-brain-search-design.md b/docs/plans/2026-05-16-brain-search-design.md
index f8564cbe..46fad5c8 100644
--- a/docs/plans/2026-05-16-brain-search-design.md
+++ b/docs/plans/2026-05-16-brain-search-design.md
@@ -407,7 +407,7 @@ Human format:
```
[1] Brain/preferences/pref-russian-replies.md • 0.81
line 12-38 • hybrid • mtime 2026-05-14
- Правила общения: всегда отвечать по-русски, кроме когда речь о…
+ Communication rules: always reply in the user's language, except when…
```
JSON format: object
diff --git a/docs/plans/2026-05-18-agent-discipline-tail-design.md b/docs/plans/2026-05-18-agent-discipline-tail-design.md
index 92520f4c..ec6f3938 100644
--- a/docs/plans/2026-05-18-agent-discipline-tail-design.md
+++ b/docs/plans/2026-05-18-agent-discipline-tail-design.md
@@ -625,8 +625,8 @@ for §D`). Each line names the trigger that would bring it back to
active planning.
`Plan/3. Agent logging discipline.md` status: `shipped`. §B / §D /
-§E sections move from `open:` to `shipped:`. The "Триггер на
-возврат" section is dropped — §30 is closed.
+§E sections move from `open:` to `shipped:`. The "return
+trigger" section is dropped — §30 is closed.
`Plan/2. Improvement.md` and `Plan/4. v0.10.5 review followups.md`
are untouched.
diff --git a/docs/plans/2026-05-18-agent-discipline-tail-impl.md b/docs/plans/2026-05-18-agent-discipline-tail-impl.md
index 7c00306b..3f16e629 100644
--- a/docs/plans/2026-05-18-agent-discipline-tail-impl.md
+++ b/docs/plans/2026-05-18-agent-discipline-tail-impl.md
@@ -3136,7 +3136,7 @@ existing `js-yaml` already on the dep tree.
- Modify: `/root/vault/Projects/OpenSecondBrain/Features/_summary.md`.
- [ ] **Step 1: Update §30 entry** — set to `✅ shipped fully in
- v0.10.7`. The current entry says `реализовано частично в
+ v0.10.7`. The current entry says `partially implemented in
v0.10.6 (§A+§C)`. Replace the closing parenthetical with
shipped-in-full and link to `[[Plan/3. Agent logging discipline]]`.
@@ -3152,7 +3152,7 @@ existing `js-yaml` already on the dep tree.
false-positive/false-negative pairs.
- [ ] **Step 3: Update the Agent-logging-discipline marker** —
- the line that previously said `реализовано частично в v0.10.6`
+ the line that previously said `partially implemented in v0.10.6`
becomes `shipped fully in v0.10.7 — §B (writer MCP split), §D
(daily discipline cron), §E (claude-memory bridge)`.
@@ -3167,7 +3167,7 @@ existing `js-yaml` already on the dep tree.
three open items move into `shipped:` (or rewrite the front-matter
to remove the `open:` field entirely).
-- [ ] **Step 2: Drop the "Триггер на возврат" section** — §30 is
+- [ ] **Step 2: Drop the "return trigger" section** — §30 is
closed; the trigger is no longer relevant.
- [ ] **Step 3: Add a tail note** — "v0.10.7 closed §B / §D / §E.
diff --git a/docs/plans/2026-05-18-brain-maturity-impl.md b/docs/plans/2026-05-18-brain-maturity-impl.md
index 21157d71..37bd9368 100644
--- a/docs/plans/2026-05-18-brain-maturity-impl.md
+++ b/docs/plans/2026-05-18-brain-maturity-impl.md
@@ -2039,8 +2039,8 @@ Expected: exit 0.
**Step 1: Replace the two deferred entries that this PR retires**
Find the deferred-work section, locate the two entries:
-- `§4 — per-runtime steering text для Claude Code и Codex.`
-- `§15 «good vs bad» SKILL-секция.`
+- `§4 — per-runtime steering text for Claude Code and Codex.`
+- `§15 «good vs bad» SKILL section.`
Replace them with one consolidated line:
diff --git a/docs/plans/2026-05-18-v0.10.6-design.md b/docs/plans/2026-05-18-v0.10.6-design.md
index 693e1888..b3db0c71 100644
--- a/docs/plans/2026-05-18-v0.10.6-design.md
+++ b/docs/plans/2026-05-18-v0.10.6-design.md
@@ -806,18 +806,18 @@ no separate `§31.3` entry needed.
- §5 — mark as fully shipped (manifest + abort drift detection
delivered in v0.10.6). Update tail text.
-- §14 — append "v0.10.6: keyboard-доступная инспекция узлов и
- layout-state persistence через localStorage". Remove the two
+- §14 — append "v0.10.6: keyboard-accessible node inspection and
+ layout-state persistence via localStorage". Remove the two
matching items from `## Deferred work` and from the §14 inline
parenthetical.
-- §22 — replace "6" header with "✅ … *(реализовано в v0.10.6)*".
-- §28 — same: "✅ … *(реализовано в v0.10.6)*".
-- §30 — "✅ Agent logging discipline — §A + §C *(реализовано в
- v0.10.6: stop-guardrail расширен на brain-events, SKILL trigger
- переформулирован; §B / §D / §E остаются открытыми)*".
-- §31 — "✅ v0.10.5 review followups *(закрыто в v0.10.6: MD040 в
- трёх planning-доках, доп. тест на пустой DYLD_LIBRARY_PATH;
- keyboard a11y делён §14 polish)*".
+- §22 — replace "6" header with "✅ … *(implemented in v0.10.6)*".
+- §28 — same: "✅ … *(implemented in v0.10.6)*".
+- §30 — "✅ Agent logging discipline — §A + §C *(implemented in
+ v0.10.6: stop-guardrail extended to brain-events, SKILL trigger
+ rephrased; §B / §D / §E remain open)*".
+- §31 — "✅ v0.10.5 review followups *(closed in v0.10.6: MD040 in
+ the three planning docs, extra test for an empty DYLD_LIBRARY_PATH;
+ keyboard a11y split into §14 polish)*".
`Projects/OpenSecondBrain/Plan/3. Agent logging discipline.md`:
diff --git a/docs/plans/2026-05-18-v0.10.6-impl.md b/docs/plans/2026-05-18-v0.10.6-impl.md
index 55844851..ee21be6d 100644
--- a/docs/plans/2026-05-18-v0.10.6-impl.md
+++ b/docs/plans/2026-05-18-v0.10.6-impl.md
@@ -1346,7 +1346,7 @@ section:
- §5: mark fully shipped (manifest + abort drift detection).
- §14: append v0.10.6 keyboard + layout-persistence note. Remove
the two matching items from `## Deferred work`.
-- §22, §28: prepend `✅ … *(реализовано в v0.10.6)*`.
+- §22, §28: prepend `✅ … *(implemented in v0.10.6)*`.
- §30: mark §A + §C shipped, leave §B / §D / §E open.
- §31: mark §31.1 + §31.2 closed, §31.3 subsumed by §14 polish.
diff --git a/docs/plans/2026-05-19-v0.10.8-impl.md b/docs/plans/2026-05-19-v0.10.8-impl.md
index f2ba7fb4..e666356e 100644
--- a/docs/plans/2026-05-19-v0.10.8-impl.md
+++ b/docs/plans/2026-05-19-v0.10.8-impl.md
@@ -1429,33 +1429,33 @@ In `/root/vault/Projects/OpenSecondBrain/Features/_summary.md`, change the headi
to:
```
-### ✅ 32. Retire `event_log_append` for agent-facing surface, single source = `Brain/log/` — 8 *(реализовано в v0.10.8: §32A облегчённый verification gate; §32B brain_note tool в always-loaded writer scope; §32C-D drop event_log_append из stop-guardrail / detector / identity-reminder; §32E hooks/README.md переписан; §32F cmdAppendEvent резолвит identity через resolveAgentName; §32G полный дроп handler из всех рантаймов (Claude/Codex/Hermes/OpenClaw); §32H drift-check операторский, без auto-watcher)*
+### ✅ 32. Retire `event_log_append` for agent-facing surface, single source = `Brain/log/` — 8 *(implemented in v0.10.8: §32A lightweight verification gate; §32B brain_note tool in the always-loaded writer scope; §32C-D drop event_log_append from stop-guardrail / detector / identity-reminder; §32E hooks/README.md rewritten; §32F cmdAppendEvent resolves identity via resolveAgentName; §32G full handler drop from all runtimes (Claude/Codex/Hermes/OpenClaw); §32H operator-driven drift-check, no auto-watcher)*
```
- [ ] **Step 14.2: Mark §23 shipped**
In the same file, change §23 (line 224) from:
-```
-### 23. Structured JSONL log параллельно с Markdown — 6
+```markdown
+### 23. Structured JSONL log alongside Markdown — 6
```
to:
-```
-### ✅ 23. Structured JSONL log параллельно с Markdown — 6 *(реализовано в v0.10.8: Brain/log/<date>.jsonl рядом с .md, atomic dual-write через proper-lockfile lock; src/core/brain/log-jsonl.ts:readLogDay - JSONL-preferred reader с markdown fallback; discipline-report переключен; no backfill - исторические дни обслуживает fallback)*
+```markdown
+### ✅ 23. Structured JSONL log alongside Markdown — 6 *(implemented in v0.10.8: Brain/log/<date>.jsonl next to the .md, atomic dual-write via a proper-lockfile lock; src/core/brain/log-jsonl.ts:readLogDay - JSONL-preferred reader with a markdown fallback; discipline-report switched over; no backfill - the fallback serves historical days)*
```
- [ ] **Step 14.3: Append Deferred-work entries**
-In `## Deferred work`, add (in the same Russian style as existing entries):
+In `## Deferred work`, add (in the same style as existing entries):
```markdown
-- **§32B CLI verb `o2b brain note`.** MCP tool отгружен в v0.10.8, но CLI verb для cron-job/shell-script use-case'а отложен. Триггер на возврат: первый случай когда заметку нужно записать вне MCP (cron, ad-hoc script). Bash-needle для detector добавляется одновременно с CLI verb.
+- **§32B CLI verb `o2b brain note`.** The MCP tool shipped in v0.10.8, but the CLI verb for the cron-job / shell-script use-case is deferred. Return trigger: the first time a note must be written outside MCP (cron, ad-hoc script). The bash needle for the detector is added together with the CLI verb.
-- **§32H auto-watcher Daily/-записей от @agent-identity.** Операторская проверка (`o2b discipline report` + грeп Daily/) сейчас достаточна. Триггер: drift-check находит повторные случаи которые операторский проход пропустил.
+- **§32H auto-watcher for Daily/ entries from @agent-identity.** The operator check (`o2b discipline report` + grep over Daily/) is sufficient for now. Trigger: drift-check finds recurring cases the operator pass missed.
-- **§23 one-shot JSONL backfill исторических markdown-дней.** Лeniвый fallback в readLogDay обслуживает все pre-v0.10.8 дни без сидекара. Триггер: машинный consumer материально выигрывает от JSONL для дней старше v0.10.8 (ни один не назван сейчас).
+- **§23 one-shot JSONL backfill of historical markdown days.** The lazy fallback in readLogDay serves all pre-v0.10.8 days without a sidecar. Trigger: a machine consumer materially benefits from JSONL for days older than v0.10.8 (none named yet).
```
- [ ] **Step 14.4: Update `CHANGELOG.md`**
@@ -1626,7 +1626,7 @@ language convention. Markdown posts, chat replies, descriptions, and
in-line comments in non-em-dash languages get ` - `. Technical
documentation and code stay as-is.
-Связано: [[feedback_always_russian]] (dialog language is the
+Related: [[feedback_always_russian]] (dialog language is the
default selector). Brain preference candidate: `pref-no-em-dashes`.
```
diff --git a/docs/plans/2026-05-20-v0.10.10-design.md b/docs/plans/2026-05-20-v0.10.10-design.md
index f15bb401..4fce7e3d 100644
--- a/docs/plans/2026-05-20-v0.10.10-design.md
+++ b/docs/plans/2026-05-20-v0.10.10-design.md
@@ -464,7 +464,7 @@ After implementation lands, the vault-side
small places (operator-confirmed edit, no automated cron writes):
- §1 shipped-in header gains "+v0.10.10 Most-applied (30d) section".
-- "Спорные / отложенные" — the `brain_context` bullet is removed
+- "Disputed / deferred" — the `brain_context` bullet is removed
(no longer disputed).
- "Deferred work" — the §32B CLI bullet moves to a shipped reference
pointing at the v0.10.10 release notes.
diff --git a/docs/plans/2026-05-20-v0.10.10-impl.md b/docs/plans/2026-05-20-v0.10.10-impl.md
index 14aabafe..7bc043d4 100644
--- a/docs/plans/2026-05-20-v0.10.10-impl.md
+++ b/docs/plans/2026-05-20-v0.10.10-impl.md
@@ -2270,9 +2270,9 @@ Wait for a yes.
If yes:
1. In the `## Tier S` section, find the `### ✅ 1.` entry. Append to
- its `*(реализовано в v0.9.1)*` parenthetical:
+ its `*(implemented in v0.9.1)*` parenthetical:
`; + v0.10.10 Most-applied (30d) section`.
-2. In the `## Спорные / отложенные` section, remove the bullet
+2. In the `## Disputed / deferred` section, remove the bullet
starting with **Pull-bootstrap MCP-tool `brain_context`/`brain_principles`**.
3. In the `## Deferred work` section, find the bullet that starts
with **§32B CLI verb `o2b brain note`** and replace it with a
diff --git a/src/core/brain/context-lanes.ts b/src/core/brain/context-lanes.ts
index 4ada194e..a6087f65 100644
--- a/src/core/brain/context-lanes.ts
+++ b/src/core/brain/context-lanes.ts
@@ -30,9 +30,6 @@ export interface ContextLaneInput {
readonly manualLane?: ContextLaneName | null;
}
-const CONSTRAINT_RE =
- /\b(?:avoid|do not|don't|must not|never|no longer|forbid|forbidden)\b|\b(?:не делай|никогда|запрещено|избегай)\b/iu;
-
export function normalizeContextLane(raw: unknown): ContextLaneName | null {
if (typeof raw !== "string") return null;
const value = raw.trim().toLowerCase();
@@ -40,10 +37,18 @@ export function normalizeContextLane(raw: unknown): ContextLaneName | null {
return null;
}
+/**
+ * Classify a source into a context lane.
+ *
+ * Language-agnostic by construction: the lane is never inferred from
+ * the words of the principle/body (an English/Russian "never/forbidden"
+ * prose scan would silently misclassify every other language). The
+ * `constraints` lane is opt-in via the explicit `context_lane:`
+ * frontmatter field (surfaced here as `manualLane`); without it, the
+ * lane follows only the structural page tier.
+ */
export function classifyContextLane(input: ContextLaneInput): ContextLaneName {
if (input.manualLane) return input.manualLane;
- const text = `${input.principle}\n${input.body}`;
- if (CONSTRAINT_RE.test(text)) return "constraints";
if (input.tier === "peripheral") return "consider";
return "directives";
}
diff --git a/src/core/brain/git/decisions.ts b/src/core/brain/git/decisions.ts
index c0b02e74..4f323b6c 100644
--- a/src/core/brain/git/decisions.ts
+++ b/src/core/brain/git/decisions.ts
@@ -2,9 +2,10 @@
* Commit-decision miner (Project History Suite, t_93d299bb).
*
* Deterministic heuristics over INGESTED commit records (never live
- * git): conventional breaking markers, BREAKING CHANGE footers, a
- * word-boundary decision keyword set, and revert shape. Same store,
- * same candidates, same order - no LLM classification anywhere.
+ * git): conventional breaking markers, BREAKING CHANGE footers, and
+ * revert shape - all fixed commit-protocol tokens, language-agnostic.
+ * Same store, same candidates, same order - no LLM classification
+ * anywhere.
*
* Each decision-shaped commit becomes one draft ADR candidate note at
* `Brain/decisions/candidates/adr-<shortsha>-<slug>.md` with the
@@ -21,20 +22,6 @@ import { atomicWriteFileSync } from "../../fs-atomic.ts";
import { listGitCommits } from "./store.ts";
import type { GitCommitRecord } from "./store.ts";
-/** Phrases that mark a commit message as decision-shaped. */
-const DECISION_KEYWORDS = [
- "decide",
- "decided",
- "decision",
- "adopt",
- "adopted",
- "switch to",
- "migrate to",
- "instead of",
- "adr",
- "rationale",
-] as const;
-
const CONVENTIONAL_BREAKING_RE = /^[a-z]+(\([^)]*\))?!:/;
const REVERT_RE = /^revert\b/i;
const SLUG_MAX = 40;
@@ -48,15 +35,15 @@ const CANDIDATE_SHA_LEN = 12;
* Empty array = not decision-shaped.
*/
export function detectDecisionSignals(subject: string, body: string): ReadonlyArray<string> {
+ // Language-agnostic by construction: only structural commit-protocol
+ // markers count as decision signals. The old English keyword set
+ // ("decide", "adopt", ...) never fired on a commit history written in
+ // any other language; conventional-commit and git-native markers
+ // (`type!:`, the `BREAKING CHANGE:` footer, the `Revert` prefix) are
+ // fixed protocol tokens git/tooling emit, not free prose.
const signals: string[] = [];
if (CONVENTIONAL_BREAKING_RE.test(subject)) signals.push("conventional_breaking");
if (/^BREAKING CHANGE\b/m.test(body)) signals.push("breaking_change_footer");
- const haystack = `${subject}\n${body}`.toLowerCase();
- for (const keyword of DECISION_KEYWORDS) {
- // Word-boundary match: 'adopt' must not fire inside 'adoption'.
- const re = new RegExp(`\\b${keyword.replace(/ /g, "\\s+")}\\b`);
- if (re.test(haystack)) signals.push(`decision_keyword:${keyword}`);
- }
if (REVERT_RE.test(subject)) signals.push("revert");
return Object.freeze(signals);
}
diff --git a/src/core/search/coverage.ts b/src/core/search/coverage.ts
index be42e555..8102fa7b 100644
--- a/src/core/search/coverage.ts
+++ b/src/core/search/coverage.ts
@@ -14,34 +14,24 @@
* in. Deterministic, no LLM, no clock.
*/
-const STOPWORDS = new Set([
- "about",
- "and",
- "are",
- "for",
- "from",
- "how",
- "the",
- "this",
- "that",
- "what",
- "when",
- "where",
- "with",
-]);
-
/** Share of the corpus a term may appear in and still count as rare. */
export const RARE_TERM_CORPUS_SHARE = 0.02;
/**
- * Significant query terms: length >= 3, not a stopword, deduplicated,
- * in query order. The same extraction the evidence pack has used since
- * PR #54 — now owned here so every consumer agrees.
+ * Significant query terms: length >= 3, deduplicated, in query order.
+ *
+ * Language-agnostic by construction: there is deliberately NO stopword
+ * list. A per-language stopword set (the old English-only one) would
+ * under-filter every other language while pretending to help. Instead,
+ * corpus-common terms are handled downstream by the IDF weighting in
+ * {@link buildCoverageReport} — a term that appears in most documents
+ * earns near-zero IDF and contributes almost nothing to the weighted
+ * coverage, in any language, without a vocabulary list.
*/
export function significantTerms(query: string): string[] {
const terms = new Set<string>();
for (const token of query.toLocaleLowerCase().split(/[^\p{L}\p{N}_-]+/u)) {
- if (token.length >= 3 && !STOPWORDS.has(token)) terms.add(token);
+ if (token.length >= 3) terms.add(token);
}
return [...terms];
}
diff --git a/src/core/search/evidence-pack.ts b/src/core/search/evidence-pack.ts
index c271be53..53e48e04 100644
--- a/src/core/search/evidence-pack.ts
+++ b/src/core/search/evidence-pack.ts
@@ -72,8 +72,29 @@ export interface EvidencePack {
readonly completeness?: CompletenessReport;
}
-const TERMINAL_STATE_RE =
- /\b(?:archived|closed|deprecated|done|resolved|retired|superseded|terminal)\b/iu;
+/**
+ * Controlled terminal-status vocabulary. These are accepted values of
+ * the frontmatter `status:` field - a controlled enum the system owns -
+ * NOT words scanned inside a note's prose. A note is terminal when its
+ * DECLARED status is one of these, so the language the note is written
+ * in is irrelevant (the old regex scanned path/title/content for these
+ * English words and false-fired on any note that merely mentioned them).
+ */
+const TERMINAL_STATUS_VALUES: ReadonlySet<string> = new Set([
+ "archived",
+ "closed",
+ "deprecated",
+ "done",
+ "resolved",
+ "retired",
+ "superseded",
+ "terminal",
+]);
+
+/** True when a frontmatter `status` value denotes a terminal state. */
+export function isTerminalStatus(status: unknown): boolean {
+ return typeof status === "string" && TERMINAL_STATUS_VALUES.has(status.trim().toLowerCase());
+}
function includesTerm(result: BrainSearchResult, term: string): boolean {
return termIncludedIn(`${result.path}\n${result.title ?? ""}\n${result.content}`, term);
@@ -87,12 +108,25 @@ function supportCoverage(
return matched.length / significant.length;
}
-export function evidenceTerminalState(result: BrainSearchResult): boolean {
- return TERMINAL_STATE_RE.test(`${result.path}\n${result.title ?? ""}\n${result.content}`);
+/**
+ * Whether a result is in a terminal state for the current query.
+ * Terminality is decided structurally by the caller (reading the
+ * frontmatter `status:` field via {@link isTerminalStatus}) and handed
+ * in as the set of terminal paths - this function never inspects the
+ * note's text.
+ */
+export function evidenceTerminalState(
+ result: BrainSearchResult,
+ terminalPaths: ReadonlySet<string>,
+): boolean {
+ return terminalPaths.has(result.path);
}
-function withTerminalReason(result: BrainSearchResult): BrainSearchResult {
- if (!evidenceTerminalState(result)) return result;
+function withTerminalReason(
+ result: BrainSearchResult,
+ terminalPaths: ReadonlySet<string>,
+): BrainSearchResult {
+ if (!evidenceTerminalState(result, terminalPaths)) return result;
if (result.reasons.some((reason) => reason.startsWith("evidence_terminal_downrank:"))) {
return result;
}
@@ -104,14 +138,17 @@ function withTerminalReason(result: BrainSearchResult): BrainSearchResult {
export function downrankTerminalEvidenceResults(
results: ReadonlyArray<BrainSearchResult>,
+ terminalPaths: ReadonlySet<string>,
): ReadonlyArray<BrainSearchResult> {
- return results.map(withTerminalReason).toSorted((left, right) => {
- const leftTerminal = evidenceTerminalState(left);
- const rightTerminal = evidenceTerminalState(right);
- if (leftTerminal !== rightTerminal) return leftTerminal ? 1 : -1;
- if (right.score !== left.score) return right.score - left.score;
- return left.chunkId - right.chunkId;
- });
+ return results
+ .map((result) => withTerminalReason(result, terminalPaths))
+ .toSorted((left, right) => {
+ const leftTerminal = evidenceTerminalState(left, terminalPaths);
+ const rightTerminal = evidenceTerminalState(right, terminalPaths);
+ if (leftTerminal !== rightTerminal) return leftTerminal ? 1 : -1;
+ if (right.score !== left.score) return right.score - left.score;
+ return left.chunkId - right.chunkId;
+ });
}
function abstentionMessage(
@@ -131,6 +168,7 @@ export function buildEvidencePack(
query: string,
results: ReadonlyArray<BrainSearchResult>,
verification?: EvidenceVerification,
+ terminalPaths: ReadonlySet<string> = new Set(),
): EvidencePack {
const significant = Object.freeze(significantTerms(query));
const matchedSet = new Set<string>();
@@ -138,7 +176,7 @@ export function buildEvidencePack(
const matched = significant.filter((term) => includesTerm(result, term));
for (const term of matched) matchedSet.add(term);
const missing = significant.filter((term) => !matched.includes(term));
- const terminalState = evidenceTerminalState(result);
+ const terminalState = evidenceTerminalState(result, terminalPaths);
const terminalDownranked = result.reasons.some((reason) =>
reason.startsWith("evidence_terminal_downrank:"),
);
diff --git a/src/core/search/query-expansion.ts b/src/core/search/query-expansion.ts
index b8c998aa..c4d6451f 100644
--- a/src/core/search/query-expansion.ts
+++ b/src/core/search/query-expansion.ts
@@ -8,9 +8,10 @@
* string work plus one read of the vault's entity registry - no local
* model, no paid call, identical output for identical input.
*
- * - lex: query tokens minus stopwords (the FTS lane is implicit AND,
- * so one stopword absent from the target note kills the match);
- * falls back to the raw tokens when everything is a stopword.
+ * - lex: the query tokens (deduped, capped), minus any the caller
+ * flags as corpus-common via `commonTokens` (high document frequency,
+ * not a hardcoded stopword list - the FTS lane is implicit AND, so a
+ * ubiquitous token would otherwise kill the match). Language-agnostic.
* - vec: the raw query, plus one entity-context line when registry
* entities match a query token - anchors the semantic lane to the
* vault's own vocabulary.
@@ -31,53 +32,16 @@ export const EXPANSION_MAX_LEX_TERMS = 8;
/** Default cap on matched registry entities woven into vec/hyde. */
export const EXPANSION_MAX_ENTITIES = 3;
-/**
- * Small closed-class English stopword set. Deliberately conservative:
- * a missed stopword only leaves one extra AND term, while an
- * over-eager list would silently drop signal.
- */
-const STOPWORDS: ReadonlySet<string> = new Set([
- "a",
- "an",
- "and",
- "are",
- "as",
- "at",
- "be",
- "but",
- "by",
- "for",
- "from",
- "how",
- "in",
- "is",
- "it",
- "me",
- "my",
- "of",
- "on",
- "or",
- "our",
- "show",
- "that",
- "the",
- "their",
- "this",
- "to",
- "was",
- "we",
- "what",
- "when",
- "where",
- "which",
- "who",
- "why",
- "with",
-]);
-
export interface ExpandQueryOptions {
readonly maxLexTerms?: number;
readonly maxEntities?: number;
+ /**
+ * Corpus-common tokens to drop from the implicit-AND lex lane. The
+ * caller derives these from document frequency (a token present in
+ * most documents carries little signal, in ANY language), never from a
+ * hardcoded stopword list. Defaults to none.
+ */
+ readonly commonTokens?: ReadonlySet<string>;
}
/**
@@ -93,11 +57,16 @@ export function expandQuery(
const maxEntities = Math.max(0, opts.maxEntities ?? EXPANSION_MAX_ENTITIES);
const trimmed = query.trim();
- const tokens = [...new Set(tokenizeForExpansion(trimmed))];
- const meaningful = tokens.filter((t) => !STOPWORDS.has(t));
- // Fall back to the raw tokens when the whole query is stopwords -
- // an empty lex lane would turn the FTS lane off entirely.
- const baseTokens = meaningful.length > 0 ? meaningful : tokens;
+ // Language-agnostic by construction: no stopword list. Corpus-common
+ // tokens (high document frequency, supplied by the caller) are dropped
+ // from the implicit-AND lex lane so one ubiquitous word cannot kill the
+ // match - in ANY language, driven by data rather than an English word
+ // list. Fall back to all tokens when every token is common, so the lex
+ // lane never goes empty.
+ const commonTokens = opts.commonTokens ?? new Set<string>();
+ const allTokens = [...new Set(tokenizeForExpansion(trimmed))];
+ const meaningful = allTokens.filter((t) => !commonTokens.has(t));
+ const baseTokens = meaningful.length > 0 ? meaningful : allTokens;
const lexTerms = baseTokens.slice(0, maxLexTerms);
const entityNames = matchEntities(vault, baseTokens).slice(0, maxEntities);
diff --git a/src/core/search/search.ts b/src/core/search/search.ts
index ab1d7087..7931abe4 100644
--- a/src/core/search/search.ts
+++ b/src/core/search/search.ts
@@ -35,7 +35,11 @@ import {
import { extractEntities } from "./entities.ts";
import { expandQueryEntities } from "./entity-alias.ts";
import { buildCoverageReport, significantTerms, termIncludedIn } from "./coverage.ts";
-import { buildEvidencePack, downrankTerminalEvidenceResults } from "./evidence-pack.ts";
+import {
+ buildEvidencePack,
+ downrankTerminalEvidenceResults,
+ isTerminalStatus,
+} from "./evidence-pack.ts";
import type { EvidenceUnionRecord, EvidenceVerification } from "./evidence-pack.ts";
import { runFtsQueryDetailed } from "./fts.ts";
import { mmrRerank } from "./mmr.ts";
@@ -215,16 +219,10 @@ export async function search(
const pathPrefix = assertSafePathPrefix(opts.pathPrefix);
const policy = resolveSemanticPolicy(config, opts);
const warnings: string[] = [];
- // Opt-in local expansion (t_2fa95db1): an explicit structured
- // document always wins; expansion only fills the gap.
- const structured =
- opts.structuredQuery ?? (expandActive === true ? expandQuery(config.vault, query) : undefined);
const sessionFocus =
opts.sessionFocus === undefined
? readActiveSessionFocus(config, opts.focusSession, Date.now())
: opts.sessionFocus;
- const keywordQuery = structuredKeywordQuery(query, structured);
- const semanticLaneQuery = structuredSemanticQuery(structured);
// Time-aware recall (recall-trust-suite): resolve since/until up front
// so invalid input fails fast, before any store I/O.
const timeRange =
@@ -232,11 +230,6 @@ export async function search(
? resolveTimeRange({ since: opts.since, until: opts.until }, Date.now())
: null;
- // Query plan (v0.20.0): one structural pass yields the intent weight
- // profile and the cache key. Pure; no I/O. Expanded terms (if any) are
- // folded in once they have been derived from the store below.
- const basePlan = buildQueryPlan(keywordQuery, [], structured?.intent);
-
// Read-only origins (cross-vault search) disable self-healing: a
// rebuild would write an index INTO the external vault. Default
// (selfHeal absent) keeps the legacy heal-and-retry behaviour.
@@ -245,6 +238,25 @@ export async function search(
? await Store.open(config, { mode: "read" })
: await openReadOrSelfHeal(config);
try {
+ // Opt-in local expansion (t_2fa95db1): an explicit structured
+ // document always wins; expansion only fills the gap. The lex lane's
+ // corpus-common tokens are derived from document frequency here
+ // (language-agnostic, no stopword list) so an implicit-AND query is
+ // not killed by a word that is ubiquitous in this vault.
+ const commonTokens =
+ opts.structuredQuery === undefined && expandActive === true
+ ? highFrequencyTokens(store, query)
+ : new Set<string>();
+ const structured =
+ opts.structuredQuery ??
+ (expandActive === true ? expandQuery(config.vault, query, { commonTokens }) : undefined);
+ const keywordQuery = structuredKeywordQuery(query, structured);
+ const semanticLaneQuery = structuredSemanticQuery(structured);
+ // Query plan (v0.20.0): one structural pass yields the intent weight
+ // profile and the cache key. Expanded terms (if any) are folded in
+ // once they have been derived from the store below.
+ const basePlan = buildQueryPlan(keywordQuery, [], structured?.intent);
+
// Persistent query cache (v0.20.0): opt-in. Keyed by the request +
// base plan hash + a config fingerprint, gated by the corpus
// generation and a TTL. A hit returns the previously computed
@@ -785,9 +797,19 @@ export async function search(
}),
)
: withCanonicalReasons;
+ // Terminal-state downrank (recall-trust-suite) is now structural and
+ // language-agnostic: a result is terminal when its frontmatter
+ // `status:` field declares a terminal value (controlled vocabulary),
+ // never because the note's prose happens to contain an English word
+ // like "done". One cached frontmatter read per candidate path, only
+ // in evidence-pack mode.
+ const terminalPaths =
+ opts.evidencePack === true
+ ? buildTerminalPaths(config.vault, withSecondPassReasons)
+ : new Set<string>();
const finalResults =
opts.evidencePack === true
- ? downrankTerminalEvidenceResults(withSecondPassReasons)
+ ? downrankTerminalEvidenceResults(withSecondPassReasons, terminalPaths)
: withSecondPassReasons;
const evidencePack =
opts.evidencePack === true
@@ -795,6 +817,7 @@ export async function search(
query,
finalResults,
buildEvidenceVerification(store, query, finalResults, pathPrefix),
+ terminalPaths,
)
: undefined;
@@ -894,6 +917,71 @@ function applyTraversal(
);
}
+/** A token in at least this share of the corpus is treated as common. */
+const COMMON_TOKEN_CORPUS_SHARE = 0.5;
+/**
+ * Floor on document frequency before a token can be "common". Below this
+ * a corpus is too small to tell a stopword-like word from a rare one, so
+ * nothing is flagged (a 2-document vault must not call a word that appears
+ * once "ubiquitous").
+ */
+const MIN_COMMON_DOCUMENT_FREQUENCY = 2;
+
+/**
+ * Corpus-common query tokens, derived from document frequency. A token
+ * present in at least {@link COMMON_TOKEN_CORPUS_SHARE} of the indexed
+ * documents (and in at least {@link MIN_COMMON_DOCUMENT_FREQUENCY} of
+ * them) carries little discriminating signal - in ANY language - so the
+ * lex expansion lane drops it rather than letting one ubiquitous word
+ * kill an implicit-AND match. Language-agnostic: no stopword list.
+ *
+ * Note: document frequency is measured through the FTS index, whose
+ * tokenization can differ slightly from `tokenizeForExpansion`. A miss
+ * only fails to flag a token as common (it stays in the lex lane), so the
+ * fallback is always the safe, non-lossy direction.
+ */
+function highFrequencyTokens(store: Store, query: string): ReadonlySet<string> {
+ const tokens = [...new Set(tokenizeForExpansion(query))];
+ if (tokens.length === 0) return new Set();
+ const documentCount = store.counts().documents;
+ if (documentCount === 0) return new Set();
+ const threshold = COMMON_TOKEN_CORPUS_SHARE * documentCount;
+ const df = store.documentFrequencies(tokens);
+ const common = new Set<string>();
+ for (const token of tokens) {
+ const freq = df.get(token) ?? 0;
+ if (freq >= MIN_COMMON_DOCUMENT_FREQUENCY && freq >= threshold) common.add(token);
+ }
+ return common;
+}
+
+/**
+ * Build the set of terminal-state paths for evidence-pack downranking.
+ * Reads each unique candidate path's frontmatter `status:` field once
+ * and includes the path when the declared status is terminal (controlled
+ * vocabulary). A missing or unreadable status is non-terminal. This is
+ * the language-agnostic replacement for scanning note prose for English
+ * status words.
+ */
+function buildTerminalPaths(
+ vault: string,
+ results: ReadonlyArray<BrainSearchResult>,
+): ReadonlySet<string> {
+ const terminal = new Set<string>();
+ const seen = new Set<string>();
+ for (const r of results) {
+ if (seen.has(r.path)) continue;
+ seen.add(r.path);
+ try {
+ const [meta] = parseFrontmatter(join(vault, r.path));
+ if (isTerminalStatus((meta as Record<string, unknown>)["status"])) terminal.add(r.path);
+ } catch {
+ // Unreadable frontmatter is non-terminal.
+ }
+ }
+ return terminal;
+}
+
/** Cap on extra records fetched per uncovered term (Feature C union). */
const UNION_RECORDS_PER_TERM = 2;
/** Cap on the total recall-union fetch per query. */
diff --git a/src/core/search/surfacing-gate.ts b/src/core/search/surfacing-gate.ts
index 74c16996..77b65ca7 100644
--- a/src/core/search/surfacing-gate.ts
+++ b/src/core/search/surfacing-gate.ts
@@ -1,13 +1,10 @@
export type SurfacingGateReason =
| "explicit"
- | "memory_question"
- | "uncertain_question"
| "duplicate"
| "empty"
- | "greeting"
| "slash_command"
| "shell_command"
- | "no_recall_intent";
+ | "default_retrieve";
export interface SurfacingGateInput {
readonly prompt: string;
@@ -20,16 +17,6 @@ export interface SurfacingGateDecision {
readonly reason: SurfacingGateReason;
}
-const GREETINGS = new Set([
- "hello",
- "hi",
- "hey",
- "привет",
- "здравствуй",
- "здравствуйте",
- "добрый день",
-]);
-
const SHELL_COMMANDS = new Set([
"awk",
"bun",
@@ -54,21 +41,10 @@ const SHELL_COMMANDS = new Set([
"yarn",
]);
-const MEMORY_PATTERNS = [
- /\b(remember|recall|memory|context|notes?|decid(?:e|ed)|decision|search|find)\b/iu,
- /\b(what did we|where did we|when did we|have we|did we)\b/iu,
- /\b(помнишь|вспомни|найди|контекст|решили|обсуждали|заметки)\b/iu,
-];
-
function normalizePrompt(prompt: string): string {
return prompt.trim().replace(/\s+/gu, " ").toLocaleLowerCase();
}
-function isGreeting(normalized: string): boolean {
- const stripped = normalized.replace(/[!.?]+$/u, "");
- return GREETINGS.has(stripped);
-}
-
function isSlashCommand(normalized: string): boolean {
return normalized.startsWith("/") && !normalized.includes(" ");
}
@@ -79,10 +55,18 @@ function isShellOnlyPrompt(normalized: string): boolean {
return SHELL_COMMANDS.has(firstToken);
}
-function hasMemoryIntent(normalized: string): boolean {
- return MEMORY_PATTERNS.some((pattern) => pattern.test(normalized));
-}
-
+/**
+ * Decide whether a prompt should trigger memory retrieval.
+ *
+ * Language-agnostic by construction: the gate never inspects prompt
+ * words against any natural-language vocabulary, so a prompt in any
+ * language is treated identically. Only structural signals suppress
+ * retrieval — an empty prompt, a verbatim repeat of the previous one, a
+ * slash command, or a single shell command (command names, not human
+ * language). Everything else FAILS OPEN: we retrieve and let ranking
+ * decide relevance, because a missed recall is worse than a cheap
+ * no-result search.
+ */
export function evaluateSurfacingGate(input: SurfacingGateInput): SurfacingGateDecision {
if (input.explicit === true) return Object.freeze({ retrieve: true, reason: "explicit" });
@@ -93,17 +77,10 @@ export function evaluateSurfacingGate(input: SurfacingGateInput): SurfacingGateD
if (previous !== null && previous === normalized) {
return Object.freeze({ retrieve: false, reason: "duplicate" });
}
- if (isGreeting(normalized)) return Object.freeze({ retrieve: false, reason: "greeting" });
if (isSlashCommand(normalized))
return Object.freeze({ retrieve: false, reason: "slash_command" });
if (isShellOnlyPrompt(normalized)) {
return Object.freeze({ retrieve: false, reason: "shell_command" });
}
- if (hasMemoryIntent(normalized)) {
- return Object.freeze({ retrieve: true, reason: "memory_question" });
- }
- if (normalized.endsWith("?") && normalized.length >= 60) {
- return Object.freeze({ retrieve: true, reason: "uncertain_question" });
- }
- return Object.freeze({ retrieve: false, reason: "no_recall_intent" });
+ return Object.freeze({ retrieve: true, reason: "default_retrieve" });
}
diff --git a/tests/cli/brain-benchmark.test.ts b/tests/cli/brain-benchmark.test.ts
index 54d8b82f..d0203bf4 100644
--- a/tests/cli/brain-benchmark.test.ts
+++ b/tests/cli/brain-benchmark.test.ts
@@ -27,7 +27,11 @@ beforeEach(async () => {
join(vault, "canary.md"),
"# Canary rollout\n\nShip one instance first, observe, expand gradually.\n",
);
- writeFileSync(join(vault, "other.md"), "# Other\n\nNothing relevant whatsoever.\n");
+ // "the" is seeded across >= 2 of the other notes (majority of the
+ // corpus) but NOT canary.md, so the DF-driven, language-agnostic
+ // common-token filter drops it from the implicit-AND lex lane.
+ writeFileSync(join(vault, "other.md"), "# Other\n\nNothing in the list is relevant here.\n");
+ writeFileSync(join(vault, "notes.md"), "# Notes\n\nThe team reviewed the plan in the meeting.\n");
datasetPath = join(tmp, "dataset.json");
writeFileSync(
datasetPath,
diff --git a/tests/cli/brain-context-pack.test.ts b/tests/cli/brain-context-pack.test.ts
index 995788b5..4b7e86bc 100644
--- a/tests/cli/brain-context-pack.test.ts
+++ b/tests/cli/brain-context-pack.test.ts
@@ -27,7 +27,9 @@ function writePref(slug: string, principle: string, extra = ""): void {
test("brain context-pack --lanes --json returns polarity lanes", async () => {
writePref("directive", "Prefer concise answers", "tier: core");
- writePref("constraint", "Never expose tokens", "tier: core");
+ // The constraints lane is opt-in via the explicit context_lane field,
+ // not inferred from prose words like "Never" (language-agnostic).
+ writePref("constraint", "Never expose tokens", "tier: core\ncontext_lane: constraints");
const out = await runCli(
["brain", "context-pack", "--vault", vault, "--max-tokens", "10000", "--lanes", "--json"],
diff --git a/tests/core/brain/context-pack.test.ts b/tests/core/brain/context-pack.test.ts
index 2deaf7d4..e984d52c 100644
--- a/tests/core/brain/context-pack.test.ts
+++ b/tests/core/brain/context-pack.test.ts
@@ -257,6 +257,10 @@ describe("packContext", () => {
topic: "security",
principle: "Never expose secret tokens",
tier: "core",
+ // The constraints lane is opt-in via the explicit context_lane
+ // field - it is no longer inferred from prose words like "Never"
+ // (that only ever worked for English/Russian).
+ context_lane: "constraints",
});
writePref("consider", {
topic: "taste",
diff --git a/tests/core/brain/git/decisions.test.ts b/tests/core/brain/git/decisions.test.ts
index c5a9d909..80d54cf1 100644
--- a/tests/core/brain/git/decisions.test.ts
+++ b/tests/core/brain/git/decisions.test.ts
@@ -46,7 +46,7 @@ afterEach(() => {
// ── signal detection ────────────────────────────────────────────────────────
-test("detectDecisionSignals matches deterministic decision shapes", () => {
+test("detectDecisionSignals matches only structural commit-protocol shapes", () => {
expect(detectDecisionSignals("feat!: drop legacy index format", "")).toContain(
"conventional_breaking",
);
@@ -56,17 +56,16 @@ test("detectDecisionSignals matches deterministic decision shapes", () => {
expect(detectDecisionSignals("feat: new api", "BREAKING CHANGE: payload renamed")).toContain(
"breaking_change_footer",
);
- expect(detectDecisionSignals("chore: migrate to bun test runner", "")).toContain(
- "decision_keyword:migrate to",
- );
- expect(
- detectDecisionSignals("docs: explain choice", "We decided to adopt JSONL instead of SQLite."),
- ).toEqual(expect.arrayContaining(["decision_keyword:decided", "decision_keyword:adopt"]));
expect(detectDecisionSignals('Revert "feat: bad idea"', "")).toContain("revert");
// Plain work is NOT decision-shaped.
expect(detectDecisionSignals("fix: off-by-one in pager", "small fix")).toEqual([]);
- // Substrings inside words do not match (no false 'adopt' in 'adoption-rate.ts').
- expect(detectDecisionSignals("chore: update adoption-rate metrics", "")).toEqual([]);
+ // Language-agnostic by construction: prose words ("decided", "adopt",
+ // "migrate to") are NOT decision signals - they only ever fired on
+ // English commit histories. Only structural protocol markers count.
+ expect(detectDecisionSignals("chore: migrate to bun test runner", "")).toEqual([]);
+ expect(
+ detectDecisionSignals("docs: explain choice", "We decided to adopt JSONL instead of SQLite."),
+ ).toEqual([]);
});
// ── mining ──────────────────────────────────────────────────────────────────
@@ -75,7 +74,7 @@ test("mineCommitDecisions writes candidates for decision commits only, with prov
appendGitRecords(vault, KEY, [
commit("a".repeat(40), "feat!: drop legacy index format", "BREAKING CHANGE: rebuild needed"),
commit("b".repeat(40), "fix: typo"),
- commit("c".repeat(40), "chore: migrate to bun test runner"),
+ commit("c".repeat(40), 'Revert "feat: half-baked migration"'),
]);
const res = mineCommitDecisions(vault, KEY);
expect(res.scanned).toBe(3);
diff --git a/tests/core/search/coverage.test.ts b/tests/core/search/coverage.test.ts
index fd3bdf2d..45a073a8 100644
--- a/tests/core/search/coverage.test.ts
+++ b/tests/core/search/coverage.test.ts
@@ -23,8 +23,16 @@ import { search } from "../../../src/core/search/search.ts";
import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";
describe("coverage engine (pure)", () => {
- test("significantTerms drops stopwords and short tokens", () => {
- expect(significantTerms("what is the alpha zephyr")).toEqual(["alpha", "zephyr"]);
+ test("significantTerms drops short tokens but keeps every word - no stopword list", () => {
+ // Language-agnostic: there is no stopword list. Tokens under length 3
+ // are dropped structurally ("is"); common words like "the"/"what" are
+ // kept here and deprioritised downstream by IDF, in any language.
+ expect(significantTerms("what is the alpha zephyr")).toEqual([
+ "what",
+ "the",
+ "alpha",
+ "zephyr",
+ ]);
});
test("idf is monotonically decreasing in document frequency", () => {
diff --git a/tests/core/search/evidence-pack.test.ts b/tests/core/search/evidence-pack.test.ts
index 8296a761..f9d20658 100644
--- a/tests/core/search/evidence-pack.test.ts
+++ b/tests/core/search/evidence-pack.test.ts
@@ -36,20 +36,36 @@ test("buildEvidencePack reports matched and missing significant terms", () => {
expect(pack.records[0]?.whyRetrieved).toEqual(["fts5_bm25: 1.000"]);
});
-test("buildEvidencePack marks terminal-state records", () => {
- const pack = buildEvidencePack("alpha", [result("done.md", "alpha superseded by another note")]);
+test("buildEvidencePack marks terminal-state records from the terminal path set", () => {
+ // Terminality is structural (declared frontmatter `status:`), handed in
+ // as the terminal-path set - never inferred from the note's prose.
+ const terminalPaths = new Set(["done.md"]);
+ const pack = buildEvidencePack(
+ "alpha",
+ [result("done.md", "alpha and more")],
+ undefined,
+ terminalPaths,
+ );
expect(pack.records[0]?.terminalState).toBe(true);
expect(pack.records[0]?.droppedCandidateReasons).toEqual([]);
});
+test("buildEvidencePack does not infer terminal state from prose", () => {
+ // "superseded" / "done" in the body must NOT make a record terminal
+ // when no terminal path set is supplied (the old regex false-fired here).
+ const pack = buildEvidencePack("alpha", [result("notes.md", "alpha superseded by another note")]);
+
+ expect(pack.records[0]?.terminalState).toBe(false);
+});
+
test("buildEvidencePack reports terminal downrank only when applied", () => {
- const pack = buildEvidencePack("alpha", [
- result("done.md", "alpha superseded by another note", [
- "fts5_bm25: 1.000",
- "evidence_terminal_downrank: true",
- ]),
- ]);
+ const pack = buildEvidencePack(
+ "alpha",
+ [result("done.md", "alpha and more", ["fts5_bm25: 1.000", "evidence_terminal_downrank: true"])],
+ undefined,
+ new Set(["done.md"]),
+ );
expect(pack.records[0]?.droppedCandidateReasons).toContain("terminal_state_downranked");
});
diff --git a/tests/core/search/query-expansion.test.ts b/tests/core/search/query-expansion.test.ts
index 7af6ac65..d62b7bda 100644
--- a/tests/core/search/query-expansion.test.ts
+++ b/tests/core/search/query-expansion.test.ts
@@ -2,9 +2,10 @@
* Deterministic query expansion producer (link-recall-intelligence,
* Task 5 / t_2fa95db1): a bare query becomes a structured lex/vec/hyde
* document for the EXISTING structured-query consumer - no model, no
- * paid call. Lex strips stopwords for the implicit-AND FTS lane, vec
- * adds an entity-context line when registry entities match, hyde is
- * one template passage. Opt-in via `search(config, {expand: true})`.
+ * paid call. Lex keeps every query token for the implicit-AND FTS lane
+ * (no stopword list - language-agnostic), vec adds an entity-context
+ * line when registry entities match, hyde is one template passage.
+ * Opt-in via `search(config, {expand: true})`.
*/
import { test, expect, beforeEach, afterEach, describe } from "bun:test";
@@ -39,7 +40,9 @@ describe("expandQuery", () => {
const first = expandQuery(vault, "the canary rollout plan");
const second = expandQuery(vault, "the canary rollout plan");
expect(first).toEqual(second);
- expect(first.lex.include).toEqual(["canary", "rollout", "plan"]);
+ // No stopword list: every token is kept (deduped, capped), in any
+ // language. Common words are deprioritised by ranking, not dropped here.
+ expect(first.lex.include).toEqual(["the", "canary", "rollout", "plan"]);
expect(first.lex.exclude).toEqual([]);
expect(first.vec[0]).toBe("the canary rollout plan");
expect(first.hyde).toHaveLength(1);
@@ -47,7 +50,7 @@ describe("expandQuery", () => {
expect(first.intent).toBeNull();
});
- test("keeps the raw tokens when every token is a stopword", () => {
+ test("keeps every token - there is no stopword list to strip", () => {
const doc = expandQuery(vault, "the and of");
expect(doc.lex.include).toEqual(["the", "and", "of"]);
});
@@ -81,7 +84,10 @@ describe("expandQuery", () => {
});
describe("search --expand", () => {
- test("stopword stripping recovers a hit the implicit-AND lane misses", async () => {
+ test("--expand annotates the lex lane and preserves real hits", async () => {
+ // Language-agnostic: expansion no longer strips a hardcoded English
+ // stopword list (that recovery only ever worked for English). A query
+ // whose own terms match is still found, with the lex lane annotated.
writeFileSync(
join(vault, "canary.md"),
"# Canary rollout\n\nShip one instance first, observe, expand gradually.\n",
@@ -89,10 +95,7 @@ describe("search --expand", () => {
writeFileSync(join(vault, "other.md"), "# Other\n\nNothing relevant here at all.\n");
await indexVault(config);
- const plain = await search(config, { query: "the canary rollout" });
- expect(plain.results).toHaveLength(0);
-
- const expanded = await search(config, { query: "the canary rollout", expand: true });
+ const expanded = await search(config, { query: "canary rollout", expand: true });
expect(expanded.results.some((r) => r.path === "canary.md")).toBe(true);
expect(
expanded.results
diff --git a/tests/core/search/search.test.ts b/tests/core/search/search.test.ts
index ff56b4d3..aaac402e 100644
--- a/tests/core/search/search.test.ts
+++ b/tests/core/search/search.test.ts
@@ -216,7 +216,13 @@ test("search degrades semantic structured lanes when semantic search is disabled
});
test("search can return an evidence pack and downrank terminal-state support", async () => {
- writeMd(vault, "Notes/terminal.md", "# Terminal\n\nalpha beta superseded by active note.");
+ // Terminal state is declared via the controlled frontmatter `status:`
+ // field, not inferred from prose - language-agnostic downranking.
+ writeMd(
+ vault,
+ "Notes/terminal.md",
+ "---\nstatus: superseded\n---\n\n# Terminal\n\nalpha beta old support.",
+ );
writeMd(vault, "Notes/active.md", "# Active\n\nalpha beta current support.");
const cfg = makeConfig({ vault, dbPath });
await indexVault(cfg);
diff --git a/tests/core/search/surfacing-gate.test.ts b/tests/core/search/surfacing-gate.test.ts
index 444cee36..3b030ebc 100644
--- a/tests/core/search/surfacing-gate.test.ts
+++ b/tests/core/search/surfacing-gate.test.ts
@@ -2,8 +2,8 @@ import { expect, test } from "bun:test";
import { evaluateSurfacingGate } from "../../../src/core/search/surfacing-gate.ts";
-test("evaluateSurfacingGate skips greetings, slash commands, shell-only prompts, and duplicates", () => {
- expect(evaluateSurfacingGate({ prompt: "hello" }).reason).toBe("greeting");
+test("evaluateSurfacingGate suppresses only structural non-recall prompts", () => {
+ expect(evaluateSurfacingGate({ prompt: "" }).reason).toBe("empty");
expect(evaluateSurfacingGate({ prompt: "/help" }).reason).toBe("slash_command");
expect(evaluateSurfacingGate({ prompt: "git status" }).reason).toBe("shell_command");
expect(
@@ -14,26 +14,28 @@ test("evaluateSurfacingGate skips greetings, slash commands, shell-only prompts,
).toBe("duplicate");
});
-test("evaluateSurfacingGate retrieves real memory questions and explicit requests", () => {
- expect(
- evaluateSurfacingGate({
- prompt: "what did we decide about recall diagnostics?",
- }),
- ).toEqual({
- retrieve: true,
- reason: "memory_question",
- });
+test("evaluateSurfacingGate fails open for any non-suppressed prompt, in any language", () => {
+ // No natural-language keyword list: an English memory question, a
+ // greeting, a long uncertain question, and a non-Latin prompt all
+ // fall through to the same language-agnostic retrieve decision.
+ for (const prompt of [
+ "what did we decide about recall diagnostics?",
+ "hello",
+ "Can you compare the options we discussed and tell me which path is less risky?",
+ "что мы решили про диагностику recall?",
+ "私たちは何を決めましたか",
+ "هل ناقشنا هذا من قبل",
+ ]) {
+ expect(evaluateSurfacingGate({ prompt })).toEqual({
+ retrieve: true,
+ reason: "default_retrieve",
+ });
+ }
+});
+
+test("evaluateSurfacingGate honours an explicit recall request", () => {
expect(evaluateSurfacingGate({ prompt: "hello", explicit: true })).toEqual({
retrieve: true,
reason: "explicit",
});
});
-
-test("evaluateSurfacingGate fails open for uncertain long questions", () => {
- expect(
- evaluateSurfacingGate({
- prompt:
- "Can you compare the options we discussed and tell me which implementation path is less risky?",
- }),
- ).toEqual({ retrieve: true, reason: "uncertain_question" });
-});
diff --git a/tests/core/search/tuning.test.ts b/tests/core/search/tuning.test.ts
index 424bd582..576f99e1 100644
--- a/tests/core/search/tuning.test.ts
+++ b/tests/core/search/tuning.test.ts
@@ -48,16 +48,24 @@ const DATASET = parseRecallBenchmarkDataset({
],
});
+// "the" is deliberately seeded into 2 of the 3 notes (canary, other) but
+// NOT into backup.md. It is therefore corpus-common (>= 50% document
+// frequency) and dropped from the implicit-AND lex lane by the
+// language-agnostic, DF-driven common-token filter - which is what lets
+// the "the nightly backup" query reach backup.md only WITH expansion.
function writeNotes(): void {
writeFileSync(
join(vault, "canary.md"),
- "# Canary rollout\n\nShip one instance first, observe, expand gradually.\n",
+ "# Canary rollout\n\nShip the first instance, observe, then expand the rollout gradually.\n",
);
writeFileSync(
join(vault, "backup.md"),
"# Nightly backup\n\nSnapshot databases every night, verify checksums offsite.\n",
);
- writeFileSync(join(vault, "other.md"), "# Other\n\nNothing relevant whatsoever.\n");
+ writeFileSync(
+ join(vault, "other.md"),
+ "# Other\n\nNothing in the list here is relevant to the task.\n",
+ );
}
describe("grid + apply", () => {
diff --git a/tests/e2e/link-recall-intelligence.integration.test.ts b/tests/e2e/link-recall-intelligence.integration.test.ts
index b0975dd4..a93f1531 100644
--- a/tests/e2e/link-recall-intelligence.integration.test.ts
+++ b/tests/e2e/link-recall-intelligence.integration.test.ts
@@ -59,9 +59,15 @@ afterEach(() => {
test("the suite composes end to end on one vault", async () => {
// -- 1. Vault: an alias owner, twin unlinked notes, one community ------
+ // "the" is seeded into the majority of the prose notes below (but NOT
+ // weekly-review.md) so that the DF-driven, language-agnostic common-token
+ // filter treats it as corpus-common and drops it from the implicit-AND
+ // lex lane - the language-neutral replacement for the old English
+ // stopword list. weekly-review.md is the recovery target, so it must
+ // omit "the" for the bare implicit-AND query to miss it.
writeFileSync(
join(vault, "project-alpha.md"),
- '---\ntitle: Project Alpha\naliases: ["PA"]\n---\n\n# Project Alpha\n\nUmbrella initiative: importer, dashboard, billing pipeline.\n',
+ '---\ntitle: Project Alpha\naliases: ["PA"]\n---\n\n# Project Alpha\n\nThe umbrella initiative covers the importer, the dashboard, and the billing pipeline.\n',
);
writeFileSync(
join(vault, "weekly-review.md"),
@@ -81,7 +87,7 @@ test("the suite composes end to end on one vault", async () => {
.filter((t) => t !== name)
.map((t) => `[[${t}]]`)
.join(" ");
- writeFileSync(join(vault, `${name}.md`), `# ${name}\n\nSee ${others}.\n`);
+ writeFileSync(join(vault, `${name}.md`), `# ${name}\n\nSee the related teams ${others}.\n`);
}
// -- 2. Alias resolution materializes at index time --------------------
diff --git a/tests/e2e/project-history.integration.test.ts b/tests/e2e/project-history.integration.test.ts
index 6c34db88..c0ea436f 100644
--- a/tests/e2e/project-history.integration.test.ts
+++ b/tests/e2e/project-history.integration.test.ts
@@ -93,7 +93,7 @@ test("ingest -> find -> mine -> architect -> re-run: one coherent project memory
expect(mined.created).toBe(1);
const candidate = readFileSync(mined.notes[0]!, "utf8");
expect(candidate).toContain("conventional_breaking");
- expect(candidate).toContain("decision_keyword:adopt");
+ expect(candidate).toContain("breaking_change_footer");
expect(candidate).toContain("migrate to jsonl store");
// ── Architect: structural notes for the same project. ──
diff --git a/tests/mcp/context-pack-tool.test.ts b/tests/mcp/context-pack-tool.test.ts
index eb2aa368..7c8a4e5b 100644
--- a/tests/mcp/context-pack-tool.test.ts
+++ b/tests/mcp/context-pack-tool.test.ts
@@ -112,7 +112,9 @@ describe("brain_context_pack tool — round trip", () => {
);
writeFileSync(
join(vault, "Brain", "preferences", "pref-constraint.md"),
- "---\nid: pref-constraint\ntopic: t\nprinciple: Never expose tokens\ntier: core\n---\n",
+ // Constraints lane is opt-in via the explicit context_lane field,
+ // not inferred from the prose word "Never" (language-agnostic).
+ "---\nid: pref-constraint\ntopic: t\nprinciple: Never expose tokens\ntier: core\ncontext_lane: constraints\n---\n",
);
const server = new MCPServer({ vault, configPath });
await initialize(server);
diff --git a/tests/mcp/recall-gate-telemetry.test.ts b/tests/mcp/recall-gate-telemetry.test.ts
index 96cf7238..59ebe92a 100644
--- a/tests/mcp/recall-gate-telemetry.test.ts
+++ b/tests/mcp/recall-gate-telemetry.test.ts
@@ -36,7 +36,7 @@ function tool(name: string) {
}
test("default off: the gate emits no telemetry", async () => {
- const decision = (await tool("brain_recall_gate").handler(ctx(), { prompt: "hello" })) as {
+ const decision = (await tool("brain_recall_gate").handler(ctx(), { prompt: "/help" })) as {
retrieve: boolean;
};
expect(decision.retrieve).toBe(false);
@@ -50,7 +50,7 @@ test("with recall_gate_telemetry on, decisions land as continuity records", asyn
telemetry_host: "hermes",
session_id: "sess-1",
});
- await tool("brain_recall_gate").handler(ctx(), { prompt: "hi" });
+ await tool("brain_recall_gate").handler(ctx(), { prompt: "/help" });
const records = listGateTelemetry(vault);
expect(records).toHaveLength(2);
expect(records.some((r) => r.payload["decision"] === "retrieve")).toBe(true);