Skip to content

Self-hosted MCP server + dashboard giving Claude Code, Cursor & Hermes agents shared long-term memory over a git-backed markdown vault. Agent-native, Obsidian-compatible second brain — markdown RAG, no vector DB.

feat(search): rank by authority, not just relevance

Ronalds Palacisddf8d32f320f
diff --git a/app/api/search/route.ts b/app/api/search/route.ts
index 2774dc0..bb0ff83 100644
--- a/app/api/search/route.ts
+++ b/app/api/search/route.ts
@@ -3,6 +3,9 @@ import { searchNotes } from "@/lib/vault/store";
export const dynamic = "force-dynamic";
export async function GET(req: Request) {
- const q = new URL(req.url).searchParams.get("q") ?? "";
- return Response.json({ results: searchNotes(q) });
+ const params = new URL(req.url).searchParams;
+ const q = params.get("q") ?? "";
+ // Humans browsing the dashboard should be able to find archived notes; they still rank
+ // far below live ones. Agents (MCP) get them excluded by default.
+ return Response.json({ results: searchNotes(q, { includeArchive: params.get("archive") !== "false" }) });
}
diff --git a/lib/config.ts b/lib/config.ts
index 5be6743..6663d74 100644
--- a/lib/config.ts
+++ b/lib/config.ts
@@ -29,6 +29,17 @@ export const VAULT_IGNORE = new Set(
.filter(Boolean),
);
+/**
+ * Top-level folders whose notes are historical: demoted in search and excluded by default.
+ * Purely a ranking hint — the notes stay readable, and nothing is hidden from brain_read.
+ */
+export const ARCHIVE_FOLDERS = new Set(
+ (process.env.ARCHIVE_FOLDERS ?? "archive,archives,_archive,trash")
+ .split(",")
+ .map((s) => s.trim().toLowerCase())
+ .filter(Boolean),
+);
+
/** Bearer token required to call the MCP endpoint. */
export const MCP_TOKEN = process.env.MCP_TOKEN ?? "";
diff --git a/lib/mcp/tools.ts b/lib/mcp/tools.ts
index dc6679d..0e21ba3 100644
--- a/lib/mcp/tools.ts
+++ b/lib/mcp/tools.ts
@@ -4,9 +4,13 @@ import {
getNote,
getTree,
listNotes,
+ listRecent,
readVaultFile,
searchNotes,
+ vaultConventions,
} from "@/lib/vault/store";
+import { authorityOf } from "@/lib/vault/authority";
+import { extractSection, listHeadings } from "@/lib/vault/parse";
import {
appendNote,
createFolder,
@@ -55,42 +59,110 @@ export const TOOLS: Tool[] = [
{
name: "brain_schema",
description:
- "Return the vault's SCHEMA.md — folder taxonomy, frontmatter conventions, wikilink model, and the write protocol. READ THIS FIRST before writing notes.",
+ "READ THIS FIRST, before searching or writing. Returns the vault's SCHEMA.md (folder taxonomy, frontmatter conventions, wikilink model, write protocol) AND a live description of this specific vault: its folders, the `status:` values actually in use, how search ranks notes by authority, and any integrity warnings. Vaults differ — never assume conventions, read them here.",
inputSchema: { type: "object", properties: {} },
- handler: () => readVaultFile("SCHEMA.md") ?? "(no SCHEMA.md in this vault)",
+ handler: () => ({
+ schema: readVaultFile("SCHEMA.md") ?? "(no SCHEMA.md in this vault)",
+ conventions: vaultConventions(),
+ }),
},
{
name: "brain_search",
- description: "Full-text + fuzzy search across all notes. Returns ranked results (path, title, folder).",
+ description:
+ "Keyword search across the vault. Returns ranked hits with `path`, `title`, `folder`, `status`, `snippet`, and `authority`.\n\n" +
+ "IMPORTANT — ranking is by keyword relevance, NOT by truth. A superseded document repeats the query words just as often as the live one, so it can outrank it. Every hit carries an `authority` field: `authoritative` (source of truth — prefer it), `current`, `provisional` (draft/proposed — never quote as settled), `superseded`, `archived`. Rank order is a suggestion; `authority` is the signal.\n\n" +
+ "Before quoting a price, guarantee, contract term, or any other single-valued fact, open the `authoritative` note and read it. If two notes disagree on such a fact, that is a defect in the vault — report it rather than averaging them.\n\n" +
+ "Archived notes are excluded unless `includeArchive` is true.",
inputSchema: {
type: "object",
- properties: { query: s("search query"), limit: { type: "number", description: "max results (default 20)" } },
+ properties: {
+ query: s("search query — a few keywords beat a full sentence"),
+ limit: { type: "number", description: "max results (default 20)" },
+ folder: s("restrict to one top-level folder, e.g. 'decisions'"),
+ includeArchive: { type: "boolean", description: "include historical notes in archive folders (default false)" },
+ },
required: ["query"],
},
- handler: ({ query, limit }) => searchNotes(String(query ?? ""), typeof limit === "number" ? limit : 20),
+ handler: ({ query, limit, folder, includeArchive }) =>
+ searchNotes(String(query ?? ""), {
+ limit: typeof limit === "number" ? limit : 20,
+ folder: folder ? String(folder) : undefined,
+ includeArchive: includeArchive === true,
+ }),
},
{
name: "brain_read",
- description: "Read a note's full markdown (frontmatter + body) plus its backlinks. Path is vault-relative, e.g. 'clients/mks/mks.md'.",
- inputSchema: { type: "object", properties: { path: s("vault-relative path") }, required: ["path"] },
- handler: ({ path }) => {
+ description:
+ "Read a note: full markdown (frontmatter + body), its `authority`, and its backlinks. Path is vault-relative, e.g. 'clients/mks/mks.md'.\n\n" +
+ "Pass `section` to read just one heading's content instead of the whole file — cheaper on long documents. If the heading isn't found you get the list of available headings back.\n\n" +
+ "Always check the returned `authority` before acting on the content: `archived` and `superseded` notes are history, not instructions.",
+ inputSchema: {
+ type: "object",
+ properties: {
+ path: s("vault-relative path"),
+ section: s("optional heading to extract, e.g. 'Pricing' — matches case-insensitively by prefix"),
+ },
+ required: ["path"],
+ },
+ handler: ({ path, section }) => {
const n = getNote(String(path));
if (!n) return { error: "not found", path };
- return {
+ const base = {
path: n.path,
title: n.title,
+ status: n.status,
+ authority: authorityOf(n),
+ tags: n.tags,
frontmatter: n.frontmatter,
- content: n.raw,
backlinks: getBacklinks(n.path).map((b) => b.path),
};
+ if (!section) return { ...base, content: n.raw };
+
+ const found = extractSection(n.body, String(section));
+ if (found === null) {
+ return { ...base, error: `section "${section}" not found`, sections: listHeadings(n.body) };
+ }
+ return { ...base, section: String(section), content: found, sections: listHeadings(n.body) };
},
},
{
name: "brain_list",
- description: "List all notes with metadata (path, title, folder, type, tags, status). Use to discover what exists.",
+ description:
+ "List every note with metadata (path, title, folder, type, tags, status, authority). Use to discover what exists. `authority` tells you which notes are source-of-truth and which are history.",
inputSchema: { type: "object", properties: {} },
handler: () =>
- listNotes().map((n) => ({ path: n.path, title: n.title, folder: n.folder, type: n.type, tags: n.tags, status: n.status })),
+ listNotes().map((n) => ({
+ path: n.path,
+ title: n.title,
+ folder: n.folder,
+ type: n.type,
+ tags: n.tags,
+ status: n.status,
+ authority: authorityOf(n),
+ })),
+ },
+ {
+ name: "brain_recent",
+ description:
+ "Notes changed most recently, newest first. Use to catch up on what humans or other agents have done since you last looked.",
+ inputSchema: {
+ type: "object",
+ properties: {
+ since: s("optional ISO date/time — only notes modified at or after this"),
+ limit: { type: "number", description: "max results (default 20)" },
+ },
+ },
+ handler: ({ since, limit }) => {
+ const t = since ? Date.parse(String(since)) : NaN;
+ return listRecent(Number.isNaN(t) ? undefined : t, typeof limit === "number" ? limit : 20).map((n) => ({
+ path: n.path,
+ title: n.title,
+ folder: n.folder,
+ status: n.status,
+ authority: authorityOf(n),
+ modified: new Date(n.mtimeMs).toISOString(),
+ }));
+ },
},
{
name: "brain_tree",
@@ -154,7 +226,8 @@ export const TOOLS: Tool[] = [
},
{
name: "brain_move",
- description: "Move or rename a note.",
+ description:
+ "Move or rename a note. This is how you retire something: when a note stops being true, move it to an archive folder (see brain_schema → conventions.ranking.archiveFolders) rather than deleting it. Archived notes are demoted in search and excluded by default, so they stop misleading agents while the reasoning trail survives. Leave a pointer in the replacement note saying what superseded what.",
inputSchema: { type: "object", properties: { from: s("current path"), to: s("new path") }, required: ["from", "to"] },
handler: async ({ from, to }) => ({ ok: true, path: await moveNote(String(from), String(to)) }),
},
@@ -173,7 +246,8 @@ export const TOOLS: Tool[] = [
},
{
name: "brain_delete",
- description: "Delete a note from the working copy (recoverable via git history).",
+ description:
+ "Delete a note (recoverable via git history). Prefer brain_move into an archive folder — deleting destroys the reasoning trail, archiving only removes it from search. Delete when the note is wrong or duplicated, archive when it is merely no longer true.",
inputSchema: { type: "object", properties: { path: s("vault-relative path") }, required: ["path"] },
handler: async ({ path }) => {
await deleteNote(String(path));
diff --git a/lib/vault/authority.ts b/lib/vault/authority.ts
new file mode 100644
index 0000000..074e447
--- /dev/null
+++ b/lib/vault/authority.ts
@@ -0,0 +1,80 @@
+import { ARCHIVE_FOLDERS } from "@/lib/config";
+
+/**
+ * Authority — how much a note should be trusted, independent of how well it matches a query.
+ *
+ * Search engines rank by *resemblance*. They cannot tell a live contract from a dead one:
+ * a superseded price list mentions "price" just as often as the real one, so it can easily
+ * outrank it. Authority is the missing axis. It is derived entirely from the files
+ * (path + frontmatter), never stored separately, so the index stays disposable.
+ */
+export type Authority = "authoritative" | "current" | "provisional" | "superseded" | "archived";
+
+/** Status/tag words that mark a note as the source of truth for whatever it covers. */
+const AUTHORITATIVE = ["locked", "canonical", "source-of-truth", "authoritative"];
+/** Words that mark a note as no longer true. */
+const SUPERSEDED = ["superseded", "deprecated", "obsolete", "replaced", "retired", "dead"];
+/** Words that mark a note as not yet decided. A `proposed` decision is not a decision. */
+const PROVISIONAL = ["draft", "proposed", "exploring", "tentative", "wip", "idea"];
+
+/** Multiplier applied to a note's search score. Files are truth; this only reorders hits. */
+const WEIGHT: Record<Authority, number> = {
+ authoritative: 3.5,
+ current: 1,
+ provisional: 0.55,
+ superseded: 0.15,
+ archived: 0.08,
+};
+
+/** One-line explanation per class, surfaced to agents via brain_schema. */
+export const AUTHORITY_MEANING: Record<Authority, string> = {
+ authoritative: "Source of truth for its subject. Quote this over anything that disagrees.",
+ current: "Live, but not declared authoritative. Fine for context.",
+ provisional: "Not decided yet (draft/proposed). Never quote as settled.",
+ superseded: "Explicitly replaced. Historical only.",
+ archived: "Lives in an archive folder. Historical only — never quote as current.",
+};
+
+/** Match `needle` as a whole token inside `hay` (so "draft" hits "draft-for-approval"). */
+function hasWord(hay: string, needle: string): boolean {
+ const esc = needle.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
+ return new RegExp(`(^|[^a-z0-9])${esc}([^a-z0-9]|$)`, "i").test(hay);
+}
+
+/** True when the note's top-level folder is an archive folder. */
+export function isArchivedPath(relPath: string): boolean {
+ return ARCHIVE_FOLDERS.has(relPath.split("/")[0].toLowerCase());
+}
+
+/**
+ * Classify a note. Order matters: a locked note inside `archive/` is still archived,
+ * and an explicitly superseded note is dead however loudly it calls itself canonical.
+ */
+export function authorityOf(meta: { path: string; status?: string; tags?: string[] }): Authority {
+ if (isArchivedPath(meta.path)) return "archived";
+ const hay = [meta.status ?? "", ...(meta.tags ?? [])].join(" ").toLowerCase();
+ if (SUPERSEDED.some((w) => hasWord(hay, w))) return "superseded";
+ if (PROVISIONAL.some((w) => hasWord(hay, w))) return "provisional";
+ if (AUTHORITATIVE.some((w) => hasWord(hay, w))) return "authoritative";
+ return "current";
+}
+
+export function weightOf(a: Authority): number {
+ return WEIGHT[a];
+}
+
+/**
+ * The live ranking contract, returned by brain_schema so an agent working against a vault
+ * it has never seen can discover the rules instead of guessing them.
+ */
+export function authorityRules() {
+ return {
+ summary:
+ "Search ranks by keyword relevance, then multiplies by authority. Authority comes from the note's folder and frontmatter — never from the text.",
+ archiveFolders: [...ARCHIVE_FOLDERS],
+ statusWords: { authoritative: AUTHORITATIVE, superseded: SUPERSEDED, provisional: PROVISIONAL },
+ weights: WEIGHT,
+ meaning: AUTHORITY_MEANING,
+ note: "Words are matched in both `status:` and `tags:`. A note with no status is `current` (weight 1) — vaults that use no conventions rank purely by relevance, exactly as before.",
+ };
+}
diff --git a/lib/vault/parse.ts b/lib/vault/parse.ts
index 8530d5d..5223f54 100644
--- a/lib/vault/parse.ts
+++ b/lib/vault/parse.ts
@@ -50,6 +50,47 @@ export function extractRawLinks(body: string, frontmatter: Record<string, unknow
return links;
}
+/** Every markdown heading in a body, in document order, without the leading #s. */
+export function listHeadings(body: string): string[] {
+ return body
+ .split("\n")
+ .filter((l) => /^#{1,6}\s+\S/.test(l))
+ .map((l) => l.replace(/^#{1,6}\s+/, "").trim());
+}
+
+/**
+ * Return one section of a note: the matching heading plus everything under it, stopping at
+ * the next heading of the same or higher level. Lets an agent read 40 lines of a 1000-line
+ * document instead of all of it. Matches case-insensitively, by prefix.
+ */
+export function extractSection(body: string, heading: string): string | null {
+ const lines = body.split("\n");
+ const want = heading.replace(/^#+\s*/, "").trim().toLowerCase();
+ if (!want) return null;
+
+ let start = -1;
+ let level = 0;
+ for (let i = 0; i < lines.length; i++) {
+ const m = /^(#{1,6})\s+(.*)$/.exec(lines[i]);
+ if (!m) continue;
+ const text = m[2].trim().toLowerCase();
+ if (text === want || text.startsWith(want)) {
+ start = i;
+ level = m[1].length;
+ break;
+ }
+ }
+ if (start === -1) return null;
+
+ const out = [lines[start]];
+ for (let i = start + 1; i < lines.length; i++) {
+ const m = /^(#{1,6})\s+/.exec(lines[i]);
+ if (m && m[1].length <= level) break;
+ out.push(lines[i]);
+ }
+ return out.join("\n").trim();
+}
+
export interface ParsedNote {
meta: Omit<NoteMeta, "mtimeMs">;
body: string;
@@ -78,6 +119,7 @@ export function parseNote(relPath: string, raw: string): ParsedNote {
folder: folderOf(relPath),
type: typeof data.type === "string" ? data.type : tags[0],
tags,
+ aliases: normalizeTags(data.aliases),
status: data.status != null ? String(data.status) : undefined,
created: data.created != null ? String(data.created) : undefined,
updated:
diff --git a/lib/vault/store.ts b/lib/vault/store.ts
index 1752c57..0a3141a 100644
--- a/lib/vault/store.ts
+++ b/lib/vault/store.ts
@@ -6,15 +6,34 @@ import { VAULT_IGNORE } from "@/lib/config";
import { activeVaultDir } from "@/lib/repos";
import { scanVault } from "./scan";
import { parseNote, stemOf } from "./parse";
+import { authorityOf, authorityRules, isArchivedPath, weightOf, type Authority } from "./authority";
import type { Graph, GraphEdge, GraphNode, Note, NoteMeta, TreeNode } from "./types";
+interface IndexDoc {
+ id: string;
+ title: string;
+ aliases: string;
+ tags: string;
+ body: string;
+ folder: string;
+ type: string;
+ /** stored, not indexed — used for filtering + authority-weighted ranking */
+ status: string;
+ authority: Authority;
+ mtimeMs: number;
+}
+
interface IndexState {
dir: string;
notes: Map<string, NoteMeta>;
+ /** path -> wikilink target stems. Kept so link/graph rebuilds need no disk IO. */
+ linksBySource: Map<string, string[]>;
stemToPath: Map<string, string>;
+ /** stems claimed by more than one file — wikilinks to these resolve to the first, silently. */
+ duplicateStems: Map<string, string[]>;
outEdges: Map<string, Set<string>>;
inEdges: Map<string, Set<string>>;
- search: MiniSearch;
+ search: MiniSearch<IndexDoc>;
builtAt: number;
}
@@ -22,52 +41,147 @@ let state: IndexState | null = null;
let watcher: FSWatcher | null = null;
let watchedDir = "";
-function buildState(): IndexState {
- const dir = activeVaultDir();
- const files = scanVault(dir);
- const notes = new Map<string, NoteMeta>();
- const stemToPath = new Map<string, string>();
- const linksBySource = new Map<string, string[]>();
- const docs: Array<{ id: string; title: string; tags: string; body: string; folder: string; type: string }> = [];
-
- for (const f of files) {
- let raw = "";
- try {
- raw = fs.readFileSync(f.abs, "utf8");
- } catch {
- continue;
- }
- const { meta, body, rawLinks } = parseNote(f.rel, raw);
- const nm: NoteMeta = { ...meta, mtimeMs: f.mtimeMs };
- notes.set(f.rel, nm);
- if (!stemToPath.has(nm.slug)) {
- stemToPath.set(nm.slug, f.rel);
- } else if (process.env.NODE_ENV !== "production") {
- console.warn(`[vault] duplicate stem "${nm.slug}" (${stemToPath.get(nm.slug)} vs ${f.rel}) — links resolve to the first.`);
+function newIndex(): MiniSearch<IndexDoc> {
+ return new MiniSearch<IndexDoc>({
+ fields: ["title", "aliases", "tags", "body", "folder", "type"],
+ storeFields: ["title", "folder", "type", "status", "authority", "mtimeMs"],
+ searchOptions: {
+ boost: { title: 3, aliases: 3, tags: 2 },
+ prefix: true,
+ fuzzy: 0.2,
+ // OR, deliberately. With AND, one query word the canonical note happens to lack
+ // ("tiers") drops it from the results entirely — and the superseded note that *does*
+ // use the word becomes the only answer. Authority weighting can only reorder what
+ // survives the combiner, so the combiner must not throw the truth away.
+ // Docs matching more terms still score higher; noise is handled by ranking, not exclusion.
+ combineWith: "OR",
+ },
+ });
+}
+
+function toDoc(meta: NoteMeta, body: string): IndexDoc {
+ return {
+ id: meta.path,
+ title: meta.title,
+ aliases: meta.aliases.join(" "),
+ tags: meta.tags.join(" "),
+ body,
+ folder: meta.folder,
+ type: meta.type ?? "",
+ status: meta.status ?? "",
+ authority: authorityOf(meta),
+ mtimeMs: meta.mtimeMs,
+ };
+}
+
+function readAndParse(
+ dir: string,
+ rel: string,
+ mtimeMs: number,
+): { meta: NoteMeta; body: string; stems: string[] } | null {
+ let raw: string;
+ try {
+ raw = fs.readFileSync(path.join(dir, rel), "utf8");
+ } catch {
+ return null;
+ }
+ const { meta, body, rawLinks } = parseNote(rel, raw);
+ return { meta: { ...meta, mtimeMs }, body, stems: rawLinks.map((l) => stemOf(l.target)) };
+}
+
+/**
+ * Rebuild stem resolution + the link graph from in-memory maps only.
+ * Cheap (no disk IO), so it can run after every single-file change.
+ */
+function recomputeLinks(s: IndexState): void {
+ s.stemToPath = new Map();
+ s.duplicateStems = new Map();
+ for (const rel of [...s.notes.keys()].sort()) {
+ const slug = s.notes.get(rel)!.slug;
+ const first = s.stemToPath.get(slug);
+ if (first === undefined) {
+ s.stemToPath.set(slug, rel);
+ } else {
+ const dupes = s.duplicateStems.get(slug) ?? [first];
+ dupes.push(rel);
+ s.duplicateStems.set(slug, dupes);
}
- linksBySource.set(f.rel, rawLinks.map((l) => stemOf(l.target)));
- docs.push({ id: f.rel, title: nm.title, tags: nm.tags.join(" "), body, folder: nm.folder, type: nm.type ?? "" });
+ }
+ for (const [slug, paths] of s.duplicateStems) {
+ console.warn(`[vault] duplicate stem "${slug}" (${paths.join(", ")}) — wikilinks resolve to the first.`);
}
- const outEdges = new Map<string, Set<string>>();
- const inEdges = new Map<string, Set<string>>();
- for (const [src, stems] of linksBySource) {
+ s.outEdges = new Map();
+ s.inEdges = new Map();
+ for (const [src, stems] of s.linksBySource) {
for (const stem of stems) {
- const tgt = stemToPath.get(stem);
+ const tgt = s.stemToPath.get(stem);
if (!tgt || tgt === src) continue;
- (outEdges.get(src) ?? outEdges.set(src, new Set()).get(src)!).add(tgt);
- (inEdges.get(tgt) ?? inEdges.set(tgt, new Set()).get(tgt)!).add(src);
+ (s.outEdges.get(src) ?? s.outEdges.set(src, new Set()).get(src)!).add(tgt);
+ (s.inEdges.get(tgt) ?? s.inEdges.set(tgt, new Set()).get(tgt)!).add(src);
}
}
+}
- const search = new MiniSearch({
- fields: ["title", "tags", "body", "folder", "type"],
- storeFields: ["title", "folder", "type"],
- searchOptions: { boost: { title: 3, tags: 2 }, prefix: true, fuzzy: 0.2, combineWith: "AND" },
- });
- search.addAll(docs);
+function buildState(): IndexState {
+ const dir = activeVaultDir();
+ const s: IndexState = {
+ dir,
+ notes: new Map(),
+ linksBySource: new Map(),
+ stemToPath: new Map(),
+ duplicateStems: new Map(),
+ outEdges: new Map(),
+ inEdges: new Map(),
+ search: newIndex(),
+ builtAt: Date.now(),
+ };
- return { dir, notes, stemToPath, outEdges, inEdges, search, builtAt: Date.now() };
+ const docs: IndexDoc[] = [];
+ for (const f of scanVault(dir)) {
+ const parsed = readAndParse(dir, f.rel, f.mtimeMs);
+ if (!parsed) continue;
+ s.notes.set(f.rel, parsed.meta);
+ s.linksBySource.set(f.rel, parsed.stems);
+ docs.push(toDoc(parsed.meta, parsed.body));
+ }
+ s.search.addAll(docs);
+ recomputeLinks(s);
+ return s;
+}
+
+/** Add or update one file in the live index. Returns false when the file is gone. */
+function upsert(s: IndexState, rel: string): boolean {
+ let mtimeMs = 0;
+ try {
+ mtimeMs = fs.statSync(path.join(s.dir, rel)).mtimeMs;
+ } catch {
+ return false;
+ }
+ const parsed = readAndParse(s.dir, rel, mtimeMs);
+ if (!parsed) return false;
+ if (s.notes.has(rel)) {
+ try {
+ s.search.discard(rel);
+ } catch {
+ /* not in the index — fall through to add */
+ }
+ }
+ s.notes.set(rel, parsed.meta);
+ s.linksBySource.set(rel, parsed.stems);
+ s.search.add(toDoc(parsed.meta, parsed.body));
+ return true;
+}
+
+function remove(s: IndexState, rel: string): void {
+ if (!s.notes.has(rel)) return;
+ try {
+ s.search.discard(rel);
+ } catch {
+ /* already gone */
+ }
+ s.notes.delete(rel);
+ s.linksBySource.delete(rel);
}
function startWatcher(dir: string) {
@@ -88,18 +202,41 @@ function startWatcher(dir: string) {
return [...VAULT_IGNORE].some((ig) => base === ig || p.includes(`${path.sep}${ig}${path.sep}`));
},
});
+
+ const pending = new Set<string>();
let t: ReturnType<typeof setTimeout> | null = null;
- const trigger = () => {
- if (t) clearTimeout(t);
- t = setTimeout(() => {
- try {
+ let fullRebuild = false;
+
+ const flush = () => {
+ try {
+ if (fullRebuild || !state || state.dir !== dir) {
state = buildState();
- } catch (e) {
- console.error("[vault] rebuild failed", e);
+ } else {
+ for (const rel of pending) if (!upsert(state, rel)) remove(state, rel);
+ recomputeLinks(state);
}
- }, 250);
+ } catch (e) {
+ console.error("[vault] index update failed", e);
+ }
+ pending.clear();
+ fullRebuild = false;
+ };
+
+ const schedule = () => {
+ if (t) clearTimeout(t);
+ t = setTimeout(flush, 250);
+ };
+ const onFile = (abs: string) => {
+ if (!abs.toLowerCase().endsWith(".md")) return;
+ pending.add(path.relative(dir, abs).split(path.sep).join("/"));
+ schedule();
+ };
+ const onDir = () => {
+ fullRebuild = true;
+ schedule();
};
- watcher.on("add", trigger).on("change", trigger).on("unlink", trigger).on("addDir", trigger).on("unlinkDir", trigger);
+
+ watcher.on("add", onFile).on("change", onFile).on("unlink", onFile).on("addDir", onDir).on("unlinkDir", onDir);
} catch (e) {
console.error("[vault] watcher failed to start", e);
}
@@ -112,16 +249,34 @@ function ensure(): IndexState {
return state;
}
-/** Force a synchronous rebuild (after a write or a workspace switch). */
+/** Force a synchronous full rebuild (workspace switch, or after a git pull changed many files). */
export function rebuildIndex(): void {
state = buildState();
startWatcher(state.dir);
}
+/** Update just these paths (created, edited, moved, or deleted). No full re-scan. */
+export function refreshPaths(relPaths: string[]): void {
+ const s = ensure();
+ for (const rel of relPaths) {
+ if (!rel.toLowerCase().endsWith(".md")) continue;
+ if (!upsert(s, rel)) remove(s, rel);
+ }
+ recomputeLinks(s);
+}
+
export function listNotes(): NoteMeta[] {
return [...ensure().notes.values()];
}
+/** Notes ordered by last modification, newest first. Powers "what changed lately". */
+export function listRecent(sinceMs?: number, limit = 20): NoteMeta[] {
+ return [...ensure().notes.values()]
+ .filter((n) => (sinceMs ? n.mtimeMs >= sinceMs : true))
+ .sort((a, b) => b.mtimeMs - a.mtimeMs)
+ .slice(0, limit);
+}
+
export function getNote(relPath: string): Note | null {
const s = ensure();
const abs = path.join(s.dir, relPath);
@@ -140,19 +295,93 @@ export interface SearchHit {
title: string;
folder: string;
type?: string;
+ status?: string;
+ /** Trust class, independent of relevance. See lib/vault/authority.ts. */
+ authority: Authority;
score: number;
+ snippet?: string;
+}
+
+export interface SearchOpts {
+ limit?: number;
+ /** Restrict to one top-level folder. */
+ folder?: string;
+ /** Include notes in archive folders (still ranked far below live ones). Default false. */
+ includeArchive?: boolean;
+ /** Attach ~200 chars of matching context per hit. Default true. */
+ snippets?: boolean;
}
-export function searchNotes(q: string, limit = 40): SearchHit[] {
+/** First body line containing a query term (or the first prose line), trimmed for display. */
+function snippetFor(dir: string, rel: string, terms: string[]): string | undefined {
+ try {
+ const raw = fs.readFileSync(path.join(dir, rel), "utf8");
+ const { body } = parseNote(rel, raw);
+ const lines = body
+ .split("\n")
+ .map((l) => l.trim())
+ .filter((l) => l.length > 0 && !l.startsWith("#") && !l.startsWith("---"));
+ const hit = lines.find((l) => {
+ const low = l.toLowerCase();
+ return terms.some((t) => low.includes(t));
+ });
+ const chosen = hit ?? lines[0];
+ if (!chosen) return undefined;
+ return chosen.length > 220 ? `${chosen.slice(0, 217)}…` : chosen;
+ } catch {
+ return undefined;
+ }
+}
+
+/**
+ * Keyword search, then reweighted by authority.
+ *
+ * The ranking is deliberately two-stage: MiniSearch says how well a note *matches*,
+ * authorityOf() says how far it can be *trusted*. Without the second stage a superseded
+ * price list outranks the locked one, because it repeats the query words just as often.
+ */
+export function searchNotes(q: string, opts: SearchOpts = {}): SearchHit[] {
const s = ensure();
if (!q.trim()) return [];
- return s.search.search(q).slice(0, limit).map((r) => ({
- path: r.id as string,
- title: (r as unknown as { title: string }).title,
- folder: (r as unknown as { folder: string }).folder,
- type: (r as unknown as { type?: string }).type || undefined,
- score: r.score,
- }));
+ const { limit = 20, folder, includeArchive = false, snippets = true } = opts;
+
+ const raw = s.search.search(q, {
+ boostDocument: (_id, _term, stored) => weightOf((stored?.authority as Authority) ?? "current"),
+ filter: (r) => {
+ const stored = r as unknown as { folder: string; authority: Authority };
+ if (!includeArchive && stored.authority === "archived") return false;
+ if (folder && stored.folder !== folder) return false;
+ return true;
+ },
+ });
+
+ const terms = q
+ .toLowerCase()
+ .split(/\s+/)
+ .map((t) => t.replace(/[^a-z0-9-]/g, ""))
+ .filter((t) => t.length >= 3);
+
+ return raw.slice(0, limit).map((r) => {
+ const hit = r as unknown as {
+ id: string;
+ title: string;
+ folder: string;
+ type?: string;
+ status?: string;
+ authority: Authority;
+ score: number;
+ };
+ return {
+ path: hit.id,
+ title: hit.title,
+ folder: hit.folder,
+ type: hit.type || undefined,
+ status: hit.status || undefined,
+ authority: hit.authority,
+ score: hit.score,
+ snippet: snippets ? snippetFor(s.dir, hit.id, terms) : undefined,
+ };
+ });
}
export function getBacklinks(relPath: string): NoteMeta[] {
@@ -165,6 +394,43 @@ export function getOutlinks(relPath: string): NoteMeta[] {
return [...(s.outEdges.get(relPath) ?? [])].map((p) => s.notes.get(p)).filter(Boolean) as NoteMeta[];
}
+/**
+ * What this vault actually looks like + how search will treat it. Returned by brain_schema so
+ * an agent meeting an unfamiliar vault discovers its conventions instead of assuming ours.
+ */
+export function vaultConventions() {
+ const s = ensure();
+ const folders = new Set<string>();
+ const statuses = new Map<string, number>();
+ const types = new Set<string>();
+ let archived = 0;
+
+ for (const n of s.notes.values()) {
+ folders.add(n.folder);
+ if (n.type) types.add(n.type);
+ if (n.status) statuses.set(n.status, (statuses.get(n.status) ?? 0) + 1);
+ if (isArchivedPath(n.path)) archived++;
+ }
+
+ return {
+ noteCount: s.notes.size,
+ archivedCount: archived,
+ folders: [...folders].sort(),
+ types: [...types].sort(),
+ statusesInUse: [...statuses.entries()]
+ .sort((a, b) => b[1] - a[1])
+ .map(([status, count]) => ({ status, count })),
+ ranking: authorityRules(),
+ integrity:
+ s.duplicateStems.size === 0
+ ? { duplicateStems: [] }
+ : {
+ duplicateStems: [...s.duplicateStems.entries()].map(([stem, paths]) => ({ stem, paths })),
+ warning: "Wikilinks to a duplicated stem resolve to the first path only. Rename one.",
+ },
+ };
+}
+
export function getGraph(folder?: string): Graph {
const s = ensure();
const inScope = (p: string) => !folder || s.notes.get(p)?.folder === folder;
diff --git a/lib/vault/types.ts b/lib/vault/types.ts
index 926e983..1a013bc 100644
--- a/lib/vault/types.ts
+++ b/lib/vault/types.ts
@@ -8,6 +8,8 @@ export interface NoteMeta {
folder: string;
type?: string;
tags: string[];
+ /** Alternate names/synonyms for this note, from `aliases:`. Indexed like the title. */
+ aliases: string[];
status?: string;
created?: string;
updated?: string;
diff --git a/lib/vault/write.ts b/lib/vault/write.ts
index 669b2d0..b22930d 100644
--- a/lib/vault/write.ts
+++ b/lib/vault/write.ts
@@ -2,7 +2,7 @@ import fsp from "node:fs/promises";
import path from "node:path";
import matter from "gray-matter";
import { activeVaultDir } from "@/lib/repos";
-import { rebuildIndex } from "./store";
+import { refreshPaths } from "./store";
import { requestSync } from "@/lib/git";
/** Resolve a vault-relative path to an absolute path in the active vault, refusing escapes. */
@@ -18,9 +18,10 @@ export function normalizeNotePath(relPath: string): string {
return /\.md$/i.test(p) ? p : `${p}.md`;
}
-function after(p: string) {
- rebuildIndex();
- requestSync(p);
+/** Re-index only what changed, then queue a git commit. `touched` are vault-relative paths. */
+function after(message: string, touched: string[]) {
+ refreshPaths(touched);
+ requestSync(message);
}
/** Write a note from a raw markdown string (frontmatter included). Used by the editor. */
@@ -29,7 +30,7 @@ export async function writeNoteRaw(relPath: string, content: string): Promise<st
const abs = safeAbs(p);
await fsp.mkdir(path.dirname(abs), { recursive: true });
await fsp.writeFile(abs, content, "utf8");
- after(`edit ${p}`);
+ after(`edit ${p}`, [p]);
return p;
}
@@ -52,7 +53,7 @@ export async function appendNote(relPath: string, text: string): Promise<string>
await fsp.mkdir(path.dirname(abs), { recursive: true });
const sep = existing && !existing.endsWith("\n") ? "\n" : "";
await fsp.writeFile(abs, `${existing}${sep}${text}\n`, "utf8");
- after(`append ${p}`);
+ after(`append ${p}`, [p]);
return p;
}
@@ -61,13 +62,13 @@ export async function moveNote(from: string, to: string): Promise<string> {
const b = safeAbs(normalizeNotePath(to));
await fsp.mkdir(path.dirname(b), { recursive: true });
await fsp.rename(a, b);
- after(`move ${from} -> ${to}`);
+ after(`move ${from} -> ${to}`, [normalizeNotePath(from), normalizeNotePath(to)]);
return normalizeNotePath(to);
}
export async function deleteNote(relPath: string): Promise<void> {
await fsp.rm(safeAbs(normalizeNotePath(relPath)));
- after(`delete ${relPath}`);
+ after(`delete ${relPath}`, [normalizeNotePath(relPath)]);
}
export async function createFolder(relPath: string): Promise<string> {
diff --git a/sample-vault/SCHEMA.md b/sample-vault/SCHEMA.md
index 1b30c84..44401b7 100644
--- a/sample-vault/SCHEMA.md
+++ b/sample-vault/SCHEMA.md
@@ -3,9 +3,13 @@
The contract every agent reads before writing (served via the `brain_schema` MCP tool).
The app is taxonomy-agnostic — it renders whatever folders exist; this documents the defaults.
+> `brain_schema` returns this file **plus a live report of your actual vault**: its folders, the
+> `status:` values in use, the exact ranking weights, and any integrity warnings. An agent never
+> has to guess your conventions — it can read them.
+
## Folders
`clients/` · `decisions/` · `projects/` · `people/` · `meetings/` · `docs/` · `research/` ·
-`inbox/` (unsorted captures). Slugs are kebab-case. New folders are allowed.
+`inbox/` (unsorted captures) · `archive/` (see below). Slugs are kebab-case. New folders are allowed.
## Frontmatter (YAML, optional)
```yaml
@@ -13,17 +17,55 @@ The app is taxonomy-agnostic — it renders whatever folders exist; this documen
title: Human title # graph node label (falls back to filename)
type: decision # decision | client | project | person | meeting | doc | research
tags: [alpha, beta]
-status: active # free text
+status: active # see "Authority" — this drives search ranking
+aliases: [win-back, churned] # synonyms; searched like the title
created: 2026-01-15
related:
- "[[projects/engram]]"
---
```
+## Authority — how search decides what to trust
+
+Search ranks by **keyword relevance**, which knows nothing about truth. A superseded price list
+uses the word "price" as often as the live one, so it will happily outrank it. Engram fixes this by
+multiplying each hit's score by an **authority** weight derived from the note itself — never from
+its text. Every search hit and every `brain_read` returns its `authority`.
+
+| authority | Triggered by | Weight | Meaning |
+|---|---|---|---|
+| `authoritative` | `status:` or a tag containing `locked`, `canonical`, `source-of-truth` | ×3.5 | Source of truth. Quote this over anything that disagrees. |
+| `current` | anything else (**including no status at all**) | ×1 | Live. Fine for context. |
+| `provisional` | `draft`, `proposed`, `exploring`, `wip`, `tentative`, `idea` | ×0.55 | Not decided. Never quote as settled. |
+| `superseded` | `superseded`, `deprecated`, `obsolete`, `replaced`, `retired`, `dead` | ×0.15 | Explicitly replaced. History only. |
+| `archived` | note lives under `archive/` (or `archives/`, `_archive/`, `trash/`) | ×0.08 | History only. **Excluded from agent search by default.** |
+
+Words are matched inside `status:` *and* `tags:`, as whole tokens — so `status: draft-for-approval`
+counts as `draft`. A vault that uses none of these conventions ranks purely by relevance, exactly as
+it would without them: **the feature is opt-in and degrades to nothing.**
+
+Override the archive folder names with the `ARCHIVE_FOLDERS` env var.
+
+### The rule this exists to enforce
+**Compose for context; route for facts.** Reading widely across notes to understand a client or a
+history is what the graph is for. But a *single-valued fact* — a price, a guarantee, a legal entity,
+a contract term — has exactly one owning note. Don't merge sources, don't take the top hit. Open the
+`authoritative` note. **If two notes disagree on such a fact, that is a defect in the vault** — report
+it rather than averaging them.
+
+### Retiring a note
+When something stops being true, **`brain_move` it into `archive/`** rather than deleting it, and add
+a pointer in the replacement saying what superseded what. Archiving removes it from search; deleting
+destroys the reasoning trail. An archived note with no superseding pointer is worse than a deleted
+one — it still looks like an answer.
+
+Mark a note `status: superseded` when it must stay in place (a decision log, where chronology matters).
+
## Links
- `[[note]]` / `[[../path/note|Alias]]` — resolve by filename stem.
- `related:` frontmatter arrays are edges too.
- Graph: nodes = notes (colored by folder, sized by degree), edges = the links above.
+- Two files sharing a basename is a bug: links resolve to the first. `brain_schema` reports these.
## Callouts
`> [!note]`, `> [!tip]`, `> [!warning]`, `> [!abstract]`, `> [!danger]`, `> [!question]`.