Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
/**
* Per-page write-time lint (evidence-at-the-boundary, task A4).
*
* A write used to land and say nothing about what it wrote. The only
* quality pass over a page was `lintConsolidate`, a vault-wide walk
* measured at 359 ms on a 2090-page vault, so an agent that wrote a
* broken wikilink or a document with no frontmatter learned about it
* from a sweep that may never run. Worse, `assertValidDocument` ran only
* under `strict` on create: update, append and every batch operation
* committed with NO document validation at all and returned a hardcoded
* success flag.
*
* This module closes that hole with the detectors that already exist,
* run against exactly the pages one call committed:
*
* - {@link validateArtifact} for the error-severity codes, so a strict
* create and a linted update agree on what a valid document is.
* - merged-link, resolved through the canonical-id resolver rather than
* through a vault-wide merge map (see `lint-consolidate.ts`).
* - broken Brain-artifact wikilink against `collectAllBasenames`, the
* readdir-only basename index the doctor already builds (6 ms).
* - {@link NEAR_DUPLICATE_CODE}, the authored body compared against the
* pages it could plausibly duplicate - the SAME directory AND the SAME
* composite scope bucket - through the shared `tokenise`/`jaccard`
* primitives (t_d30c0548). A bounded, deterministic candidate set per
* write; the finding rides the receipt and never gates the write.
* - {@link UNDATED_QUANTITY_CODE}, the authored body scanned for a count
* or amount with no structural date on the page (t_fb442a24). The
* trigger shapes live in `undated-quantity.ts`; this module owns the
* escape hatches - `valid_from` in frontmatter, an ISO date in the
* body's claim-bearing prose, or a `[Source: <name>, YYYY-MM-DD]`
* marker - and the finding quotes the lexeme and its body offset,
* counted after frontmatter removal and named as such, never the
* line. Advisory only.
*
* `demote-stale-stable` is deliberately EXCLUDED, not overlooked: its
* trigger is the page's creation age against the staleness cap, so it can
* never fire on a page written this turn. Reporting it here would be a
* check that is structurally incapable of finding anything.
*
* ## What this envelope does NOT cover, stated rather than implied
*
* - `brain_write_session` keeps its own `errors[]` channel and its own
* correction prompt. It validates BEFORE committing and refuses, which
* is a different contract from an advisory computed after the fact;
* folding it in would give one tool two error channels.
* - `write-session/engine.ts` and `distill/distill-source.ts` write with
* their own lexical validation and bypass the note-write path
* entirely, so nothing here sees their bytes.
* - Log appends (`append_log_line`, `apply_evidence`) name no note path
* and are not linted: the log line is machine-composed, not authored.
* - Body normalization on the write boundary is a recorded REFUSAL
* (t_d30c0548), not an omission: every candidate set (whitespace
* collapse, smart-quote folding, trailing-space trim) changes authored
* bytes that receipts, dedup and byte-stability tests pin. The
* near-duplicate score is computed over the body exactly as authored,
* and frontmatter is never compared and never touched.
*
* ## Shape
*
* The payload composes two conventions already in the codebase rather
* than inventing a third: `WriteSessionError {code, path, message}` plus
* a `severity` from `DoctorSeverity` plus the `next_command` idiom
* resolved through `nextCommandField`. One field is added - `page` - for
* the reason a batch needs it: one report spans several files, and a
* finding that cannot name its file is unusable. `path` keeps its
* `WriteSessionError` meaning, the location WITHIN the document.
*
* The report always carries `total`, `returned`, `truncated` and
* `skipped`, so a capped list can never be read as a complete one, and a
* page too large to validate lands in `skipped` with a reason instead of
* vanishing. A page the lint THREW on lands there too: the failure is one
* page's, and reporting it as a whole-report `unavailable` discarded the
* findings already collected for the pages before it.
*
* Every string in the report is composed here from identifiers and
* integers. A raw errno message would name an absolute path, which is the
* operator's home directory in a write receipt - the leak
* `writeFrontmatterAtomic` was rewritten to close in the same release.
*
* ## Attachment
*
* Modelled field for field on `write-advisory.ts`: computed AROUND the
* write, never gating it, surfaced as an additive key that is ABSENT
* ENTIRELY when there is nothing to say. Failure of the lint itself - of
* the whole call, before any page has been read - degrades to a named
* {@link PageLintUnavailable} on the key rather than to a missing key: an
* absent `lint` key must mean clean, and only that.
*/
import { lstatSync, readFileSync, readdirSync, statSync } from "node:fs";
import { join, posix, resolve } from "node:path";
import { vaultRelative } from "../path-safety.ts";
import { ROUTE_STAGE, timeStageSync } from "../route-scope.ts";
import { compositeScopeKey, scopeFromFrontmatter } from "../scope-key.ts";
import { parseFrontmatterText } from "../vault.ts";
import type { FrontmatterMap } from "../types.ts";
import { memoryFromStore, storeMemoryRel } from "../store/read/memory-ids.ts";
import { collectAllBasenames } from "./doctor/records.ts";
import {
LINT_CONSOLIDATE_KIND,
createMergedLinkResolver,
type MergedLinkResolution,
type MergedLinkResolver,
} from "./lint-consolidate.ts";
import { nextCommandField, type NextCommandField } from "./next-step.ts";
import { pageIdToPath, reportMergeChain } from "./page-meta/page-id.ts";
import { brainDirs } from "./paths.ts";
import { loadSchemaPack } from "./schema-pack.ts";
import type { BrainSchemaVocabulary } from "./schema-vocab.ts";
import {
NEAR_DUPLICATE_THRESHOLDS,
findNearDuplicates,
type NearDuplicatePoolEntry,
type ReadableRef,
} from "./near-duplicate.ts";
import { VALID_FROM_KEY } from "./lifecycle/temporal-replace.ts";
import { tokenise } from "./similarity.ts";
import { parseCitations } from "./temporal/citations.ts";
import type { DoctorSeverity } from "./types.ts";
import {
containsIsoDate,
containsProseIsoDate,
findUndatedQuantities,
} from "./undated-quantity.ts";
import {
WIKILINK_TARGET_RE,
isBrainArtifactId,
isLiveWikilinkSpan,
normaliseWikilinkTarget,
} from "./wikilink.ts";
import { ARTIFACT_MAX_BYTES, validateArtifact } from "./write-session/validate.ts";
/** The additive JSON key every write surface carries the report in. */
export const PAGE_LINT_KEY = "lint";
/**
* Longest finding list a write receipt carries. A write response is read
* inline by the agent that made the write; past a couple of dozen lines
* it stops being a fix list and becomes a payload. Everything beyond the
* cap is still COUNTED - see `total` and `truncated`.
*/
export const PAGE_LINT_MAX_FINDINGS = 25;
/** Code the lint reports itself under when it could not run at all. */
export const PAGE_LINT_UNAVAILABLE_CODE = "page-lint-unavailable";
/** The doctor code an unresolvable Brain wikilink is already reported as. */
const BROKEN_WIKILINK_CODE = "broken-wikilink";
/**
* Near-duplicate bar (t_d30c0548): the jaccard score at or above which the
* authored body counts as closely matching an existing page. Deliberately
* high - near-DUPLICATE claims near-identity, unlike the 0.5 contradiction
* precedent that answers "same subject". An alias of the kernel's one
* threshold table, so the write hint cannot drift from it.
*/
export const NEAR_DUPLICATE_JACCARD = NEAR_DUPLICATE_THRESHOLDS.writeHint;
/** Code the near-duplicate detector reports under. */
export const NEAR_DUPLICATE_CODE = "near-duplicate";
/**
* Code the undated-quantity detector reports under (write-honesty, F4).
* A count or amount with no structural date on the page answers as true at
* every instant, because an absent `valid_from` is unbounded. The code is
* registered in `diagnostics.ts` so the finding carries a next command.
*/
export const UNDATED_QUANTITY_CODE = "undated-quantity";
/**
* Most sibling pages one directory contributes as near-duplicate
* candidates: the newest by mtime. Every sibling is stat'ed (cheap), but
* only these are read and tokenised, so a write into a folder of
* thousands of notes costs a bounded read. Siblings past the cap are
* counted on the report as `candidates_skipped`, never dropped silently.
*/
export const NEAR_DUPLICATE_MAX_CANDIDATES = 200;
/** Extension every candidate page on disk carries. */
const MARKDOWN_EXT = ".md";
/**
* Canonical prefix of a vault-relative spelling that leaves the vault.
* The write kernel refuses traversal, so such a page can only reach this
* module from a caller - but the candidate walk must not depend on that:
* enumerating outside the vault would read the operator's filesystem for
* a receipt about the vault.
*/
const VAULT_ESCAPE_PREFIX = "../";
/** Absent-directory case for a candidate lookup: an empty list, never a miss. */
const NO_CANDIDATES: ReadonlyArray<NearDuplicateCandidate> = Object.freeze([]);
/** Why a written page was not linted. Never a silent omission. */
export const PAGE_LINT_SKIP_REASON = Object.freeze({
overByteCap: "page-over-byte-cap",
unreadable: "page-unreadable",
/** The page was read and the lint itself threw on it. See {@link lintOnePage}. */
lintFailed: "page-lint-failed",
} as const);
export type PageLintSkipReason = (typeof PAGE_LINT_SKIP_REASON)[keyof typeof PAGE_LINT_SKIP_REASON];
/** Membership list, in the order a page meets the three gates. */
export const PAGE_LINT_SKIP_REASONS: ReadonlyArray<PageLintSkipReason> = Object.freeze([
PAGE_LINT_SKIP_REASON.overByteCap,
PAGE_LINT_SKIP_REASON.unreadable,
PAGE_LINT_SKIP_REASON.lintFailed,
]);
/**
* Narrow a string read back across a tool boundary. The reason crosses the
* MCP wire on all four write tools, where a caller decides from it whether
* the silence about that page means clean or means unread, so a reader needs
* to be able to reject a value this build does not understand.
*/
export function isPageLintSkipReason(value: unknown): value is PageLintSkipReason {
return (
typeof value === "string" && (PAGE_LINT_SKIP_REASONS as ReadonlyArray<string>).includes(value)
);
}
/** What a failure is called when it carries neither an errno nor a name. */
const UNNAMED_FAILURE_CODE = "unknown";
/**
* The identity of a failure, with nothing of the operator's filesystem in it:
* the errno code when the kernel supplied one, the error's class otherwise.
*
* The rejected alternative is `err.message`, which is what shipped. Node
* renders an errno as `ENOENT: no such file or directory, stat
* '/home/<user>/<vault>/Brain/x.md'` - an absolute home path, in a key that
* rides back on every note-write response. The same release rewrote
* `writeFrontmatterAtomic` (`src/core/vault.ts`) for precisely this reason:
* identifiers and integers cross this boundary, never a path and never an OS
* message.
*/
export function failureCode(err: unknown): string {
const code = (err as NodeJS.ErrnoException | null)?.code;
if (typeof code === "string" && code.length > 0) return code;
const name = (err as Error | null)?.name;
return typeof name === "string" && name.length > 0 ? name : UNNAMED_FAILURE_CODE;
}
/** Rank by severity: errors are what a strict create would have refused. */
const SEVERITY_RANK: Readonly<Record<DoctorSeverity, number>> = Object.freeze({
error: 0,
warning: 1,
});
/**
* One finding about one written page. `code`, `path` and `message` are
* the `WriteSessionError` triple verbatim; `severity` and `next_command`
* are the two conventions layered on top; `page` names the file.
*/
export interface PageLintFinding extends NextCommandField {
readonly severity: DoctorSeverity;
readonly code: string;
/** Vault-relative path of the page the finding is about. */
readonly page: string;
/**
* Location within the document: `body`, `frontmatter`, `tags`, or the
* thing the finding points at - a link target for the link detectors,
* the resembling page's vault-relative path for {@link NEAR_DUPLICATE_CODE}
* (evidence in the same spirit the write-conflict advisory renders
* `[[pref-id]] jaccard=…` into its log events).
*/
readonly path: string;
readonly message: string;
}
/** A written page the lint did not read, and why. */
export interface PageLintSkip {
/** Vault-relative path of the page, never an absolute one. */
readonly page: string;
readonly reason: PageLintSkipReason;
/**
* The measurement or the errno CODE behind the skip, so the reason is
* checkable. Identifiers and integers only - see {@link failureCode}.
*/
readonly detail: string;
}
/** The lint itself failed. Reported, never swallowed into an absent key. */
export interface PageLintUnavailable {
readonly code: string;
readonly message: string;
}
export interface PageLintReport {
/** Ranked, capped at {@link PAGE_LINT_MAX_FINDINGS}. */
readonly findings: ReadonlyArray<PageLintFinding>;
/** Findings detected, including those the cap dropped. */
readonly total: number;
/** Findings actually carried in {@link findings}. */
readonly returned: number;
/** True iff `total > returned`. */
readonly truncated: boolean;
/** Written pages that were not linted, each with its reason. */
readonly skipped: ReadonlyArray<PageLintSkip>;
/** Present iff the lint could not run; the counters above are then zero. */
readonly unavailable?: PageLintUnavailable;
/**
* Sibling pages the near-duplicate check did not compare against
* because their directory exceeded {@link NEAR_DUPLICATE_MAX_CANDIDATES}.
* Present only when the cap applied.
*/
readonly candidates_skipped?: number;
/** Sibling pages the near-duplicate check could not read, each named. Present only when any. */
readonly candidates_unreadable?: ReadonlyArray<PageLintCandidateSkip>;
/**
* Whether the near-duplicate check looked beyond the written pages'
* directories. Present only when widening is on; `used` alone is not
* something to say, `index_unavailable` is.
*/
readonly widening?: NearDuplicateWideningStatus;
/**
* Why widening reported `index_unavailable`: the search error code, the
* errno code or the error name ({@link failureCode}), so a missing index,
* a locked or corrupt one and a defect read differently. Present only
* with that status.
*/
readonly widening_detail?: string;
}
/** A sibling page the near-duplicate check could not read, and the errno code why. */
export interface PageLintCandidateSkip {
readonly page: string;
readonly detail: string;
}
/**
* Whether the near-duplicate hint looked beyond the written pages' own
* directories: `used` when the keyword index answered, `index_unavailable`
* when it could not be opened or queried. Absent from a report when
* widening is off.
*/
export type NearDuplicateWideningStatus = "used" | "index_unavailable";
/** How one call to {@link lintWrittenPages} is bounded. */
export interface LintWrittenPagesOptions {
/**
* The caller's reach. Required, never defaulted: a sibling page the
* caller may not read is dropped before it is scored and before the
* census counts it, and a wikilink target or merge-chain hop the caller
* may not read answers as absent, so neither a finding nor a census
* number can disclose it. An operator-reach caller passes `READ_ALL_REFS`.
*/
readonly readable: ReadableRef;
/**
* Candidates from outside the written pages' directories, collected by
* `page-lint-widening.ts`. Scored under the same scope-bucket and reach
* rules; a page already in a written page's directory counts once.
*/
readonly extraCandidates?: ReadonlyArray<NearDuplicateCandidate>;
/** Whether widening ran; absent when it is off. */
readonly widening?: NearDuplicateWideningStatus;
/** Why widening was unavailable; see {@link PageLintReport.widening_detail}. */
readonly wideningDetail?: string;
}
/** What building the near-duplicate candidate index left out, by count and by name. */
export interface NearDuplicateCensus {
readonly candidatesSkipped: number;
readonly unreadable: ReadonlyArray<PageLintCandidateSkip>;
/** Whether widening ran; absent when it is off. */
readonly widening?: NearDuplicateWideningStatus;
/** Why widening was unavailable; see {@link PageLintReport.widening_detail}. */
readonly wideningDetail?: string;
}
/**
* Rank findings: errors ahead of warnings (an error is what a strict
* create would have refused), then page, then code, then location. Named
* and exported once so a CLI surface and an MCP surface cannot drift into
* two orderings of one list.
*/
export function comparePageLintFindings(a: PageLintFinding, b: PageLintFinding): number {
return (
SEVERITY_RANK[a.severity] - SEVERITY_RANK[b.severity] ||
a.page.localeCompare(b.page) ||
a.code.localeCompare(b.code) ||
a.path.localeCompare(b.path)
);
}
function finding(
severity: DoctorSeverity,
code: string,
page: string,
path: string,
message: string,
): PageLintFinding {
return Object.freeze({ severity, code, page, path, message, ...nextCommandField(code) });
}
/**
* The schema type the document declares, or `null`. Mirrors
* `assertValidDocument`: the validator judges the declared type against
* the pack. Fed the frontmatter {@link parseFrontmatterText} already
* produced for the near-duplicate scan, so one page parses exactly once;
* that parse is total (a document without frontmatter parses to an empty
* map), so the null case is simply a document that declares nothing.
*/
function declaredSchemaType(meta: FrontmatterMap): string | null {
const declared = meta["type"];
return typeof declared === "string" && declared.trim() !== "" ? declared : null;
}
/**
* Every distinct wikilink target on the page, in first-appearance order.
* Distinct rather than per-occurrence: a page that names one dead target
* five times has one thing wrong with it, and five identical rows would
* spend the finding cap saying so. A bracket behind an odd backslash run
* is escaped data and opens or closes no link - the same rule
* `extractWikilinks` applies, so a `\[[literal]]` span raises no
* broken-link row.
*/
function distinctLinkTargets(raw: string): ReadonlyArray<string> {
const seen = new Set<string>();
for (const match of raw.matchAll(WIKILINK_TARGET_RE)) {
if (!isLiveWikilinkSpan(raw, match.index!, match.index! + match[0].length)) continue;
const target = normaliseWikilinkTarget(match[1] ?? "");
if (target !== "") seen.add(target);
}
return [...seen];
}
/**
* One page that can serve the near-duplicate detector as a candidate: its
* vault-relative path, composite scope bucket, and body tokens, read once
* per call. The written pages themselves are in here too - a batch can
* write two near-identical pages into one directory and each should see
* the other - so self-comparison is excluded at detection time, not here.
*/
export interface NearDuplicateCandidate {
/** Vault-relative path of the candidate page. */
readonly page: string;
/** The page's composite scope bucket, `compositeScopeKey` spelling. */
readonly scopeKey: string;
/** The authored body through the shared `tokenise` primitive. */
readonly tokens: ReadonlySet<string>;
}
/**
* Everything the lint computes once per call and threads per page.
*
* Exported with {@link lintPagesWithContext} so the per-page failure path can
* be exercised: no filesystem state reaches it - every reader inside
* {@link lintOnePage} either cannot throw or catches its own errors - and an
* accumulation that is only correct in theory is the kind this release keeps
* finding. A caller already holding the indexes can lint with them too.
*/
export interface LintContext {
readonly basenames: ReadonlySet<string>;
readonly vocabulary: BrainSchemaVocabulary;
readonly mergedLinks: MergedLinkResolver;
/**
* The near-duplicate candidates, keyed by the vault-relative directory
* they live in. Built once per call from exactly the directories the
* written pages landed in - the bounded candidate set (same directory,
* same scope bucket) needs nothing vault-wide.
*/
readonly nearDuplicateCandidates: ReadonlyMap<string, ReadonlyArray<NearDuplicateCandidate>>;
/**
* Candidates from outside the written pages' directories (widening).
* Every written page is compared with them as well as with its own
* directory's candidates.
*/
readonly extraCandidates?: ReadonlyArray<NearDuplicateCandidate>;
/**
* The caller's reach, applied again by the kernel before any score, and
* by the wikilink and merged-link checks, so a withheld page answers as
* an absent one on every finding.
*/
readonly readable: ReadableRef;
/** What the candidate index left out; absent means nothing was. */
readonly nearDuplicateCensus?: NearDuplicateCensus;
}
/**
* The near-duplicate findings for one written page: the authored body
* against every candidate in the same directory - plus every widened
* candidate - in the same composite scope bucket, scored through the
* shared kernel at {@link NEAR_DUPLICATE_JACCARD}. The page itself is
* never its own candidate - a batch can write two lookalikes, but a
* single write cannot resemble itself - and the score rides the message
* so an operator judgement call is informed.
*
* A same-directory finding keeps the message shipped since v1.64.0; a
* widened one appends `method=<method>`, so a reader can tell the hint
* came from the keyword index. One deliberate change from v1.73.0: the
* kernel scores nothing below `NEAR_DUPLICATE_MIN_TOKENS` body
* tokens, on either side, so two short stubs that share their few words
* by accident (formerly a jaccard=1.000 finding) are no longer reported.
* The kernel's own match order never reaches the receipt, which is ranked
* by {@link comparePageLintFindings}. The pool is bounded upstream (the
* directory cap is counted in `candidates_skipped`, the widening pull by
* its top-k), so the kernel's own cap is set to the pool size and never
* drops a candidate unreported.
*/
function nearDuplicateFindings(
ctx: LintContext,
page: string,
meta: FrontmatterMap,
body: string,
): PageLintFinding[] {
const local = ctx.nearDuplicateCandidates.get(posix.dirname(page)) ?? NO_CANDIDATES;
const widened = new Set<string>();
const pool: NearDuplicatePoolEntry[] = local.map(poolEntry);
const inPool = new Set(local.map((candidate) => candidate.page));
for (const candidate of ctx.extraCandidates ?? NO_CANDIDATES) {
if (inPool.has(candidate.page)) continue;
inPool.add(candidate.page);
widened.add(candidate.page);
pool.push(poolEntry(candidate));
}
const { matches } = findNearDuplicates({ ref: page, tokens: tokenise(body) }, pool, {
threshold: NEAR_DUPLICATE_JACCARD,
readable: ctx.readable,
bucket: compositeScopeKey(scopeFromFrontmatter(meta)),
cap: pool.length,
});
return matches.map((match) => {
const evidence = `body resembles [[${match.ref}]] jaccard=${match.score.toFixed(3)} (threshold ${NEAR_DUPLICATE_JACCARD})`;
const message = widened.has(match.ref) ? `${evidence} method=${match.method}` : evidence;
return finding("warning", NEAR_DUPLICATE_CODE, page, match.ref, message);
});
}
function poolEntry(candidate: NearDuplicateCandidate): NearDuplicatePoolEntry {
return { ref: candidate.page, tokens: candidate.tokens, bucket: candidate.scopeKey };
}
const NOT_MERGED: MergedLinkResolution = Object.freeze({ canonical: null, unresolvable: null });
/** Whether `id` names a Brain page on disk the caller may not read. */
function isWithheldPageId(vault: string, ctx: LintContext, id: string): boolean {
const path = pageIdToPath(vault, id);
return path !== null && !ctx.readable(vaultRelative(path, vault));
}
/**
* The merge-chain answer for one wikilink target at the caller's reach.
*
* The vault-wide resolver reads the `merged_into:` pointer of every page
* on the chain. A page the caller may not read answers as an absent one
* would: the walk ends at its id, the way it ends at a page with no file.
* So a withheld target is not merged at all, and a chain that reaches a
* withheld page names that page's id as the canonical end and nothing it
* points at. Only a link the resolver found merged pays the second walk.
*/
function mergedAtReach(vault: string, ctx: LintContext, target: string): MergedLinkResolution {
const merged = ctx.mergedLinks.resolve(target);
if (merged.canonical === null && merged.unresolvable === null) return merged;
const { visited } = reportMergeChain(vault, target);
const withheld = visited.findIndex((id) => isWithheldPageId(vault, ctx, id));
if (withheld === -1) return merged;
if (withheld === 0) return NOT_MERGED;
return Object.freeze({ canonical: visited[withheld]!, unresolvable: null });
}
/**
* Whether a Brain artifact named `target` exists where the caller may read
* it. `basenames` answers the vault-wide question once per call; a hit is
* asked again per directory `collectAllBasenames` indexes, so an artifact
* the caller may not read answers as an absent one. Only a linked name
* that exists pays the per-directory check.
*/
function hasReadableArtifact(vault: string, ctx: LintContext, target: string): boolean {
if (!ctx.basenames.has(target)) return false;
const dirs = brainDirs(vault);
const memoryDirs = [dirs.inbox, dirs.processed, dirs.archived, dirs.preferences, dirs.retired];
const fromStore = memoryFromStore(vault);
if (fromStore) {
// Store-only (task 2.10): a preference or a signal is a fold record,
// judged by the path its page would have. The fold's answer is final
// for the memory folders: a stale page the frozen legacy tree still
// holds there names nothing live.
const rel = storeMemoryRel(vault, target);
if (rel !== null && ctx.readable(rel)) return true;
}
for (const dir of [dirs.brain, ...(fromStore ? [] : memoryDirs), dirs.log]) {
const absolute = join(dir, `${target}.md`);
if (!ctx.readable(vaultRelative(absolute, vault))) continue;
if (lstatSync(absolute, { throwIfNoEntry: false })?.isFile() === true) return true;
}
return false;
}
/**
* Whether the page supplies a structural date that places its quantities
* in time. Three hatches, all structural: a `valid_from` boundary in
* frontmatter, an ISO date or instant in the body's claim-bearing prose,
* or a `[Source: <name>, YYYY-MM-DD]` marker. The citation marker always
* carries its own ISO date, so the date hatch already covers a well-formed
* one; it is checked explicitly through the canonical parser so the
* contract reads as three hatches rather than a coincidence.
*
* The body hatch reads the same masked view the quantity scanner does
* (`containsProseIsoDate`): a date inside a fenced block, an inline code
* span, a table row, a URL or a wikilink target is not a claim about the
* vault, so it must not quiet the undated quantities around it. The scope
* stays page-wide - one prose date anywhere still dates the page.
*/
function hasDatedQuantityContext(meta: FrontmatterMap, body: string): boolean {
const declared = meta[VALID_FROM_KEY];
// The boundary hatch is structural like the other two: the value must
// NAME a date. Any non-empty string ("TBD", "unknown", "soon") places
// nothing in time, and accepting it let a placeholder quiet the lint.
if (typeof declared === "string" && containsIsoDate(declared)) return true;
if (containsProseIsoDate(body)) return true;
return parseCitations(body).markers.length > 0;
}
/**
* The undated-quantity findings for one written page: one warning per
* lexeme the body states with no date on the page. The lexeme and its
* offset are the whole evidence - the authored line is not quoted, so a
* receipt cannot carry prose the caller did not ask it to repeat. The
* offset is counted within the post-frontmatter body the scanner sees
* (neither a file offset nor an index into the raw write), and the
* message names that coordinate system so a reader can locate the
* lexeme honestly.
*/
function undatedQuantityFindings(
page: string,
meta: FrontmatterMap,
body: string,
): PageLintFinding[] {
if (hasDatedQuantityContext(meta, body)) return [];
return findUndatedQuantities(body).map(({ lexeme, offset }) =>
finding(
"warning",
UNDATED_QUANTITY_CODE,
page,
"body",
`quantity ${JSON.stringify(lexeme)} at body offset ${offset} (after frontmatter) carries no date`,
),
);
}
function lintOnePage(
vault: string,
ctx: LintContext,
page: string,
raw: string,
): PageLintFinding[] {
const out: PageLintFinding[] = [];
const [meta, body] = parseFrontmatterText(raw);
const schemaType = declaredSchemaType(meta);
for (const violation of validateArtifact(raw, { schemaType, vocabulary: ctx.vocabulary })) {
out.push(finding("error", violation.code, page, violation.path, violation.message));
}
for (const target of distinctLinkTargets(raw)) {
// A link outside the Brain id space is user prose or a cross-layer
// note, and is nobody's here to flag - the same boundary the doctor
// and the orphaned-reference fixer draw.
if (!isBrainArtifactId(target)) continue;
const merged = mergedAtReach(vault, ctx, target);
if (merged.unresolvable !== null) {
out.push(
finding(
"warning",
LINT_CONSOLIDATE_KIND.mergedLink,
page,
target,
`wikilink [[${target}]] has a merge chain with no canonical end: ${merged.unresolvable}`,
),
);
continue;
}
if (merged.canonical !== null) {
out.push(
finding(
"warning",
LINT_CONSOLIDATE_KIND.mergedLink,
page,
target,
`wikilink [[${target}]] points at a page merged into ${merged.canonical}`,
),
);
continue;
}
if (!hasReadableArtifact(vault, ctx, target)) {
out.push(
finding(
"warning",
BROKEN_WIKILINK_CODE,
page,
target,
`wikilink [[${target}]] resolves to no Brain artifact`,
),
);
}
}
out.push(...undatedQuantityFindings(page, meta, body));
out.push(
...timeStageSync(ROUTE_STAGE.nearDuplicateLookup, () =>
nearDuplicateFindings(ctx, page, meta, body),
),
);
return out;
}
function skip(page: string, reason: PageLintSkipReason, detail: string): PageLintSkip {
return Object.freeze({ page, reason, detail });
}
/** The empty report: nothing detected, nothing skipped, nothing to say. */
function emptyReport(unavailable?: PageLintUnavailable): PageLintReport {
return Object.freeze({
findings: Object.freeze([]),
total: 0,
returned: 0,
truncated: false,
skipped: Object.freeze([]),
...(unavailable !== undefined ? { unavailable } : {}),
});
}
/**
* The vault-relative spelling of a page, whatever spelling the caller
* handed in. The ONE derivation for every string that names a page on
* this report - findings, skips, and near-duplicate candidates alike -
* so the map keys and the page strings compared against them can never
* drift into two normalizations.
*/
function canonicalPage(vault: string, page: string): string {
return vaultRelative(resolve(vault, page), vault);
}
/**
* The near-duplicate candidate index for one call: every Markdown page in
* exactly the directories the written pages landed in, keyed by that
* directory, each carrying its composite scope bucket and body tokens.
*
* Only the newest {@link NEAR_DUPLICATE_MAX_CANDIDATES} siblings per
* directory are read; the rest are counted in the census. A candidate
* that cannot be read is not a candidate - it cannot provide evidence -
* but it is NAMED in the census with its errno code, never skipped
* silently. A candidate over the artifact byte cap is
* skipped for the same cost reason a written page over the cap is skipped
* rather than validated. The directory enumerations themselves are NOT
* guarded: the write just committed into them, so a failure to list one
* is a failure of the lint's inputs and belongs in the report's
* `unavailable` slot like every other context-building failure. A page
* spelled outside the vault ({@link VAULT_ESCAPE_PREFIX}) collects no
* candidates at all: the walk stops at the vault boundary.
*
* A sibling outside the caller's reach (`readable`) is dropped at the
* listing, before it is stat'ed: it is never scored, never named as
* unreadable and never counted against the cap.
*/
function collectNearDuplicateCandidates(
vault: string,
pages: ReadonlyArray<string>,
readable: ReadableRef,
): {
readonly index: ReadonlyMap<string, ReadonlyArray<NearDuplicateCandidate>>;
readonly census: NearDuplicateCensus;
} {
const directories = new Set<string>();
for (const named of pages) {
const canonical = canonicalPage(vault, named);
if (canonical.startsWith(VAULT_ESCAPE_PREFIX)) continue;
directories.add(posix.dirname(canonical));
}
const index = new Map<string, NearDuplicateCandidate[]>();
const unreadable: PageLintCandidateSkip[] = [];
let candidatesSkipped = 0;
for (const directory of directories) {
// Stat every sibling (cheap), then read only the newest few.
const siblings: Array<{ page: string; absolute: string; size: number; mtimeMs: number }> = [];
for (const name of readdirSync(resolve(vault, directory))) {
if (!name.endsWith(MARKDOWN_EXT)) continue;
const absolute = resolve(vault, directory, name);
const page = canonicalPage(vault, absolute);
if (!readable(page)) continue;
try {
const stat = statSync(absolute);
siblings.push({ page, absolute, size: stat.size, mtimeMs: stat.mtimeMs });
} catch (err) {
unreadable.push({ page, detail: failureCode(err) });
}
}
const newest = siblings.toSorted((a, b) => b.mtimeMs - a.mtimeMs);
candidatesSkipped += Math.max(0, newest.length - NEAR_DUPLICATE_MAX_CANDIDATES);
const candidates: NearDuplicateCandidate[] = [];
for (const sibling of newest.slice(0, NEAR_DUPLICATE_MAX_CANDIDATES)) {
// Size from the inode: reading an over-cap page to find that out is
// the cost the cap exists to avoid.
if (sibling.size > ARTIFACT_MAX_BYTES) continue;
try {
const [meta, body] = parseFrontmatterText(readFileSync(sibling.absolute, "utf8"));
candidates.push({
page: sibling.page,
scopeKey: compositeScopeKey(scopeFromFrontmatter(meta)),
tokens: tokenise(body),
});
} catch (err) {
unreadable.push({ page: sibling.page, detail: failureCode(err) });
}
}
index.set(directory, candidates);
}
return { index, census: { candidatesSkipped, unreadable } };
}
/**
* Lint exactly the pages a write committed, reading what is on disk.
*
* Runs AFTER the commit on purpose: the batch kernel wrote the bytes and
* the handler never composed them, so the only honest subject is the
* file. It never gates the write and it NEVER throws: a failure that
* leaves nothing to lint AGAINST is reported as
* {@link PageLintReport.unavailable}, and a failure on one page is a
* {@link PAGE_LINT_SKIP_REASON.lintFailed} entry beside the results of the
* pages that did lint.
*
* Cost discipline: the basename index and the schema pack are computed
* ONCE per call and threaded, the merge resolver memoises per call, and
* nothing here walks vault content beyond the written pages' own
* directories - the near-duplicate candidate index reads each of those
* once per call, sized against the same artifact cap as the pages.
*
* Every finding answers at the caller's reach (`opts.readable`): a
* near-duplicate candidate, a wikilink target and a merge-chain hop the
* caller may not read all read as absent. The written pages themselves
* are always linted.
*/
export function lintWrittenPages(
vault: string,
pages: ReadonlyArray<string>,
opts: LintWrittenPagesOptions,
): PageLintReport {
if (pages.length === 0) return emptyReport();
let ctx: LintContext;
try {
// The candidate walk and the per-page scoring are the near-duplicate
// lookup; under an open route scope both add to that one stage.
const nearDuplicates = timeStageSync(ROUTE_STAGE.nearDuplicateLookup, () =>
collectNearDuplicateCandidates(vault, pages, opts.readable),
);
ctx = {
basenames: collectAllBasenames(vault),
vocabulary: loadSchemaPack(vault).vocabulary,
mergedLinks: createMergedLinkResolver(vault),
nearDuplicateCandidates: nearDuplicates.index,
...(opts.extraCandidates !== undefined ? { extraCandidates: opts.extraCandidates } : {}),
readable: opts.readable,
nearDuplicateCensus: {
...nearDuplicates.census,
...(opts.widening !== undefined ? { widening: opts.widening } : {}),
...(opts.wideningDetail !== undefined ? { wideningDetail: opts.wideningDetail } : {}),
},
};
} catch (err) {
return emptyReport({
code: PAGE_LINT_UNAVAILABLE_CODE,
message: `write-time page lint could not start: ${failureCode(err)}`,
});
}
return lintPagesWithContext(vault, ctx, pages);
}
/**
* Lint pages against indexes that are already built. The body of
* {@link lintWrittenPages}, split at the one seam that matters: everything
* above it fails the whole report (there are no indexes to lint against),
* everything below it fails at most one page.
*/
export function lintPagesWithContext(
vault: string,
ctx: LintContext,
pages: ReadonlyArray<string>,
): PageLintReport {
const detected: PageLintFinding[] = [];
const skipped: PageLintSkip[] = [];
for (const named of pages) {
// Rendered through the same derivation every other reporting surface
// uses, so the page a finding names is vault-relative whichever
// spelling the caller handed in - an absolute path here would be the
// operator's home directory in a write receipt.
const page = canonicalPage(vault, named);
const absolute = resolve(vault, page);
let raw: string;
try {
// Size first, from the inode: an over-cap page must be REPORTED as
// skipped, and reading it to find that out is the cost the cap
// exists to avoid.
const bytes = statSync(absolute).size;
if (bytes > ARTIFACT_MAX_BYTES) {
skipped.push(
skip(
page,
PAGE_LINT_SKIP_REASON.overByteCap,
`${bytes} bytes exceeds the ${ARTIFACT_MAX_BYTES}-byte artifact cap`,
),
);
continue;
}
raw = readFileSync(absolute, "utf8");
} catch (err) {
skipped.push(skip(page, PAGE_LINT_SKIP_REASON.unreadable, failureCode(err)));
continue;
}
try {
detected.push(...lintOnePage(vault, ctx, page, raw));
} catch (err) {
// One page's failure is one page's news. Reporting it as `unavailable`
// threw away the findings already collected for EARLIER pages and the
// skip list with them, and left the pages after it silently unlinted -
// a report that reads clean about work that was never done. The report
// already has an honest shape for "this page was not linted".
skipped.push(skip(page, PAGE_LINT_SKIP_REASON.lintFailed, failureCode(err)));
}
}
const ranked = detected.toSorted(comparePageLintFindings);
const returned = ranked.slice(0, PAGE_LINT_MAX_FINDINGS);
const census = ctx.nearDuplicateCensus;
return Object.freeze({
findings: Object.freeze(returned),
total: ranked.length,
returned: returned.length,
truncated: ranked.length > returned.length,
skipped: Object.freeze(skipped),
...(census !== undefined && census.candidatesSkipped > 0
? { candidates_skipped: census.candidatesSkipped }
: {}),
...(census !== undefined && census.unreadable.length > 0
? { candidates_unreadable: Object.freeze([...census.unreadable]) }
: {}),
...(census?.widening !== undefined ? { widening: census.widening } : {}),
...(census?.wideningDetail !== undefined ? { widening_detail: census.wideningDetail } : {}),
});
}
export type PageLintField = { readonly [PAGE_LINT_KEY]?: PageLintReport };
/** Absent case: nothing to say contributes no key whatsoever. */
const NO_PAGE_LINT: PageLintField = Object.freeze({});
/** Whether a report carries anything a caller needs to see. */
function hasSomethingToSay(report: PageLintReport): boolean {
return (
report.total > 0 ||
report.skipped.length > 0 ||
report.unavailable !== undefined ||
report.candidates_skipped !== undefined ||
report.candidates_unreadable !== undefined ||
report.widening === "index_unavailable"
);
}
/**
* Resolve a report into the object to spread onto a write receipt. A
* clean report - and `null` - yields the empty object, so the key is
* ABSENT rather than present-and-empty. That absence is the whole
* contract: it means the lint ran and found nothing, which is why a lint
* that could NOT run reports `unavailable` instead of returning nothing.
*/
export function pageLintField(report: PageLintReport | null): PageLintField {
if (report === null || !hasSomethingToSay(report)) return NO_PAGE_LINT;
return Object.freeze({ [PAGE_LINT_KEY]: report });
}