Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript
41.0 KB
Raw Blame
/**
 * Per-page write-time lint (evidence-at-the-boundary, task A4).
 *
 * A write used to land and say nothing about what it wrote. The only
 * quality pass over a page was `lintConsolidate`, a vault-wide walk
 * measured at 359 ms on a 2090-page vault, so an agent that wrote a
 * broken wikilink or a document with no frontmatter learned about it
 * from a sweep that may never run. Worse, `assertValidDocument` ran only
 * under `strict` on create: update, append and every batch operation
 * committed with NO document validation at all and returned a hardcoded
 * success flag.
 *
 * This module closes that hole with the detectors that already exist,
 * run against exactly the pages one call committed:
 *
 *   - {@link validateArtifact} for the error-severity codes, so a strict
 *     create and a linted update agree on what a valid document is.
 *   - merged-link, resolved through the canonical-id resolver rather than
 *     through a vault-wide merge map (see `lint-consolidate.ts`).
 *   - broken Brain-artifact wikilink against `collectAllBasenames`, the
 *     readdir-only basename index the doctor already builds (6 ms).
 *   - {@link NEAR_DUPLICATE_CODE}, the authored body compared against the
 *     pages it could plausibly duplicate - the SAME directory AND the SAME
 *     composite scope bucket - through the shared `tokenise`/`jaccard`
 *     primitives (t_d30c0548). A bounded, deterministic candidate set per
 *     write; the finding rides the receipt and never gates the write.
 *   - {@link UNDATED_QUANTITY_CODE}, the authored body scanned for a count
 *     or amount with no structural date on the page (t_fb442a24). The
 *     trigger shapes live in `undated-quantity.ts`; this module owns the
 *     escape hatches - `valid_from` in frontmatter, an ISO date in the
 *     body's claim-bearing prose, or a `[Source: <name>, YYYY-MM-DD]`
 *     marker - and the finding quotes the lexeme and its body offset,
 *     counted after frontmatter removal and named as such, never the
 *     line. Advisory only.
 *
 * `demote-stale-stable` is deliberately EXCLUDED, not overlooked: its
 * trigger is the page's creation age against the staleness cap, so it can
 * never fire on a page written this turn. Reporting it here would be a
 * check that is structurally incapable of finding anything.
 *
 * ## What this envelope does NOT cover, stated rather than implied
 *
 *   - `brain_write_session` keeps its own `errors[]` channel and its own
 *     correction prompt. It validates BEFORE committing and refuses, which
 *     is a different contract from an advisory computed after the fact;
 *     folding it in would give one tool two error channels.
 *   - `write-session/engine.ts` and `distill/distill-source.ts` write with
 *     their own lexical validation and bypass the note-write path
 *     entirely, so nothing here sees their bytes.
 *   - Log appends (`append_log_line`, `apply_evidence`) name no note path
 *     and are not linted: the log line is machine-composed, not authored.
 *   - Body normalization on the write boundary is a recorded REFUSAL
 *     (t_d30c0548), not an omission: every candidate set (whitespace
 *     collapse, smart-quote folding, trailing-space trim) changes authored
 *     bytes that receipts, dedup and byte-stability tests pin. The
 *     near-duplicate score is computed over the body exactly as authored,
 *     and frontmatter is never compared and never touched.
 *
 * ## Shape
 *
 * The payload composes two conventions already in the codebase rather
 * than inventing a third: `WriteSessionError {code, path, message}` plus
 * a `severity` from `DoctorSeverity` plus the `next_command` idiom
 * resolved through `nextCommandField`. One field is added - `page` - for
 * the reason a batch needs it: one report spans several files, and a
 * finding that cannot name its file is unusable. `path` keeps its
 * `WriteSessionError` meaning, the location WITHIN the document.
 *
 * The report always carries `total`, `returned`, `truncated` and
 * `skipped`, so a capped list can never be read as a complete one, and a
 * page too large to validate lands in `skipped` with a reason instead of
 * vanishing. A page the lint THREW on lands there too: the failure is one
 * page's, and reporting it as a whole-report `unavailable` discarded the
 * findings already collected for the pages before it.
 *
 * Every string in the report is composed here from identifiers and
 * integers. A raw errno message would name an absolute path, which is the
 * operator's home directory in a write receipt - the leak
 * `writeFrontmatterAtomic` was rewritten to close in the same release.
 *
 * ## Attachment
 *
 * Modelled field for field on `write-advisory.ts`: computed AROUND the
 * write, never gating it, surfaced as an additive key that is ABSENT
 * ENTIRELY when there is nothing to say. Failure of the lint itself - of
 * the whole call, before any page has been read - degrades to a named
 * {@link PageLintUnavailable} on the key rather than to a missing key: an
 * absent `lint` key must mean clean, and only that.
 */

import { lstatSync, readFileSync, readdirSync, statSync } from "node:fs";
import { join, posix, resolve } from "node:path";

import { vaultRelative } from "../path-safety.ts";
import { ROUTE_STAGE, timeStageSync } from "../route-scope.ts";
import { compositeScopeKey, scopeFromFrontmatter } from "../scope-key.ts";
import { parseFrontmatterText } from "../vault.ts";
import type { FrontmatterMap } from "../types.ts";
import { memoryFromStore, storeMemoryRel } from "../store/read/memory-ids.ts";
import { collectAllBasenames } from "./doctor/records.ts";
import {
  LINT_CONSOLIDATE_KIND,
  createMergedLinkResolver,
  type MergedLinkResolution,
  type MergedLinkResolver,
} from "./lint-consolidate.ts";
import { nextCommandField, type NextCommandField } from "./next-step.ts";
import { pageIdToPath, reportMergeChain } from "./page-meta/page-id.ts";
import { brainDirs } from "./paths.ts";
import { loadSchemaPack } from "./schema-pack.ts";
import type { BrainSchemaVocabulary } from "./schema-vocab.ts";
import {
  NEAR_DUPLICATE_THRESHOLDS,
  findNearDuplicates,
  type NearDuplicatePoolEntry,
  type ReadableRef,
} from "./near-duplicate.ts";
import { VALID_FROM_KEY } from "./lifecycle/temporal-replace.ts";
import { tokenise } from "./similarity.ts";
import { parseCitations } from "./temporal/citations.ts";
import type { DoctorSeverity } from "./types.ts";
import {
  containsIsoDate,
  containsProseIsoDate,
  findUndatedQuantities,
} from "./undated-quantity.ts";
import {
  WIKILINK_TARGET_RE,
  isBrainArtifactId,
  isLiveWikilinkSpan,
  normaliseWikilinkTarget,
} from "./wikilink.ts";
import { ARTIFACT_MAX_BYTES, validateArtifact } from "./write-session/validate.ts";

/** The additive JSON key every write surface carries the report in. */
export const PAGE_LINT_KEY = "lint";

/**
 * Longest finding list a write receipt carries. A write response is read
 * inline by the agent that made the write; past a couple of dozen lines
 * it stops being a fix list and becomes a payload. Everything beyond the
 * cap is still COUNTED - see `total` and `truncated`.
 */
export const PAGE_LINT_MAX_FINDINGS = 25;

/** Code the lint reports itself under when it could not run at all. */
export const PAGE_LINT_UNAVAILABLE_CODE = "page-lint-unavailable";

/** The doctor code an unresolvable Brain wikilink is already reported as. */
const BROKEN_WIKILINK_CODE = "broken-wikilink";

/**
 * Near-duplicate bar (t_d30c0548): the jaccard score at or above which the
 * authored body counts as closely matching an existing page. Deliberately
 * high - near-DUPLICATE claims near-identity, unlike the 0.5 contradiction
 * precedent that answers "same subject". An alias of the kernel's one
 * threshold table, so the write hint cannot drift from it.
 */
export const NEAR_DUPLICATE_JACCARD = NEAR_DUPLICATE_THRESHOLDS.writeHint;

/** Code the near-duplicate detector reports under. */
export const NEAR_DUPLICATE_CODE = "near-duplicate";

/**
 * Code the undated-quantity detector reports under (write-honesty, F4).
 * A count or amount with no structural date on the page answers as true at
 * every instant, because an absent `valid_from` is unbounded. The code is
 * registered in `diagnostics.ts` so the finding carries a next command.
 */
export const UNDATED_QUANTITY_CODE = "undated-quantity";

/**
 * Most sibling pages one directory contributes as near-duplicate
 * candidates: the newest by mtime. Every sibling is stat'ed (cheap), but
 * only these are read and tokenised, so a write into a folder of
 * thousands of notes costs a bounded read. Siblings past the cap are
 * counted on the report as `candidates_skipped`, never dropped silently.
 */
export const NEAR_DUPLICATE_MAX_CANDIDATES = 200;

/** Extension every candidate page on disk carries. */
const MARKDOWN_EXT = ".md";

/**
 * Canonical prefix of a vault-relative spelling that leaves the vault.
 * The write kernel refuses traversal, so such a page can only reach this
 * module from a caller - but the candidate walk must not depend on that:
 * enumerating outside the vault would read the operator's filesystem for
 * a receipt about the vault.
 */
const VAULT_ESCAPE_PREFIX = "../";

/** Absent-directory case for a candidate lookup: an empty list, never a miss. */
const NO_CANDIDATES: ReadonlyArray<NearDuplicateCandidate> = Object.freeze([]);

/** Why a written page was not linted. Never a silent omission. */
export const PAGE_LINT_SKIP_REASON = Object.freeze({
  overByteCap: "page-over-byte-cap",
  unreadable: "page-unreadable",
  /** The page was read and the lint itself threw on it. See {@link lintOnePage}. */
  lintFailed: "page-lint-failed",
} as const);

export type PageLintSkipReason = (typeof PAGE_LINT_SKIP_REASON)[keyof typeof PAGE_LINT_SKIP_REASON];

/** Membership list, in the order a page meets the three gates. */
export const PAGE_LINT_SKIP_REASONS: ReadonlyArray<PageLintSkipReason> = Object.freeze([
  PAGE_LINT_SKIP_REASON.overByteCap,
  PAGE_LINT_SKIP_REASON.unreadable,
  PAGE_LINT_SKIP_REASON.lintFailed,
]);

/**
 * Narrow a string read back across a tool boundary. The reason crosses the
 * MCP wire on all four write tools, where a caller decides from it whether
 * the silence about that page means clean or means unread, so a reader needs
 * to be able to reject a value this build does not understand.
 */
export function isPageLintSkipReason(value: unknown): value is PageLintSkipReason {
  return (
    typeof value === "string" && (PAGE_LINT_SKIP_REASONS as ReadonlyArray<string>).includes(value)
  );
}

/** What a failure is called when it carries neither an errno nor a name. */
const UNNAMED_FAILURE_CODE = "unknown";

/**
 * The identity of a failure, with nothing of the operator's filesystem in it:
 * the errno code when the kernel supplied one, the error's class otherwise.
 *
 * The rejected alternative is `err.message`, which is what shipped. Node
 * renders an errno as `ENOENT: no such file or directory, stat
 * '/home/<user>/<vault>/Brain/x.md'` - an absolute home path, in a key that
 * rides back on every note-write response. The same release rewrote
 * `writeFrontmatterAtomic` (`src/core/vault.ts`) for precisely this reason:
 * identifiers and integers cross this boundary, never a path and never an OS
 * message.
 */
export function failureCode(err: unknown): string {
  const code = (err as NodeJS.ErrnoException | null)?.code;
  if (typeof code === "string" && code.length > 0) return code;
  const name = (err as Error | null)?.name;
  return typeof name === "string" && name.length > 0 ? name : UNNAMED_FAILURE_CODE;
}

/** Rank by severity: errors are what a strict create would have refused. */
const SEVERITY_RANK: Readonly<Record<DoctorSeverity, number>> = Object.freeze({
  error: 0,
  warning: 1,
});

/**
 * One finding about one written page. `code`, `path` and `message` are
 * the `WriteSessionError` triple verbatim; `severity` and `next_command`
 * are the two conventions layered on top; `page` names the file.
 */
export interface PageLintFinding extends NextCommandField {
  readonly severity: DoctorSeverity;
  readonly code: string;
  /** Vault-relative path of the page the finding is about. */
  readonly page: string;
  /**
   * Location within the document: `body`, `frontmatter`, `tags`, or the
   * thing the finding points at - a link target for the link detectors,
   * the resembling page's vault-relative path for {@link NEAR_DUPLICATE_CODE}
   * (evidence in the same spirit the write-conflict advisory renders
   * `[[pref-id]] jaccard=…` into its log events).
   */
  readonly path: string;
  readonly message: string;
}

/** A written page the lint did not read, and why. */
export interface PageLintSkip {
  /** Vault-relative path of the page, never an absolute one. */
  readonly page: string;
  readonly reason: PageLintSkipReason;
  /**
   * The measurement or the errno CODE behind the skip, so the reason is
   * checkable. Identifiers and integers only - see {@link failureCode}.
   */
  readonly detail: string;
}

/** The lint itself failed. Reported, never swallowed into an absent key. */
export interface PageLintUnavailable {
  readonly code: string;
  readonly message: string;
}

export interface PageLintReport {
  /** Ranked, capped at {@link PAGE_LINT_MAX_FINDINGS}. */
  readonly findings: ReadonlyArray<PageLintFinding>;
  /** Findings detected, including those the cap dropped. */
  readonly total: number;
  /** Findings actually carried in {@link findings}. */
  readonly returned: number;
  /** True iff `total > returned`. */
  readonly truncated: boolean;
  /** Written pages that were not linted, each with its reason. */
  readonly skipped: ReadonlyArray<PageLintSkip>;
  /** Present iff the lint could not run; the counters above are then zero. */
  readonly unavailable?: PageLintUnavailable;
  /**
   * Sibling pages the near-duplicate check did not compare against
   * because their directory exceeded {@link NEAR_DUPLICATE_MAX_CANDIDATES}.
   * Present only when the cap applied.
   */
  readonly candidates_skipped?: number;
  /** Sibling pages the near-duplicate check could not read, each named. Present only when any. */
  readonly candidates_unreadable?: ReadonlyArray<PageLintCandidateSkip>;
  /**
   * Whether the near-duplicate check looked beyond the written pages'
   * directories. Present only when widening is on; `used` alone is not
   * something to say, `index_unavailable` is.
   */
  readonly widening?: NearDuplicateWideningStatus;
  /**
   * Why widening reported `index_unavailable`: the search error code, the
   * errno code or the error name ({@link failureCode}), so a missing index,
   * a locked or corrupt one and a defect read differently. Present only
   * with that status.
   */
  readonly widening_detail?: string;
}

/** A sibling page the near-duplicate check could not read, and the errno code why. */
export interface PageLintCandidateSkip {
  readonly page: string;
  readonly detail: string;
}

/**
 * Whether the near-duplicate hint looked beyond the written pages' own
 * directories: `used` when the keyword index answered, `index_unavailable`
 * when it could not be opened or queried. Absent from a report when
 * widening is off.
 */
export type NearDuplicateWideningStatus = "used" | "index_unavailable";

/** How one call to {@link lintWrittenPages} is bounded. */
export interface LintWrittenPagesOptions {
  /**
   * The caller's reach. Required, never defaulted: a sibling page the
   * caller may not read is dropped before it is scored and before the
   * census counts it, and a wikilink target or merge-chain hop the caller
   * may not read answers as absent, so neither a finding nor a census
   * number can disclose it. An operator-reach caller passes `READ_ALL_REFS`.
   */
  readonly readable: ReadableRef;
  /**
   * Candidates from outside the written pages' directories, collected by
   * `page-lint-widening.ts`. Scored under the same scope-bucket and reach
   * rules; a page already in a written page's directory counts once.
   */
  readonly extraCandidates?: ReadonlyArray<NearDuplicateCandidate>;
  /** Whether widening ran; absent when it is off. */
  readonly widening?: NearDuplicateWideningStatus;
  /** Why widening was unavailable; see {@link PageLintReport.widening_detail}. */
  readonly wideningDetail?: string;
}

/** What building the near-duplicate candidate index left out, by count and by name. */
export interface NearDuplicateCensus {
  readonly candidatesSkipped: number;
  readonly unreadable: ReadonlyArray<PageLintCandidateSkip>;
  /** Whether widening ran; absent when it is off. */
  readonly widening?: NearDuplicateWideningStatus;
  /** Why widening was unavailable; see {@link PageLintReport.widening_detail}. */
  readonly wideningDetail?: string;
}

/**
 * Rank findings: errors ahead of warnings (an error is what a strict
 * create would have refused), then page, then code, then location. Named
 * and exported once so a CLI surface and an MCP surface cannot drift into
 * two orderings of one list.
 */
export function comparePageLintFindings(a: PageLintFinding, b: PageLintFinding): number {
  return (
    SEVERITY_RANK[a.severity] - SEVERITY_RANK[b.severity] ||
    a.page.localeCompare(b.page) ||
    a.code.localeCompare(b.code) ||
    a.path.localeCompare(b.path)
  );
}

function finding(
  severity: DoctorSeverity,
  code: string,
  page: string,
  path: string,
  message: string,
): PageLintFinding {
  return Object.freeze({ severity, code, page, path, message, ...nextCommandField(code) });
}

/**
 * The schema type the document declares, or `null`. Mirrors
 * `assertValidDocument`: the validator judges the declared type against
 * the pack. Fed the frontmatter {@link parseFrontmatterText} already
 * produced for the near-duplicate scan, so one page parses exactly once;
 * that parse is total (a document without frontmatter parses to an empty
 * map), so the null case is simply a document that declares nothing.
 */
function declaredSchemaType(meta: FrontmatterMap): string | null {
  const declared = meta["type"];
  return typeof declared === "string" && declared.trim() !== "" ? declared : null;
}

/**
 * Every distinct wikilink target on the page, in first-appearance order.
 * Distinct rather than per-occurrence: a page that names one dead target
 * five times has one thing wrong with it, and five identical rows would
 * spend the finding cap saying so. A bracket behind an odd backslash run
 * is escaped data and opens or closes no link - the same rule
 * `extractWikilinks` applies, so a `\[[literal]]` span raises no
 * broken-link row.
 */
function distinctLinkTargets(raw: string): ReadonlyArray<string> {
  const seen = new Set<string>();
  for (const match of raw.matchAll(WIKILINK_TARGET_RE)) {
    if (!isLiveWikilinkSpan(raw, match.index!, match.index! + match[0].length)) continue;
    const target = normaliseWikilinkTarget(match[1] ?? "");
    if (target !== "") seen.add(target);
  }
  return [...seen];
}

/**
 * One page that can serve the near-duplicate detector as a candidate: its
 * vault-relative path, composite scope bucket, and body tokens, read once
 * per call. The written pages themselves are in here too - a batch can
 * write two near-identical pages into one directory and each should see
 * the other - so self-comparison is excluded at detection time, not here.
 */
export interface NearDuplicateCandidate {
  /** Vault-relative path of the candidate page. */
  readonly page: string;
  /** The page's composite scope bucket, `compositeScopeKey` spelling. */
  readonly scopeKey: string;
  /** The authored body through the shared `tokenise` primitive. */
  readonly tokens: ReadonlySet<string>;
}

/**
 * Everything the lint computes once per call and threads per page.
 *
 * Exported with {@link lintPagesWithContext} so the per-page failure path can
 * be exercised: no filesystem state reaches it - every reader inside
 * {@link lintOnePage} either cannot throw or catches its own errors - and an
 * accumulation that is only correct in theory is the kind this release keeps
 * finding. A caller already holding the indexes can lint with them too.
 */
export interface LintContext {
  readonly basenames: ReadonlySet<string>;
  readonly vocabulary: BrainSchemaVocabulary;
  readonly mergedLinks: MergedLinkResolver;
  /**
   * The near-duplicate candidates, keyed by the vault-relative directory
   * they live in. Built once per call from exactly the directories the
   * written pages landed in - the bounded candidate set (same directory,
   * same scope bucket) needs nothing vault-wide.
   */
  readonly nearDuplicateCandidates: ReadonlyMap<string, ReadonlyArray<NearDuplicateCandidate>>;
  /**
   * Candidates from outside the written pages' directories (widening).
   * Every written page is compared with them as well as with its own
   * directory's candidates.
   */
  readonly extraCandidates?: ReadonlyArray<NearDuplicateCandidate>;
  /**
   * The caller's reach, applied again by the kernel before any score, and
   * by the wikilink and merged-link checks, so a withheld page answers as
   * an absent one on every finding.
   */
  readonly readable: ReadableRef;
  /** What the candidate index left out; absent means nothing was. */
  readonly nearDuplicateCensus?: NearDuplicateCensus;
}

/**
 * The near-duplicate findings for one written page: the authored body
 * against every candidate in the same directory - plus every widened
 * candidate - in the same composite scope bucket, scored through the
 * shared kernel at {@link NEAR_DUPLICATE_JACCARD}. The page itself is
 * never its own candidate - a batch can write two lookalikes, but a
 * single write cannot resemble itself - and the score rides the message
 * so an operator judgement call is informed.
 *
 * A same-directory finding keeps the message shipped since v1.64.0; a
 * widened one appends `method=<method>`, so a reader can tell the hint
 * came from the keyword index. One deliberate change from v1.73.0: the
 * kernel scores nothing below `NEAR_DUPLICATE_MIN_TOKENS` body
 * tokens, on either side, so two short stubs that share their few words
 * by accident (formerly a jaccard=1.000 finding) are no longer reported.
 * The kernel's own match order never reaches the receipt, which is ranked
 * by {@link comparePageLintFindings}. The pool is bounded upstream (the
 * directory cap is counted in `candidates_skipped`, the widening pull by
 * its top-k), so the kernel's own cap is set to the pool size and never
 * drops a candidate unreported.
 */
function nearDuplicateFindings(
  ctx: LintContext,
  page: string,
  meta: FrontmatterMap,
  body: string,
): PageLintFinding[] {
  const local = ctx.nearDuplicateCandidates.get(posix.dirname(page)) ?? NO_CANDIDATES;
  const widened = new Set<string>();
  const pool: NearDuplicatePoolEntry[] = local.map(poolEntry);
  const inPool = new Set(local.map((candidate) => candidate.page));
  for (const candidate of ctx.extraCandidates ?? NO_CANDIDATES) {
    if (inPool.has(candidate.page)) continue;
    inPool.add(candidate.page);
    widened.add(candidate.page);
    pool.push(poolEntry(candidate));
  }
  const { matches } = findNearDuplicates({ ref: page, tokens: tokenise(body) }, pool, {
    threshold: NEAR_DUPLICATE_JACCARD,
    readable: ctx.readable,
    bucket: compositeScopeKey(scopeFromFrontmatter(meta)),
    cap: pool.length,
  });
  return matches.map((match) => {
    const evidence = `body resembles [[${match.ref}]] jaccard=${match.score.toFixed(3)} (threshold ${NEAR_DUPLICATE_JACCARD})`;
    const message = widened.has(match.ref) ? `${evidence} method=${match.method}` : evidence;
    return finding("warning", NEAR_DUPLICATE_CODE, page, match.ref, message);
  });
}

function poolEntry(candidate: NearDuplicateCandidate): NearDuplicatePoolEntry {
  return { ref: candidate.page, tokens: candidate.tokens, bucket: candidate.scopeKey };
}

const NOT_MERGED: MergedLinkResolution = Object.freeze({ canonical: null, unresolvable: null });

/** Whether `id` names a Brain page on disk the caller may not read. */
function isWithheldPageId(vault: string, ctx: LintContext, id: string): boolean {
  const path = pageIdToPath(vault, id);
  return path !== null && !ctx.readable(vaultRelative(path, vault));
}

/**
 * The merge-chain answer for one wikilink target at the caller's reach.
 *
 * The vault-wide resolver reads the `merged_into:` pointer of every page
 * on the chain. A page the caller may not read answers as an absent one
 * would: the walk ends at its id, the way it ends at a page with no file.
 * So a withheld target is not merged at all, and a chain that reaches a
 * withheld page names that page's id as the canonical end and nothing it
 * points at. Only a link the resolver found merged pays the second walk.
 */
function mergedAtReach(vault: string, ctx: LintContext, target: string): MergedLinkResolution {
  const merged = ctx.mergedLinks.resolve(target);
  if (merged.canonical === null && merged.unresolvable === null) return merged;
  const { visited } = reportMergeChain(vault, target);
  const withheld = visited.findIndex((id) => isWithheldPageId(vault, ctx, id));
  if (withheld === -1) return merged;
  if (withheld === 0) return NOT_MERGED;
  return Object.freeze({ canonical: visited[withheld]!, unresolvable: null });
}

/**
 * Whether a Brain artifact named `target` exists where the caller may read
 * it. `basenames` answers the vault-wide question once per call; a hit is
 * asked again per directory `collectAllBasenames` indexes, so an artifact
 * the caller may not read answers as an absent one. Only a linked name
 * that exists pays the per-directory check.
 */
function hasReadableArtifact(vault: string, ctx: LintContext, target: string): boolean {
  if (!ctx.basenames.has(target)) return false;
  const dirs = brainDirs(vault);
  const memoryDirs = [dirs.inbox, dirs.processed, dirs.archived, dirs.preferences, dirs.retired];
  const fromStore = memoryFromStore(vault);
  if (fromStore) {
    // Store-only (task 2.10): a preference or a signal is a fold record,
    // judged by the path its page would have. The fold's answer is final
    // for the memory folders: a stale page the frozen legacy tree still
    // holds there names nothing live.
    const rel = storeMemoryRel(vault, target);
    if (rel !== null && ctx.readable(rel)) return true;
  }
  for (const dir of [dirs.brain, ...(fromStore ? [] : memoryDirs), dirs.log]) {
    const absolute = join(dir, `${target}.md`);
    if (!ctx.readable(vaultRelative(absolute, vault))) continue;
    if (lstatSync(absolute, { throwIfNoEntry: false })?.isFile() === true) return true;
  }
  return false;
}

/**
 * Whether the page supplies a structural date that places its quantities
 * in time. Three hatches, all structural: a `valid_from` boundary in
 * frontmatter, an ISO date or instant in the body's claim-bearing prose,
 * or a `[Source: <name>, YYYY-MM-DD]` marker. The citation marker always
 * carries its own ISO date, so the date hatch already covers a well-formed
 * one; it is checked explicitly through the canonical parser so the
 * contract reads as three hatches rather than a coincidence.
 *
 * The body hatch reads the same masked view the quantity scanner does
 * (`containsProseIsoDate`): a date inside a fenced block, an inline code
 * span, a table row, a URL or a wikilink target is not a claim about the
 * vault, so it must not quiet the undated quantities around it. The scope
 * stays page-wide - one prose date anywhere still dates the page.
 */
function hasDatedQuantityContext(meta: FrontmatterMap, body: string): boolean {
  const declared = meta[VALID_FROM_KEY];
  // The boundary hatch is structural like the other two: the value must
  // NAME a date. Any non-empty string ("TBD", "unknown", "soon") places
  // nothing in time, and accepting it let a placeholder quiet the lint.
  if (typeof declared === "string" && containsIsoDate(declared)) return true;
  if (containsProseIsoDate(body)) return true;
  return parseCitations(body).markers.length > 0;
}

/**
 * The undated-quantity findings for one written page: one warning per
 * lexeme the body states with no date on the page. The lexeme and its
 * offset are the whole evidence - the authored line is not quoted, so a
 * receipt cannot carry prose the caller did not ask it to repeat. The
 * offset is counted within the post-frontmatter body the scanner sees
 * (neither a file offset nor an index into the raw write), and the
 * message names that coordinate system so a reader can locate the
 * lexeme honestly.
 */
function undatedQuantityFindings(
  page: string,
  meta: FrontmatterMap,
  body: string,
): PageLintFinding[] {
  if (hasDatedQuantityContext(meta, body)) return [];
  return findUndatedQuantities(body).map(({ lexeme, offset }) =>
    finding(
      "warning",
      UNDATED_QUANTITY_CODE,
      page,
      "body",
      `quantity ${JSON.stringify(lexeme)} at body offset ${offset} (after frontmatter) carries no date`,
    ),
  );
}

function lintOnePage(
  vault: string,
  ctx: LintContext,
  page: string,
  raw: string,
): PageLintFinding[] {
  const out: PageLintFinding[] = [];
  const [meta, body] = parseFrontmatterText(raw);
  const schemaType = declaredSchemaType(meta);
  for (const violation of validateArtifact(raw, { schemaType, vocabulary: ctx.vocabulary })) {
    out.push(finding("error", violation.code, page, violation.path, violation.message));
  }
  for (const target of distinctLinkTargets(raw)) {
    // A link outside the Brain id space is user prose or a cross-layer
    // note, and is nobody's here to flag - the same boundary the doctor
    // and the orphaned-reference fixer draw.
    if (!isBrainArtifactId(target)) continue;
    const merged = mergedAtReach(vault, ctx, target);
    if (merged.unresolvable !== null) {
      out.push(
        finding(
          "warning",
          LINT_CONSOLIDATE_KIND.mergedLink,
          page,
          target,
          `wikilink [[${target}]] has a merge chain with no canonical end: ${merged.unresolvable}`,
        ),
      );
      continue;
    }
    if (merged.canonical !== null) {
      out.push(
        finding(
          "warning",
          LINT_CONSOLIDATE_KIND.mergedLink,
          page,
          target,
          `wikilink [[${target}]] points at a page merged into ${merged.canonical}`,
        ),
      );
      continue;
    }
    if (!hasReadableArtifact(vault, ctx, target)) {
      out.push(
        finding(
          "warning",
          BROKEN_WIKILINK_CODE,
          page,
          target,
          `wikilink [[${target}]] resolves to no Brain artifact`,
        ),
      );
    }
  }
  out.push(...undatedQuantityFindings(page, meta, body));
  out.push(
    ...timeStageSync(ROUTE_STAGE.nearDuplicateLookup, () =>
      nearDuplicateFindings(ctx, page, meta, body),
    ),
  );
  return out;
}

function skip(page: string, reason: PageLintSkipReason, detail: string): PageLintSkip {
  return Object.freeze({ page, reason, detail });
}

/** The empty report: nothing detected, nothing skipped, nothing to say. */
function emptyReport(unavailable?: PageLintUnavailable): PageLintReport {
  return Object.freeze({
    findings: Object.freeze([]),
    total: 0,
    returned: 0,
    truncated: false,
    skipped: Object.freeze([]),
    ...(unavailable !== undefined ? { unavailable } : {}),
  });
}

/**
 * The vault-relative spelling of a page, whatever spelling the caller
 * handed in. The ONE derivation for every string that names a page on
 * this report - findings, skips, and near-duplicate candidates alike -
 * so the map keys and the page strings compared against them can never
 * drift into two normalizations.
 */
function canonicalPage(vault: string, page: string): string {
  return vaultRelative(resolve(vault, page), vault);
}

/**
 * The near-duplicate candidate index for one call: every Markdown page in
 * exactly the directories the written pages landed in, keyed by that
 * directory, each carrying its composite scope bucket and body tokens.
 *
 * Only the newest {@link NEAR_DUPLICATE_MAX_CANDIDATES} siblings per
 * directory are read; the rest are counted in the census. A candidate
 * that cannot be read is not a candidate - it cannot provide evidence -
 * but it is NAMED in the census with its errno code, never skipped
 * silently. A candidate over the artifact byte cap is
 * skipped for the same cost reason a written page over the cap is skipped
 * rather than validated. The directory enumerations themselves are NOT
 * guarded: the write just committed into them, so a failure to list one
 * is a failure of the lint's inputs and belongs in the report's
 * `unavailable` slot like every other context-building failure. A page
 * spelled outside the vault ({@link VAULT_ESCAPE_PREFIX}) collects no
 * candidates at all: the walk stops at the vault boundary.
 *
 * A sibling outside the caller's reach (`readable`) is dropped at the
 * listing, before it is stat'ed: it is never scored, never named as
 * unreadable and never counted against the cap.
 */
function collectNearDuplicateCandidates(
  vault: string,
  pages: ReadonlyArray<string>,
  readable: ReadableRef,
): {
  readonly index: ReadonlyMap<string, ReadonlyArray<NearDuplicateCandidate>>;
  readonly census: NearDuplicateCensus;
} {
  const directories = new Set<string>();
  for (const named of pages) {
    const canonical = canonicalPage(vault, named);
    if (canonical.startsWith(VAULT_ESCAPE_PREFIX)) continue;
    directories.add(posix.dirname(canonical));
  }
  const index = new Map<string, NearDuplicateCandidate[]>();
  const unreadable: PageLintCandidateSkip[] = [];
  let candidatesSkipped = 0;
  for (const directory of directories) {
    // Stat every sibling (cheap), then read only the newest few.
    const siblings: Array<{ page: string; absolute: string; size: number; mtimeMs: number }> = [];
    for (const name of readdirSync(resolve(vault, directory))) {
      if (!name.endsWith(MARKDOWN_EXT)) continue;
      const absolute = resolve(vault, directory, name);
      const page = canonicalPage(vault, absolute);
      if (!readable(page)) continue;
      try {
        const stat = statSync(absolute);
        siblings.push({ page, absolute, size: stat.size, mtimeMs: stat.mtimeMs });
      } catch (err) {
        unreadable.push({ page, detail: failureCode(err) });
      }
    }
    const newest = siblings.toSorted((a, b) => b.mtimeMs - a.mtimeMs);
    candidatesSkipped += Math.max(0, newest.length - NEAR_DUPLICATE_MAX_CANDIDATES);
    const candidates: NearDuplicateCandidate[] = [];
    for (const sibling of newest.slice(0, NEAR_DUPLICATE_MAX_CANDIDATES)) {
      // Size from the inode: reading an over-cap page to find that out is
      // the cost the cap exists to avoid.
      if (sibling.size > ARTIFACT_MAX_BYTES) continue;
      try {
        const [meta, body] = parseFrontmatterText(readFileSync(sibling.absolute, "utf8"));
        candidates.push({
          page: sibling.page,
          scopeKey: compositeScopeKey(scopeFromFrontmatter(meta)),
          tokens: tokenise(body),
        });
      } catch (err) {
        unreadable.push({ page: sibling.page, detail: failureCode(err) });
      }
    }
    index.set(directory, candidates);
  }
  return { index, census: { candidatesSkipped, unreadable } };
}

/**
 * Lint exactly the pages a write committed, reading what is on disk.
 *
 * Runs AFTER the commit on purpose: the batch kernel wrote the bytes and
 * the handler never composed them, so the only honest subject is the
 * file. It never gates the write and it NEVER throws: a failure that
 * leaves nothing to lint AGAINST is reported as
 * {@link PageLintReport.unavailable}, and a failure on one page is a
 * {@link PAGE_LINT_SKIP_REASON.lintFailed} entry beside the results of the
 * pages that did lint.
 *
 * Cost discipline: the basename index and the schema pack are computed
 * ONCE per call and threaded, the merge resolver memoises per call, and
 * nothing here walks vault content beyond the written pages' own
 * directories - the near-duplicate candidate index reads each of those
 * once per call, sized against the same artifact cap as the pages.
 *
 * Every finding answers at the caller's reach (`opts.readable`): a
 * near-duplicate candidate, a wikilink target and a merge-chain hop the
 * caller may not read all read as absent. The written pages themselves
 * are always linted.
 */
export function lintWrittenPages(
  vault: string,
  pages: ReadonlyArray<string>,
  opts: LintWrittenPagesOptions,
): PageLintReport {
  if (pages.length === 0) return emptyReport();
  let ctx: LintContext;
  try {
    // The candidate walk and the per-page scoring are the near-duplicate
    // lookup; under an open route scope both add to that one stage.
    const nearDuplicates = timeStageSync(ROUTE_STAGE.nearDuplicateLookup, () =>
      collectNearDuplicateCandidates(vault, pages, opts.readable),
    );
    ctx = {
      basenames: collectAllBasenames(vault),
      vocabulary: loadSchemaPack(vault).vocabulary,
      mergedLinks: createMergedLinkResolver(vault),
      nearDuplicateCandidates: nearDuplicates.index,
      ...(opts.extraCandidates !== undefined ? { extraCandidates: opts.extraCandidates } : {}),
      readable: opts.readable,
      nearDuplicateCensus: {
        ...nearDuplicates.census,
        ...(opts.widening !== undefined ? { widening: opts.widening } : {}),
        ...(opts.wideningDetail !== undefined ? { wideningDetail: opts.wideningDetail } : {}),
      },
    };
  } catch (err) {
    return emptyReport({
      code: PAGE_LINT_UNAVAILABLE_CODE,
      message: `write-time page lint could not start: ${failureCode(err)}`,
    });
  }
  return lintPagesWithContext(vault, ctx, pages);
}

/**
 * Lint pages against indexes that are already built. The body of
 * {@link lintWrittenPages}, split at the one seam that matters: everything
 * above it fails the whole report (there are no indexes to lint against),
 * everything below it fails at most one page.
 */
export function lintPagesWithContext(
  vault: string,
  ctx: LintContext,
  pages: ReadonlyArray<string>,
): PageLintReport {
  const detected: PageLintFinding[] = [];
  const skipped: PageLintSkip[] = [];
  for (const named of pages) {
    // Rendered through the same derivation every other reporting surface
    // uses, so the page a finding names is vault-relative whichever
    // spelling the caller handed in - an absolute path here would be the
    // operator's home directory in a write receipt.
    const page = canonicalPage(vault, named);
    const absolute = resolve(vault, page);
    let raw: string;
    try {
      // Size first, from the inode: an over-cap page must be REPORTED as
      // skipped, and reading it to find that out is the cost the cap
      // exists to avoid.
      const bytes = statSync(absolute).size;
      if (bytes > ARTIFACT_MAX_BYTES) {
        skipped.push(
          skip(
            page,
            PAGE_LINT_SKIP_REASON.overByteCap,
            `${bytes} bytes exceeds the ${ARTIFACT_MAX_BYTES}-byte artifact cap`,
          ),
        );
        continue;
      }
      raw = readFileSync(absolute, "utf8");
    } catch (err) {
      skipped.push(skip(page, PAGE_LINT_SKIP_REASON.unreadable, failureCode(err)));
      continue;
    }
    try {
      detected.push(...lintOnePage(vault, ctx, page, raw));
    } catch (err) {
      // One page's failure is one page's news. Reporting it as `unavailable`
      // threw away the findings already collected for EARLIER pages and the
      // skip list with them, and left the pages after it silently unlinted -
      // a report that reads clean about work that was never done. The report
      // already has an honest shape for "this page was not linted".
      skipped.push(skip(page, PAGE_LINT_SKIP_REASON.lintFailed, failureCode(err)));
    }
  }

  const ranked = detected.toSorted(comparePageLintFindings);
  const returned = ranked.slice(0, PAGE_LINT_MAX_FINDINGS);
  const census = ctx.nearDuplicateCensus;
  return Object.freeze({
    findings: Object.freeze(returned),
    total: ranked.length,
    returned: returned.length,
    truncated: ranked.length > returned.length,
    skipped: Object.freeze(skipped),
    ...(census !== undefined && census.candidatesSkipped > 0
      ? { candidates_skipped: census.candidatesSkipped }
      : {}),
    ...(census !== undefined && census.unreadable.length > 0
      ? { candidates_unreadable: Object.freeze([...census.unreadable]) }
      : {}),
    ...(census?.widening !== undefined ? { widening: census.widening } : {}),
    ...(census?.wideningDetail !== undefined ? { widening_detail: census.wideningDetail } : {}),
  });
}

export type PageLintField = { readonly [PAGE_LINT_KEY]?: PageLintReport };

/** Absent case: nothing to say contributes no key whatsoever. */
const NO_PAGE_LINT: PageLintField = Object.freeze({});

/** Whether a report carries anything a caller needs to see. */
function hasSomethingToSay(report: PageLintReport): boolean {
  return (
    report.total > 0 ||
    report.skipped.length > 0 ||
    report.unavailable !== undefined ||
    report.candidates_skipped !== undefined ||
    report.candidates_unreadable !== undefined ||
    report.widening === "index_unavailable"
  );
}

/**
 * Resolve a report into the object to spread onto a write receipt. A
 * clean report - and `null` - yields the empty object, so the key is
 * ABSENT rather than present-and-empty. That absence is the whole
 * contract: it means the lint ran and found nothing, which is why a lint
 * that could NOT run reports `unavailable` instead of returning nothing.
 */
export function pageLintField(report: PageLintReport | null): PageLintField {
  if (report === null || !hasSomethingToSay(report)) return NO_PAGE_LINT;
  return Object.freeze({ [PAGE_LINT_KEY]: report });
}