Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript
22.7 KB
Raw Blame
/**
 * Where a source came from, decided once for both intake callers.
 *
 * The source-ingest pipeline used to stamp one hardcoded provenance level on
 * every source it saw, which made a scraped URL and a file in the operator's
 * own vault indistinguishable at the moment they entered the brain. There was
 * no untrusted signal at intake to join to, so this module originates one.
 *
 * The decision is STRUCTURAL - it reads the shape of the source identity and
 * nothing else. No word list, no host list, no scheme list: a URI scheme is
 * recognised by its grammar (RFC 3986 §3.1, via {@link hasUriScheme}) and
 * containment is decided by the same vault-boundary check every writer in
 * this repository already funnels through.
 *
 * POLARITY. The first version of this module asked "is this identity provably
 * outside the vault?" and trusted everything else, which handed trust to every
 * identity it simply could not parse: `evil.com/article` and
 * `www.example.com/x` are not absolute, carry no scheme and do not climb out
 * of the vault, so they entered as if the operator had written them, and an
 * agent reading a hostile page could be told by that page to drop the scheme.
 * The question asked here is the other one: is this identity SHAPED like a
 * location inside this vault? Only a yes is trusted. A shape that establishes
 * nothing is not a shape that establishes authority.
 *
 * WHAT THE SHAPE CANNOT SETTLE. `evil.com` and `readme.md` are the same
 * structure - dot-separated labels, no separator - because a hostname and a
 * vault-relative filename are drawn from the same character set. Nothing short
 * of asking the filesystem tells them apart, and asking it would make a note
 * the operator has not written yet untrusted, which confuses absence with
 * authority. So a single-segment identity is resolved by the one structural
 * signal available: it is trusted when it carries this vault's note
 * extension, and untrusted otherwise. `evil.com` therefore lands untrusted,
 * `readme.md` trusted, and the residual ambiguity (`.md` is also a country
 * TLD) costs nothing an attacker did not already have - an identity is a
 * claim, and an attacker willing to write `evil.md` was equally free to write
 * `primer.md`. The shape gate stops the identities that could not be ours; it
 * was never able to stop a lie about one that could.
 *
 * A COLON IS NOT A SCHEME. {@link hasUriScheme} is shared with the markdown
 * link reader, where its input is a link target and a leading `word:` can only
 * be a scheme. A source identity is a FILENAME, a different domain: this
 * project's own vault names notes `Meeting: Q3 planning.md`. Reused unguarded
 * the predicate read that colon as a scheme and quarantined the operator's own
 * note - and quarantine is one-way, so the entities of that intake left every
 * ordinary read with nothing but a success response to show for it. The fix is
 * the reuse, not the predicate: it is applied here only where a scheme can
 * legally appear (before the first separator) and only to a segment that could
 * be a URI at all (a URI admits no whitespace, RFC 3986 §2).
 *
 * THE SHAPE IS NECESSARY, NOT SUFFICIENT (GitHub #160). The shape gate above
 * is the first half of the decision; {@link classifySourceOrigin} then asks
 * the filesystem whether the named file is there. See its docblock for why
 * that question is this module's to ask, and for the limit of the answer.
 */

import { isAbsolute, join, relative, sep } from "node:path";
import {
  closeSync,
  constants as fsConstants,
  fstatSync,
  openSync,
  readSync,
  realpathSync,
  statSync,
} from "node:fs";

import {
  canonicalNotePath,
  ensureInsideVault,
  hasUriScheme,
  realpathInsideVault,
} from "../../path-safety.ts";
import { hashBytes } from "../ingest/content-manifest.ts";
import { INTAKE_TRUST, type IntakeTrust } from "../trust/untrusted-provenance.ts";

/** `[[Articles/x.md]]` - the wikilink form the NER tool's `source` arrives in. */
const WIKILINK_WRAPPER_RE = /^\[\[(.*)\]\]$/s;

/** `[[note|Alias]]` - the display text, not part of the target. */
const WIKILINK_ALIAS_SEPARATOR = "|";

/** `[[note#Section]]` - a position within the target, not a different target. */
const WIKILINK_ANCHOR_SEPARATOR = "#";

/**
 * `stat` errnos that answer "there is nothing here", which is a trust verdict.
 * ENOTDIR is the same answer through a different route: a path segment that
 * exists but is a file, so what follows it cannot exist either.
 */
const ERRNO_NO_SUCH_ENTRY = "ENOENT";
const ERRNO_NOT_A_DIRECTORY = "ENOTDIR";
/** A symlink met where `O_NOFOLLOW` forbids one: not this vault's file. */
const ERRNO_SYMLINK_LOOP = "ELOOP";

/**
 * How the one read opens a source: never through a final-component symlink,
 * never blocking (a FIFO swapped in after the stat must not hang the read).
 * Both flags are POSIX; where the platform lacks one it is simply absent.
 */
const SOURCE_OPEN_FLAGS =
  fsConstants.O_RDONLY | (fsConstants.O_NOFOLLOW ?? 0) | (fsConstants.O_NONBLOCK ?? 0);

/** What a failure is called when it carries neither an errno nor a name. */
const UNNAMED_ERRNO = "unknown";

/**
 * Largest source this classifier will read in order to record its digest.
 *
 * The hash is an audit record of the bytes an extraction claims to come from,
 * and the material an agent can have read is bounded by the context it read it
 * into: a million-token window is a few megabytes of text, so 8 MiB is above
 * anything an extraction could honestly cite while still refusing to pull a
 * dataset or a video into memory. The rejected alternative was no ceiling at
 * all, which is what shipped: the classifier read the whole file before it had
 * decided anything the caller could act on.
 */
export const SOURCE_HASH_MAX_BYTES = 8_388_608;

/**
 * The filesystem would not answer for a source this vault does own.
 *
 * A distinct class because the two things it must NOT be are both close by: it
 * is not an {@link INTAKE_TRUST.untrusted} verdict (that would quarantine the
 * operator's own note over a chmod, one-way, while reporting success), and it
 * is not the caller's malformed payload. The message carries the vault-RELATIVE
 * identity and the errno code and nothing else. Rethrowing Node's own error put
 * `EACCES: permission denied, statx '/home/<user>/<vault>/x.md'` into an MCP
 * error on a path the CALLER supplied, which is an existence-and-permission
 * oracle over the operator's filesystem; the same release rewrote
 * `writeFrontmatterAtomic` to keep absolute paths out of exactly this channel.
 */
export class SourceTrustError extends Error {
  constructor(message: string) {
    super(message);
    this.name = "SourceTrustError";
  }
}

/** A source identity, and the bytes it was found to stand for. */
export interface SourceOrigin {
  /** The lane this source commits in. */
  readonly trust: IntakeTrust;
  /**
   * SHA-256 of the source file's bytes, present only when the verdict is
   * trusted - an untrusted identity has no file of ours to hash, and a hash
   * of nothing would be a claim rather than a record.
   */
  readonly contentHash?: string;
}

/**
 * A source origin together with the bytes its digest was computed over.
 * `bytes` is present exactly when `contentHash` is: both come from one read,
 * so a check that runs on these bytes runs on the bytes the page records.
 */
export interface SourceOriginWithBytes extends SourceOrigin {
  readonly bytes?: Uint8Array;
}

/**
 * The origin of a source with no local bytes: the untrusted lane, no digest.
 * Also the answer for a vault file the caller may not read at its reach, so
 * a hidden page and an absent one are indistinguishable.
 */
export const UNTRUSTED_ORIGIN: SourceOriginWithBytes = Object.freeze({
  trust: INTAKE_TRUST.untrusted,
});

/** Everything before the first occurrence of `separator`, or the whole string. */
function cutAt(value: string, separator: string): string {
  const at = value.indexOf(separator);
  return at === -1 ? value : value.slice(0, at);
}

/** The separator {@link canonicalNotePath} normalises every identity to. */
const PATH_SEPARATOR = "/";

/** The no-op path segment, dropped before the shape is read. */
const SAME_DIR_SEGMENT = ".";

/** The climbing segment, left in place for the vault-boundary check to judge. */
const PARENT_DIR_SEGMENT = "..";

/**
 * The extension a note in this vault carries. Used for ONE decision - telling
 * a single-segment filename apart from a bare hostname - because that is the
 * only place where the path shape alone cannot decide.
 */
const VAULT_NOTE_EXTENSION = ".md";

/** Any whitespace character; a URI contains none (RFC 3986 §2). */
const WHITESPACE_RE = /\s/;

/**
 * Could this segment be a URI prefix? Whitespace rules it out before the
 * scheme grammar is consulted, so a filename that merely contains a colon
 * (`Meeting: Q3 planning.md`) is never mistaken for `mailto:`.
 */
function bearsUriScheme(segment: string): boolean {
  return !WHITESPACE_RE.test(segment) && hasUriScheme(segment);
}

/**
 * Is this leading segment shaped like an authority rather than a directory?
 * A dot in the FIRST segment is the residue of a scheme-less address
 * (`evil.com/article`); vault directories are named, not dotted. `..` is
 * excluded because it is a path operator, and the vault-boundary check below
 * is the one entitled to answer for it.
 */
function isAuthorityShaped(segment: string): boolean {
  return segment !== PARENT_DIR_SEGMENT && segment.includes(".");
}

/** Does this segment name a note file in this vault's own extension? */
function hasVaultNoteExtension(segment: string): boolean {
  return segment.toLowerCase().endsWith(VAULT_NOTE_EXTENSION);
}

/**
 * The one canonical form of a source identity, for every caller that needs to
 * decide whether two spellings name the same source.
 *
 * The wikilink wrapper, the alias pipe and the anchor fragment are all
 * DECORATION on one target: `[[Articles/x.md]]`, `[[Articles/x.md|Primer]]`
 * and `[[Articles/x.md#Intro]]` are the same file, and Obsidian writes all
 * three. Handling only the wrapper - the state this repository was in - left
 * `note|Alias` as a single dotless segment, which the shape gate reads as a
 * bare hostname and quarantines: a legitimate operator note lost to a link
 * style, and quarantine is one-way.
 *
 * The alias and anchor are stripped ONLY inside the wrapper, because `|` and
 * `#` mean alias and anchor only in wikilink grammar. Outside it they are
 * ordinary characters of the identity - a URL fragment is part of the address
 * the agent read - and cutting them there would silently rewrite the caller's
 * source into a different one.
 */
export function normalizeSourceIdentity(source: string): string {
  const trimmed = source.trim();
  const inner = WIKILINK_WRAPPER_RE.exec(trimmed);
  if (inner === null) return canonicalNotePath(trimmed);
  const target = cutAt(cutAt(inner[1] ?? "", WIKILINK_ALIAS_SEPARATOR), WIKILINK_ANCHOR_SEPARATOR);
  return canonicalNotePath(target.trim());
}

/**
 * Is this identity SHAPED like a location inside this vault? The first half
 * of the decision, and the half that has not changed: it rules out every
 * identity that could not be ours. Returns the vault-absolute path when the
 * shape holds, `null` when it does not.
 */
function resolveVaultShapedPath(vault: string, canonical: string): string | null {
  if (canonical.length === 0) return null;
  if (isAbsolute(canonical)) return null;

  const segments = canonical.split(PATH_SEPARATOR).filter((s) => s !== SAME_DIR_SEGMENT);
  // An empty segment is a doubled separator: the `//host` authority marker,
  // the remains of `scheme://host`, or a trailing slash naming a directory.
  // None of the three is a note identity this vault owns.
  if (segments.some((segment) => segment.length === 0)) return null;

  // A scheme precedes the path, so only the segments before the last one can
  // carry it. The final segment is a basename and its colons are its own.
  if (segments.slice(0, -1).some(bearsUriScheme)) return null;

  const head = segments[0] ?? "";
  const establishedShape =
    segments.length > 1 ? !isAuthorityShaped(head) : hasVaultNoteExtension(head);
  if (!establishedShape) return null;

  try {
    return ensureInsideVault(join(vault, canonical), vault);
  } catch {
    // The identity climbs out of the vault (or escapes it through a
    // symlink). It names a location this vault does not own, which is the
    // same verdict as naming one on another host.
    return null;
  }
}

/**
 * The vault-relative identity of a source SHAPED like a location inside this
 * vault, in POSIX form with every `.` and `..` segment resolved, or `null`
 * when the shape does not hold. Touches no file: it is the identity a reach
 * predicate is asked about BEFORE anything is stat-ed or read, so a refusal
 * of the stat or the size ceiling never names a page the caller may not read.
 */
export function vaultShapedIdentity(vault: string, sourcePath: string): string | null {
  const abs = resolveVaultShapedPath(vault, normalizeSourceIdentity(sourcePath));
  return abs === null ? null : relative(vault, abs).split(sep).join(PATH_SEPARATOR);
}

/**
 * Does `readable` refuse this source? Asked on {@link vaultShapedIdentity},
 * before any filesystem question about the source; a source with no
 * vault-shaped identity is never asked about, and without a predicate
 * nothing is refused.
 */
export function isSourceHidden(
  vault: string,
  sourcePath: string,
  readable: ((rel: string) => boolean) | undefined,
): boolean {
  if (readable === undefined) return false;
  const rel = vaultShapedIdentity(vault, sourcePath);
  return rel !== null && !readable(rel);
}

/** Is this the filesystem saying "nothing is there", rather than refusing? */
function isAbsenceErrno(cause: unknown): boolean {
  const code = (cause as NodeJS.ErrnoException | null)?.code;
  return code === ERRNO_NO_SUCH_ENTRY || code === ERRNO_NOT_A_DIRECTORY;
}

/**
 * The errno code alone, or the error's class when the kernel named none.
 * Exported for sibling refusals that must name a host fs failure without
 * its message: a Node fs error's `message` embeds the absolute path the
 * call failed on, which never crosses a wire surface.
 */
export function errnoCode(cause: unknown): string {
  const code = (cause as NodeJS.ErrnoException | null)?.code;
  if (typeof code === "string" && code.length > 0) return code;
  const name = (cause as Error | null)?.name;
  return typeof name === "string" && name.length > 0 ? name : UNNAMED_ERRNO;
}

/** The refusal above, composed from identifiers only. */
function refusal(identity: string, cause: unknown): SourceTrustError {
  return new SourceTrustError(
    `source ${identity} is inside this vault but could not be read (${errnoCode(cause)})`,
  );
}

/** A source identity that resolved to a file this vault holds. */
interface VaultSourceFile {
  /** The canonical, vault-relative identity - safe to name in a message. */
  readonly identity: string;
  readonly abs: string;
  /** Size from the same `stat` the existence question was answered with. */
  readonly size: number;
}

/**
 * The file a source identity stands for, or `null` when there is nothing
 * there. Shape gate first, then the one filesystem question both callers
 * need, so the lane and the digest are decided from a single `stat`.
 */
function resolveVaultSourceFile(vault: string, sourcePath: string): VaultSourceFile | null {
  const identity = normalizeSourceIdentity(sourcePath);
  const abs = resolveVaultShapedPath(vault, identity);
  if (abs === null) return null;

  let stat: ReturnType<typeof statSync>;
  try {
    stat = statSync(abs);
  } catch (cause) {
    if (isAbsenceErrno(cause)) return null;
    throw refusal(identity, cause);
  }
  // A directory (or a socket, or a device) is not material an extraction can
  // have been read from, even though it is genuinely inside the vault.
  if (!stat.isFile()) return null;
  return { identity, abs, size: stat.size };
}

/**
 * The lane a source commits in, without reading its bytes.
 *
 * Split out for the caller that cites SEVERAL sources: it will discard every
 * digest (see `resolveIntakeOrigin`), so hashing to reach a lane reads whole
 * files for nothing. The lane and the digest ask the same question of the
 * filesystem; only the digest needs the file's contents.
 */
export function classifySourceTrust(vault: string, sourcePath: string): IntakeTrust {
  return resolveVaultSourceFile(vault, sourcePath) === null
    ? INTAKE_TRUST.untrusted
    : INTAKE_TRUST.trusted;
}

/**
 * Classify where a source came from: the lane it commits in, and - when it is
 * ours - the bytes it stands for.
 *
 * Trusted requires two things now. The identity must be SHAPED like a
 * location inside this vault, which is the only authority this system has;
 * and that location must hold a readable file.
 *
 * WHY EXISTENCE IS PART OF THE QUESTION. The docstring this replaces argued
 * the opposite - that whether the file is there says nothing about who was
 * entitled to name the location, and that "a source the operator has not
 * written yet is still inside their own namespace". Answered on its own
 * terms: this classifier does not serve a namespace, it serves an assertion.
 * `brain_intake_entities` asserts that THESE ENTITIES WERE EXTRACTED FROM
 * THIS MATERIAL, and a path with no bytes behind it cannot have produced an
 * extraction. Absence here is not "not yet written", it is "there was nothing
 * to read". Without the check the caller supplying the assertion is the same
 * agent that read the material, so hostile material only had to tell it which
 * plausible path to name (GitHub #160).
 *
 * WHAT THIS DOES NOT BUY, stated plainly because the release note repeats it:
 * an attacker forced to name a real file names `README.md`. The recorded hash
 * is then the digest of the bytes that were CLAIMED, not of the bytes that
 * produced the entities. This removes the free bypass - a plausible-looking
 * string no longer suffices, the caller must name something the operator
 * actually holds - and it makes the claim auditable afterwards. It does not
 * stop the lie, and nothing available on this side of the boundary can: the
 * project already established there is no unforgeable caller identity in the
 * MCP surface (`src/core/write-binding/index.ts`).
 *
 * AN UNREADABLE FILE IS NOT A VERDICT. Only "nothing is there" answers the
 * trust question; every other errno - a permission denial, an I/O failure -
 * is the filesystem refusing to answer, and folding it into `untrusted` would
 * quarantine the operator's own note over a chmod, one-way, while reporting
 * success. Those propagate - as a {@link SourceTrustError} naming the
 * vault-relative identity and the errno code, never the kernel's sentence,
 * because this refusal reaches an MCP caller who chose the path.
 *
 * An EMPTY identity is untrusted here rather than an error, because this
 * function answers about an identity it was given. "The caller named no
 * source at all" is a different failure with a different remedy - the caller
 * can be asked - and it belongs to the boundaries that can still ask.
 */
export function classifySourceOrigin(vault: string, sourcePath: string): SourceOrigin {
  const { trust, contentHash } = readSourceOrigin(vault, sourcePath);
  return contentHash === undefined ? UNTRUSTED_ORIGIN : { trust, contentHash };
}

/**
 * {@link classifySourceOrigin}, keeping the bytes the digest was computed
 * over. Same verdicts, same refusals ({@link SourceTrustError}, the
 * {@link SOURCE_HASH_MAX_BYTES} ceiling), and ONE read of the file: the
 * digest and the returned bytes cannot describe two different versions of
 * the source, which a hash followed by a second read for the caller could.
 */
export function readSourceOrigin(vault: string, sourcePath: string): SourceOriginWithBytes {
  const file = resolveVaultSourceFile(vault, sourcePath);
  if (file === null) return UNTRUSTED_ORIGIN;

  if (file.size > SOURCE_HASH_MAX_BYTES) {
    // Refused rather than committed trusted with no digest: absence of a hash
    // already MEANS "no single set of bytes was recorded" (the several-sources
    // case), so reusing it here would make an unrecordable source look like a
    // multi-source intake and quietly drop the audit record this classifier
    // exists to produce.
    throw ceilingRefusal(file.identity, file.size);
  }

  let bytes: Uint8Array | null;
  try {
    bytes = readBoundedSource(vault, file);
  } catch (cause) {
    if (cause instanceof SourceTrustError) throw cause;
    // The file went away (or was swapped for a symlink) between the stat and
    // the read. That is the same answer the stat itself would have given a
    // moment later - there is nothing of ours there - and a race is not a
    // reason to fail an intake that a retry would classify cleanly.
    if (isAbsenceErrno(cause) || errnoCode(cause) === ERRNO_SYMLINK_LOOP) return UNTRUSTED_ORIGIN;
    throw refusal(file.identity, cause);
  }
  if (bytes === null) return UNTRUSTED_ORIGIN;
  // The one digest function in this repository, so the digest a summary page
  // records and the digest an entity page records cannot drift apart.
  return { trust: INTAKE_TRUST.trusted, contentHash: hashBytes(bytes), bytes };
}

/** The size refusal, composed from the identity and integers only. */
function ceilingRefusal(identity: string, size: number): SourceTrustError {
  return new SourceTrustError(
    `source ${identity} is ${size} bytes, past the ${SOURCE_HASH_MAX_BYTES}-byte ` +
      "ceiling on a source this classifier will read to record its digest",
  );
}

/**
 * Read a resolved source through ONE descriptor, so the checks that admitted
 * it hold for the bytes read: the path is resolved to its real location and
 * must still be inside the vault, the open refuses a symlink in the final
 * component, the descriptor must be a regular file, and no more than the
 * ceiling is read however large the file has grown since the stat. `null`
 * when what is there now is not a regular file of this vault.
 */
function readBoundedSource(vault: string, file: VaultSourceFile): Uint8Array | null {
  const real = realpathSync(file.abs);
  if (!realpathInsideVault(real, vault)) return null;
  const fd = openSync(real, SOURCE_OPEN_FLAGS);
  try {
    const stat = fstatSync(fd);
    if (!stat.isFile()) return null;
    if (stat.size > SOURCE_HASH_MAX_BYTES) throw ceilingRefusal(file.identity, stat.size);
    // One byte past the expected size, so growth since the fstat is seen.
    let buffer = Buffer.allocUnsafe(stat.size + 1);
    let filled = 0;
    for (;;) {
      if (filled === buffer.length) {
        if (filled > SOURCE_HASH_MAX_BYTES) throw ceilingRefusal(file.identity, filled);
        const grown = Buffer.allocUnsafe(Math.min(buffer.length * 2, SOURCE_HASH_MAX_BYTES + 1));
        buffer.copy(grown, 0, 0, filled);
        buffer = grown;
      }
      const read = readSync(fd, buffer, filled, buffer.length - filled, null);
      if (read === 0) break;
      filled += read;
    }
    return buffer.subarray(0, filled);
  } finally {
    closeSync(fd);
  }
}