Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript
24.8 KB
Raw Blame
/**
 * Extraction intake for the pledge family (dated-commitments wave).
 *
 * One validated, idempotent path turns an agent-posted typed pledge
 * extraction (a source page plus an array of items: title, due, evidence
 * quotes, optional text and counterparty names) into `Brain/pledges/`
 * pages, mirroring the extraction-intake discipline of
 * `intake/extract-intake.ts`. The kernel never calls a model: the model
 * that produced the extraction lives on the agent side of the MCP
 * boundary, and this module validates, anchors and commits.
 *
 *   - Validate EVERYTHING before any write. Structure, source
 *     readability at the caller's reach, and quote containment for every
 *     item are checked first; one invalid item refuses the whole intake
 *     and leaves the vault untouched (no partial batches). The create arm
 *     re-checks each slug under its own write lock, so a concurrent writer
 *     can still take a slug between validation and commit: that failure
 *     removes the pages this intake already created (best effort) and
 *     reports both what was created and what was rolled back. The residual
 *     window is a rollback that cannot remove a page (reported by name, not
 *     hidden) and any re-anchor rewrites made before the failing item.
 *   - Identity is idempotent. A pledge's identity is the sha256 prefix
 *     (first 12 hex) over the normalized `(title, due, source)` triple.
 *     A re-post of the same identity with the same normalized payload is
 *     a reported duplicate no-op; the same identity with a different
 *     payload throws a named {@link PledgeIntakeError} mismatch - the
 *     preference idempotency-key precedent.
 *   - Counterparties are anchored server-side. The kernel never trusts
 *     caller-supplied entity ids: names are resolved against canonical-
 *     scope entities the caller can read (`entityStatusInScope` plus the
 *     reach predicate - a withheld record is absent, so it can neither
 *     match nor block a match) by whole-name equality over the shared
 *     quality-gated match forms (`entityNameEqualsAnchorForm` - a name is
 *     resolved, not scanned for, so a name that merely contains an
 *     entity's name anchors nowhere). A name that resolves to exactly one
 *     entity becomes a `promised_to` edge; a name that resolves nowhere -
 *     or that two canonical records claim - is reported on the result and
 *     stored nowhere, so a pledge exists without edges rather than with
 *     fake ones.
 *   - The re-anchor arm. Re-posting the same identity and payload after
 *     the entity has since been registered computes the edge the first
 *     post could not, rewrites the page's `promised_to` frontmatter (the
 *     frontmatter-relations ingest projects the typed edges) and reports
 *     the slug under `reanchored`; a third identical post is a plain
 *     duplicate again. The arm is additive: it merges the stored and the
 *     freshly anchored ids and rewrites only when that union is larger,
 *     so an entity leaving canonical scope is reported under `staleEdges`
 *     and never turns an identical repost into an edge deletion.
 */

import { readFileSync } from "node:fs";
import { join, relative, sep } from "node:path";

import { parseFrontmatterText } from "../vault.ts";
import { entityNameEqualsAnchorForm } from "./atomic-facts.ts";
import { buildEntityIndex } from "./entities/index-builder.ts";
import { resolveEntityLexicon } from "./entities/lexicon.ts";
import { ENTITY_STATUS_SCOPE, entityStatusInScope } from "./entities/status-scope.ts";
import {
  createPledgePage,
  isSourcePage,
  NOTES_HEADER,
  normalizePledgeSource,
  pledgeExists,
  pledgeIdentity,
  pledgeSlugCandidates,
  rewritePledgeFrontmatter,
  showPledge,
  type PledgePage,
} from "./pledges.ts";
import { validateIsoDate } from "./paths.ts";
import { strictestVisibilityOf } from "../graph/visibility.ts";
import { remoteDenyVisibilityTokens } from "../graph/privacy-markers.ts";
import { PROMISED_TO_RELATION } from "../graph/relation-vocab.ts";
import { normalizeForQuoteComparison, quoteHaystack, spanOccursIn } from "./distill/quote-spans.ts";
import {
  acquireLaneLock,
  assertLaneWrite,
  removeLanePage,
  writeLaneText,
} from "../store/lane-pages.ts";

/** A pledge-intake payload or anchor failed validation; nothing was written. */
export class PledgeIntakeError extends Error {
  constructor(message: string) {
    super(message);
    this.name = "PledgeIntakeError";
  }
}

/** One pledge the agent extracted from a source. */
export interface PledgeIntakeItem {
  readonly title: string;
  readonly due: string;
  readonly evidence: ReadonlyArray<string>;
  readonly text?: string;
  readonly counterparties?: ReadonlyArray<string>;
}

/** A typed pledge extraction the agent supplies for intake. */
export interface PledgeIntakeInput {
  /** The vault-relative page every item was extracted from and cites. */
  readonly source: string;
  readonly pledges: ReadonlyArray<PledgeIntakeItem>;
}

export interface PledgeIntakeResult {
  /** Slugs of pledges created by this intake, in input order. */
  readonly created: ReadonlyArray<string>;
  /** Slugs whose identity and payload were already stored (no-op). */
  readonly duplicates: ReadonlyArray<string>;
  /** Slugs whose `promised_to` edges this post added (the re-anchor arm). */
  readonly reanchored: ReadonlyArray<string>;
  /**
   * Slugs with a stored `promised_to` edge the readable canonical
   * registry can no longer reproduce (the entity left canonical scope,
   * or became unreadable). Reported, never removed: the page is left
   * byte-identical, because an edge that was valid when computed is not
   * the caller's to delete, and the re-anchor arm does not grow a new
   * edge beside a dead one.
   */
  readonly staleEdges: ReadonlyArray<string>;
  /**
   * Counterparty names that anchored to no canonical-scope entity, first
   * seen first, deduplicated. Stored nowhere: the pledge carries only the
   * edges that anchored.
   */
  readonly unanchoredCounterparties: ReadonlyArray<string>;
}

export interface PledgeIntakeOptions {
  /** Agent identity stamped as the page `agent`. */
  readonly agent: string;
  /** Injected clock for deterministic stamps. */
  readonly now: Date;
  /**
   * May the caller read the vault page at this vault-relative path? The
   * cited source is refused up front when the predicate says no, exactly
   * like an absent page - nothing is read from it, nothing is written
   * from it, and nothing about it leaks. The same predicate governs the
   * stored pledge lookup and the entity pages counterparties may anchor
   * to, so on those read surfaces a withheld page is identical to an
   * absent one.
   *
   * The create path is the deliberate exception: a slug already held by a
   * page the caller cannot read is refused at any reach ("pledge already
   * exists"), because the identity behind the gate cannot be compared and
   * creating there would replace the withheld page - the same discipline
   * `brain_pledge` documents for its add arm. A readable page held by a
   * DIFFERENT identity is not that case: the new pledge files under its
   * identity suffix instead. A caller can therefore learn that some page
   * holds a slug it cannot read; it can never read or overwrite it.
   */
  readonly readable?: (rel: string) => boolean;
}

/**
 * Hard cap on the pledge items one intake may post. Every distinct
 * counterparty name scans the canonical entity registry synchronously
 * inside one handler, so an unbounded batch multiplies that work; the
 * number follows the family's write-batch ceiling (`MAX_BATCH_OPERATIONS`,
 * 100) rather than inventing a second scale.
 */
export const MAX_INTAKE_PLEDGES = 100;

/** Per-item cap on counterparty names: each name is one registry scan. */
export const MAX_INTAKE_COUNTERPARTIES = 32;

/**
 * Longest counterparty name the intake accepts. The schema caps a pledge
 * title at 200 characters and a name is matched by the same occurrence
 * kernel, so one bound governs both.
 */
export const MAX_INTAKE_COUNTERPARTY_NAME_LENGTH = 200;

/** Collapse whitespace runs and trim: the intake's normalization for text. */
function collapse(value: string): string {
  return value.replaceAll(/\s+/gu, " ").trim();
}

/** A vault-relative POSIX path from an absolute one, whatever the host separator. */
function vaultRel(vault: string, abs: string): string {
  return relative(vault, abs).split(sep).join("/");
}

/**
 * Normalize one item's counterparty names and enforce the per-item caps
 * at the point the work happens: every name is a registry scan, and a
 * name is matched by the same occurrence kernel that bounds a title, so
 * a longer or more numerous list is refused with a named error before
 * anything is anchored or written.
 */
function prepareCounterpartyNames(slug: string, raw: ReadonlyArray<string> | undefined): string[] {
  const names = (raw ?? []).map((name) => name.trim()).filter((name) => name.length > 0);
  if (names.length > MAX_INTAKE_COUNTERPARTIES) {
    throw new PledgeIntakeError(
      `pledge intake item names more than ${MAX_INTAKE_COUNTERPARTIES} counterparties: ${slug} - ` +
        "keep the counterparty list to the people the promise is owed to",
    );
  }
  for (const name of names) {
    if (name.length > MAX_INTAKE_COUNTERPARTY_NAME_LENGTH) {
      throw new PledgeIntakeError(
        `pledge intake counterparty name is longer than ${MAX_INTAKE_COUNTERPARTY_NAME_LENGTH} ` +
          `characters (${name.length}): ${slug}`,
      );
    }
  }
  return names;
}

/**
 * The normalized payload a stored pledge must match for a re-post to read
 * as a duplicate: the promise text and the evidence quote set, order-
 * insensitive over the quotes. Counterparty names are deliberately not
 * part of the payload - they are anchor requests the server resolves, and
 * a re-post that names a since-registered entity is the re-anchor arm,
 * not a mismatch. The notes section is deliberately not part of it either:
 * the intake never writes one, so a note added to a stored page by hand
 * must not turn an otherwise identical re-post into a mismatch (the
 * re-anchor rewrite preserves such a note untouched).
 */
function payloadKey(text: string, evidence: ReadonlyArray<string>): string {
  return JSON.stringify([collapse(text), evidence.map((quote) => collapse(quote)).toSorted()]);
}

function storedPayloadKey(page: PledgePage): string {
  return payloadKey(page.text, page.evidence);
}

/** The item shape the validation pass hands to the write pass. */
interface PreparedIntakeItem {
  readonly slug: string;
  readonly title: string;
  readonly due: string;
  readonly evidence: string[];
  readonly text?: string;
  readonly counterparties: string[];
  /** The stored page under the same slug, or null when the slug is free. */
  readonly existing: PledgePage | null;
}

/**
 * Intake a typed pledge extraction into the pledge family. Validates the
 * full payload first (throwing {@link PledgeIntakeError} with no write on
 * failure), anchors counterparties server-side, then commits: creates new
 * pledges, reports stored identity-and-payload repeats as duplicates,
 * adds edges through the re-anchor arm when an entity registered between
 * two identical posts, and reports - without writing - a stored edge the
 * anchor can no longer reproduce under `staleEdges`.
 */
export function intakePledges(
  vault: string,
  input: PledgeIntakeInput,
  opts: PledgeIntakeOptions,
): PledgeIntakeResult {
  // Lane write guard, the one every pledge writer passes: the vault
  // identity guard on an unswitched machine, the store identity check on a
  // store-only one (design D14 item 13).
  assertLaneWrite(vault);

  // The batch ceiling comes before any read: an oversized extraction is
  // refused whole, so a caller cannot buy unbounded registry scans.
  if (input.pledges.length > MAX_INTAKE_PLEDGES) {
    throw new PledgeIntakeError(
      `a pledge intake may contain at most ${MAX_INTAKE_PLEDGES} pledges, got ${input.pledges.length}`,
    );
  }

  // --- Validation pass: everything, before any write. ---
  const source = normalizePledgeSource(input.source);
  if (source.length === 0 || source.startsWith("..")) {
    throw new PledgeIntakeError(
      `pledge intake source must name a vault page: ${JSON.stringify(input.source)}`,
    );
  }
  // The reach question comes first, before any stat or read, so a refused
  // source never even has its existence on disk confirmed. The reach
  // predicate answers false for an absent path too, so a withheld and an
  // absent source both refuse right here, under one message - identical to
  // absent; the not-found arm below is the local reach's own diagnostic.
  if (opts.readable !== undefined && !opts.readable(source)) {
    throw new PledgeIntakeError(
      `pledge intake source page is not readable at the caller's reach: ${source}`,
    );
  }
  const sourceAbs = join(vault, source);
  if (!isSourcePage(sourceAbs)) {
    throw new PledgeIntakeError(
      `pledge intake source page not found: ${source} - a pledge cites bytes the vault holds`,
    );
  }
  const sourceText = readFileSync(sourceAbs, "utf8");
  const haystack = quoteHaystack(normalizeForQuoteComparison(sourceText));
  // Read once, derived twice: the containment haystack above and the
  // visibility the created pages inherit. The write pass hands both to
  // `createPledgePage`, so a full batch never re-reads the source.
  const [sourceMeta] = parseFrontmatterText(sourceText);
  // The full union, so a source reserving itself under an informal key
  // (`confidentiality: secret`) reserves the derived pledge pages too -
  // their evidence fields quote the source, so the quotes inherit the
  // reservation.
  const visibility = strictestVisibilityOf([remoteDenyVisibilityTokens(sourceMeta)]);

  const prepared: PreparedIntakeItem[] = [];
  const seenSlugs = new Set<string>();
  const seenIdentities = new Map<string, string>();
  const counterpartyNames: string[] = [];
  const seenNames = new Set<string>();
  for (const item of input.pledges) {
    const title = collapse(item.title);
    if (title.length === 0) {
      throw new PledgeIntakeError("pledge intake item title must not be empty");
    }
    let due: string;
    try {
      due = validateIsoDate(item.due);
    } catch {
      throw new PledgeIntakeError(
        `pledge intake due is not a valid calendar date: ${JSON.stringify(item.due)}`,
      );
    }
    const evidence = item.evidence.map((quote) => quote.trim()).filter((quote) => quote.length > 0);
    if (evidence.length === 0) {
      throw new PledgeIntakeError(
        "pledge intake evidence must cite at least one quote from the source page (an uncited pledge is refused)",
      );
    }
    for (const quote of evidence) {
      if (!spanOccursIn(quote, haystack)) {
        throw new PledgeIntakeError(
          `evidence quote does not occur in the source page ${source}: ${JSON.stringify(quote)}`,
        );
      }
    }
    const identity = pledgeIdentity(title, due, source);
    // Slug resolution, exactly the order the create arm will try: the title
    // slug, then its identity suffix. A candidate held by a DIFFERENT
    // identity is skipped, not refused - titles that differ only in
    // non-ASCII letters fold to one slug, and promises to different people
    // are the family's core case. A repeat of the SAME identity inside one
    // batch keeps the duplicated-post refusal.
    const candidates = pledgeSlugCandidates(title, due, source);
    let slug = candidates[0]!;
    let existing: PledgePage | null = null;
    // The first candidate no stored page occupies. The scan records it and
    // keeps going: a LATER candidate may hold this exact identity (the
    // suffixed page a since-removed slug-holder once forced), and the same
    // promise must win over a free slot or a re-post would file a second
    // copy under the freed slug.
    let free: string | null = null;
    let resolved = false;
    for (const candidate of candidates) {
      if (seenSlugs.has(candidate)) {
        if (seenIdentities.get(candidate) === identity) {
          throw new PledgeIntakeError(
            `pledge intake repeats a slug within one batch: ${candidate} - post each pledge once per intake`,
          );
        }
        continue;
      }
      const stored = showPledge(vault, {
        slug: candidate,
        // The stored-page lookup answers at the caller's reach: a withheld
        // page is identical to an absent one, so nothing is compared,
        // classified or rewritten behind the gate.
        ...(opts.readable !== undefined ? { readable: opts.readable } : {}),
      });
      if (stored !== null) {
        if (pledgeIdentity(stored.title, stored.due, stored.source) === identity) {
          slug = candidate;
          existing = stored;
          resolved = true;
          break;
        }
        continue;
      }
      // A withheld page passes the lookup above as absent; the create arm
      // refuses that taken slug, and so does the validation pass, before any
      // write - a create would otherwise replace a page the caller cannot
      // read.
      if (pledgeExists(vault, candidate)) {
        throw new PledgeIntakeError(
          `pledge already exists: ${candidate} (remove it first to recreate)`,
        );
      }
      if (free === null) free = candidate;
    }
    if (!resolved && free !== null) {
      slug = free;
      existing = null;
      resolved = true;
    }
    if (!resolved) {
      throw new PledgeIntakeError(
        `pledge intake slug already held by a different pledge: ${candidates[0]} - remove the existing pledge first`,
      );
    }
    seenSlugs.add(slug);
    seenIdentities.set(slug, identity);
    if (existing !== null && storedPayloadKey(existing) !== payloadKey(item.text ?? "", evidence)) {
      throw new PledgeIntakeError(
        `pledge intake identity mismatch: ${slug} already stores this identity with a different ` +
          "payload - repost the stored payload or remove the pledge first",
      );
    }
    // The text arm of the same promise: a notes-section header is refused by
    // `addPledge` when the page is rendered, pulled into the validation pass
    // so one failing item cannot strand the items before it on disk.
    if ((item.text ?? "").split(/\r?\n/u).some((line) => line.trim() === NOTES_HEADER)) {
      throw new PledgeIntakeError(
        `pledge intake item text must not contain the ${NOTES_HEADER} section header: ${slug}`,
      );
    }
    const counterparties = prepareCounterpartyNames(slug, item.counterparties);
    for (const name of counterparties) {
      if (!seenNames.has(name)) {
        seenNames.add(name);
        counterpartyNames.push(name);
      }
    }
    prepared.push({
      slug,
      title,
      due,
      evidence,
      ...(item.text !== undefined ? { text: item.text.trim() } : {}),
      counterparties,
      existing,
    });
  }

  // --- Anchoring pass: read-only, server-side, never caller-supplied ids. ---
  const lexicon = resolveEntityLexicon(vault);
  // Canonical scope AND the caller's reach: a record the caller cannot
  // read is absent, so it can neither match a name nor block the one
  // readable record that does (the brief's stated rule). At local reach
  // the predicate is undefined and the per-entity check is skipped.
  const canonical = buildEntityIndex(vault).entities.filter(
    (entity) =>
      entityStatusInScope(entity.status, ENTITY_STATUS_SCOPE.canonical) &&
      (opts.readable === undefined || opts.readable(vaultRel(vault, entity.path))),
  );
  const canonicalIds = new Set(canonical.map((entity) => entity.id));
  const verdictByName = new Map<string, string | null>();
  const unanchoredCounterparties: string[] = [];
  for (const name of counterpartyNames) {
    // Whole-name equality, not occurrence: a counterparty NAME is resolved
    // to the entity it names, so a name that merely contains an entity's
    // name ("the Ada Lovelace team") is a different party and anchors
    // nowhere, reported below instead of stored as a false edge.
    const matches = canonical.filter((entity) =>
      entityNameEqualsAnchorForm(name, [entity.name, ...entity.aliases], { lexicon }),
    );
    verdictByName.set(name, matches.length === 1 ? matches[0]!.id : null);
    if (matches.length !== 1) unanchoredCounterparties.push(name);
  }

  // --- Write pass: nothing above wrote; everything below is decided. ---
  const created: string[] = [];
  const createdPages: PledgePage[] = [];
  const duplicates: string[] = [];
  const reanchored: string[] = [];
  const staleEdges: string[] = [];
  try {
    for (const prep of prepared) {
      const anchoredIds = [
        ...new Set(
          prep.counterparties
            .map((name) => verdictByName.get(name))
            .filter((id): id is string => typeof id === "string"),
        ),
      ].toSorted();
      if (prep.existing === null) {
        const page = createPledgePage(vault, {
          title: prep.title,
          due: prep.due,
          source,
          sourceText,
          visibility,
          evidence: prep.evidence,
          ...(prep.text !== undefined ? { text: prep.text } : {}),
          counterparties: anchoredIds,
          agent: opts.agent,
          now: opts.now,
        });
        createdPages.push(page);
        // The page's own slug, not the validation pass's prediction: the
        // create arm re-resolves the candidates under its lock, and the
        // report must name the page that is actually on disk.
        created.push(page.slug);
        continue;
      }
      const storedIds = [...new Set(prep.existing.counterparties)].toSorted();
      // Staleness is a fact about each STORED edge versus the readable
      // canonical registry, never about the names this particular post
      // carried: names are anchor requests, not part of the payload, so a
      // repost that omits some (or all) of them asked for no anchoring and
      // says nothing about the stored edges. A slug with a stale edge is
      // reported and left byte-identical - no edge is removed, and the
      // re-anchor arm does not grow a new edge beside a dead one.
      if (storedIds.some((id) => !canonicalIds.has(id))) {
        staleEdges.push(prep.slug);
        continue;
      }
      const mergedIds = [...new Set([...storedIds, ...anchoredIds])].toSorted();
      if (mergedIds.length === storedIds.length) {
        duplicates.push(prep.slug);
        continue;
      }
      rewritePledgeCounterparties(vault, prep.existing, mergedIds);
      reanchored.push(prep.slug);
    }
  } catch (err) {
    // Validation checked every slug, but a create takes its lock and checks
    // again at write time: a concurrent add or a second intake can take a
    // slug in between, so item N can fail after items 1..N-1 were written.
    // Remove the pages this intake created (best effort) and report both
    // lists, so the no-partial-batches promise holds rather than leaving a
    // silent half-batch on disk. Nothing created means nothing to undo: the
    // original refusal (a lock collision included) surfaces unchanged.
    if (createdPages.length === 0) throw err;
    const removed: string[] = [];
    const stuck: string[] = [];
    for (const page of createdPages) {
      try {
        removeLanePage(vault, page.path);
        removed.push(page.slug);
      } catch {
        stuck.push(page.slug);
      }
    }
    const reason = err instanceof Error ? err.message : String(err);
    throw new PledgeIntakeError(
      `pledge intake failed after creating ${createdPages.length} pledge(s): ${reason}; ` +
        `rolled back ${removed.length > 0 ? removed.join(", ") : "none"}` +
        (stuck.length > 0 ? ` - could not remove ${stuck.join(", ")}` : ""),
    );
  }

  return Object.freeze({
    created: Object.freeze(created),
    duplicates: Object.freeze(duplicates),
    reanchored: Object.freeze(reanchored),
    staleEdges: Object.freeze(staleEdges),
    unanchoredCounterparties: Object.freeze(unanchoredCounterparties),
  });
}

/**
 * Rewrite one stored pledge page's `promised_to` frontmatter in place
 * through the shared family helper ({@link rewritePledgeFrontmatter}):
 * the stored map is parsed back whole, only `promised_to` is mutated, and
 * the page is re-rendered with its body bytes untouched - a `visibility`
 * a private pledge carries, or any key an operator added by hand, survives
 * the rewrite. A fixed-key render would be correct only for creation,
 * where there are no prior bytes to preserve; a rewrite has them, so it
 * mutates rather than regenerates.
 */
function rewritePledgeCounterparties(
  vault: string,
  page: PledgePage,
  counterparties: ReadonlyArray<string>,
): void {
  // Same lock discipline as the family's other writers: the rewrite is a
  // read-modify-write of the whole frontmatter, so a re-anchor racing a
  // fulfill (or a second re-anchor) must never overwrite either edit.
  const handle = acquireLaneLock(vault, page.path);
  try {
    writeLaneText(
      vault,
      page.path,
      rewritePledgeFrontmatter(vault, page, (meta) => {
        meta[PROMISED_TO_RELATION] = [...counterparties];
      }),
    );
  } finally {
    handle.release();
  }
}