Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript
22.6 KB
Raw Blame
/**
 * The attestation verification kernel (delivery trust, card t_3bdb82b0).
 *
 * The operator question is "did the state these surfaces stamp change
 * since the attestation", and the answer is a REPORT, per surface:
 *
 *   - `intact`  - the shard's current head is still the attested head;
 *   - `appended`- the head moved, and walking forward from the attested
 *     head through `prev` links reaches the new head with every row
 *     hashing to itself: the shard grew legitimately since signing;
 *   - `diverged`- the attested head is gone, or the forward walk meets a
 *     row whose hash does not recompute, whose `prev` does not link, or
 *     whose links were stripped: an edit, deletion or rewrite after the
 *     signed point;
 *   - `missing` - the shard file is gone.
 *
 * ## Report-only, and forward-only, exactly like the log chain
 *
 * Nothing here writes, and nothing here throws - "could not verify" is
 * never folded into "verified clean" (a store whose manifest cannot be
 * recomputed reads as `storeDigestMatches: false`, which is the honest
 * answer the contract's shape can carry). The walk starts AT the attested
 * head on purpose: an attestation anchors a point in each shard's
 * history, not the whole history, and integrity of the rows BEFORE that
 * point is the log-chain verifier's standing question (`o2b brain log
 * verify`), not this one. The attested row itself and everything after
 * it are this walk's question, held to the full standard: the attested
 * row's own hash must recompute from its fields, and a row at or after
 * it that the walk cannot chain (unparseable, stripped links) reads as
 * `diverged`, never as silence between chained rows.
 *
 * The per-shard classification reads lock-free, like `verifyLogChain`:
 * a verifier's misread is a report the operator re-runs, where a wrong
 * head gets SIGNED and a signature is permanent. The manifest recompute
 * for the store digest is the one place a verify waits - it rides the
 * collector's bounded-retry log lock, so an append in progress delays a
 * report by at most that retry window, and a lock that is still held
 * when the retries end is a store this run could not measure, reported
 * as `storeDigestMatches: false` beside the intact shard verdicts.
 *
 * ## What a legitimately advanced store reads as
 *
 * Appends after signing are the NORMAL case. The verdict says so per
 * shard, and `storeDigestMatches` reports `false` because the recomputed
 * manifest really does differ from the attested one - with the appended
 * shards naming why. The record's own seal and signature still verify;
 * a false store digest beside nothing but `appended` shards is a store
 * that moved, not a store that was broken.
 *
 * ## Legacy day files only (design D14 item 10)
 *
 * An attestation binds the legacy log day files and the ledger files under
 * the vault, and this kernel verifies those only; every verification says
 * so in `covers`. A store-only machine writes its log and ledgers as brain
 * store records, which no attestation names: their integrity is each
 * record's checksum plus the partition state digests the doctor compares
 * (`STORE_LOG_INTEGRITY` in `log-chain.ts`).
 */

import { existsSync, lstatSync } from "node:fs";
import { join, relative, sep } from "node:path";

import {
  attestationStoreDigest,
  collectAttestationManifest,
  isGrammarValidName,
  ledgerGrowthGrammars,
  type AttestationManifest,
  type CollectAttestationManifestOptions,
} from "./digest.ts";
import type { LedgerShardGrammar } from "../ledger-shards.ts";
import { verifyAttestationSignature } from "./signing.ts";
import type { AttestationRecord } from "./record.ts";
import { LOG_CHAIN_FIELDS, logChainHash } from "../log.ts";
import { scanJsonlRows } from "../log-jsonl.ts";
import { BRAIN_LOG_REL } from "../path-constants.ts";
import { gitStoreRootDir } from "../git/store.ts";
import { brainDirs } from "../paths.ts";
import { SafeguardAbortError, type Safeguard } from "../safeguard.ts";
import type { ProgressSink } from "../progress.ts";

/** The state of one attested shard, as verification reports it. */
export type AttestationShardState = "intact" | "appended" | "diverged" | "missing";

/** What {@link verifyAttestation} covers: the legacy day files and ledger files of the vault. */
export const ATTESTATION_VERIFY_COVERS = "legacy-day-files";

export interface AttestationVerification {
  /** Always {@link ATTESTATION_VERIFY_COVERS}: brain store records are not attested. */
  readonly covers: typeof ATTESTATION_VERIFY_COVERS;
  readonly recordId: string;
  readonly createdAt: string;
  readonly signatureValid: boolean;
  /** One verdict per ATTESTED shard, in shard-name order. */
  readonly shards: ReadonlyArray<{
    readonly shardId: string;
    readonly state: AttestationShardState;
  }>;
  /**
   * Whether the manifest recomputed NOW still digests to the record's
   * `storeDigest`. False for a legitimately advanced store too - the
   * per-shard states are what distinguish "moved" from "broken".
   */
  readonly storeDigestMatches: boolean;
  /**
   * Whether the recomputed manifest differs from the attested one in a
   * way normal growth cannot explain: a vault-identity or index change,
   * ledger damage (an attested file gone, an invalid name, an addition
   * to a strict directory), or growth neither the new-shard rule nor the
   * ledger grammars cover. The one boolean both faces exit and warn on,
   * so the CLI and the doctor cannot drift about what counts as damage.
   */
  readonly unexplainedDrift: boolean;
  /**
   * Whether the manifest could be recomputed at all. False for a run
   * that could not measure - a log lock still held after the retries, a
   * directory that could not be listed, an index that could not be
   * read - which is a different fact from drift, and the faces word it
   * differently: "re-run" for the CLI, the uncertain stream for the
   * doctor, never ledger-damage prose.
   */
  readonly storeMeasured: boolean;
}

export interface VerifyAttestationOptions {
  /**
   * The public half to verify the signature against. When offered, the
   * explicit-path contract applies: a missing or unreadable file is an
   * invalid signature, never a fall back to deriving the key from the
   * private seed - agent-facing callers name the vault's exported half
   * so a verification they trigger never reads operator key material.
   * The operator CLI omits it, because deriving the public half from
   * the operator's own seed on the operator's own machine is what makes
   * a missing export file survivable there.
   */
  readonly publicPath?: string;
  /** Cooperative deadline guard, consulted per shard and per manifest surface. */
  readonly safeguard?: Safeguard;
  /**
   * Accepted per the kernel contract. Nothing is emitted today: a
   * `ProgressEvent` names an operation from the closed `OPERATION`
   * vocabulary, and no attestation member exists there yet - a
   * mislabelled operation would put an unresolvable name on every live
   * progress stream. Wiring one member into `OPERATION` plus a
   * `progressCounter` here is the whole change when the vocabulary
   * opens to attestation.
   */
  readonly onProgress?: ProgressSink;
}

/** One shard row as the chain walk needs it, or null when it is not that shape. */
interface ChainRow {
  readonly h: string;
  readonly prev: string | null;
  readonly ts: string;
  readonly kind: string;
  readonly payload: Readonly<Record<string, string | ReadonlyArray<string>>>;
}

/** Strict projection: the exact shape `renderJsonlLine` can emit. */
function chainRow(row: Readonly<Record<string, unknown>>): ChainRow | null {
  const prev = row[LOG_CHAIN_FIELDS.prev];
  if (prev !== null && typeof prev !== "string") return null;
  const ts = row["ts"];
  const kind = row["kind"];
  const payload = row["payload"];
  if (typeof ts !== "string" || typeof kind !== "string") return null;
  if (payload === null || typeof payload !== "object" || Array.isArray(payload)) return null;
  const projected: Record<string, string | ReadonlyArray<string>> = {};
  for (const [key, value] of Object.entries(payload as Record<string, unknown>)) {
    if (typeof value === "string") {
      projected[key] = value;
      continue;
    }
    if (Array.isArray(value) && value.every((item) => typeof item === "string")) {
      projected[key] = value as string[];
      continue;
    }
    return null;
  }
  const h = row[LOG_CHAIN_FIELDS.h];
  if (typeof h !== "string" || h === "") return null;
  return { h, prev, ts, kind, payload: projected };
}

/**
 * Classify one attested shard. Never throws: an unreadable shard is
 * damage to name (`diverged`), never a crash and never a clean bill.
 */
function classifyShard(
  vault: string,
  shardName: string,
  attestedHead: string,
): AttestationShardState {
  let logDir: string;
  try {
    logDir = brainDirs(vault).log;
  } catch {
    return "missing";
  }
  const path = join(logDir, shardName);
  if (!existsSync(path)) return "missing";
  const scan = scanJsonlRows(path);
  if (!scan.readable) return "diverged";

  const rows: ChainRow[] = [];
  let currentHead: string | null = null;
  let anchorIndex = -1;
  for (const line of scan.lines) {
    const row = line.row === null ? null : chainRow(line.row);
    if (row === null) {
      // Before the attested point a row the projection refuses is
      // pre-chain history - the log-chain verifier's standing question,
      // not this walk's. At or after it, a row that cannot be chained
      // (unparseable JSON, stripped links, a payload shape the
      // projection refuses) is an edit: skipping it would read a
      // garbled or rewritten row as legitimate growth.
      if (anchorIndex >= 0) return "diverged";
      continue;
    }
    rows.push(row);
    currentHead = row.h;
    if (anchorIndex < 0 && row.h === attestedHead) anchorIndex = rows.length - 1;
  }
  if (anchorIndex < 0) return "diverged";

  // The attested row is what the attestation vouches for, so its stored
  // hash must recompute from its own fields: a row edited after signing
  // with its `h` field left in place would otherwise ride both the
  // `intact` verdict and the forward walk's anchor.
  const anchor = rows[anchorIndex]!;
  if (logChainHash(anchor.prev, anchor.ts, anchor.kind, anchor.payload) !== attestedHead) {
    return "diverged";
  }
  if (currentHead === attestedHead) return "intact";

  // Forward walk from the attested head: every row after it must hash to
  // itself and link to its predecessor, ending exactly at the head.
  let previous = attestedHead;
  for (let i = anchorIndex + 1; i < rows.length; i++) {
    const row = rows[i]!;
    if (row.prev !== previous) return "diverged";
    if (logChainHash(row.prev, row.ts, row.kind, row.payload) !== row.h) return "diverged";
    previous = row.h;
  }
  return previous === currentHead ? "appended" : "diverged";
}

/**
 * Verify one attestation against the store as it stands now.
 *
 * Performs no writes and reports every attested shard even when the
 * signature is invalid - the operator needs the whole picture, and a
 * signature check alone would hide how far the store moved. Every
 * failure mode lands in the report; the one throw is the caller's OWN
 * {@link SafeguardAbortError}, which propagates from a deadline rather
 * than reading as a verdict.
 */
export function verifyAttestation(
  vault: string,
  record: AttestationRecord,
  opts: VerifyAttestationOptions = {},
): AttestationVerification {
  const checkpoint = (): void => opts.safeguard?.checkpoint();
  const signatureValid =
    opts.publicPath !== undefined
      ? verifyAttestationSignature(record, vault, { publicPath: opts.publicPath })
      : verifyAttestationSignature(record, vault);
  checkpoint();

  const shards = Object.entries(record.manifest.logHeads)
    .toSorted(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0))
    .map(([shardId, head]) => {
      checkpoint();
      return { shardId, state: classifyShard(vault, shardId, head) };
    });
  checkpoint();

  let storeDigestMatches = false;
  let storeMeasured = false;
  let unexplainedDrift = false;
  try {
    const recomputeOptions: CollectAttestationManifestOptions = { safeguard: opts.safeguard };
    const recomputed = collectAttestationManifest(vault, recomputeOptions);
    storeDigestMatches = attestationStoreDigest(recomputed) === record.storeDigest;
    unexplainedDrift = unexplainedManifestDrift(vault, record.manifest, recomputed);
    storeMeasured = true;
  } catch (err) {
    // The caller's own deadline is not a store fact: a run the safeguard
    // aborts must surface as the abort, never as a normal-looking report
    // whose digest happens to differ.
    if (err instanceof SafeguardAbortError) throw err;
    // A manifest that cannot be recomputed (a held log lock, an
    // unlistable ledger directory) is a store this run could not
    // measure. "Not measured" is not "matching", and it is not drift
    // either - it is its own fact, and the faces word it as one.
    storeDigestMatches = false;
  }

  return Object.freeze({
    covers: ATTESTATION_VERIFY_COVERS,
    recordId: record.id,
    createdAt: record.createdAt,
    signatureValid,
    shards: Object.freeze(shards),
    storeDigestMatches,
    unexplainedDrift,
    storeMeasured,
  });
}

function sameFiles(a: ReadonlyArray<string>, b: ReadonlyArray<string>): boolean {
  return a.length === b.length && a.every((name, i) => name === b[i]);
}

/**
 * The kind of one entry of an inventoried directory, read through the
 * directory's inventory key, or null when the entry cannot be stated
 * (it vanished or cannot be read between the listing and this check) or
 * is not a regular file or directory - a symlink, FIFO or socket is not
 * something this build's writers produce, and a kind read through the
 * link's TARGET would let a planted link ride a lawful name as growth.
 * An entry this run cannot measure is never explained growth.
 */
function entryKind(vault: string, dirKey: string, name: string): "file" | "directory" | null {
  try {
    const dir = join(vault, ...dirKey.split("/"));
    const st = lstatSync(join(dir, name));
    if (st.isDirectory()) return "directory";
    return st.isFile() ? "file" : null;
  } catch {
    return null;
  }
}

/**
 * Whether one inventoried directory's entry list is explained: every
 * attested entry must still be there with the kind its NAME says the
 * producer gave it, and every name that was not attested must be normal
 * growth for that directory - a file parsing under the directory's
 * ledger grammar (a new ISO-week audit shard, a new continuity month, a
 * new metric surface) or a known ledger subdirectory created beneath it
 * (a first work-identity directory inside `Brain/.state`, a first
 * per-preference trail inside the preference-audit root). The name
 * alone does not settle it: an entry whose kind is not the producer's
 * (a directory spelled as a shard, or a file where a known ledger
 * subdirectory belongs) is not something this build writes, and a kind
 * this run cannot read is not provable growth either. The same
 * name-derived expectation judges the ATTESTED entries - a shard name
 * was signed as a file, a known subdirectory name as a directory - so
 * an attested file swapped for a same-name directory is still drift,
 * while a subdirectory the producers legitimately made before signing
 * is not. An attested name under no grammar at all was signed as a
 * file, the only kind this build's writers make at the file level. A
 * directory with no grammar entry is strict - additions there are not
 * something this build's writers produce.
 */
function ledgerGrowthExplained(
  vault: string,
  grammars: ReadonlyMap<string, ReadonlyArray<LedgerShardGrammar>>,
  dir: string,
  attestedFiles: ReadonlyArray<string>,
  now: ReadonlyArray<string>,
): boolean {
  const layout = grammars.get(dir);
  if (layout === undefined) return sameFiles(attestedFiles, now);
  const attestedSet = new Set(attestedFiles);
  const kind = (name: string): string | null => entryKind(vault, dir, name);
  const wanted = (name: string): string =>
    isGrammarValidName(name, layout) || !grammars.has(`${dir}/${name}`) ? "file" : "directory";
  return (
    attestedFiles.every((name) => now.includes(name) && kind(name) === wanted(name)) &&
    now.every((name) => {
      if (attestedSet.has(name)) return kind(name) === wanted(name);
      if (isGrammarValidName(name, layout)) return kind(name) === "file";
      return grammars.has(`${dir}/${name}`) && kind(name) === "directory";
    })
  );
}

/**
 * Whether the recomputed manifest differs from the attested one beyond
 * what normal growth explains. Growth is the NORMAL case, and every
 * append-only producer has it: a log shard appended after signing is
 * classified `appended`, a shard created after signing joins the log
 * directory's inventory and `logHeads`, and the append-only ledger
 * directories gain their time-window shards (a new hook-audit week, a
 * new continuity month, a new metric surface) - allowed exactly when the
 * new names parse under the producer's own grammar and carry the kind
 * the producer writes (files; a known subdirectory where a nested
 * ledger belongs), with every attested entry still present with the
 * kind its name says it was signed as. Ledger directories and known
 * subdirectories that were created entirely after signing follow the
 * same rule against an empty attested list - a first metric surface, a
 * first hook-audit week on a vault signed before the hooks ran, a first
 * work-identity directory. Everything else is drift no verdict names:
 * the vault identity, the index stamp, an attested entry gone, an
 * invalid name, or ANY addition to `Brain/preferences`, whose files are
 * authored state.
 */
function unexplainedManifestDrift(
  vault: string,
  attested: AttestationManifest,
  recomputed: AttestationManifest,
): boolean {
  if (attested.vaultIdentityId !== recomputed.vaultIdentityId) return true;
  if (attested.indexStamp !== recomputed.indexStamp) return true;
  const logDirKey = BRAIN_LOG_REL;
  const grammars = ledgerGrowthGrammars(vault);
  const newShards = Object.keys(recomputed.logHeads)
    .filter((shard) => !(shard in attested.logHeads))
    .toSorted();
  // A new JSONL shard lands beside its markdown twin (the appender
  // writes the pair), and only the JSONL half carries a chain head - so
  // the log directory's allowed growth is both files of each new shard.
  const newLogFiles = newShards
    .flatMap((shard) =>
      shard.endsWith(".jsonl") ? [shard, `${shard.slice(0, -".jsonl".length)}.md`] : [shard],
    )
    .toSorted();
  // The log directory's inventory also names the ledger subdirectories
  // under it (the audit families, the schema-mutation audit, the
  // pref-audit root), so its allowed set grows by exactly those known
  // names.
  const logLedgerSubdirs = [...grammars.keys()].flatMap((key) =>
    key.startsWith(`${BRAIN_LOG_REL}/`) && !key.slice(BRAIN_LOG_REL.length + 1).includes("/")
      ? [key.slice(BRAIN_LOG_REL.length + 1)]
      : [],
  );
  for (const [dir, attestedFiles] of Object.entries(attested.ledgerInventories)) {
    const now = recomputed.ledgerInventories[dir];
    if (now === undefined) return true; // an inventoried directory vanished
    if (dir === logDirKey) {
      if (
        !explainedLogGrowth(vault, attestedFiles, newShards, newLogFiles, logLedgerSubdirs, now)
      ) {
        return true;
      }
      continue;
    }
    if (!ledgerGrowthExplained(vault, grammars, dir, attestedFiles, now)) return true;
  }
  for (const dir of Object.keys(recomputed.ledgerInventories)) {
    if (dir in attested.ledgerInventories) continue;
    // A directory that gained ledger status since signing is only
    // explained when it is the log directory filling with the new
    // shards and known ledger subdirectories, one of those known
    // subdirectories itself (a first continuity month, a first audit
    // week), a directory whose producer declares a grammar (the
    // metrics, hook-audit, watchdog, idempotency, continuity and
    // lineage-state ledgers all live outside `Brain/log` and may be
    // created entirely after signing), or a first repository store
    // directory - each judged against its producer's grammar, so every
    // name it brings must parse and an unrelated file stays drift.
    if (
      dir === logDirKey &&
      explainedLogGrowth(
        vault,
        [],
        newShards,
        newLogFiles,
        logLedgerSubdirs,
        recomputed.ledgerInventories[dir] ?? [],
      )
    ) {
      continue;
    }
    if (
      (grammars.has(dir) ||
        dir.startsWith(`${BRAIN_LOG_REL}/`) ||
        isFirstRepoStoreDir(vault, dir)) &&
      ledgerGrowthExplained(vault, grammars, dir, [], recomputed.ledgerInventories[dir] ?? [])
    ) {
      continue;
    }
    return true;
  }
  return false;
}

/**
 * Whether `dir` (an inventory key) is one repository store directory
 * directly under the store root: the shape a repository's first link
 * creates, which may happen entirely after signing. One level only -
 * the sweep inventories exactly these directories, and the layout rule
 * (watermark, digest, commits shards) belongs to the store, not to
 * anything nested deeper.
 */
function isFirstRepoStoreDir(vault: string, dir: string): boolean {
  const rootRel = relative(vault, gitStoreRootDir(vault)).split(sep).join("/");
  return dir.startsWith(`${rootRel}/`) && !dir.slice(rootRel.length + 1).includes("/");
}

/**
 * Whether the log directory's entry list is explained by growth, in set
 * semantics: every attested file and every new shard's JSONL must still
 * be there with the kind its name says the producer gave it, the new
 * shards' markdown twins MAY be there (a twin can predate its JSONL on
 * a legacy md-only day, and the appender's own partial-failure path can
 * leave a JSONL without one), the known ledger subdirectories under the
 * log MAY appear as the DIRECTORIES they are (their own inventories are
 * judged by the ledger rule; a regular file riding a subdirectory's
 * name is an impostor no inventory catches any other way), and nothing
 * else may appear or vanish.
 */
function explainedLogGrowth(
  vault: string,
  attestedFiles: ReadonlyArray<string>,
  newShards: ReadonlyArray<string>,
  newLogFiles: ReadonlyArray<string>,
  ledgerSubdirs: ReadonlyArray<string>,
  now: ReadonlyArray<string>,
): boolean {
  const allowed = new Set([...attestedFiles, ...newLogFiles, ...ledgerSubdirs]);
  const subdirs = new Set(ledgerSubdirs);
  const kind = (name: string): string | null => entryKind(vault, BRAIN_LOG_REL, name);
  const wanted = (name: string): string => (subdirs.has(name) ? "directory" : "file");
  const required = [...attestedFiles, ...newShards];
  return (
    required.every((name) => now.includes(name) && kind(name) === wanted(name)) &&
    now.every((name) => allowed.has(name) && kind(name) === wanted(name))
  );
}