Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript
18.6 KB
Raw Blame
/**
 * Write-ahead journal for the skill-proposal accept sequence
 * (no-dead-ends, Unit I).
 *
 * Accepting a proposal is four filesystem steps against three different
 * trees: write the accepted archive, materialize the procedure, remove the
 * pending copy, rebuild the procedural projections. Each individual write
 * is already atomic and exclusive (`writeFrontmatterAtomic` over
 * `atomicCreateFileSyncExclusive`), so the risk is never a torn file - it
 * is a process that stops BETWEEN two of those steps and leaves a
 * duplicate (pending plus accepted), an orphan (an accepted archive with
 * no procedure) or a stale projection.
 *
 * An exception can be compensated in a `catch`; a crash cannot. So the
 * sequence records its intent here BEFORE each step, and the next accept
 * resolves whatever it finds. The journal states the step the writer was
 * ABOUT to perform, never the one it completed - that is what makes a
 * crash inside a step recoverable:
 *
 *   - `archive`     - about to write the accepted archive
 *   - `materialize` - about to write the procedure
 *   - `commit`      - about to remove the pending copy, then reproject
 *
 * `acceptedExisted` / `materializedExisted` are captured before the
 * sequence starts so a rollback never deletes a file the sequence did not
 * create.
 *
 * The materialize step writes ONE artifact, but not always the same kind
 * of one: a mined or declared proposal materializes a procedure inside the
 * vault, and a `mature_page` proposal materializes a `SKILL.md` under the
 * configured skills root, which may sit OUTSIDE the vault. So the journal
 * records the absolute path the step is about to write
 * ({@link SkillAcceptJournalEntry.materializedPath}) rather than leaving
 * the resolver to re-derive it from the slug - a rollback that guessed the
 * wrong tree would leave the real artifact behind and claim it had not.
 * The field is optional: a marker written before it existed names the
 * procedure path implicitly, and the resolver falls back to that.
 *
 * The same guess problem applies to the TREE the path sits in: a caller
 * (CLI flag, MCP server config) can resolve the skills root differently
 * from what recovery would resolve on its next run. So a skill
 * materialization also records the root its accept resolved
 * ({@link SkillAcceptJournalEntry.skillsRoot}); the field is optional and
 * absent on legacy markers and on procedure materializations, which never
 * consult the skills tree.
 *
 * The journal is a repair marker, not history: it is removed as soon as
 * the sequence completes or is resolved. A journal left on disk means a
 * sequence is still outstanding.
 *
 * The marker file itself is agent-writable, and its `materializedPath`
 * drives the rollback's deletion and the restore's rewrite, so the parser
 * refuses (security audit M4, the deletion sink) a `materialized_path` the
 * accept sequence cannot have produced on a marker whose target did not
 * exist - the shape the rollback's `rmSync` reads with no at-use guard in
 * front of it: a relative value (the module writes only absolute paths), or
 * an absolute path resolving outside the vault and the skills root the
 * materialize step used - the root the marker itself records when present,
 * else the default resolution (a legacy marker names no root). On a legacy
 * marker, the one shape left outside both roots stays admissible - a
 * `mature_page` materialization under a caller's override skills root,
 * which the marker cannot name - recognized as an absolute path ending in
 * the skill page filename. On a marker whose target DID exist, the path
 * reaches only the restore branch, which validates its own target at use;
 * there the parser stays permissive so diagnosis can still read the marker.
 * A refused marker is unreadable data, exactly like a corrupt one: named,
 * retained, and blocking every accept until an operator removes it. A
 * hand-edited journal is not a recovery input.
 *
 * A marker that cannot be READ is a third state, and it used to be a dead
 * end: the sweep runs before every accept, so one unparseable file blocked
 * the verb permanently and the refusal named no way out. There are now two
 * readers - {@link readSkillAcceptJournals}, which reports unreadable
 * markers as data for the diagnosis surface, and
 * {@link listSkillAcceptJournals}, which still refuses, now by name and
 * naming the command that clears them.
 */

import { existsSync, mkdirSync, readdirSync, readFileSync, rmSync } from "node:fs";
import { basename, dirname, isAbsolute, join, resolve, sep } from "node:path";

import { atomicWriteFileSync } from "../fs-atomic.ts";
import { machineStatePath, machineStateRoot } from "../state/machine-state.ts";

import { resolveSkillsDir } from "../config.ts";
import { ensureInsideVault } from "../path-safety.ts";
import { SKILL_FILE_NAME } from "../surface/skills.ts";
import { requireNextStep } from "./next-step.ts";
import {
  BRAIN_SKILL_ACCEPT_JOURNAL_REL,
  skillAcceptJournalPath,
  skillAcceptPriorContentPath,
} from "./paths.ts";
import { assertLaneWrite } from "../store/lane-pages.ts";
import { defaultSkillsDir } from "../surface/skills-dir.ts";

/** Step the accept sequence was about to perform when the journal was stamped. */
export type SkillAcceptPhase = "archive" | "materialize" | "commit";

export interface SkillAcceptJournalEntry {
  readonly slug: string;
  readonly id: string;
  readonly phase: SkillAcceptPhase;
  readonly startedAt: string;
  /** The accepted archive was already on disk before the sequence started. */
  readonly acceptedExisted: boolean;
  /** The materialize target was already on disk before the sequence started. */
  readonly materializedExisted: boolean;
  /**
   * Absolute path the materialize step writes. Absent on a marker written
   * before this field existed, where the target is the procedure path the
   * slug derives.
   */
  readonly materializedPath?: string;
  /**
   * The skills root the accept sequence resolved for a skill
   * materialization: the caller's override, the configured `skills_dir`,
   * or the vault's own skills subtree. Recovery and the parse-time guard
   * check the journal's materialized path against this root rather than
   * re-deriving one. Absent on legacy markers and on procedure
   * materializations, which never consult the skills tree; both fall back
   * to the default resolved root.
   */
  readonly skillsRoot?: string;
  /**
   * Vault-side copy of the incumbent bytes, captured before a scored
   * rewrite's materialize step OVERWRITES an existing SKILL.md. Present
   * only when the sequence was about to displace content; the rollback
   * restores the incumbent BODY from it, not just its existence. Absent
   * on markers written before rewrites existed, and on sequences whose
   * materialize target did not exist.
   */
  readonly priorContentPath?: string;
}

const JOURNAL_SCHEMA_VERSION = 1;
const JOURNAL_SUFFIX = ".json";
const PHASES: ReadonlySet<string> = new Set<SkillAcceptPhase>(["archive", "materialize", "commit"]);

/** Stamp (or re-stamp) the journal for `entry.slug`. Overwrites by design. */
export function writeSkillAcceptJournal(vault: string, entry: SkillAcceptJournalEntry): void {
  // Vault-identity write guard (context-integrity-gates, Unit J).
  assertLaneWrite(vault);
  const path = skillAcceptJournalPath(vault, entry.slug);
  const payload = {
    schema_version: JOURNAL_SCHEMA_VERSION,
    slug: entry.slug,
    id: entry.id,
    phase: entry.phase,
    started_at: entry.startedAt,
    accepted_existed: entry.acceptedExisted,
    // Written under the historical key so a marker this build leaves behind
    // is still readable by the build that shipped the key.
    procedure_existed: entry.materializedExisted,
    ...(entry.materializedPath !== undefined ? { materialized_path: entry.materializedPath } : {}),
    ...(entry.skillsRoot !== undefined ? { skills_root: entry.skillsRoot } : {}),
    ...(entry.priorContentPath !== undefined ? { prior_content_path: entry.priorContentPath } : {}),
  };
  // A device-local repair marker (machine state), a plain file in both
  // modes: never a lane record, which would sync and outlive its removal.
  mkdirSync(dirname(path), { recursive: true });
  atomicWriteFileSync(path, `${JSON.stringify(payload, null, 2)}\n`);
}

/**
 * Remove one marker file by path. Idempotent, vault-guarded and confined
 * to the vault. Exists for the unreadable case, where the slug cannot be
 * trusted to come from the file's own contents.
 */
export function removeSkillAcceptJournalFile(vault: string, path: string): void {
  // Vault-identity write guard (context-integrity-gates, Unit J).
  assertLaneWrite(vault);
  // Confined to where the journals live: the vault on an unswitched
  // machine, the device-local state once switched.
  rmSync(ensureInsideVault(path, machineStateRoot(vault)), { force: true });
}

/**
 * Remove the journal for `slug`, and the prior-content copy beside it
 * when a rewrite captured one. Idempotent: both removals are forced, so
 * clearing a journal that never captured anything is not an error.
 */
export function clearSkillAcceptJournal(vault: string, slug: string): void {
  // Vault-identity write guard (context-integrity-gates, Unit J).
  assertLaneWrite(vault);
  rmSync(skillAcceptJournalPath(vault, slug), { force: true });
  rmSync(skillAcceptPriorContentPath(vault, slug), { force: true });
}

/** Registry code for a marker file this module cannot parse. */
export const SKILL_ACCEPT_JOURNAL_UNREADABLE_CODE = "skill-accept-journal-unreadable";

/**
 * Resolved once, at module scope, so a registry that lost the entry fails
 * at import rather than inside the very refusal it is meant to explain.
 */
const UNREADABLE_EXIT = requireNextStep(SKILL_ACCEPT_JOURNAL_UNREADABLE_CODE).nextCommand;

/** One marker file on disk that cannot be parsed back into an entry. */
export interface UnreadableSkillAcceptJournal {
  readonly path: string;
  /** Why it could not be read - a parse failure or a missing field. */
  readonly detail: string;
}

/** Outstanding markers, split into the ones that read and the ones that do not. */
export interface SkillAcceptJournalState {
  readonly entries: ReadonlyArray<SkillAcceptJournalEntry>;
  readonly unreadable: ReadonlyArray<UnreadableSkillAcceptJournal>;
}

/**
 * A marker file that cannot be parsed back into a complete entry.
 *
 * The write is atomic, so this shape never comes from a torn write of
 * ours - it arrives by hand edit, a sync merge or media damage, and it
 * blocks every accept until it is gone. The message therefore names both
 * the file and the command that removes it; without that the operator's
 * only route was reading this module.
 */
export class SkillAcceptJournalUnreadableError extends Error {
  readonly path: string;
  readonly detail: string;
  constructor(entry: UnreadableSkillAcceptJournal) {
    super(
      `skill accept journal is unreadable: ${entry.path} (${entry.detail}). ` +
        `Every accept is blocked until it is resolved: ${UNREADABLE_EXIT}`,
    );
    this.name = "SkillAcceptJournalUnreadableError";
    this.path = entry.path;
    this.detail = entry.detail;
  }
}

/**
 * Every outstanding journal, slug-ordered, partitioned by whether it
 * parses.
 *
 * This is the DIAGNOSIS surface: it reports an unreadable marker as data
 * so an operator can be shown what is wrong and act on it. The transaction
 * paths use {@link listSkillAcceptJournals} instead, which refuses - a
 * writer must never mistake an unreadable marker for "no outstanding work".
 */
export function readSkillAcceptJournals(vault: string): SkillAcceptJournalState {
  // Where the writers put them: machine state (the vault unswitched, the
  // device-local state once switched).
  const dir = machineStatePath(vault, BRAIN_SKILL_ACCEPT_JOURNAL_REL);
  if (!existsSync(dir)) {
    return Object.freeze({ entries: Object.freeze([]), unreadable: Object.freeze([]) });
  }
  // Resolved once per read: the DEFAULT skills root, used by the
  // parse-time materialized-path guard when a marker records no root of
  // its own (legacy markers, procedure materializations).
  const defaultSkillsRoot = resolveJournalSkillsRoot(vault);
  const entries: SkillAcceptJournalEntry[] = [];
  const unreadable: UnreadableSkillAcceptJournal[] = [];
  for (const name of readdirSync(dir).toSorted()) {
    if (!name.endsWith(JOURNAL_SUFFIX)) continue;
    const path = join(dir, name);
    const parsed = parseJournal(path, vault, defaultSkillsRoot);
    if ("detail" in parsed) unreadable.push(parsed);
    else entries.push(parsed);
  }
  return Object.freeze({ entries: Object.freeze(entries), unreadable: Object.freeze(unreadable) });
}

/**
 * Every outstanding journal, slug-ordered. A file this module cannot parse
 * back into a complete entry is a defect, not a soft skip: it is reported
 * by throwing {@link SkillAcceptJournalUnreadableError} so the caller
 * cannot mistake an unreadable marker for "no outstanding work".
 */
export function listSkillAcceptJournals(vault: string): ReadonlyArray<SkillAcceptJournalEntry> {
  const state = readSkillAcceptJournals(vault);
  const first = state.unreadable[0];
  if (first !== undefined) throw new SkillAcceptJournalUnreadableError(first);
  return state.entries;
}

function parseJournal(
  path: string,
  vault: string,
  defaultSkillsRoot: string,
): SkillAcceptJournalEntry | UnreadableSkillAcceptJournal {
  let raw: Record<string, unknown>;
  try {
    raw = JSON.parse(readFileSync(path, "utf8")) as Record<string, unknown>;
  } catch (cause) {
    return Object.freeze({
      path,
      detail: cause instanceof Error ? cause.message : String(cause),
    });
  }
  const slug = raw["slug"];
  const id = raw["id"];
  const phase = raw["phase"];
  const startedAt = raw["started_at"];
  if (
    typeof slug !== "string" ||
    typeof id !== "string" ||
    typeof phase !== "string" ||
    !PHASES.has(phase) ||
    typeof startedAt !== "string"
  ) {
    return Object.freeze({ path, detail: "a required field is missing or of the wrong type" });
  }
  const materializedPath = raw["materialized_path"];
  const priorContentPath = raw["prior_content_path"];
  const skillsRoot = raw["skills_root"];
  const recordedSkillsRoot =
    typeof skillsRoot === "string" && skillsRoot.length > 0 ? skillsRoot : undefined;
  const materializedExisted = raw["procedure_existed"] === true;
  // The rollback's rmSync is the one sink a hand-edited path reaches with
  // no at-use guard in front of it. The restore branch validates its own
  // target (security audit M4), and once the target existed mid-sequence
  // the path feeds nothing else - so the parse boundary refuses an
  // unproducible path exactly where the deletion sink reads it, against
  // the skills root the marker itself records when present, else the
  // default resolution.
  if (!materializedExisted && typeof materializedPath === "string" && materializedPath.length > 0) {
    const refusal = refusedMaterializedPathDetail(
      materializedPath,
      vault,
      recordedSkillsRoot ?? defaultSkillsRoot,
      recordedSkillsRoot !== undefined,
    );
    if (refusal !== null) return Object.freeze({ path, detail: refusal });
  }
  return Object.freeze({
    slug,
    id,
    phase: phase as SkillAcceptPhase,
    startedAt,
    acceptedExisted: raw["accepted_existed"] === true,
    materializedExisted,
    ...(typeof materializedPath === "string" && materializedPath.length > 0
      ? { materializedPath }
      : {}),
    ...(recordedSkillsRoot !== undefined ? { skillsRoot: recordedSkillsRoot } : {}),
    ...(typeof priorContentPath === "string" && priorContentPath.length > 0
      ? { priorContentPath }
      : {}),
  });
}

/**
 * The DEFAULT skills root, used by the parse-time guard when a marker
 * records no root of its own: the configured override, else the vault's
 * own skills subtree. The same resolution a legacy accept performed, so
 * the guard admits exactly the paths such an accept can write.
 */
function resolveJournalSkillsRoot(vault: string): string {
  return resolveSkillsDir() ?? defaultSkillsDir(vault);
}

/**
 * The parse-time refusal for a journal-supplied `materialized_path` on a
 * marker whose target did not exist (the rollback's deletion shape), or
 * null when the accept sequence itself could have written it.
 *
 * `skillsRoot` is the root the marker itself recorded when it names one,
 * else the default resolution; `skillsRootRecorded` says which. A value
 * the module cannot have produced is refused at the boundary both readers
 * share, before any recovery touches the filesystem: a relative value (the
 * module writes only absolute paths), or an absolute path resolving
 * outside the vault and that root - except on a LEGACY marker, where the
 * one shape recovery cannot re-derive stays admissible: a `mature_page`
 * materialization under a caller's override skills root the marker cannot
 * name, recognized as an absolute path ending in the skill page filename
 * (this also keeps legacy markers readable across a skills-dir config
 * change between accept and recovery). A marker that records its root
 * needs no such escape: its recorded root bounds the materialize tree, and
 * only the module's own shape inside it is admitted.
 */
function refusedMaterializedPathDetail(
  value: string,
  vault: string,
  skillsRoot: string,
  skillsRootRecorded: boolean,
): string | null {
  if (!isAbsolute(value)) {
    return (
      `materialized_path '${value}' is not an absolute path; the accept sequence writes only ` +
      "absolute paths, so this marker was hand-edited and is not a recovery input"
    );
  }
  if (isLexicallyInside(value, vault)) return null;
  if (isLexicallyInside(value, skillsRoot)) {
    if (!skillsRootRecorded) return null;
    if (basename(value) === SKILL_FILE_NAME) return null;
  } else if (!skillsRootRecorded && basename(value) === SKILL_FILE_NAME) {
    return null;
  }
  return skillsRootRecorded
    ? `materialized_path '${value}' resolves outside the vault and the skills root the accept ` +
        `recorded (${skillsRoot}) and is not a materialized ${SKILL_FILE_NAME} path; ` +
        "a hand-edited journal is not a recovery input"
    : `materialized_path '${value}' resolves outside the vault and the skills root ` +
        `(${skillsRoot}) and is not a materialized ${SKILL_FILE_NAME} path; a hand-edited ` +
        "journal is not a recovery input";
}

/**
 * Lexical containment of `target` inside `root`, mirroring the rollback
 * confinement predicate in `skill-proposals` (`isLexicallyInside` there):
 * `path.resolve` normalisation plus the platform separator, so the
 * parse-time guard and the at-use guard admit the same paths.
 */
function isLexicallyInside(target: string, root: string): boolean {
  const resolvedTarget = resolve(target);
  const resolvedRoot = resolve(root) + sep;
  return resolvedTarget.startsWith(resolvedRoot);
}