Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript
17.1 KB
Raw Blame
/**
 * The one call every export path makes before its bytes leave the vault.
 *
 * ## Why truncation is a REFUSAL here and not elsewhere
 *
 * `redactRawOutput` fails closed on oversized input: it scans the prefix
 * that fits the window, drops the tail, and stamps
 * `SCAN_TRUNCATED_MARKER` so the artifact reads as unverified rather than
 * clean. Two consumers already answer that differently, and both are
 * right for where they sit.
 *
 * `src/mcp/artifact-store.ts` disables the window entirely
 * (`maxInput: Number.POSITIVE_INFINITY`). Its whole job is to preserve a
 * full tool payload for a later fetch, its destination is a dot-directory
 * INSIDE the vault, and truncating would silently clip a payload the
 * agent is about to ask for by id. Nothing crosses a trust boundary, so
 * scanning everything and keeping everything is the honest answer.
 *
 * An export is the opposite case on both axes. The bytes leave the
 * machine, and the operator cannot inspect what they no longer have. So
 * this guard neither truncates silently nor scans without bound: it keeps
 * the window - an unbounded scan over an operator-supplied payload is the
 * denial-of-service bound the window exists for - and turns the marker
 * into a refusal. Writing a file that says "partially scanned" at the
 * bottom is the misleading success this project forbids: the export
 * either scanned its payload or it did not go.
 *
 * The refusal is per PAYLOAD, not per export, because the scan runs over
 * each string leaf rather than the serialised document. A whole-vault
 * bundle far larger than the window still exports; a single field larger
 * than the window is what refuses, which is a field no legitimate note
 * carries.
 *
 * ## What it redacts, and what it does not
 *
 * `redactTokens` is ON. At an egress boundary the asymmetry inverts: a
 * false positive mangles an id in a copy while the vault still holds the
 * original, and a false negative is unrecoverable because the bytes are
 * gone. That is the reverse of the receipt boundary the redactor's
 * defaults are tuned for, which is why the flag is set here and not
 * there.
 *
 * `redactInfra` is OFF, and only its credential half
 * (`redactUrlCredentials`) is taken. A `user:pass@host` URL is a
 * credential; a `host:port` or a public IP in an authored note is a
 * reference a portable knowledge bundle has to keep. Redacting those
 * would break the interchange contract for a class of exposure the
 * operator already controls by choosing the destination. That gap is
 * stated here rather than left to be discovered.
 *
 * ## Identifiers are refused, never rewritten
 *
 * Redaction replaces a value. That is harmless for a payload - the vault
 * still holds the original - and destructive for an IDENTIFIER: a redacted
 * filename merges every page whose name matched onto one bundle path, a
 * redacted mapping key renames the field, and a redacted path segment
 * hands a support person a path that does not exist. So `redactStructured`
 * reports secret-shaped identifiers instead of replacing them, and this
 * guard turns that report into the second refusal
 * ({@link EGRESS_OUTCOME.refusedSecretIdentifier}), on the same reasoning
 * as the truncation one: an export that cannot be made safe does not go.
 *
 * The cost is a false refusal on a legitimately-named file that begins
 * with a vendor credential prefix. That is recoverable by renaming; a
 * merged page is not.
 */

import { MAX_REDACTOR_INPUT, redactStructured, type RedactRawOutputOptions } from "../redactor.ts";
import { sortedDistinctLiterals } from "../secret-ref.ts";
import { EGRESS_SITES, type EgressSiteId } from "./registry.ts";

/** How a payload offered to the boundary came back. */
export const EGRESS_OUTCOME = Object.freeze({
  /** Scanned in full and cleared to be written. */
  released: "released",
  /** Some field exceeded the scan window, so the payload is unverified. */
  refusedScanTruncated: "refused_scan_truncated",
  /**
   * An identifier - a filename, an id, a mapping key - is secret-shaped.
   * Redacting it is not available (it would merge the pages a path names,
   * or rename a field), so the payload is refused instead.
   */
  refusedSecretIdentifier: "refused_secret_identifier",
} as const);

export type EgressOutcome = (typeof EGRESS_OUTCOME)[keyof typeof EGRESS_OUTCOME];

export const EGRESS_OUTCOMES: ReadonlyArray<EgressOutcome> = Object.freeze([
  EGRESS_OUTCOME.released,
  EGRESS_OUTCOME.refusedScanTruncated,
  EGRESS_OUTCOME.refusedSecretIdentifier,
]);

/** Takes `unknown`: an outcome may be read back out of a serialised result. */
export function isEgressOutcome(value: unknown): value is EgressOutcome {
  return typeof value === "string" && (EGRESS_OUTCOMES as ReadonlyArray<string>).includes(value);
}

export interface EgressReleased<T> {
  readonly outcome: typeof EGRESS_OUTCOME.released;
  /** The redacted payload, same shape as the input. */
  readonly payload: T;
  /** True when anything changed, so the verb can say the copy differs. */
  readonly redacted: boolean;
}

export interface EgressRefused {
  readonly outcome:
    | typeof EGRESS_OUTCOME.refusedScanTruncated
    | typeof EGRESS_OUTCOME.refusedSecretIdentifier;
  /** Operator-facing sentence naming the verb and what to do about it. */
  readonly detail: string;
  /**
   * The tree locations the structural scan reported as secret-shaped
   * identifiers, already inside {@link detail} and carried structurally as
   * well. Empty on the truncation arm.
   *
   * A caller composing its own prefix around `detail` has to know WHICH
   * identifier was refused before it can decide what is safe to say. The
   * transcript export is the case: it named each refused record by its
   * `session_id`, which is the transcript's basename - so when the
   * basename was itself the secret-shaped identifier, the refusal printed
   * the exact value it had just declined to write. Reading the list is
   * what lets that caller withhold the name on that one branch and keep
   * printing it on every other.
   */
  readonly secretIdentifiers: ReadonlyArray<string>;
}

/**
 * Released or refused. A union rather than a nullable payload, so a
 * caller cannot write a refusal's empty text by forgetting a check.
 */
export type EgressVerdict<T> = EgressReleased<T> | EgressRefused;

/**
 * The redaction policy for every path that writes outside the vault. One
 * constant, because six paths with six policies is the drift this unit
 * exists to remove.
 *
 * Exported for the one in-vault writer that needs the same grade without
 * an egress site: the continuity promotion re-scans ledger text before it
 * becomes an indexed page, and a second policy spelled next to this one
 * would drift exactly the way the shared constant exists to prevent.
 */
export const EGRESS_REDACTION_OPTIONS: RedactRawOutputOptions = Object.freeze({
  redactTokens: true,
  redactUrlCredentials: true,
});

/** Bytes, rendered for an operator who has to act on the number. */
const SCAN_WINDOW_MIB = MAX_REDACTOR_INPUT / (1024 * 1024);

/**
 * Emitted on stderr by every verb whose payload the guard changed, so
 * stdout stays byte-identical and parseable. An export that quietly
 * differs from the vault it came from is a surprise the operator finds
 * later, in the copy they already shared.
 */
export const EGRESS_REDACTION_NOTICE =
  "note: secret-shaped content was redacted; this export no longer matches the vault byte for byte\n";

/**
 * Emitted on stderr by a verb run with the content-class override, so an
 * operator who passes `--include-prompt-derived` is told, on the run,
 * that the copy carries material the default leaves out. Like the
 * redaction notice it goes to stderr, so stdout stays parseable.
 */
export const EGRESS_CONTENT_CLASS_INCLUDE_NOTICE =
  "note: prompt-derived content is INCLUDED by override; this export carries material the default leaves out\n";

/**
 * The sentence naming every item the default content-class policy left
 * out of an export, in the shape the knowledge pack's `blocked` list
 * established: each omission named, with the reason beside it. An
 * omission that is not named is a silently smaller export - the exact
 * dishonesty the class policy exists to remove - so every verb that
 * omits writes this line to stderr.
 */
export function contentClassOmissionNotice(omitted: ReadonlyArray<string>): string {
  const list = omitted.map((name) => `${name} (content_class: prompt-derived)`).join(", ");
  return (
    `note: ${omitted.length} item(s) left out as prompt-derived: ${list}; ` +
    "pass --include-prompt-derived to export them\n"
  );
}

/**
 * The operator-overridable half of the egress policy: what happens to
 * pages whose `content_class` frontmatter carries the reserved
 * `prompt-derived` value.
 *
 * The DEFAULT is exclusion, and it is not represented as a field here -
 * an absent `contentClass` means the default applies, the same way an
 * absent `foreignIdentifiers` means the default identifier rule does.
 * Setting `include` is the per-verb operator override: the flag is
 * spelled `--include-prompt-derived` on every export verb that offers
 * it, and a run that sets it emits
 * {@link EGRESS_CONTENT_CLASS_INCLUDE_NOTICE} on stderr.
 */
export interface EgressContentClassPolicy {
  /** Carry class-marked items; the default leaves them out and names them. */
  readonly include?: boolean;
}

/**
 * What the default content-class policy did to one collection of items a
 * payload was composed from. `kept` is what the payload should carry;
 * `omitted` holds each excluded item's operator-facing name, ready for
 * {@link contentClassOmissionNotice}. Under the override the split is the
 * identity and `omitted` is empty - an override run has nothing to name.
 */
export interface EgressContentClassSplit<T> {
  readonly kept: ReadonlyArray<T>;
  readonly omitted: ReadonlyArray<string>;
}

/**
 * Apply the content-class policy to the items a shared export's payload
 * is composed from. This is the exclusion seam: every export that
 * carries pages splits its page collection through here BEFORE the
 * payload is composed and redacted, so a class-marked page is not in the
 * bytes the guard scans rather than redacted inside them. The census
 * (`egress-census.test.ts`) pins the page-carrying sites to this call by
 * name, in both directions.
 *
 * The predicate comes from the caller because the guard is shape-blind
 * by design - it cannot know whether a payload's items are pages, nodes
 * or contracts, and pretending otherwise would put a vault walk inside
 * the one module that stays pure of the filesystem.
 */
export function egressContentClassSplit<T>(
  items: ReadonlyArray<T>,
  policy: EgressPolicy,
  item: {
    /** The operator-facing name an omission is disclosed under. */
    readonly name: (entry: T) => string;
    /** Whether this item carries the reserved class. */
    readonly isPromptDerived: (entry: T) => boolean;
  },
): EgressContentClassSplit<T> {
  if (policy.contentClass?.include === true) return { kept: [...items], omitted: [] };
  const kept: T[] = [];
  const omitted: string[] = [];
  for (const entry of items) {
    if (item.isPromptDerived(entry)) omitted.push(item.name(entry));
    else kept.push(entry);
  }
  return { kept, omitted };
}

/**
 * What a call site may vary about the one policy above.
 *
 * Deliberately not a second policy constant: the redaction options stay
 * single, and this names the one decision that cannot be answered once for
 * every site because it depends on WHERE the payload's identifiers were
 * named rather than on where they are going.
 */
export interface EgressPolicy {
  /**
   * The payload's identifiers were named outside this vault, so judge them
   * with the full bare-token detector rather than vendor prefixes alone.
   * Set by the transcript corpus and nothing else - see
   * `foreignIdentifierCarriesSecret` in the redactor for why the default
   * is right for every vault-authored export.
   */
  readonly foreignIdentifiers?: boolean;
  /**
   * Values already resolved from `$secret:` references at a use site
   * (the named-secret resolver). The structural passes judge SHAPES - a
   * key name, a vendor prefix - and a resolved credential under a quiet
   * key name carries neither, so the boundary needs the values themselves.
   * Each occurrence is replaced with the standard marker, longest value
   * first so a value containing another cannot have its match destroyed
   * and leak its head as a fragment, and before the truncation guard so
   * no cut leaves a partial credential in the kept prefix. Absent (the
   * default) the composition is byte-identical to the pre-literal guard.
   *
   * The wired boundaries are the ones that KNOW the values: the MCP
   * error redaction and the config-mapping status surfaces feed
   * `resolvedSecretLiterals(vault)` for the vault they serve, and the
   * decision-model adapters pass the key they hold. A boundary that
   * cannot know the vault must not pretend to redact: for those the
   * option stays absent, which is the named decision, not an omission.
   */
  readonly resolvedLiterals?: ReadonlyArray<string>;
  /**
   * What happens to pages carrying the reserved `prompt-derived` content
   * class. Absent, the default applies: such pages are left out of the
   * payload a shared export composes - split out at
   * {@link egressContentClassSplit} BEFORE composition, so they are
   * absent rather than redacted - and every omission is named on stderr.
   * `include: true` is the per-verb `--include-prompt-derived` override
   * and turns the split into the identity, with
   * {@link EGRESS_CONTENT_CLASS_INCLUDE_NOTICE} on stderr.
   */
  readonly contentClass?: EgressContentClassPolicy;
}

/**
 * The one policy plus what the call site may vary. The literals pass
 * through the shared substitution-safety ordering
 * ({@link sortedDistinctLiterals}); the redactor applies them verbatim
 * ahead of every pattern pass.
 */
function composeRedactionOptions(policy: EgressPolicy): RedactRawOutputOptions {
  return {
    ...EGRESS_REDACTION_OPTIONS,
    ...(policy.foreignIdentifiers === true ? { foreignIdentifiers: true } : {}),
    ...(policy.resolvedLiterals === undefined
      ? {}
      : { literals: sortedDistinctLiterals(policy.resolvedLiterals) }),
  };
}

/**
 * Redact `payload` for `site`, or refuse it. The payload may be a string
 * or any JSON-shaped tree; see `redactStructured` for why the tree form
 * exists rather than a pass over serialised bytes.
 */
export function redactForEgress<T>(
  site: EgressSiteId,
  payload: T,
  policy: EgressPolicy = {},
): EgressVerdict<T> {
  const scanned = redactStructured(payload, composeRedactionOptions(policy));
  if (scanned.truncated) {
    return {
      outcome: EGRESS_OUTCOME.refusedScanTruncated,
      secretIdentifiers: Object.freeze([]),
      detail:
        `${EGRESS_SITES[site].verb} refused to write: a field exceeded the redactor ` +
        `scan window (${SCAN_WINDOW_MIB} MiB), so only part of it was scanned and this ` +
        "export cannot be reported as clean. Nothing was written. Shrink the oversized " +
        "field or export a narrower selection.",
    };
  }
  if (scanned.secretIdentifiers.length > 0) {
    return {
      outcome: EGRESS_OUTCOME.refusedSecretIdentifier,
      secretIdentifiers: scanned.secretIdentifiers,
      detail:
        `${EGRESS_SITES[site].verb} refused to write: an identifier is secret-shaped, and an ` +
        "identifier is never rewritten - a redacted filename merges the pages it names and a " +
        "redacted mapping key renames the field, so redacting here would destroy content " +
        "rather than protect it. Nothing was written. Rename or exclude what these tree " +
        `locations point at, then re-run: ${scanned.secretIdentifiers.join(", ")}. ` +
        "(Locations, not values: the identifier is the secret.)",
    };
  }
  return {
    outcome: EGRESS_OUTCOME.released,
    payload: scanned.value as T,
    redacted: scanned.redacted,
  };
}

/**
 * Redact a configuration mapping for a surface outside the vault - the
 * config snapshot in `export-config`, `second_brain_status` over MCP, and
 * the OpenClaw status tool.
 *
 * This is what `config.ts`'s `redactMapping` became. That copy matched
 * five substrings against key NAMES only and never inspected a value, so
 * a credential under a name it did not recognise passed straight through.
 * Both halves now come from the shared redactor.
 *
 * No refusal arm: a status tool's answer is not a file leaving the
 * machine, and a config value past the scan window carries the marker in
 * place, which is visible rather than silent. For the same reason it does
 * not act on `secretIdentifiers`: the resolved PATHS are what this surface
 * exists to report, they are returned verbatim, and a credential stored as
 * a directory name is left visible rather than turned into a status call
 * that refuses to answer. Path values reach the caller unchanged, which is
 * the point - the previous behaviour replaced any 24-character segment of
 * the operator's own path with a placeholder.
 */
export function redactConfigMapping(
  data: Readonly<Record<string, unknown>>,
  policy: EgressPolicy = {},
): Record<string, unknown> {
  return redactStructured(data, composeRedactionOptions(policy)).value as Record<string, unknown>;
}