Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
/**
* The one call every export path makes before its bytes leave the vault.
*
* ## Why truncation is a REFUSAL here and not elsewhere
*
* `redactRawOutput` fails closed on oversized input: it scans the prefix
* that fits the window, drops the tail, and stamps
* `SCAN_TRUNCATED_MARKER` so the artifact reads as unverified rather than
* clean. Two consumers already answer that differently, and both are
* right for where they sit.
*
* `src/mcp/artifact-store.ts` disables the window entirely
* (`maxInput: Number.POSITIVE_INFINITY`). Its whole job is to preserve a
* full tool payload for a later fetch, its destination is a dot-directory
* INSIDE the vault, and truncating would silently clip a payload the
* agent is about to ask for by id. Nothing crosses a trust boundary, so
* scanning everything and keeping everything is the honest answer.
*
* An export is the opposite case on both axes. The bytes leave the
* machine, and the operator cannot inspect what they no longer have. So
* this guard neither truncates silently nor scans without bound: it keeps
* the window - an unbounded scan over an operator-supplied payload is the
* denial-of-service bound the window exists for - and turns the marker
* into a refusal. Writing a file that says "partially scanned" at the
* bottom is the misleading success this project forbids: the export
* either scanned its payload or it did not go.
*
* The refusal is per PAYLOAD, not per export, because the scan runs over
* each string leaf rather than the serialised document. A whole-vault
* bundle far larger than the window still exports; a single field larger
* than the window is what refuses, which is a field no legitimate note
* carries.
*
* ## What it redacts, and what it does not
*
* `redactTokens` is ON. At an egress boundary the asymmetry inverts: a
* false positive mangles an id in a copy while the vault still holds the
* original, and a false negative is unrecoverable because the bytes are
* gone. That is the reverse of the receipt boundary the redactor's
* defaults are tuned for, which is why the flag is set here and not
* there.
*
* `redactInfra` is OFF, and only its credential half
* (`redactUrlCredentials`) is taken. A `user:pass@host` URL is a
* credential; a `host:port` or a public IP in an authored note is a
* reference a portable knowledge bundle has to keep. Redacting those
* would break the interchange contract for a class of exposure the
* operator already controls by choosing the destination. That gap is
* stated here rather than left to be discovered.
*
* ## Identifiers are refused, never rewritten
*
* Redaction replaces a value. That is harmless for a payload - the vault
* still holds the original - and destructive for an IDENTIFIER: a redacted
* filename merges every page whose name matched onto one bundle path, a
* redacted mapping key renames the field, and a redacted path segment
* hands a support person a path that does not exist. So `redactStructured`
* reports secret-shaped identifiers instead of replacing them, and this
* guard turns that report into the second refusal
* ({@link EGRESS_OUTCOME.refusedSecretIdentifier}), on the same reasoning
* as the truncation one: an export that cannot be made safe does not go.
*
* The cost is a false refusal on a legitimately-named file that begins
* with a vendor credential prefix. That is recoverable by renaming; a
* merged page is not.
*/
import { MAX_REDACTOR_INPUT, redactStructured, type RedactRawOutputOptions } from "../redactor.ts";
import { sortedDistinctLiterals } from "../secret-ref.ts";
import { EGRESS_SITES, type EgressSiteId } from "./registry.ts";
/** How a payload offered to the boundary came back. */
export const EGRESS_OUTCOME = Object.freeze({
/** Scanned in full and cleared to be written. */
released: "released",
/** Some field exceeded the scan window, so the payload is unverified. */
refusedScanTruncated: "refused_scan_truncated",
/**
* An identifier - a filename, an id, a mapping key - is secret-shaped.
* Redacting it is not available (it would merge the pages a path names,
* or rename a field), so the payload is refused instead.
*/
refusedSecretIdentifier: "refused_secret_identifier",
} as const);
export type EgressOutcome = (typeof EGRESS_OUTCOME)[keyof typeof EGRESS_OUTCOME];
export const EGRESS_OUTCOMES: ReadonlyArray<EgressOutcome> = Object.freeze([
EGRESS_OUTCOME.released,
EGRESS_OUTCOME.refusedScanTruncated,
EGRESS_OUTCOME.refusedSecretIdentifier,
]);
/** Takes `unknown`: an outcome may be read back out of a serialised result. */
export function isEgressOutcome(value: unknown): value is EgressOutcome {
return typeof value === "string" && (EGRESS_OUTCOMES as ReadonlyArray<string>).includes(value);
}
export interface EgressReleased<T> {
readonly outcome: typeof EGRESS_OUTCOME.released;
/** The redacted payload, same shape as the input. */
readonly payload: T;
/** True when anything changed, so the verb can say the copy differs. */
readonly redacted: boolean;
}
export interface EgressRefused {
readonly outcome:
| typeof EGRESS_OUTCOME.refusedScanTruncated
| typeof EGRESS_OUTCOME.refusedSecretIdentifier;
/** Operator-facing sentence naming the verb and what to do about it. */
readonly detail: string;
/**
* The tree locations the structural scan reported as secret-shaped
* identifiers, already inside {@link detail} and carried structurally as
* well. Empty on the truncation arm.
*
* A caller composing its own prefix around `detail` has to know WHICH
* identifier was refused before it can decide what is safe to say. The
* transcript export is the case: it named each refused record by its
* `session_id`, which is the transcript's basename - so when the
* basename was itself the secret-shaped identifier, the refusal printed
* the exact value it had just declined to write. Reading the list is
* what lets that caller withhold the name on that one branch and keep
* printing it on every other.
*/
readonly secretIdentifiers: ReadonlyArray<string>;
}
/**
* Released or refused. A union rather than a nullable payload, so a
* caller cannot write a refusal's empty text by forgetting a check.
*/
export type EgressVerdict<T> = EgressReleased<T> | EgressRefused;
/**
* The redaction policy for every path that writes outside the vault. One
* constant, because six paths with six policies is the drift this unit
* exists to remove.
*
* Exported for the one in-vault writer that needs the same grade without
* an egress site: the continuity promotion re-scans ledger text before it
* becomes an indexed page, and a second policy spelled next to this one
* would drift exactly the way the shared constant exists to prevent.
*/
export const EGRESS_REDACTION_OPTIONS: RedactRawOutputOptions = Object.freeze({
redactTokens: true,
redactUrlCredentials: true,
});
/** Bytes, rendered for an operator who has to act on the number. */
const SCAN_WINDOW_MIB = MAX_REDACTOR_INPUT / (1024 * 1024);
/**
* Emitted on stderr by every verb whose payload the guard changed, so
* stdout stays byte-identical and parseable. An export that quietly
* differs from the vault it came from is a surprise the operator finds
* later, in the copy they already shared.
*/
export const EGRESS_REDACTION_NOTICE =
"note: secret-shaped content was redacted; this export no longer matches the vault byte for byte\n";
/**
* Emitted on stderr by a verb run with the content-class override, so an
* operator who passes `--include-prompt-derived` is told, on the run,
* that the copy carries material the default leaves out. Like the
* redaction notice it goes to stderr, so stdout stays parseable.
*/
export const EGRESS_CONTENT_CLASS_INCLUDE_NOTICE =
"note: prompt-derived content is INCLUDED by override; this export carries material the default leaves out\n";
/**
* The sentence naming every item the default content-class policy left
* out of an export, in the shape the knowledge pack's `blocked` list
* established: each omission named, with the reason beside it. An
* omission that is not named is a silently smaller export - the exact
* dishonesty the class policy exists to remove - so every verb that
* omits writes this line to stderr.
*/
export function contentClassOmissionNotice(omitted: ReadonlyArray<string>): string {
const list = omitted.map((name) => `${name} (content_class: prompt-derived)`).join(", ");
return (
`note: ${omitted.length} item(s) left out as prompt-derived: ${list}; ` +
"pass --include-prompt-derived to export them\n"
);
}
/**
* The operator-overridable half of the egress policy: what happens to
* pages whose `content_class` frontmatter carries the reserved
* `prompt-derived` value.
*
* The DEFAULT is exclusion, and it is not represented as a field here -
* an absent `contentClass` means the default applies, the same way an
* absent `foreignIdentifiers` means the default identifier rule does.
* Setting `include` is the per-verb operator override: the flag is
* spelled `--include-prompt-derived` on every export verb that offers
* it, and a run that sets it emits
* {@link EGRESS_CONTENT_CLASS_INCLUDE_NOTICE} on stderr.
*/
export interface EgressContentClassPolicy {
/** Carry class-marked items; the default leaves them out and names them. */
readonly include?: boolean;
}
/**
* What the default content-class policy did to one collection of items a
* payload was composed from. `kept` is what the payload should carry;
* `omitted` holds each excluded item's operator-facing name, ready for
* {@link contentClassOmissionNotice}. Under the override the split is the
* identity and `omitted` is empty - an override run has nothing to name.
*/
export interface EgressContentClassSplit<T> {
readonly kept: ReadonlyArray<T>;
readonly omitted: ReadonlyArray<string>;
}
/**
* Apply the content-class policy to the items a shared export's payload
* is composed from. This is the exclusion seam: every export that
* carries pages splits its page collection through here BEFORE the
* payload is composed and redacted, so a class-marked page is not in the
* bytes the guard scans rather than redacted inside them. The census
* (`egress-census.test.ts`) pins the page-carrying sites to this call by
* name, in both directions.
*
* The predicate comes from the caller because the guard is shape-blind
* by design - it cannot know whether a payload's items are pages, nodes
* or contracts, and pretending otherwise would put a vault walk inside
* the one module that stays pure of the filesystem.
*/
export function egressContentClassSplit<T>(
items: ReadonlyArray<T>,
policy: EgressPolicy,
item: {
/** The operator-facing name an omission is disclosed under. */
readonly name: (entry: T) => string;
/** Whether this item carries the reserved class. */
readonly isPromptDerived: (entry: T) => boolean;
},
): EgressContentClassSplit<T> {
if (policy.contentClass?.include === true) return { kept: [...items], omitted: [] };
const kept: T[] = [];
const omitted: string[] = [];
for (const entry of items) {
if (item.isPromptDerived(entry)) omitted.push(item.name(entry));
else kept.push(entry);
}
return { kept, omitted };
}
/**
* What a call site may vary about the one policy above.
*
* Deliberately not a second policy constant: the redaction options stay
* single, and this names the one decision that cannot be answered once for
* every site because it depends on WHERE the payload's identifiers were
* named rather than on where they are going.
*/
export interface EgressPolicy {
/**
* The payload's identifiers were named outside this vault, so judge them
* with the full bare-token detector rather than vendor prefixes alone.
* Set by the transcript corpus and nothing else - see
* `foreignIdentifierCarriesSecret` in the redactor for why the default
* is right for every vault-authored export.
*/
readonly foreignIdentifiers?: boolean;
/**
* Values already resolved from `$secret:` references at a use site
* (the named-secret resolver). The structural passes judge SHAPES - a
* key name, a vendor prefix - and a resolved credential under a quiet
* key name carries neither, so the boundary needs the values themselves.
* Each occurrence is replaced with the standard marker, longest value
* first so a value containing another cannot have its match destroyed
* and leak its head as a fragment, and before the truncation guard so
* no cut leaves a partial credential in the kept prefix. Absent (the
* default) the composition is byte-identical to the pre-literal guard.
*
* The wired boundaries are the ones that KNOW the values: the MCP
* error redaction and the config-mapping status surfaces feed
* `resolvedSecretLiterals(vault)` for the vault they serve, and the
* decision-model adapters pass the key they hold. A boundary that
* cannot know the vault must not pretend to redact: for those the
* option stays absent, which is the named decision, not an omission.
*/
readonly resolvedLiterals?: ReadonlyArray<string>;
/**
* What happens to pages carrying the reserved `prompt-derived` content
* class. Absent, the default applies: such pages are left out of the
* payload a shared export composes - split out at
* {@link egressContentClassSplit} BEFORE composition, so they are
* absent rather than redacted - and every omission is named on stderr.
* `include: true` is the per-verb `--include-prompt-derived` override
* and turns the split into the identity, with
* {@link EGRESS_CONTENT_CLASS_INCLUDE_NOTICE} on stderr.
*/
readonly contentClass?: EgressContentClassPolicy;
}
/**
* The one policy plus what the call site may vary. The literals pass
* through the shared substitution-safety ordering
* ({@link sortedDistinctLiterals}); the redactor applies them verbatim
* ahead of every pattern pass.
*/
function composeRedactionOptions(policy: EgressPolicy): RedactRawOutputOptions {
return {
...EGRESS_REDACTION_OPTIONS,
...(policy.foreignIdentifiers === true ? { foreignIdentifiers: true } : {}),
...(policy.resolvedLiterals === undefined
? {}
: { literals: sortedDistinctLiterals(policy.resolvedLiterals) }),
};
}
/**
* Redact `payload` for `site`, or refuse it. The payload may be a string
* or any JSON-shaped tree; see `redactStructured` for why the tree form
* exists rather than a pass over serialised bytes.
*/
export function redactForEgress<T>(
site: EgressSiteId,
payload: T,
policy: EgressPolicy = {},
): EgressVerdict<T> {
const scanned = redactStructured(payload, composeRedactionOptions(policy));
if (scanned.truncated) {
return {
outcome: EGRESS_OUTCOME.refusedScanTruncated,
secretIdentifiers: Object.freeze([]),
detail:
`${EGRESS_SITES[site].verb} refused to write: a field exceeded the redactor ` +
`scan window (${SCAN_WINDOW_MIB} MiB), so only part of it was scanned and this ` +
"export cannot be reported as clean. Nothing was written. Shrink the oversized " +
"field or export a narrower selection.",
};
}
if (scanned.secretIdentifiers.length > 0) {
return {
outcome: EGRESS_OUTCOME.refusedSecretIdentifier,
secretIdentifiers: scanned.secretIdentifiers,
detail:
`${EGRESS_SITES[site].verb} refused to write: an identifier is secret-shaped, and an ` +
"identifier is never rewritten - a redacted filename merges the pages it names and a " +
"redacted mapping key renames the field, so redacting here would destroy content " +
"rather than protect it. Nothing was written. Rename or exclude what these tree " +
`locations point at, then re-run: ${scanned.secretIdentifiers.join(", ")}. ` +
"(Locations, not values: the identifier is the secret.)",
};
}
return {
outcome: EGRESS_OUTCOME.released,
payload: scanned.value as T,
redacted: scanned.redacted,
};
}
/**
* Redact a configuration mapping for a surface outside the vault - the
* config snapshot in `export-config`, `second_brain_status` over MCP, and
* the OpenClaw status tool.
*
* This is what `config.ts`'s `redactMapping` became. That copy matched
* five substrings against key NAMES only and never inspected a value, so
* a credential under a name it did not recognise passed straight through.
* Both halves now come from the shared redactor.
*
* No refusal arm: a status tool's answer is not a file leaving the
* machine, and a config value past the scan window carries the marker in
* place, which is visible rather than silent. For the same reason it does
* not act on `secretIdentifiers`: the resolved PATHS are what this surface
* exists to report, they are returned verbatim, and a credential stored as
* a directory name is left visible rather than turned into a status call
* that refuses to answer. Path values reach the caller unchanged, which is
* the point - the previous behaviour replaced any 24-character segment of
* the operator's own path with a placeholder.
*/
export function redactConfigMapping(
data: Readonly<Record<string, unknown>>,
policy: EgressPolicy = {},
): Record<string, unknown> {
return redactStructured(data, composeRedactionOptions(policy)).value as Record<string, unknown>;
}