Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
/**
* Best-effort secret redactor + text-field normaliser shared across
* the Brain writers.
*
* The redactor catches six secret-bearing keys in four assignment
* shapes:
*
* key=value env-style assignments, including a
* prefixed name (`ANTHROPIC_API_KEY=…`)
* key: value YAML / log lines / single-line `key: token`
* "key": "value" JSON object entries
* Authorization: Bearer <token> HTTP authorization header (special case)
*
* Each match keeps the key (and surrounding quoting) and replaces the
* value with the literal `***REDACTED***`. The transform is
* intentionally narrow — receipts and signals carry a disclaimer that
* the agent must visually inspect output before posting externally.
*
* An optional infra-topology pass (`redactInfra`) additionally scrubs
* bare network coordinates that carry no `key=value` shape — public
* IPv4/IPv6 literals, `user:pass@host` URL credentials, `host:port`
* endpoints, and internal hostnames. It is off by default (the key/value
* passes suffice for receipts) and enabled on the artifact store, which
* persists full tool payloads where a bare IP or internal FQDN is the
* most common topology leak. Every infra regex is bounded (no nested
* unbounded quantifiers), so the pass stays linear on large inputs.
*
* Oversized input FAILS CLOSED: rather than silently dropping the tail
* past the scan window as if it were clean, {@link redactRawOutput}
* appends {@link SCAN_TRUNCATED_MARKER}. {@link wasScanTruncated} lets a
* downstream consumer holding only bytes off disk detect that marker and
* demote/exclude the artifact instead of trusting a partially-scanned
* payload. A caller that RAN the scan reads truncation from
* {@link scanRawOutput} instead - content is free to quote the marker,
* so the text is not evidence about the scan that produced it.
*
* `redactStructured` is the tree form, used at the export boundary and by
* every surface that hands a configuration mapping outside the vault. It
* redacts each string leaf and replaces any value whose KEY NAME declares
* a credential, because a serialised document cannot be scanned safely -
* the `key: value` pass runs to end of line and would eat a closing JSON
* quote - and because a bare token leaf carries no assignment shape for
* the value passes to see.
*
* `normaliseTextField` is the shared input sanitiser for fields that
* land in YAML frontmatter or single-line Markdown bullets. It strips
* C0 control characters (except `\n` and `\t`), folds the unicode line
* separators `U+2028` / `U+2029` to `\n`, NFC-normalises, and caps
* length to `maxLen`. The function never throws — out-of-spec input
* is silently coerced into something safe to persist. A misrecorded
* signal is worse than a missed one (the dream pass picks up patterns
* from repeats); a YAML-poisoning signal is worse than either.
*/
/**
* The single replacement token every redaction in this project emits.
* Exported because it is now the ONE spelling: `src/core/config.ts` used
* to carry a private `redactMapping` that wrote `[REDACTED]` instead,
* matched five substrings against key names only, and never looked at a
* value - three answers to "is this a secret" where there should be one.
* That copy was collapsed into {@link redactStructured}, and its callers
* onto this token.
*/
export const REDACTION_PLACEHOLDER = "***REDACTED***";
const PLACEHOLDER = REDACTION_PLACEHOLDER;
export const PRIVATE_REGION_PLACEHOLDER = "***PRIVATE***";
/**
* Maximum input size scanned by `redactRawOutput`. Receipts have no
* legitimate reason to embed multi-megabyte payloads — a runaway pipe
* of server logs is the realistic cause of an oversize input. The
* regex pipeline is linear, so the window is a DoS bound rather than a
* correctness limit; 1 MiB is wide enough that a secret rarely lands
* past it, and anything that does trips the fail-closed marker below
* rather than being dropped as if it were clean.
*/
export const MAX_REDACTOR_INPUT = 1024 * 1024;
/**
* Stable, machine-detectable sentinel embedded in {@link SCAN_TRUNCATED_MARKER}.
* {@link wasScanTruncated} matches on this so downstream consumers can
* demote/exclude a partially-scanned artifact without parsing prose.
*/
const SCAN_TRUNCATED_SENTINEL = "***SCAN_TRUNCATED***";
/**
* Appended when input exceeds the scan window. The tail past the window
* is dropped *and* flagged: because it was never scanned, the whole
* payload must be treated as unverified rather than clean.
*/
export const SCAN_TRUNCATED_MARKER =
`\n\n${SCAN_TRUNCATED_SENTINEL} [redactor scan window exceeded (> 1 MiB); the unscanned tail was dropped. ` +
`This payload was only partially scanned — treat it as unverified and inspect the raw source before sharing.]\n`;
/**
* True when `text` carries the fail-closed marker appended by
* {@link redactRawOutput} on oversized input. Consumers use this to
* demote or exclude an artifact that could not be fully scanned instead
* of trusting the redactor's output as complete.
*/
export function wasScanTruncated(text: string): boolean {
return typeof text === "string" && text.includes(SCAN_TRUNCATED_SENTINEL);
}
/**
* An open tag's attributes stop at the next `<` as well as at `>`: with
* `[^>]*` a run of `<private ` with no closing bracket made every attempt
* scan to the end of the text, quadratic in its length. A tag whose
* attributes reach a `<` or the end of the text still opens a region
* there, so the bound never narrows what counts as private: the rule
* fails closed and stays linear.
*/
const PRIVATE_OPEN_TAG_RE = /<private\b[^<>]*(?:>|(?=<)|$)/gi;
const PRIVATE_CLOSE_TAG_RE = /<\/private>/gi;
/**
* Canonical list of secret-bearing field names. Each entry is the
* underscore-separated canonical form; the regex builder below makes
* `_` and `-` interchangeable and `_` optional, so a single entry
* `api_key` covers `api_key` / `apikey` / `api-key` automatically.
* Don't add the visual variants here — they're already covered.
*/
export const SECRET_KEYS: ReadonlyArray<string> = [
"api_key",
"token",
"access_token",
"refresh_token",
"bearer",
"secret",
"client_secret",
"authorization",
"private_key",
"password",
"passwd",
"pwd",
"credential",
"credentials",
"session_token",
];
const KEY_PATTERN = SECRET_KEYS.map((k) => k.replace(/[-_]/g, "[-_]?")).join("|");
/**
* Credential field-identifier fragments matched ONLY against a mapping's
* key NAME, never against free text. `key` lives here rather than in
* {@link SECRET_KEYS} because a bare `key: value` line in prose is not a
* credential assignment and redacting every one of them would mangle
* ordinary text, while a configuration entry whose identifier contains
* `key` (`openai_key`, `keyfile`, `signing_key`) reliably is one. It is a
* field identifier, not a prose word - the same class as every other
* literal in this module.
*/
const SECRET_KEY_NAME_FRAGMENTS: ReadonlyArray<string> = ["key"];
/**
* Matches anywhere inside an identifier, because configuration keys
* compose (`telegram_bot_token`, `openrouter_api_key`).
*/
const SECRET_KEY_NAME_RE = new RegExp(
`(?:${KEY_PATTERN}|${SECRET_KEY_NAME_FRAGMENTS.join("|")})`,
"i",
);
/**
* True when a mapping key's NAME declares its value to be a credential.
* Takes `unknown` so a key read back off disk needs no pre-check.
*/
export function isSecretKeyName(name: unknown): boolean {
return typeof name === "string" && SECRET_KEY_NAME_RE.test(name);
}
// `key=value` (env-style): value runs to whitespace or end of line.
//
// The key may be the SUFFIX of a longer identifier, and the boundary here
// has to say so explicitly. `\b` was what stood in this position, and `_`
// is a word character, so the boundary never fired after one:
// `export ANTHROPIC_API_KEY=…` and `MY_DB_PASSWORD=…` - the single most
// common shape a credential takes in a shell line, a `.env` file or an
// agent transcript - both passed through this pass untouched while the
// unprefixed `API_KEY=…` was caught. What still came out redacted in the
// common cases came out that way through the vendor-prefix rule on the
// VALUE (`sk-…`, `ghp_…`), not through this key-name pass, so a
// non-vendor secret under a prefixed name left verbatim.
//
// `(?<![A-Za-z0-9])` is the replacement: the key may begin at the start of
// the identifier or immediately after a `_` / `-` separator, which is how
// a prefixed environment variable is spelled, and nowhere else. It stays
// deliberately narrow at both ends. `--max-tokens=4096` keeps its value,
// because `token` there is followed by `s` rather than by the assignment.
// `MYTOKEN=…` also keeps its value, because `TOKEN` starts mid-identifier
// with no separator in front of it - the same under-match `\b` already
// had, left in place rather than widened into every word that happens to
// end in a secret name.
const ENV_RE = new RegExp(`(?<![A-Za-z0-9])(${KEY_PATTERN})(\\s*=\\s*)([^\\s\\r\\n]+)`, "gi");
// `key: value` outside of JSON quoting. Excludes the `"key": ...` JSON
// shape and the `Authorization: Bearer X` header (handled below).
//
// The separator is horizontal whitespace ONLY. With `\s*` it matched
// across a newline, so `password:\nnext_key: kept` read the FOLLOWING
// line as this key's value and deleted a key that was never a secret. A
// value on the next line is not a value; a value indented under the key
// is a block, and {@link YAML_SECRET_BLOCK_RE} owns that case.
const COLON_VALUE_RE = new RegExp(
`(?<!")\\b(${KEY_PATTERN})([ \\t]*:[ \\t]*)("[^"]*"|'[^']*'|[^\\r\\n]+)`,
"gi",
);
/**
* A secret key whose value CONTINUES on more-indented lines: a block
* scalar (`token: |`), a block list, or a nested mapping. The whole
* continuation is replaced as one unit, because replacing the first line
* alone orphans the rest - `token: |` became `token: ***REDACTED***`
* followed by its still-readable indented block, and
* `token:\n - a\n - b` became `token:\n ***REDACTED***\n - b`.
*
* The backreference pins the continuation to a deeper indent than the
* key, so a following key at the same level is never swallowed. Every
* quantifier is bounded to a single line, so the pass stays linear.
*/
const YAML_SECRET_BLOCK_RE = new RegExp(
`^([ \\t]*)(${KEY_PATTERN})[ \\t]*:[ \\t]*(?:[|>][+-]?\\d{0,2})?[ \\t]*\\r?\\n` +
"(?:\\1[ \\t]+[^\\r\\n]*\\r?\\n?)+",
"gim",
);
// `"key": "value"` JSON entries.
const JSON_ENTRY_RE = new RegExp(
`("(?:${KEY_PATTERN})"\\s*:\\s*)("(?:[^"\\\\]|\\\\.)*"|true|false|null|-?\\d+(?:\\.\\d+)?)`,
"gi",
);
// `Authorization: Bearer <token>` header. COLON_VALUE_RE already
// redacts `authorization: ...` lines, but the canonical HTTP header is
// common enough that we preserve the `Bearer ` prefix for readability
// and only replace the token portion.
const BEARER_RE = /\b(Bearer\s+)([A-Za-z0-9._\-+/=]+)/gi;
// A bare JSON Web Token: three base64url segments joined by dots. The
// shape carries no key=value assignment for the passes above and no
// vendor prefix for the token pass, and the canonical 20-character
// header slips HIGH_ENTROPY_TOKEN_RE's 24-character gate - so under the
// token pass the payload and signature were eaten while the header
// stayed, announcing a credential and naming its algorithm. Default-on:
// a JWT is a bearer credential wherever it appears, not only where a
// key names it.
//
// The header prefix anchors the match, while matching any three
// dot-separated base64url runs would also claim ordinary dotted
// identifiers. A compact header `{"` encodes to `eyJ`; a header written
// with whitespace after the brace encodes to `eyA` (`{ `), `ewo` (`{\n`),
// `ew0` (`{\r`) or `ewk` (`{\t`), so all five open the match. The header
// segment must be at least 12 characters in total (a real header is far
// longer), which keeps short dotted words such as `ewok.a.b` prose.
// Segments are bounded (64 KiB each, room for a large claims payload),
// so the pass stays linear on large inputs.
const JWT_RE =
/\b(?:eyJ|eyA|ewo|ew0|ewk)[A-Za-z0-9_-]{9,65533}(?:\.[A-Za-z0-9_-]{4,65536}){2}(?![A-Za-z0-9_-])/g;
// ----- Infra-topology detectors (opt-in via `redactInfra`) ------------------
//
// These scrub network coordinates that carry no key=value shape, so the
// assignment passes above never see them. Every regex uses only bounded
// repetition ({m,n}) — no nested unbounded quantifiers — so the pass is
// linear and cannot be driven into catastrophic backtracking (ReDoS) by
// a large adversarial input.
/** A single dotted-quad octet (0-255), used to compose IPv4 patterns. */
const IPV4_OCTET = "(?:25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)";
/** A syntactically-valid IPv4 literal (four octets). */
const IPV4 = `${IPV4_OCTET}(?:\\.${IPV4_OCTET}){3}`;
// `scheme://user:pass@host` — strip the embedded credentials but keep the
// scheme and `@host` for readability. Run first so the host that follows
// is still available to the host/port passes below.
//
// The password class is `[^\s@]+`, not `[^\s/@]+`: a generated database
// password routinely carries `/`, and with the old class the regex failed
// the match ENTIRELY - both the user and the password left verbatim, a
// wider leak than if the pass had never run. The username may be EMPTY -
// a password-only authority is a spelling connection strings really use
// (`scheme://:pass@host`) - while the colon between the halves stays
// mandatory and adjacent, so a URL whose path carries an `@` but no
// userinfo colon is untouched. Every class still cannot cross whitespace,
// the `://` scheme anchor and the `@` anchor are kept. The anchors alone do
// not keep the pass linear: a run that never meets its terminator is
// rescanned from every word boundary, which is quadratic on a long line of
// `a://b:` or `a.` repeats. Each run therefore has a length bound (scheme
// 32, user 256, password 4096 characters), so a start position costs at
// most a fixed number of steps; a password longer than the bound is not
// recognised by this pass.
//
// Because the password class crosses `/`, a `host:port/path@x` URL would
// read the port colon as the userinfo colon and swallow the host, the port
// and half the path (`http://localhost:5173/@vite/client`). The lookahead
// rejects a colon followed by 1-5 digits and then `/`: that is a port and a
// path, not a password. Two accepted trade-offs, both as on main: a slash
// password that begins with 1-5 digits (`bob:8080/x@host`) is missed, and a
// query or fragment directly after a port with an `@` in it
// (`example.com:443?x@y`) is still read as userinfo - far rarer than a
// password such as `123?secret`, which this keeps redacted.
const BASIC_AUTH_URL_RE =
/\b([a-zA-Z][a-zA-Z0-9+.-]{0,31}:\/\/)([^\s/:@]{0,256}):(?!\d{1,5}\/)([^\s@]{1,4096})@/g;
// `ipv4:port` — a reachable service endpoint. Redacted whole regardless of
// whether the address is public or private (the port is what leaks the
// service). Runs before the bare-IPv4 pass so the port form is caught first.
const IPV4_PORT_RE = new RegExp(`\\b${IPV4}:\\d{1,5}\\b`, "g");
// `fqdn:port` — a named service endpoint (`db.example.com:5432`). The final
// label is alphabetic, so this never collides with `ipv4:port`. A negative
// lookahead skips source-file extensions (`index.js:42`, `app.ts:128`) so
// diagnostics and stack frames are not mistaken for service endpoints when
// `redactInfra` runs over tool output (e.g. ArtifactStore.put).
const FQDN_PORT_SOURCE_EXTS =
"js|ts|tsx|jsx|py|json|rs|go|java|rb|php|c|cc|cpp|cxx|h|hpp|css|scss|sass|less|" +
"html|htm|xml|yaml|yml|toml|ini|cfg|md|markdown|sh|bash|sql|vue|svelte|gradle|" +
"kt|swift|scala|clj|ex|exs|erl|elm|dart|lua|pl|pm|r|jl|tf|lock|map|txt|csv|log";
const FQDN_PORT_RE = new RegExp(
"\\b(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\\.)+(?!(" +
FQDN_PORT_SOURCE_EXTS +
"):\\d)[a-zA-Z]{2,63}:\\d{1,5}\\b",
"g",
);
// Internal hostnames — FQDNs under a private/self-hosted suffix
// (`db.internal`, `svc.cluster.local`, `host.corp`, …). These reveal
// internal topology even without a port.
const INTERNAL_HOST_RE =
/\b(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+(?:internal|intranet|localdomain|local|lan|corp|home)\b/gi;
// Bare IPv6 literals: either a full 8-group address or any `::`-compressed
// form. Requiring `::` or 8 groups keeps `HH:MM:SS`-style timestamps (only
// two colons, no `::`) from being mistaken for an address. Every quantifier
// is bounded.
const IPV6_RE = new RegExp(
"(?<![\\w:.])(?:" +
// full 8 groups
"(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}" +
"|" +
// `::`-compressed with leading groups (e.g. 2001:db8::1, fe80::)
"(?:[0-9A-Fa-f]{1,4}:){0,6}[0-9A-Fa-f]{1,4}::(?:[0-9A-Fa-f]{1,4}:){0,6}[0-9A-Fa-f]{0,4}" +
"|" +
// leading `::` (e.g. ::1, ::ffff:1.2.3.4-style prefix)
"::(?:[0-9A-Fa-f]{1,4}:){0,6}[0-9A-Fa-f]{1,4}" +
")(?![\\w:.])",
"g",
);
// Bare IPv4 literal not part of a longer dotted run (excludes version
// strings like `1.2.3.4.5` and `v1.2.3`). Public-only: the callback skips
// private/reserved ranges.
const IPV4_BARE_RE = new RegExp(`(?<![\\w.])${IPV4}(?![\\w.])`, "g");
/** RFC 1918 / loopback / link-local / CGNAT / multicast+reserved IPv4. */
function isPrivateOrReservedIPv4(ip: string): boolean {
const octets = ip.split(".");
const a = Number.parseInt(octets[0] ?? "", 10);
const b = Number.parseInt(octets[1] ?? "", 10);
if (a === 0 || a === 10 || a === 127) return true; // this-network, private, loopback
if (a === 172 && b >= 16 && b <= 31) return true; // 172.16.0.0/12
if (a === 192 && b === 168) return true; // 192.168.0.0/16
if (a === 169 && b === 254) return true; // link-local
if (a === 100 && b >= 64 && b <= 127) return true; // CGNAT 100.64.0.0/10
if (a >= 224) return true; // multicast + reserved (224+)
return false;
}
/** Loopback / unspecified / link-local (fe80::/10) / unique-local (fc00::/7). */
function isPrivateOrReservedIPv6(ip: string): boolean {
const lower = ip.toLowerCase();
if (lower === "::" || lower === "::1") return true;
if (/^fe[89ab]/.test(lower)) return true; // link-local
if (/^f[cd]/.test(lower)) return true; // unique-local
return false;
}
// ----- Bare high-entropy token detector (opt-in via `redactTokens`) ---------
//
// Credential tokens passed as bare positional values carry no key=value
// shape for the assignment passes to latch onto (e.g. an argv like
// `["mytool", "sk-abc123"]`). Two complementary shapes are scrubbed:
// well-known vendor-prefixed keys, and long mixed-class runs that look
// like an API key or hash. Every quantifier is bounded, so the pass stays
// linear and cannot be driven into catastrophic backtracking.
/**
* Vendor-prefixed credential tokens (OpenAI/Stripe `sk-`/`sk_`/`rk_`/`pk_`,
* GitHub `ghp_`/`gho_`/…/`github_pat_`, Slack `xox?-`, AWS `AKIA…`, Google
* `AIza…`, GitLab `glpat-`). The recognizable prefix is what lets a short
* token like `sk-abc123` be caught without a length gate that would also
* hit ordinary words.
*/
const VENDOR_TOKEN_RE = new RegExp(
[
"\\b(?:sk|rk|pk)[-_][A-Za-z0-9._-]{3,200}",
"\\b(?:ghp|gho|ghu|ghs|ghr)_[A-Za-z0-9]{6,255}",
"\\bgithub_pat_[A-Za-z0-9_]{6,255}",
"\\bxox[baprs]-[A-Za-z0-9-]{6,200}",
"\\bAKIA[0-9A-Z]{16}\\b",
"\\bAIza[0-9A-Za-z._-]{10,100}",
"\\bglpat-[A-Za-z0-9_-]{6,100}",
].join("|"),
"g",
);
/**
* A long bare run that mixes letters and digits — the shape of an
* unprefixed API key, session id, or hash. Length-gated (≥ 24) so ordinary
* words and short ids are never touched, with bounded repetition so the
* lookaheads stay linear.
*/
const HIGH_ENTROPY_TOKEN_RE =
/\b(?=[A-Za-z0-9_-]{24,200}\b)(?=[A-Za-z0-9_-]{0,199}[A-Za-z])(?=[A-Za-z0-9_-]{0,199}\d)[A-Za-z0-9_-]{24,200}\b/g;
/**
* A CONTENT ADDRESS: a hexadecimal run, optionally dash-grouped. Digests
* (`sha256`, git object ids), canonical uuids and the hashed directory
* names they end up in all have this shape, and all three are identifiers
* a knowledge bundle has to carry unchanged - a wikilink target
* `[[Brain/artifacts/<sha256>.json]]` that comes back as a placeholder is
* a corrupted restore on the export/import round trip, not a mangled copy.
*
* Excluding the shape costs the high-entropy pass any credential that is
* pure hexadecimal. That is the narrower risk: credentials in this
* ecosystem carry a vendor prefix (caught by {@link VENDOR_TOKEN_RE}
* regardless of alphabet) or mix alphabets beyond hexadecimal, while
* content addresses are pervasive in every vault this exports.
*/
const CONTENT_ADDRESS_RE = /^[0-9a-fA-F]+(?:-[0-9a-fA-F]+)*$/;
function isContentAddress(run: string): boolean {
return CONTENT_ADDRESS_RE.test(run);
}
/**
* A SIGNAL ID: `sig-<YYYY-MM-DD>-<slug>`, the filename and frontmatter
* `id` every taste signal in a Brain vault carries, and the target every
* preference's `_evidenced_by` wikilink points at. The same carve-out
* argument as {@link CONTENT_ADDRESS_RE}, arriving through a second
* shape: 26 characters of `[A-Za-z0-9_-]` mixing letters and digits, not
* pure hexadecimal, so `HIGH_ENTROPY_TOKEN_RE` claimed it. A slug of nine
* characters is enough to cross the gate, which is routine, so an export
* shipped `_evidenced_by: ["[[***REDACTED***]]"]` and the import wrote
* that back verbatim - reporting a full success while every preference
* that lost a signal id landed on ONE dangling wikilink. A placeholder is
* a constant, so replacing an identifier merges the things it identified.
*
* The carve-out is the SHAPE, not the prefix: an ISO date, then lowercase
* alphanumeric runs joined by dashes, which is exactly what `slugify`
* emits plus the `-<n>` collision suffix `allocateAndCreate` appends. A
* credential wearing a `sig-` prefix is still redacted whenever it fails
* either half - no ISO date, or the mixed case, underscore or other
* punctuation a slug cannot carry - which covers every credential format
* this redactor was written against.
*
* It is not an absolute, and the earlier wording here said it was. A
* secret that is entirely lowercase alphanumeric AND is written behind
* `sig-` AND behind an ISO date passes: `sig-2026-08-16-<secret>` is
* indistinguishable from a signal id whose slug happens to be that
* string, because at that point it IS one by shape. Nothing narrower
* would do without keying on the prefix, which is what the paragraph
* above rejects - and a key wearing three disguises to reach a Brain
* signal filename is a different threat from the one this pass is for.
*/
const SIGNAL_ID_RE = /^sig-\d{4}-\d{2}-\d{2}-[a-z0-9]+(?:-[a-z0-9]+)*$/;
function isSignalId(run: string): boolean {
return SIGNAL_ID_RE.test(run);
}
/**
* True when a high-entropy run is an IDENTIFIER this boundary must hand
* through unchanged rather than a payload it may replace. One predicate
* so the two passes that ask the question - the bare-token replacer and
* the identifier scan - cannot drift into disagreeing about what an
* identifier is.
*/
function isPreservedIdentifier(run: string): boolean {
return isContentAddress(run) || isSignalId(run);
}
/**
* A base64 credential: the shape `HIGH_ENTROPY_TOKEN_RE` cannot see,
* because its class excludes `+` and `/`. An AWS secret access key
* (`wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY`) splits into sub-runs of
* 13, 21 and 5 under that class - each below the 24-gate - so it passed
* verbatim while its paired `AKIA…` id was caught, leaking the half that
* matters.
*
* Narrow on purpose, because this class of run is also what long paths
* and embedded blobs look like:
* - the MAXIMAL run is 32 to 64 characters (a blob runs to thousands
* and is left whole; a shorter run is not a credential);
* - it carries at least one `+`, `/` or `=`, since a pure-alphanumeric
* run is already {@link HIGH_ENTROPY_TOKEN_RE}'s job;
* - it mixes upper, lower and digit;
* - it neither starts nor ends with `/`, so a rooted path cannot match
* from its first character.
*
* What remains reachable is a long extension-less mixed-case path inside
* prose - a `.`, `-` or `_` anywhere in it breaks the run - and at an
* egress boundary that copy is recoverable where a leaked key is not.
*/
const BASE64_SECRET_RE = new RegExp(
"(?<![A-Za-z0-9+/=])" +
"(?=[A-Za-z0-9+/=]{32,64}(?![A-Za-z0-9+/=]))" +
"(?=[A-Za-z0-9+/=]{0,63}[a-z])" +
"(?=[A-Za-z0-9+/=]{0,63}[A-Z])" +
"(?=[A-Za-z0-9+/=]{0,63}\\d)" +
"(?=[A-Za-z0-9+/=]{0,63}[+/=])" +
"[A-Za-z0-9+=][A-Za-z0-9+/=]{30,62}[A-Za-z0-9+=]",
"g",
);
function redactBareTokens(text: string): string {
return text
.replace(VENDOR_TOKEN_RE, PLACEHOLDER)
.replace(BASE64_SECRET_RE, PLACEHOLDER)
.replace(HIGH_ENTROPY_TOKEN_RE, (run: string) =>
isPreservedIdentifier(run) ? run : PLACEHOLDER,
);
}
/**
* The credential half of the infra pass, split out so a caller can take
* it WITHOUT the topology half. `user:pass@host` is a credential by any
* reading; a bare public IP or an internal FQDN is topology. An export of
* authored knowledge wants the first and not the second, because
* redacting every `host:port` in a knowledge bundle mangles legitimate
* references for a class of leak the operator already controls by
* choosing the destination.
*
* Also the first step of {@link redactSpecifierCredentials}.
*/
function redactUrlCredentials(text: string): string {
return text.replace(BASIC_AUTH_URL_RE, (_m, scheme: string) => `${scheme}${PLACEHOLDER}@`);
}
/**
* Query parameter keys whose value is a credential in a module source or
* import specifier: the Terraform git getter's `sshkey`, S3 and GCS
* signing and access-key parameters, and the generic token, password and
* signature names. Matched without regard to case.
*/
export const CREDENTIAL_QUERY_KEYS: ReadonlyArray<string> = Object.freeze([
"sshkey",
"token",
"access_token",
"password",
"secret",
"signature",
"sig",
"key",
"aws_access_key_id",
"aws_access_key_secret",
"aws_secret_access_key",
"aws_access_token",
"x-amz-signature",
"x-amz-credential",
"x-amz-security-token",
"x-goog-signature",
"x-goog-credential",
]);
const CREDENTIAL_QUERY_KEY_SET: ReadonlySet<string> = new Set(CREDENTIAL_QUERY_KEYS);
/** A Terraform go-getter forcing prefix (`git::`, `s3::`, `gcs::`), kept as written. */
const GETTER_PREFIX_RE = /^[a-z0-9]+::/i;
/** Schemes whose userinfo has no conventional login: a bare user there is a token. */
const TOKEN_USERINFO_SCHEMES: ReadonlySet<string> = new Set([
"http:",
"https:",
"git+http:",
"git+https:",
]);
/**
* A user name that reads as a login (`git`, `deploy`) rather than a token;
* kept on the schemes where a bare login is conventional (`ssh://git@`).
*/
const PLAUSIBLE_LOGIN_RE = /^[a-z_][a-z0-9._-]{0,31}$/;
function decodedKey(raw: string): string {
try {
return decodeURIComponent(raw.replaceAll("+", " ")).toLowerCase();
} catch {
return raw.toLowerCase();
}
}
/** `search` with the value of every credential key replaced, or null when none matched. */
function redactCredentialQuery(search: string): string | null {
if (search.length <= 1) return null;
let changed = false;
const pairs = search
.slice(1)
.split("&")
.map((pair) => {
const eq = pair.indexOf("=");
if (eq < 0 || !CREDENTIAL_QUERY_KEY_SET.has(decodedKey(pair.slice(0, eq)))) return pair;
changed = true;
return `${pair.slice(0, eq + 1)}${PLACEHOLDER}`;
});
return changed ? `?${pairs.join("&")}` : null;
}
/**
* The credential query pass for a specifier the URL parser refuses (the
* scp-like `git@host:org/repo?sshkey=...` form, a host without a scheme):
* the text between the first `?` and the first `#` after it is read as the
* query.
*/
function redactUnparsedQuery(specifier: string): string {
const start = specifier.indexOf("?");
if (start < 0) return specifier;
const hash = specifier.indexOf("#", start);
const end = hash < 0 ? specifier.length : hash;
const search = redactCredentialQuery(specifier.slice(start, end));
return search === null
? specifier
: `${specifier.slice(0, start)}${search}${specifier.slice(end)}`;
}
/**
* The credential pass for a module source or import specifier. Covered:
* the `user:password@` pair of any scheme ({@link redactUrlCredentials}),
* a password containing `@`, a bare userinfo of an http(s) or git+http(s) URL (a token,
* with or without a go-getter prefix such as `git::`), a userinfo of any
* other scheme that does not read as a login (`ssh://git@` is kept), and
* the value of every {@link CREDENTIAL_QUERY_KEYS} parameter. A specifier
* that does not parse as a URL gets the `user:password@` pass and the
* query pass ({@link redactUnparsedQuery}), not the bare-userinfo one; a
* credential in a path segment, a fragment or an unnamed query key is not
* recognised. A specifier with nothing to redact is returned
* byte-identical; a redacted one is re-serialised by the URL parser.
*/
export function redactSpecifierCredentials(specifier: string): string {
const basic = redactUrlCredentials(specifier);
const prefix = GETTER_PREFIX_RE.exec(basic)?.[0] ?? "";
let url: URL;
try {
url = new URL(basic.slice(prefix.length));
} catch {
return redactUnparsedQuery(basic);
}
let changed = false;
if (url.password !== "") {
url.username = PLACEHOLDER;
url.password = "";
changed = true;
} else if (
url.username !== "" &&
url.username !== PLACEHOLDER &&
(TOKEN_USERINFO_SCHEMES.has(url.protocol) || !PLAUSIBLE_LOGIN_RE.test(url.username))
) {
url.username = PLACEHOLDER;
changed = true;
}
const search = redactCredentialQuery(url.search);
if (search !== null) {
url.search = search;
changed = true;
}
return changed ? `${prefix}${url.href}` : basic;
}
function redactInfraTopology(text: string): string {
// Credentials first, so the host that follows is still available to the
// host/port passes below - the order this pass has always run in.
let out = redactUrlCredentials(text);
out = out.replace(IPV4_PORT_RE, PLACEHOLDER);
out = out.replace(FQDN_PORT_RE, PLACEHOLDER);
out = out.replace(INTERNAL_HOST_RE, PLACEHOLDER);
out = out.replace(IPV6_RE, (match: string) =>
isPrivateOrReservedIPv6(match) ? match : PLACEHOLDER,
);
out = out.replace(IPV4_BARE_RE, (match: string) =>
isPrivateOrReservedIPv4(match) ? match : PLACEHOLDER,
);
return out;
}
// ----- Keeping a redacted frontmatter block parseable -----------------------
//
// The placeholder opens with `*`, which YAML reads as an ALIAS node. A
// placeholder written unquoted into a mapping value therefore makes the
// whole block unparseable to Obsidian and to every spec-compliant reader -
// while this repository's own lenient parser accepts it, which is how the
// defect shipped. Any pass can put a placeholder there (the key-name rule,
// the bare-token pass), so the quoting is applied once at the end over the
// frontmatter block rather than at each producer.
const PLACEHOLDER_PATTERN = PLACEHOLDER.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
/** Leading `---` fenced block of a markdown document, captured in parts. */
const FRONTMATTER_BLOCK_RE = /^(---\r?\n)([\s\S]*?)(\r?\n---(?:\r?\n|$))/;
/** `key: ***REDACTED***…` at the start of a mapping value. */
const FRONTMATTER_SCALAR_RE = new RegExp(
`^([ \\t]*[^\\s#][^:\\r\\n]*:[ \\t]*)(${PLACEHOLDER_PATTERN}[^\\r\\n]*)$`,
"gm",
);
/** `- ***REDACTED***…` at the start of a block-list item. */
const FRONTMATTER_ITEM_RE = new RegExp(
`^([ \\t]*-[ \\t]+)(${PLACEHOLDER_PATTERN}[^\\r\\n]*)$`,
"gm",
);
/**
* `[***REDACTED***]` - a placeholder as an item of a FLOW sequence or
* mapping. `formatFrontmatter` writes lists inline, so `aliases:` comes
* back as `[…]` and the two line-anchored rules above never see it. Same
* alias node, same parse error, one position further in.
*/
const FRONTMATTER_FLOW_ITEM_RE = new RegExp(
`([[{,][ \\t]*)(${PLACEHOLDER_PATTERN})(?=[ \\t]*[,\\]}])`,
"g",
);
function quoteYamlScalar(value: string): string {
return `"${value.replace(/\\/g, "\\\\").replace(/"/g, '\\"')}"`;
}
function quoteRedactedFrontmatter(text: string): string {
if (!text.startsWith("---")) return text;
return text.replace(
FRONTMATTER_BLOCK_RE,
(_match, open: string, body: string, close: string) =>
open +
body
.replace(FRONTMATTER_SCALAR_RE, (_m, prefix: string, value: string) =>
value.startsWith('"') ? `${prefix}${value}` : `${prefix}${quoteYamlScalar(value)}`,
)
.replace(FRONTMATTER_ITEM_RE, (_m, prefix: string, value: string) =>
value.startsWith('"') ? `${prefix}${value}` : `${prefix}${quoteYamlScalar(value)}`,
)
.replace(
FRONTMATTER_FLOW_ITEM_RE,
(_m, prefix: string, value: string) => `${prefix}${quoteYamlScalar(value)}`,
) +
close,
);
}
/** One outermost private region: `[start, end)`, tags included. */
interface PrivateRegionSpan {
readonly start: number;
readonly end: number;
}
/**
* Every outermost `<private>` region of `text`, in order. Nested open tags
* deepen the region; an unclosed region runs to the end of the text and
* ends the walk. Linear: the next open and the next close are each found
* once and reused until the scan passes them, so a run of nested opens
* before one far close no longer re-scans to that close per open.
*/
function privateRegionSpans(text: string): PrivateRegionSpan[] {
const spans: PrivateRegionSpan[] = [];
const open = new RegExp(PRIVATE_OPEN_TAG_RE.source, "gi");
const close = new RegExp(PRIVATE_CLOSE_TAG_RE.source, "gi");
const nextAt = (re: RegExp, from: number): RegExpExecArray | null => {
re.lastIndex = from;
return re.exec(text);
};
let cursor = 0;
while (cursor < text.length) {
const first = nextAt(open, cursor);
if (!first) break;
let depth = 1;
let scan = first.index + first[0].length;
let nextOpen = nextAt(open, scan);
let nextClose = nextAt(close, scan);
while (depth > 0) {
if (!nextClose) {
spans.push({ start: first.index, end: text.length });
return spans;
}
if (nextOpen && nextOpen.index < nextClose.index) {
depth += 1;
scan = nextOpen.index + nextOpen[0].length;
} else {
depth -= 1;
scan = nextClose.index + nextClose[0].length;
}
if (nextOpen && nextOpen.index < scan) nextOpen = nextAt(open, scan);
if (nextClose.index < scan) nextClose = nextAt(close, scan);
}
spans.push({ start: first.index, end: scan });
cursor = scan;
}
return spans;
}
export function stripPrivateRegions(text: string): string {
if (!text) return text;
let output = "";
let cursor = 0;
for (const span of privateRegionSpans(text)) {
output += text.slice(cursor, span.start) + PRIVATE_REGION_PLACEHOLDER;
cursor = span.end;
}
return output + text.slice(cursor);
}
/**
* The text of every outermost `<private>` region in `text`, tags included,
* using the same nesting rule as {@link stripPrivateRegions}. An unclosed
* region runs to the end of the text. A caller holding only a slice of a
* page (a search chunk) uses this over the whole page to tell whether the
* slice carries private text whose tags fell outside the slice.
*/
export function privateRegionTexts(text: string): string[] {
if (!text) return [];
return privateRegionSpans(text).map((span) => text.slice(span.start, span.end));
}
export interface RedactRawOutputOptions {
/**
* Maximum input length before the truncation guard fires. Defaults to
* {@link MAX_REDACTOR_INPUT} (1 MiB) - the right cap for receipts,
* where a multi-megabyte payload is a runaway log pipe. Callers that
* must redact-without-losing-data (the MCP artifact store, whose whole
* job is to preserve the full payload for later fetch) pass
* `Number.POSITIVE_INFINITY` to disable truncation while still scrubbing
* secrets.
*/
readonly maxInput?: number;
/**
* Known secret values to scrub verbatim (write-time-integrity-
* governance, secret custody): every literal occurrence is replaced
* before the pattern passes run, so a credential injected into a
* subprocess env can never travel back through captured output even
* when no key=value shape surrounds it.
*/
readonly literals?: ReadonlyArray<string>;
/**
* When `true`, also run the infra-topology pass (public IPv4/IPv6,
* `user:pass@host` URL credentials, `host:port` endpoints, internal
* hostnames). Off by default — the key/value passes suffice for
* receipts, and blanket IP/host redaction would mangle legitimate
* prose. Enabled on the artifact store, whose full tool payloads are
* where a bare coordinate is the likeliest topology leak.
*/
readonly redactInfra?: boolean;
/**
* When `true`, also scrub bare high-entropy credential tokens that carry
* no key=value shape — vendor-prefixed keys (`sk-…`, `ghp_…`, `AKIA…`)
* and long mixed letter+digit runs. Off by default (over-redacting prose
* is worse than the narrow key/value passes). Enabled on the secret-exec
* audit trail, whose long-lived log records a full argv that may carry a
* foreign credential passed as a positional argument.
*/
readonly redactTokens?: boolean;
/**
* When `true`, scrub credentials embedded in a URL authority
* (`scheme://user:pass@host`) while leaving network topology alone.
* Implied by {@link redactInfra}, which takes the whole infra pass.
* The export boundary takes this half on its own: a URL password is a
* credential, whereas a `host:port` in an authored note is a reference
* a portable bundle has to keep.
*/
readonly redactUrlCredentials?: boolean;
}
/**
* A scan and what it could and could not see. `truncated` is a fact about
* the SCAN - the input was longer than the window - and never a fact read
* back off the output text. {@link wasScanTruncated} exists for a consumer
* holding only bytes off disk; a caller that ran the scan itself must use
* this, because a payload is free to quote the marker verbatim and a
* substring test on the output would then let an oversized string report
* itself as fully scanned.
*/
export interface RawScanResult {
readonly text: string;
readonly truncated: boolean;
}
/**
* The scanning form of {@link redactRawOutput}, returning the redacted
* text together with whether the scan window was exceeded. Every caller
* that has to REFUSE on a partial scan (the egress guard, through
* {@link redactStructured}) reads truncation from here.
*/
export function scanRawOutput(text: string, opts: RedactRawOutputOptions = {}): RawScanResult {
if (!text) return { text, truncated: false };
// Scrub known literals BEFORE the truncation guard: a secret value
// straddling the cut boundary must not survive as a partial
// fragment in the kept prefix.
let out = text;
for (const literal of opts.literals ?? []) {
if (literal.length === 0) continue;
out = out.split(literal).join(PLACEHOLDER);
}
// Fail closed on oversized input: scan the prefix that fits the window,
// drop the unscanned tail, and flag the result so a downstream consumer
// treats it as unverified rather than trusting it as fully scanned.
const maxInput = opts.maxInput ?? MAX_REDACTOR_INPUT;
const truncated = out.length > maxInput;
if (truncated) out = out.slice(0, maxInput) + SCAN_TRUNCATED_MARKER;
out = stripPrivateRegions(out);
// Order matters: handle JSON entries first so the COLON_VALUE_RE
// doesn't also match inside JSON pairs (the negative-lookbehind
// keeps it off the `"key":` portion, but if we ran COLON_VALUE_RE
// first, a value like `"token": "abc123"` could be partially
// mangled).
out = out.replace(JSON_ENTRY_RE, (_match, keyPart: string, value: string) => {
if (value.startsWith('"')) return `${keyPart}"${PLACEHOLDER}"`;
return `${keyPart}${PLACEHOLDER}`;
});
out = out.replace(ENV_RE, (_match, key: string, sep: string) => {
return `${key}${sep}${PLACEHOLDER}`;
});
// Bearer headers BEFORE the generic colon rule.
out = out.replace(BEARER_RE, (_match, prefix: string) => `${prefix}${PLACEHOLDER}`);
// Bare JWTs BEFORE the opt-in passes: the token pass would eat only the
// payload and signature segments (the header slips the entropy gate) and
// leave a half-standing credential, and the URL pass must see a
// JWT-in-a-password already collapsed.
out = out.replace(JWT_RE, PLACEHOLDER);
// Indented continuations BEFORE the single-line rule: `token: |` has a
// same-line value the colon rule would consume, leaving its block behind.
out = out.replace(
YAML_SECRET_BLOCK_RE,
(_match, indent: string, key: string) => `${indent}${key}: "${PLACEHOLDER}"\n`,
);
out = out.replace(COLON_VALUE_RE, (match, key: string, sep: string, value: string) => {
if (value.includes(PLACEHOLDER)) return match;
if (value.startsWith('"') && value.endsWith('"')) {
return `${key}${sep}"${PLACEHOLDER}"`;
}
if (value.startsWith("'") && value.endsWith("'")) {
return `${key}${sep}'${PLACEHOLDER}'`;
}
// Quoted even when the source value was bare: the placeholder opens
// with `*`, which YAML reads as an alias node, so an unquoted
// replacement turns a valid mapping into a parse error.
return `${key}${sep}"${PLACEHOLDER}"`;
});
// Infra-topology pass last: it runs on values the key/value passes
// already left untouched (bare coordinates), and any value they redacted
// is now a placeholder with no IP/host shape left to match.
if (opts.redactTokens) out = redactBareTokens(out);
if (opts.redactInfra) out = redactInfraTopology(out);
else if (opts.redactUrlCredentials) out = redactUrlCredentials(out);
// Last: a placeholder any pass above put into frontmatter value position
// has to be quoted or the block stops parsing.
out = quoteRedactedFrontmatter(out);
return { text: out, truncated };
}
export function redactRawOutput(text: string, opts: RedactRawOutputOptions = {}): string {
return scanRawOutput(text, opts).text;
}
// ----- Identifiers, which are checked rather than rewritten ----------------
//
// A payload can be replaced: the vault still holds the original and the
// copy is merely poorer. An IDENTIFIER cannot. Redacting a filename merges
// the pages it names (three notes collapsed onto `concepts/***REDACTED***`,
// two bodies destroyed); redacting a mapping key renames a field; redacting
// a path segment hands a support person a path that does not exist. So an
// identifier is scanned and REPORTED, never rewritten - and a caller at a
// trust boundary decides what a report is worth.
/**
* Key names whose value NAMES something rather than carrying content.
* Matched on the last underscore/dash-separated segment, because
* identifier keys compose (`bundle_path`, `session_id`, `config_path`).
*/
const IDENTIFIER_KEY_RE =
/(^|[_-])(id|ids|uuid|uuids|guid|path|paths|slug|slugs|filename|filenames|basename)$/i;
/** True when a mapping key declares its value to be an identity. */
export function isIdentifierKeyName(name: unknown): boolean {
return typeof name === "string" && IDENTIFIER_KEY_RE.test(name);
}
/** Rooted path forms, where internal whitespace is still a path. */
const PATH_ANCHOR_RE = /^(?:[/\\]|~[/\\]|\.{1,2}[/\\]|[A-Za-z]:[/\\])/;
/**
* True when a whole string leaf is a filesystem path. A URL authority is
* deliberately excluded (`://`, `@`): `scheme://user:pass@host` is a
* credential the url-credential pass must still reach.
*/
export function isPathLikeValue(value: string): boolean {
if (value.length === 0 || value.length > 4096) return false;
if (value.includes("@") || value.includes("://")) return false;
if (!value.includes("/") && !value.includes("\\")) return false;
return PATH_ANCHOR_RE.test(value) || !/\s/.test(value);
}
/** Non-global copy: `.test` on a `/g` regex carries `lastIndex` between calls. */
const VENDOR_TOKEN_TEST_RE = new RegExp(VENDOR_TOKEN_RE.source);
/**
* Does an identifier VALUE carry a credential? Vendor prefixes only. A
* long mixed run is a guess, which is the right trade for a payload and
* the wrong one for an identity: record ids, slugs and build ids are long
* mixed runs BY CONSTRUCTION (`ctn_20260815120000_a1b2c3d4e5f6a7b8`), and
* refusing every export that contains one would refuse every export.
*/
function identifierCarriesSecret(value: string): boolean {
return VENDOR_TOKEN_TEST_RE.test(value);
}
/**
* The FULL detector set applied to one string: a vendor prefix anywhere in
* it, or a high-entropy run that is not one of the identifier shapes this
* boundary hands through unchanged.
*
* One predicate rather than two copies, because the two positions that ask
* for it - a mapping key name, and an identifier a foreign runtime named -
* are asking the same question and must not drift into answering it
* differently.
*/
function carriesBareCredential(value: string): boolean {
if (identifierCarriesSecret(value)) return true;
for (const match of value.matchAll(HIGH_ENTROPY_TOKEN_RE)) {
if (!isPreservedIdentifier(match[0])) return true;
}
return false;
}
/**
* Does a mapping KEY carry a credential? The full detector set, because a
* key name is authored vocabulary rather than a generated identity: a
* 24-character mixed run in key position is anomalous where the same run
* in an id is ordinary. The OKF manifest's `producer_meta` is built from a
* page's `x-*` frontmatter keys, which is how a token reaches this
* position at all.
*/
function keyNameCarriesSecret(name: string): boolean {
return carriesBareCredential(name);
}
/**
* Does an identifier VALUE that a FOREIGN runtime named carry a
* credential? The full detector set again, and the reason is that
* {@link identifierCarriesSecret}'s narrowing does not reach here.
*
* That narrowing - vendor prefixes only, because "record ids, slugs and
* build ids are long mixed runs BY CONSTRUCTION" - is an argument about
* identities THIS vault generates, whose construction this code knows.
* `session_id` and `turn_id` on a transcript record are neither: they are
* a foreign harness's filename and a foreign harness's turn uuid, and
* nothing in this build constrains their shape. A bare high-entropy run
* in one of them is therefore not "an id by construction", it is an
* unexplained secret-shaped string, and treating it as the former let a
* transcript named `Xk7Qp2Rm9Wz4Tn6Yb8Vc3Ld5.jsonl` export verbatim under
* exit 0 while its vendor-prefixed sibling refused the whole run.
*
* Opt-in ({@link RedactStructuredOptions.foreignIdentifiers}) rather than
* the default, because the narrowing is still correct everywhere the
* identifiers ARE vault-authored: turning this on for the preference and
* OKF exports would refuse them over their own content addresses and
* container ids.
*/
function foreignIdentifierCarriesSecret(value: string): boolean {
return carriesBareCredential(value);
}
/** Bound on the reported list, so a refusal message stays readable. */
const MAX_REPORTED_IDENTIFIERS = 12;
// ----- Structured (mapping / tree) redaction --------------------------------
/**
* Outcome of {@link redactStructured}: the transformed value plus the two
* facts a caller at a trust boundary has to act on.
*/
export interface StructuredRedaction {
/** The redacted value. Same shape as the input. */
readonly value: unknown;
/** True when any leaf changed - nothing was silently altered. */
readonly redacted: boolean;
/**
* True when some string was larger than the scan window, so only its
* prefix was examined. The value is NOT clean and NOT provably dirty;
* a caller about to hand it outside must refuse rather than report
* success. {@link wasScanTruncated} is the underlying reader.
*/
readonly truncated: boolean;
/**
* Tree locations (`manifest.pages[2].bundle_path`, `producer_meta#0`)
* where an IDENTIFIER is secret-shaped. Each one was left verbatim -
* rewriting it would merge or rename what it identifies - so a caller
* handing these bytes outside must act on this list rather than assume
* the value is clean. Locations only: the identifier itself is the
* secret, and echoing it into a message would leak what the refusal is
* refusing to write.
*/
readonly secretIdentifiers: ReadonlyArray<string>;
}
/**
* {@link redactStructured}'s options: the raw-scan policy every string leaf
* is scanned under, plus the one decision that only exists for a tree.
*/
export interface RedactStructuredOptions extends RedactRawOutputOptions {
/**
* When `true`, an IDENTIFIER value is judged by the full bare-token
* detector ({@link foreignIdentifierCarriesSecret}) instead of by vendor
* prefixes alone. Off by default, and set only by a boundary whose
* identifiers were named OUTSIDE this vault - the transcript corpus,
* whose `session_id` is a foreign harness's filename and whose `turn_id`
* is its turn uuid. See {@link foreignIdentifierCarriesSecret} for why
* the default narrowing is right everywhere else.
*/
readonly foreignIdentifiers?: boolean;
}
/** Walked as data; anything else (Date, Map, class instance) passes through. */
function isPlainContainer(value: object): boolean {
const proto = Object.getPrototypeOf(value) as object | null;
return proto === Object.prototype || proto === null;
}
/**
* True for the sibling-pair shape configuration formats use for
* environment entries: `{ name: "DB_PASSWORD", value: … }`. The credential
* NAME sits in a SIBLING member, so the key-name rule - which reads only
* the CURRENT key, and here sees `value`, which says nothing - never
* fires, and a literal secret rode past the walk untouched.
*
* The matcher is the same {@link isSecretKeyName} the object branch
* applies to a key; no new secret-name vocabulary. An entry without a
* literal `value` member - the ECS `valueFrom` reference shape - is not a
* literal secret and stays with the normal walk.
*/
function siblingPairDeclaresSecret(item: unknown): boolean {
if (typeof item !== "object" || item === null) return false;
if (!isPlainContainer(item)) return false;
const record = item as Record<string, unknown>;
return "value" in record && isSecretKeyName(record["name"]);
}
/**
* Redact a JSON-shaped value tree: every string leaf through
* {@link redactRawOutput}, and every value whose KEY NAME declares a
* credential ({@link isSecretKeyName}) replaced whole.
*
* This exists because redacting a SERIALISED document is unsafe. The
* `key: value` pass consumes to end of line, so a note body reading
* `my token: abc` inside a pretty-printed JSON string would lose the
* closing quote and the document would stop parsing. Redacting the tree
* and serialising afterwards has neither problem, and it is also the only
* way the key-name rule can fire at all - a bare `sk-…` leaf carries no
* assignment shape for the value passes to latch onto.
*
* `null` and `undefined` under a credential key are left alone on
* purpose: absence is not a secret, and stamping a placeholder over it
* would report a credential that is not configured as one that is.
*/
export function redactStructured(
input: unknown,
opts: RedactStructuredOptions = {},
): StructuredRedaction {
let redacted = false;
let truncated = false;
const secretIdentifiers = new Set<string>();
const record = (location: string): void => {
if (secretIdentifiers.size < MAX_REPORTED_IDENTIFIERS) secretIdentifiers.add(location);
};
/**
* Walk a plain object's entries under `location`, marking each value
* secret when `secretKey` says its key declares one. One loop for both
* positions that walk an object - the object branch, and the array
* branch's sibling `{name, value}` pair - so the key reporting and the
* location spelling cannot drift between them.
*/
const walkEntries = (
source: Record<string, unknown>,
location: string,
secretKey: (key: string) => boolean,
): Record<string, unknown> => {
const out: Record<string, unknown> = {};
let index = 0;
for (const [key, child] of Object.entries(source)) {
// The key is reported by POSITION, never by name: the name is the
// secret in this case.
if (keyNameCarriesSecret(key)) record(`${location === "" ? "" : location}#${index}`);
const childLocation = location === "" ? key : `${location}.${key}`;
out[key] = walk(child, childLocation, secretKey(key), isIdentifierKeyName(key));
index += 1;
}
return out;
};
const walk = (
value: unknown,
location: string,
underSecretKey: boolean,
underIdentifierKey: boolean,
): unknown => {
if (underSecretKey) {
if (value === null || value === undefined) return value;
redacted = true;
return PLACEHOLDER;
}
if (typeof value === "string" && (underIdentifierKey || isPathLikeValue(value))) {
// Checked, never collapsed: see the identifier section above.
//
// The wider detector is applied only under an identifier KEY, never
// to a leaf that merely looks path-shaped: `foreignIdentifiers` is a
// statement about where a NAME came from, and a path-shaped run of
// prose carries no such provenance.
const secretShaped =
opts.foreignIdentifiers === true && underIdentifierKey
? foreignIdentifierCarriesSecret(value)
: identifierCarriesSecret(value);
if (secretShaped) record(location);
// One exception, and it is not a collapse: a `user:pass@host`
// authority is not part of what an identifier identifies. Stripping
// it leaves the identifier pointing at the same resource, where
// replacing the identifier would point it at nothing. Without this,
// a URL under a key named `source_path` would be exempt from the
// credential pass that catches the same URL one key over.
const cleaned =
opts.redactInfra === true || opts.redactUrlCredentials === true
? redactUrlCredentials(value)
: value;
if (cleaned !== value) redacted = true;
return cleaned;
}
if (typeof value === "string") {
// Truncation comes from the scan, never from a substring test on the
// output: a leaf is free to quote the marker verbatim, and reading
// the answer back out of the text let such a leaf release at any
// size with its unscanned tail silently dropped.
const scan = scanRawOutput(value, opts);
if (scan.text !== value) redacted = true;
if (scan.truncated) truncated = true;
return scan.text;
}
if (Array.isArray(value)) {
return value.map((item, index) => {
const itemLocation = `${location}[${index}]`;
if (!siblingPairDeclaresSecret(item)) {
return walk(item, itemLocation, false, underIdentifierKey);
}
// A `{name, value}` pair whose NAME declares a credential: the
// literal rides in the sibling `value` member, so that member is
// walked under a secret key. Every other member keeps the normal
// per-key decision.
return walkEntries(item as Record<string, unknown>, itemLocation, (key) =>
key === "value" ? true : isSecretKeyName(key),
);
});
}
if (typeof value === "object" && value !== null) {
if (!isPlainContainer(value)) return value;
return walkEntries(value as Record<string, unknown>, location, isSecretKeyName);
}
return value;
};
return {
value: walk(input, "", false, false),
redacted,
truncated,
secretIdentifiers: Object.freeze([...secretIdentifiers].toSorted()),
};
}
// ----- Text-field normaliser ------------------------------------------------
/**
* C0 control characters (U+0000…U+001F) are illegal in YAML scalars
* except for `\t` (` `) and `\n` (`
`). U+007F (DEL) is
* similarly hazardous. Strip everything in that range outside the
* two allowed control bytes — those are what we encounter in normal
* text and want to preserve verbatim.
*/
const FORBIDDEN_C0_RE = /[\x00-\x08\x0B-\x0C\x0E-\x1F\x7F]/g;
/**
* The Unicode line separator (U+2028) and paragraph separator
* (U+2029) are technically legal but render as line breaks in most
* editors and confuse one-line YAML scalars. Fold both to `\n` so
* a downstream Markdown reader sees normal line breaks.
*/
const UNICODE_LINE_SEP_RE = /[\u2028\u2029]/g;
export interface NormaliseTextFieldOptions {
/** Hard upper bound on output length in UTF-16 code units. */
readonly maxLen: number;
/**
* When `true`, also strip newlines and tabs — appropriate for
* single-line fields like `principle` or `scope` where a stray
* newline would corrupt the YAML scalar.
*/
readonly singleLine?: boolean;
}
/**
* Normalise a free-form text field for safe persistence in Brain
* frontmatter or apply-evidence log payloads. Never throws — invalid
* input is coerced to a safe shape (empty string for non-strings,
* truncation for over-length input).
*
* Pipeline:
* 1. Coerce non-string to empty.
* 2. Strip forbidden C0 controls (everything except `\t`/`\n`).
* 3. Fold U+2028 / U+2029 to `\n`.
* 4. If `singleLine`, collapse `\n`/`\r`/`\t` runs to a single space.
* 5. NFC-normalise so combining characters don't trip the length cap.
* 6. Truncate to `maxLen`.
*
* Trim is left to the caller — the writer for a given field decides
* whether leading / trailing whitespace is significant.
*/
export function normaliseTextField(value: unknown, opts: NormaliseTextFieldOptions): string {
if (typeof value !== "string") return "";
let s = value.replace(FORBIDDEN_C0_RE, "");
s = s.replace(UNICODE_LINE_SEP_RE, "\n");
if (opts.singleLine) {
s = s.replace(/[\r\n\t]+/g, " ");
} else {
// Normalise CRLF to LF so multi-line fields don't carry Windows
// line endings into YAML or Markdown.
s = s.replace(/\r\n/g, "\n").replace(/\r/g, "\n");
}
s = s.normalize("NFC");
if (s.length > opts.maxLen) {
s = s.slice(0, opts.maxLen);
}
return s;
}
/**
* Convenience: redact + normalise in one call. Used by the Brain
* writers (`writeSignal`, `appendApplyEvidence`) to keep field
* sanitisation consistent across surfaces.
*/
export function sanitiseTextField(value: unknown, opts: NormaliseTextFieldOptions): string {
if (typeof value !== "string") return "";
return normaliseTextField(redactRawOutput(value), opts);
}