Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(secret-scan): claim keys after a JSON or percent escape; vendor values join the echo dictionary (#7978)

garrytan2bf98da2ac06
diff --git a/docs/architecture/key-files/secret-scan.md b/docs/architecture/key-files/secret-scan.md
index a6997cd9d2..324cddf531 100644
--- a/docs/architecture/key-files/secret-scan.md
+++ b/docs/architecture/key-files/secret-scan.md
@@ -6,6 +6,6 @@ The format-based secret detector shared by `sources push`, bootstrap verify, the
compiled-context sensitivity scan and the transcripts redactor, and the composed
sensitivity detector built on it.
-- `src/core/secret-scan.ts` — pattern-based secret detection shared by `sources push`, bootstrap verify, the compiled-context sensitivity scan and the transcripts redactor. Detection is FORMAT-based, not prefix-only: vendor shapes (anthropic, openai, voyage, github, gitlab, slack, aws AKIA/ASIA, google, stripe, sendgrid, twilio, supabase sb_secret_/sbp_, npm, huggingface, `digitalocean` `do[opr]_v1_` + 64 hex, the AI-provider keys `openrouter` `sk-or-v1-` + 64 lowercase hex, `groq` `gsk_` + 52, `xai` `xai-` + 80, `perplexity` `pplx-` + 48 and `replicate` `r8_` + 37, all fixed-length with a hard right edge, gbrain), JWTs, private keys (whole-text claims, below), plus four catch-alls appended LAST (`bearer` — prebuilt two-group layout so only the token is the value; `db_url_credentials` — the `scheme://user:pass@` span of database URLs, host/db kept; `url_credentials` — the same span for `http(s)` (scheme case-insensitive, `gi`), user 0–128 stopping at `:`/`/`, password 1–256 excluding whitespace, `@`, `"`, `'`, `/`, `?`, `#`, with a `validate` hook that rejects placeholder passwords (`<...>`, `${...}`, `$VAR`, all `*`, all `x`); `basic_auth` — two prebuilt case-insensitive forms, `Authorization: Basic` with a 4–2048 char value and bare `Basic` with 16–2048, both with a hard right edge and a `validate` that requires the value to base64-decode to printable ASCII containing `:`; it is NOT in the bearer family, so it joins neither the echo dictionary nor `secret:bearer`). The pattern contract: `CorePattern` carries optional `flags` (default `'g'`), `precheck`, `validate(value)` (O(value), run on the whole match before the claim and the union split; a rejected match is not re-tried from inside) and `since` (the gbrain version that added or last changed the rule; `SECURITY_WAVE_SINCE` = `0.60.31.0` on `digitalocean`, `url_credentials`, `basic_auth` and `private_key_pem`, `PROVIDER_KEYS_SINCE` on the five AI-provider rules (#7959), kept in step with the release that ships a rule change). Compiled patterns keep group 1 = anchor, group 2 = value. `patternSince(pattern)` reads the map; every finding of such a pattern carries `SecretFinding.since`, which the push and bootstrap refusals render. Fixed-length shapes carry a trailing negative lookahead. Per-line claimed-span dedupe is first-wins for ATTRIBUTION (order matters: `Bearer <vendor key>` keeps the vendor name, a fully covered later span is skipped) but coverage is the UNION: every unclaimed slice of a later overlapping span is claimed as a hit of the later pattern (own value, own fingerprint), so a database URL whose username is a vendor-shaped id still loses its password. The transcript-only `labeled_credential` rule (`ScanOpts.labeledCredentials`) lives in `secret-scan-labeled.ts` (Security). The opt-in `high_entropy_assignment` rule (`ScanOpts.highEntropy`) requires a 12+ char value that contains a digit and a non-digit AND clears 3.5 bits/char on the bias-corrected (Miller-Madow) estimate `correctedEntropy`, since raw per-character entropy is capped at log2(n) and rejected most random 12-16 char secrets. It is two compiled patterns sharing the name: QUOTED (after `"` or `'`, any run of non-quote, non-whitespace characters) and UNQUOTED (the legacy `[A-Za-z0-9+/_=-]` class plus `!#$%*@^~.?<>`; never `&`, `()[]{}`, `:`, `,` or `;`, so query strings, calls and SQL `SET` lists stay intact, and a password carrying `,`/`;` matches only quoted; the last character excludes `.?!`, so a sentence's full stop is not claimed). Its `validate` rejects values that start with `scheme://`, `./`, `~/`, `${`, `$(` or `<REDACTED:`, dotted identifier paths (`process.env.X`) and `/`-paths holding a character outside the legacy class. Consumers with it on: retrieval output, transcript ingest, hooks and the hook heartbeat, the atoms drain, `embed-facts.ts` error text and the context-engine relay re-scan. Off: the push gate, bootstrap verify/repo and the compiled-context sensitivity scan. Own minimal allowlist + `<ws>/.gbrain-scan-allow` per-finding overrides — deliberately NOT the repo's `.gitleaks.toml` (a public-repo CI fixture policy); the allowlist block (`loadWorkspaceAllowlist`, the glob dialect `globToRegExp`/`matchesGlob`, `valueAllowlisted`/`pathAllowlisted`, `SCAN_ALLOW_FILENAME`, the 16-hex fingerprint floor) lives in `src/core/secret-scan-allowlist.ts` since wave 16 (G4.1, the 1500-line module cap) and is re-exported here unchanged (`test/secret-scan-allowlist.test.ts`). `correctedEntropy` is one histogram pass (distinct count = histogram size; byte-identical to the former `shannonEntropy + Set` form, `test/secret-scan-entropy-equivalence.test.ts`; ~80 → ~25 ms per 250 × 4096-char segment). `ScanOpts.work` / `RedactOpts.work` takes a `ScanWork` accumulator (`src/core/secret-scan-work.ts`: lines, regex execs, candidates per pattern, chars handed to the entropy gate counted inside `correctedEntropy`, validate calls, claims, union splits, echo sweeps) pinned exactly on the perf fixture by `test/secret-scan-work-counters.test.ts`; the perf file's absolute bounds stay the only guard on per-character cost. Deliberately NOT patterns: `sb_publishable_`, Supabase project refs (identifiers, not credentials). Never spell a literal database or `http(s)` URL with userinfo, or any credential, in this file's comments or in fixtures (assemble test values at runtime; `scripts/check-pg-url-redaction.sh` greps `src/` for every `db_url_credentials` scheme). Long lines are scanned in full (no length cap — a secret on a 1 MB minified line must still redact); what keeps that linear is bounded quantifiers on every shape whose value class could otherwise run to end-of-line — `db_url_credentials` (user ≤128 stopping at `:`/`/`; password 1–256 stopping only at whitespace, `@` and the string delimiters `"`/`'`, so unescaped `/ { } < > | ^ \` and backtick passwords redact), `jwt` (prebuilt so its boundary excludes `-` as well as `_`; segments ≤4096), `high_entropy_assignment` (≤64 identifier chars after the keyword, value ≤4096), `url_credentials` (user ≤128, password ≤256), `basic_auth` (value ≤2048, so its decode is constant work) and the private-key walks (below) — plus a per-pattern `precheck` substring gate on the catch-alls (`@` for both URL patterns; `earer`/`EARER`, since an all-caps `BEARER` scheme is accepted; a case-insensitive `basic`), a carried next-newline position in `scanPemBlocks` (sticky -1, so the header-line count is O(text) even when thousands of hits sit ahead of a newline-free multi-megabyte tail), a lazily-allocated per-line claimed-char bitmap behind the first-wins dedupe (O(value) per hit), and a windowed `buildPreview` (whole line ≤160 chars; otherwise hit−40…+80, edges snapped outward to whole spans, every claimed span in the window spliced by position, so a preview never carries a sibling secret or a fragment). `redactFindings` (= `planRedaction` + `applyRedaction`) rebuilds the text by SPAN-SPLICE in one O(text + hits) pass over the claimed spans, and fills the UNCLAIMED gaps with a bounded echo pass: the values claimed by `bearer` and `high_entropy_assignment` form an echo dictionary (value → pattern; floors `ECHO_MIN_CHARS_BEARER` = 20 / `ECHO_MIN_CHARS_ENTROPY` = 12, values ≤ `ECHO_MAX_VALUE_CHARS` = 512, the first `ECHO_MAX_UNIQUE` = 64 unique values in claim order across BOTH patterns — a value past either cap is redacted at its claim, its echoes are the accepted miss), matched by a streaming merge over the ORIGINAL text — per value, the next occurrence at or beyond the emit cursor (`indexOf` resuming at the cursor, re-sought only once the cursor passes it), leftmost-longest at each step, emitted incrementally into the unclaimed gaps — so retained state is O(values) however long the text is (occurrences are never collected into a list) (never an alternation RegExp, never over emitted tokens). Echoes add no findings. `planRedaction` feeds a caller-supplied shared map (`RedactOpts.echoValues`) and `applyRedaction` reads it at apply time, so a multi-field document scrubs an echo in one field of a value claimed in another. Every other pattern redacts exactly what `scanText` reports (a value embedded past its boundary is left as-is; vendor shapes re-match on their own). PRIVATE KEYS: `pemClaims` finds `BEGIN`/`END … PRIVATE KEY` fences in two global passes and merges them in text order. BLOCK: a BEGIN whose next fence is an END within `PEM_BODY_MAX_CHARS` (16384) is claimed header through END; a block never swallows a later BEGIN. FORWARD (BEGIN with no END in view): the header plus the body lines that follow; a header followed by no body line (a marker quoted in prose) is claimed alone. BACKWARD (END with no BEGIN in view): the body lines before the fence plus the fence; an END with no body line before it is no claim. A body line is a whole-line base64 run (`A-Za-z0-9+/=`) indented by up to 16 spaces/tabs; line ends are `\n`, `\r\n` or the literal JSON escapes; interior lines are 16–128 chars, the line next to a fence and a partial line at a cut 1–128; optional RFC 1421 `Proc-Type:`/`DEK-Info:` lines and one blank line may follow BEGIN; a line with anything else, or a run over 128 chars, ends the body. Total claimed body stays within `PEM_BODY_MAX_CHARS`. Each walk stops at the neighbouring fence or the previous claim, so the pass is linear (20k unterminated BEGINs, 20k END-only fences and 1 MB of base64 lines are perf-pinned). Per-line hits that fall wholly inside a key claim are dropped (one key, one finding). A FORWARD claim's finding also carries `legacyFingerprint`, the header-only fingerprint earlier releases reported, so `workspace-push.ts` can name a stale allowlist entry; a truncated key's fingerprint covers its body and therefore depends on where the text was cut, so it is not a stable allowlist key. `privateKeySpans(text)` exports the same claims as sorted, disjoint `{start, end}` spans. Accepted misses: body on the header's own line, space-joined single-line keys, blockquoted (`> `) keys, base64 runs over 128 chars, body past 16384 chars, `http(s)` passwords containing a literal `/`, `?` or `#`, Basic values over 2048 chars, digitless assignment values, dotted passwords whose every segment starts with a letter, and unquoted passwords containing `,` or `;`. False-positive budget: `scripts/secret-scan-fp-budget.ts` diffs findings against a base ref's scanner (`--base`, default `origin/master`) (`--dir <brain>` reports per-pattern counts only, never content) and `test/secret-scan-fp-budget.test.ts` pins a synthetic brain-like fixture (zero findings on its non-secret part in both modes, every planted secret redacted). Shape tests: `test/secret-scan-credential-shapes.test.ts`, `test/secret-scan-private-key-spans.test.ts`. Perf pins: `test/secret-scan-perf.test.ts`. Retrieval output is another consumer of its canonical scanner; that output boundary does not sanitize internal evaluation/provider inputs.
+- `src/core/secret-scan.ts` — pattern-based secret detection shared by `sources push`, bootstrap verify, the compiled-context sensitivity scan and the transcripts redactor. Detection is FORMAT-based, not prefix-only: vendor shapes (anthropic, openai, voyage, github, gitlab, slack, aws AKIA/ASIA, google, stripe, sendgrid, twilio, supabase sb_secret_/sbp_, npm, huggingface, `digitalocean` `do[opr]_v1_` + 64 hex, the AI-provider keys `openrouter` `sk-or-v1-` + 64 lowercase hex, `groq` `gsk_` + 52, `xai` `xai-` + 80, `perplexity` `pplx-` + 48 and `replicate` `r8_` + 37, all fixed-length with a hard right edge, gbrain), JWTs, private keys (whole-text claims, below), plus four catch-alls appended LAST (`bearer` — prebuilt two-group layout so only the token is the value; `db_url_credentials` — the `scheme://user:pass@` span of database URLs, host/db kept; `url_credentials` — the same span for `http(s)` (scheme case-insensitive, `gi`), user 0–128 stopping at `:`/`/`, password 1–256 excluding whitespace, `@`, `"`, `'`, `/`, `?`, `#`, with a `validate` hook that rejects placeholder passwords (`<...>`, `${...}`, `$VAR`, all `*`, all `x`); `basic_auth` — two prebuilt case-insensitive forms, `Authorization: Basic` with a 4–2048 char value and bare `Basic` with 16–2048, both with a hard right edge and a `validate` that requires the value to base64-decode to printable ASCII containing `:`; it is NOT in the bearer family, so it joins neither the echo dictionary nor `secret:bearer`). The pattern contract: `CorePattern` carries optional `flags` (default `'g'`), `precheck`, `validate(value)` (O(value), run on the whole match before the claim and the union split; a rejected match is not re-tried from inside) and `since` (the gbrain version that added or last changed the rule; `SECURITY_WAVE_SINCE` = `0.60.31.0` on `digitalocean`, `url_credentials`, `basic_auth` and `private_key_pem`, `PROVIDER_KEYS_SINCE` on the five AI-provider rules (#7959), kept in step with the release that ships a rule change). Compiled patterns keep group 1 = anchor, group 2 = value. Every core rule and both `high_entropy_assignment` forms also have an escape-boundary twin (#7978), compiled right after the rule under the same name and anchored on a JSON escape (`\n`, `\r`, `\t`, `\b`, `\f`, `\uXXXX`) or a percent-escape (`%XX`), whose last character the default boundary reads as a word character; a twin runs only on a line `ESCAPE_IN_LINE_RE` matches, so other lines run the old list, and a prebuilt rule carries its twin as `CorePattern.escaped`. `patternSince(pattern)` reads the map; every finding of such a pattern carries `SecretFinding.since`, which the push and bootstrap refusals render. Fixed-length shapes carry a trailing negative lookahead. Per-line claimed-span dedupe is first-wins for ATTRIBUTION (order matters: `Bearer <vendor key>` keeps the vendor name, a fully covered later span is skipped) but coverage is the UNION: every unclaimed slice of a later overlapping span is claimed as a hit of the later pattern (own value, own fingerprint), so a database URL whose username is a vendor-shaped id still loses its password. The transcript-only `labeled_credential` rule (`ScanOpts.labeledCredentials`) lives in `secret-scan-labeled.ts` (Security). The opt-in `high_entropy_assignment` rule (`ScanOpts.highEntropy`; its shapes and gates live in `src/core/secret-scan-assignment.ts`, split out at the 1500-line module cap) requires a 12+ char value that contains a digit and a non-digit AND clears 3.5 bits/char on the bias-corrected (Miller-Madow) estimate `correctedEntropy`, since raw per-character entropy is capped at log2(n) and rejected most random 12-16 char secrets. It is two compiled patterns sharing the name: QUOTED (after `"` or `'`, any run of non-quote, non-whitespace characters) and UNQUOTED (the legacy `[A-Za-z0-9+/_=-]` class plus `!#$%*@^~.?<>`; never `&`, `()[]{}`, `:`, `,` or `;`, so query strings, calls and SQL `SET` lists stay intact, and a password carrying `,`/`;` matches only quoted; the last character excludes `.?!`, so a sentence's full stop is not claimed). Its `validate` rejects values that start with `scheme://`, `./`, `~/`, `${`, `$(` or `<REDACTED:`, dotted identifier paths (`process.env.X`) and `/`-paths holding a character outside the legacy class. Consumers with it on: retrieval output, transcript ingest, hooks and the hook heartbeat, the atoms drain, `embed-facts.ts` error text and the context-engine relay re-scan. Off: the push gate, bootstrap verify/repo and the compiled-context sensitivity scan. Own minimal allowlist + `<ws>/.gbrain-scan-allow` per-finding overrides — deliberately NOT the repo's `.gitleaks.toml` (a public-repo CI fixture policy); the allowlist block (`loadWorkspaceAllowlist`, the glob dialect `globToRegExp`/`matchesGlob`, `valueAllowlisted`/`pathAllowlisted`, `SCAN_ALLOW_FILENAME`, the 16-hex fingerprint floor) lives in `src/core/secret-scan-allowlist.ts` since wave 16 (G4.1, the 1500-line module cap) and is re-exported here unchanged (`test/secret-scan-allowlist.test.ts`). `correctedEntropy` is one histogram pass (distinct count = histogram size; byte-identical to the former `shannonEntropy + Set` form, `test/secret-scan-entropy-equivalence.test.ts`; ~80 → ~25 ms per 250 × 4096-char segment). `ScanOpts.work` / `RedactOpts.work` takes a `ScanWork` accumulator (`src/core/secret-scan-work.ts`: lines, regex execs, candidates per pattern, chars handed to the entropy gate counted inside `correctedEntropy`, validate calls, claims, union splits, echo sweeps) pinned exactly on the perf fixture by `test/secret-scan-work-counters.test.ts`; the perf file's absolute bounds stay the only guard on per-character cost. Deliberately NOT patterns: `sb_publishable_`, Supabase project refs (identifiers, not credentials). Never spell a literal database or `http(s)` URL with userinfo, or any credential, in this file's comments or in fixtures (assemble test values at runtime; `scripts/check-pg-url-redaction.sh` greps `src/` for every `db_url_credentials` scheme). Long lines are scanned in full (no length cap — a secret on a 1 MB minified line must still redact); what keeps that linear is bounded quantifiers on every shape whose value class could otherwise run to end-of-line — `db_url_credentials` (user ≤128 stopping at `:`/`/`; password 1–256 stopping only at whitespace, `@` and the string delimiters `"`/`'`, so unescaped `/ { } < > | ^ \` and backtick passwords redact), `jwt` (prebuilt so its boundary excludes `-` as well as `_`; segments ≤4096), `high_entropy_assignment` (≤64 identifier chars after the keyword, value ≤4096), `url_credentials` (user ≤128, password ≤256), `basic_auth` (value ≤2048, so its decode is constant work) and the private-key walks (below) — plus a per-pattern `precheck` substring gate on the catch-alls (`@` for both URL patterns; `earer`/`EARER`, since an all-caps `BEARER` scheme is accepted; a case-insensitive `basic`), a carried next-newline position in `scanPemBlocks` (sticky -1, so the header-line count is O(text) even when thousands of hits sit ahead of a newline-free multi-megabyte tail), a lazily-allocated per-line claimed-char bitmap behind the first-wins dedupe (O(value) per hit), and a windowed `buildPreview` (whole line ≤160 chars; otherwise hit−40…+80, edges snapped outward to whole spans, every claimed span in the window spliced by position, so a preview never carries a sibling secret or a fragment). `redactFindings` (= `planRedaction` + `applyRedaction`) rebuilds the text by SPAN-SPLICE in one O(text + hits) pass over the claimed spans, and fills the UNCLAIMED gaps with a bounded echo pass: the values claimed by `bearer`, `high_entropy_assignment`, `labeled_credential` and every vendor or `jwt` rule (#7978; not `basic_auth` or the two URL userinfo catch-alls) form an echo dictionary (value → pattern; floors `ECHO_MIN_CHARS_BEARER` = `ECHO_MIN_CHARS_VENDOR` = 20 / `ECHO_MIN_CHARS_ENTROPY` = 12, values ≤ `ECHO_MAX_VALUE_CHARS` = 512, the first `ECHO_MAX_UNIQUE` = 64 unique values in claim order across every pattern — a value past either cap is redacted at its claim, its echoes are the accepted miss), matched by a streaming merge over the ORIGINAL text — per value, the next occurrence at or beyond the emit cursor (`indexOf` resuming at the cursor, re-sought only once the cursor passes it), leftmost-longest at each step, emitted incrementally into the unclaimed gaps — so retained state is O(values) however long the text is (occurrences are never collected into a list) (never an alternation RegExp, never over emitted tokens). Echoes add no findings. `planRedaction` feeds a caller-supplied shared map (`RedactOpts.echoValues`) and `applyRedaction` reads it at apply time, so a multi-field document scrubs an echo in one field of a value claimed in another. `basic_auth` and the URL userinfo catch-alls redact exactly what `scanText` reports; a vendor value embedded past its boundary (`token_<key>`) is scrubbed as an echo when the same value is claimed elsewhere in the document, and left as-is otherwise. PRIVATE KEYS: `pemClaims` finds `BEGIN`/`END … PRIVATE KEY` fences in two global passes and merges them in text order. BLOCK: a BEGIN whose next fence is an END within `PEM_BODY_MAX_CHARS` (16384) is claimed header through END; a block never swallows a later BEGIN. FORWARD (BEGIN with no END in view): the header plus the body lines that follow; a header followed by no body line (a marker quoted in prose) is claimed alone. BACKWARD (END with no BEGIN in view): the body lines before the fence plus the fence; an END with no body line before it is no claim. A body line is a whole-line base64 run (`A-Za-z0-9+/=`) indented by up to 16 spaces/tabs; line ends are `\n`, `\r\n` or the literal JSON escapes; interior lines are 16–128 chars, the line next to a fence and a partial line at a cut 1–128; optional RFC 1421 `Proc-Type:`/`DEK-Info:` lines and one blank line may follow BEGIN; a line with anything else, or a run over 128 chars, ends the body. Total claimed body stays within `PEM_BODY_MAX_CHARS`. Each walk stops at the neighbouring fence or the previous claim, so the pass is linear (20k unterminated BEGINs, 20k END-only fences and 1 MB of base64 lines are perf-pinned). Per-line hits that fall wholly inside a key claim are dropped (one key, one finding). A FORWARD claim's finding also carries `legacyFingerprint`, the header-only fingerprint earlier releases reported, so `workspace-push.ts` can name a stale allowlist entry; a truncated key's fingerprint covers its body and therefore depends on where the text was cut, so it is not a stable allowlist key. `privateKeySpans(text)` exports the same claims as sorted, disjoint `{start, end}` spans. Accepted misses: body on the header's own line, space-joined single-line keys, blockquoted (`> `) keys, base64 runs over 128 chars, body past 16384 chars, `http(s)` passwords containing a literal `/`, `?` or `#`, Basic values over 2048 chars, digitless assignment values, dotted passwords whose every segment starts with a letter, and unquoted passwords containing `,` or `;`. False-positive budget: `scripts/secret-scan-fp-budget.ts` diffs findings against a base ref's scanner (`--base`, default `origin/master`) (`--dir <brain>` reports per-pattern counts only, never content) and `test/secret-scan-fp-budget.test.ts` pins a synthetic brain-like fixture (zero findings on its non-secret part in both modes, every planted secret redacted). Shape tests: `test/secret-scan-credential-shapes.test.ts`, `test/secret-scan-private-key-spans.test.ts`. Perf pins: `test/secret-scan-perf.test.ts`. Retrieval output is another consumer of its canonical scanner; that output boundary does not sanitize internal evaluation/provider inputs.
- `src/core/secret-scan-stream.ts` — `scanTextStream(chunks, opts)`, the windowed scan of a byte stream too large to hold as one string (the push gate's staged blobs over 25 MiB). Bytes are decoded by a streaming `TextDecoder` (`ignoreBOM`, so a split multi-byte character joins and invalid bytes become U+FFFD as `Buffer.toString` makes them) and handed to `scanTextSpans` (`secret-scan.ts`: `scanText` plus each claim's absolute span, allowlisted claims kept with `finding: null`) in windows of at most `STREAM_SCAN_WINDOW_CHARS` (8 Mi UTF-16 units). Each next window starts at a line start at least `STREAM_SCAN_OVERLAP_CHARS` (64 Ki, above the longest claim: a private key's 16384-char body plus fences, a jwt's 3 × 4096) before the previous end, or mid-line at exactly the overlap (never inside a surrogate pair) when the line is too long. A window keeps the claims that start before the next window's start, the last window keeps the rest, and a claim wholly inside a span an earlier window kept (a seam's partial view, or an exact duplicate) is dropped; lines are 1-based over the whole stream. `streamScanWork` (@internal) counts windows, bytes, scanned chars and the largest window. `test/secret-scan-stream.test.ts` pins equality with `scanText` across both seams of a window, mid-line splits, split multi-byte characters, invalid UTF-8 and a BOM.
- `src/core/context/sensitivity-scan.ts` — composed detector for compiled-context entries (secret-scan → PII → path shapes → operator blocklist → operator pattern file). secret-scan OWNS credential-shaped families it also matches: a PII finding whose value (raw, or with the `Bearer ` keyword stripped) matches a secret finding's fingerprint is not reported twice; short bearer tokens under secret-scan's floor still surface as `pii:bearer`. Every family honors the same `.gbrain-scan-allow` fingerprint escape hatch. `formatSensitivityDrop(drop, workspaceRoot)` renders the one content-free stderr line `gbrain compile-context` prints per dropped entry: reason `sensitivity_scan`, the pattern family, the fingerprint and the absolute `.gbrain-scan-allow` path, never the value (`test/sensitivity-scan.test.ts`).
diff --git a/src/core/secret-scan-assignment.ts b/src/core/secret-scan-assignment.ts
new file mode 100644
index 0000000000..be950d69cb
--- /dev/null
+++ b/src/core/secret-scan-assignment.ts
@@ -0,0 +1,107 @@
+/**
+ * secret-scan-assignment.ts: the opt-in `high_entropy_assignment` rule's
+ * shapes and gates (`ScanOpts.highEntropy`), split out of secret-scan.ts at
+ * its 1500-line module cap (#7978). secret-scan.ts compiles these sources,
+ * plain and behind its ESCAPE_BOUNDARY, and runs the gates; nothing else
+ * imports this module.
+ */
+
+// Opt-in: `secret|token|password|api key`-shaped assignment whose value has
+// high Shannon entropy. Keyword-anchored (compiled inline below so the group
+// layout matches the core patterns: group 1 = anchor, group 2 = value).
+//
+// The left boundary is [^A-Za-z0-9] rather than the core patterns'
+// [^A-Za-z0-9_]: `_` has to read as a SEPARATOR here, or `api_access_token`
+// and `SMTP_PASSWORD` cannot match their own keyword. The core patterns keep
+// `_` as a word character on purpose (a vendor prefix inside a longer
+// identifier must not fire); this rule is keyword-anchored, so the same
+// reasoning inverts. The keyword is also allowed trailing identifier
+// segments before the assignment, or `AWS_SECRET_ACCESS_KEY=` cannot match:
+// the keyword `SECRET` is not adjacent to the `=`, `_ACCESS_KEY` is. The
+// entropy gate still decides, so a wider anchor costs nothing on prose.
+//
+// The value floor is 12, not 20. A 16-character SMTP password sat under the
+// old floor and stayed plaintext in the receipt on disk even once the keyword
+// matched — the value length was doing gating the entropy check is there to
+// do. Real passwords are frequently 12-16 characters; secrets that long with
+// 3.5 bits/char of entropy are not prose.
+//
+// The value must ALSO contain at least one digit. Identifier-shaped values
+// (`DefaultAzureCredential`, `/usr/local/bin/aws-vault`, an env-var NAME
+// assigned to an `apiKeyEnvVar`) clear the entropy gate on mixed case and
+// separators alone and were being redacted out of ordinary code. Real
+// machine-minted secrets essentially always carry digits; a digitless
+// passphrase is the accepted miss. A 40-hex git sha assigned to a `token:`
+// key still redacts (digits + entropy) — documented, acceptable.
+//
+// Both quantifiers after the keyword are BOUNDED. The trailing identifier
+// segments used to be `[A-Za-z0-9_-]*`: because the left boundary admits `-`
+// and `_` and that class contains them too, every `-`/`_` in a run like
+// `-apikey-apikey…` or `_token_token…` was a fresh keyword start whose
+// suffix ran to end-of-line and backtracked — quadratic (210 KB of
+// `-apikey` took ~35 s, 180 KB of `_token` ~15 s). At `{0,64}` a start does
+// constant work; no real credential key carries 64 identifier characters
+// after its keyword. The value is capped at 4096 for the same
+// constant-work-per-occurrence guarantee (the jwt segments share the cap):
+// a value longer than that is redacted only through its first 4096 chars —
+// the accepted miss, well above any real token or base64 key blob.
+//
+// The 3.5 bits/char floor is judged on the bias-corrected (Miller-Madow)
+// estimate, not the raw per-character Shannon entropy. The raw figure of an
+// n-character string can never exceed log2(n) (3.58 at 12 characters, 3.91
+// at 15) and drops sharply with a single repeated character, so the raw
+// floor rejected 68% of uniformly random 12-character alphanumeric secrets,
+// 9% at 15 and 5% at 16: exactly the 12-16 character passwords the 12-char
+// value floor exists to catch (a 15-character value such as `!dcG4Gmw1qGRbR3`
+// after a password key stayed plaintext).
+// Miller-Madow adds (distinct - 1) / (2 n ln 2), which cuts those misses
+// below 1% at every length from 12 up and is negligible on long values.
+// Because the correction can lift an all-digit value past the floor, the
+// value must also carry a non-digit: counters, timestamps and numeric ids
+// (`token_count=1234567890123`) never reached 3.5 raw bits (log2(10) = 3.32)
+// and still never redact.
+export const HIGH_ENTROPY_MIN_BITS_PER_CHAR = 3.5;
+export const HIGH_ENTROPY_REQUIRES_DIGIT_RE = /[0-9]/;
+export const HIGH_ENTROPY_REQUIRES_NON_DIGIT_RE = /[^0-9]/;
+
+// PUNCTUATED VALUES. The rule is two compiled patterns sharing the name and
+// the group-2-is-value contract. QUOTED: after an opening `"` or `'`, the
+// value is any run of non-quote, non-whitespace characters (a password like
+// `ab!#%…<…` is one value). UNQUOTED: the legacy class plus the password
+// punctuation `!#$%*@^~.?<>` — not `&` (it would swallow sibling query
+// parameters), not `()[]{}` (so `token = getToken(x)` never matches), not `:`,
+// and not `,` / `;` (the false-positive measurement found SQL `SET` lists
+// such as `execution_token=NULL,claim_expires_at=NULL,error_code=$3` read as
+// one value; a password carrying them is matched only when quoted) — and the
+// last character excludes `.?!`, so `TOKEN=<tok>.` claims the token without
+// the sentence's full stop and its bare echoes still scrub.
+// Both keep the 12-4096 bounds and the digit + entropy gates. Each also has
+// an ESCAPE_BOUNDARY twin (#7978): `\\npassword=…` in escaped tool output.
+export const assignmentKey = (boundary: string) =>
+ `${boundary}(?:secret|token|passwd|password|passphrase|credential|api[_-]?key|apikey)[A-Za-z0-9_-]{0,64}["']?\\s*[:=]\\s*`;
+export const ASSIGNMENT_QUOTED_SOURCE = (key: string) => `(${key}["'])([^"'\\s]{12,4096})`;
+export const ASSIGNMENT_UNQUOTED_SOURCE = (key: string) =>
+ `(${key})([A-Za-z0-9+/_=\\-!#$%*@^~.?<>]{11,4095}[A-Za-z0-9+/_=\\-#$%*@^~<>])`;
+/** The legacy (pre-punctuation) value class, whole-value form. */
+const LEGACY_ASSIGNMENT_VALUE_RE = /^[A-Za-z0-9+/_=-]+$/;
+/** A dotted member path in code (`process.env.DB_PASSWORD_2`): every segment an identifier. */
+const DOTTED_IDENTIFIER_RE = /^[A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)+$/;
+
+/**
+ * Assignment-value validate: reject values that are references, not secrets
+ * — a URL (`scheme://`), a relative or home path (`./`, `~/`), a shell
+ * expansion (`${`, `$(`), an already-redacted token, an absolute path (`/`
+ * plus a character outside the legacy class, such as the `.` of a file
+ * name), and a dotted member path whose every segment is an identifier
+ * (`process.env.X`, `config.secrets.key2`: the `.` admitted for passwords
+ * would otherwise turn code into a value). A leading `/` on a pure base64
+ * value still fires: one random base64 secret in 64 starts with `/`, and the
+ * legacy rule redacted those. A dotted password whose every segment starts
+ * with a letter is the accepted miss.
+ */
+export function assignmentValueIsSecretShaped(value: string): boolean {
+ if (/^[A-Za-z][A-Za-z0-9+.-]{0,31}:\/\//.test(value)) return false;
+ if (value.startsWith('./') || value.startsWith('~/') || value.startsWith('${') || value.startsWith('$(')) return false;
+ if (value.startsWith('<REDACTED:') || DOTTED_IDENTIFIER_RE.test(value)) return false;
+ return !(value.startsWith('/') && !LEGACY_ASSIGNMENT_VALUE_RE.test(value));
+}
diff --git a/src/core/secret-scan-stream.ts b/src/core/secret-scan-stream.ts
index a7ac59a49b..123c0f1afa 100644
--- a/src/core/secret-scan-stream.ts
+++ b/src/core/secret-scan-stream.ts
@@ -16,7 +16,7 @@
*
* OWNERSHIP. A claim belongs to the window holding the start of the match that
* made it, anchor included (`from`: the `Bearer ` or `Authorization: Basic `
- * keyword, an assignment key, or the boundary character before a value), not
+ * keyword, an assignment key, or the boundary character or escape before a value), not
* to the window holding the value's first character: a mid-line cut between
* an anchor and its value leaves the next window a value with no anchor, which
* no pattern claims. A window keeps a claim whose match starts before the next
diff --git a/src/core/secret-scan.ts b/src/core/secret-scan.ts
index 9c21ea6c80..3bd0cb9bc1 100644
--- a/src/core/secret-scan.ts
+++ b/src/core/secret-scan.ts
@@ -97,6 +97,10 @@ export {
} from './secret-scan-allowlist.ts';
import { valueAllowlisted, pathAllowlisted } from './secret-scan-allowlist.ts';
import { PUSH_SCAN_MAX_FILE_BYTES } from './page-size-policy.ts';
+import {
+ ASSIGNMENT_QUOTED_SOURCE, ASSIGNMENT_UNQUOTED_SOURCE, HIGH_ENTROPY_MIN_BITS_PER_CHAR, HIGH_ENTROPY_REQUIRES_DIGIT_RE,
+ HIGH_ENTROPY_REQUIRES_NON_DIGIT_RE, assignmentKey, assignmentValueIsSecretShaped,
+} from './secret-scan-assignment.ts';
import type { ScanWork } from './secret-scan-work.ts';
export { createScanWork, type ScanWork } from './secret-scan-work.ts';
@@ -104,7 +108,7 @@ export { createScanWork, type ScanWork } from './secret-scan-work.ts';
//
// Each core pattern is compiled with a left boundary group `(^|[^A-Za-z0-9_])`
// so prefixes embedded inside longer identifiers ("risk-assessment…",
-// "task-…") never fire. Order matters: 'anthropic' precedes 'openai' and the
+// "task-…") never fire, plus an escape-boundary twin (ESCAPE_BOUNDARY, #7978). Order matters: 'anthropic' precedes 'openai' and the
// per-line claimed-span set prevents a `sk-ant-…` key from double-reporting
// as a generic `sk-` match.
//
@@ -133,6 +137,8 @@ interface CompiledPattern {
continuation?: 'single' | 'pair';
/** The text the regex reads instead of the line, the same length (offsets and reported values come from the line). */
subject?: (line: string) => string;
+ /** An ESCAPE_BOUNDARY twin: runs only on a line ESCAPE_IN_LINE_RE matches. */
+ afterEscape?: boolean;
}
interface CorePattern {
@@ -145,6 +151,11 @@ interface CorePattern {
* literal keyword (e.g. `Bearer `) that must NOT be part of the value.
*/
prebuilt?: boolean;
+ /**
+ * Prebuilt rules only: the same two-group source anchored on ESCAPE_BOUNDARY
+ * instead (#7978). A default-boundary rule gets that twin automatically.
+ */
+ escaped?: string;
/** RegExp flags. Default `'g'`; a case-insensitive shape uses `'gi'`. */
flags?: string;
/**
@@ -182,6 +193,22 @@ const SECURITY_WAVE_SINCE = '0.60.31.0';
*/
const PROVIDER_KEYS_SINCE = '0.60.502.0';
+/**
+ * #7978: a JSON escape (`\\n`, `\\t`, `\\u0022`) or a percent-escape (`%22`)
+ * ends on a word character, so the default boundary never fired after one and
+ * a key opening an escaped JSON string or a URL-encoded value went unclaimed
+ * in every lane. Each rule gets a twin anchored here, compiled right after it
+ * under the same name (attribution order unchanged) and run only on lines
+ * holding such an escape, so every other line runs the old pattern list. A
+ * backslash or `%` that does not start an escape is not a boundary.
+ */
+const ESCAPE_BOUNDARY = '(?:\\\\(?:[bfnrt]|u[0-9A-Fa-f]{4})|%[0-9A-Fa-f]{2})';
+const ESCAPE_IN_LINE_RE = /\\[bfnrtu]|%[0-9A-Fa-f]{2}/;
+const JWT_VALUE = '(eyJ[A-Za-z0-9_-]{8,4096}\\.[A-Za-z0-9_-]{8,4096}\\.[A-Za-z0-9_-]{8,4096})';
+const BEARER_ANCHOR = '[Bb](?:earer|EARER)\\s+)([A-Za-z0-9._~+/=-]{20,})';
+const BASIC_HEADER = 'authorization\\s{0,16}:\\s{0,16}basic\\s{1,16})([A-Za-z0-9+/=]{4,2048})(?![A-Za-z0-9+/=])';
+const BASIC_BARE = 'basic\\s{1,16})([A-Za-z0-9+/=]{16,2048})(?![A-Za-z0-9+/=])';
+
/**
* Placeholder passwords in documentation URLs: `<password>`, `${VAR}`,
* `$VAR`, all-`*` and all-`x` masks. Never a live credential, so
@@ -283,10 +310,9 @@ const CORE_PATTERNS: ReadonlyArray<CorePattern> = [
// front of it); a segment over 4096 chars is the accepted miss.
{
name: 'jwt',
- source:
- '(^|[^A-Za-z0-9_-])' +
- '(eyJ[A-Za-z0-9_-]{8,4096}\\.[A-Za-z0-9_-]{8,4096}\\.[A-Za-z0-9_-]{8,4096})',
+ source: `(^|[^A-Za-z0-9_-])${JWT_VALUE}`,
prebuilt: true,
+ escaped: `(${ESCAPE_BOUNDARY})${JWT_VALUE}`,
},
// ── Catch-alls: LAST, so vendor/JWT attribution above claims the span first.
// Bearer: prebuilt two-group layout — the `Bearer ` keyword is the anchor
@@ -298,8 +324,9 @@ const CORE_PATTERNS: ReadonlyArray<CorePattern> = [
// skipped before the regex runs.
{
name: 'bearer',
- source: '((?:^|[^A-Za-z0-9_])[Bb](?:earer|EARER)\\s+)([A-Za-z0-9._~+/=-]{20,})',
+ source: `((?:^|[^A-Za-z0-9_])${BEARER_ANCHOR}`,
prebuilt: true,
+ escaped: `(${ESCAPE_BOUNDARY}${BEARER_ANCHOR}`,
precheck: (line) => line.includes('earer') || line.includes('EARER'),
},
// Connection strings with inline credentials: the value is exactly the
@@ -369,8 +396,9 @@ const CORE_PATTERNS: ReadonlyArray<CorePattern> = [
// family.
{
name: 'basic_auth',
- source: '((?:^|[^A-Za-z0-9_])authorization\\s{0,16}:\\s{0,16}basic\\s{1,16})([A-Za-z0-9+/=]{4,2048})(?![A-Za-z0-9+/=])',
+ source: `((?:^|[^A-Za-z0-9_])${BASIC_HEADER}`,
prebuilt: true,
+ escaped: `(${ESCAPE_BOUNDARY}${BASIC_HEADER}`,
flags: 'gi',
precheck: (line) => BASIC_PRECHECK_RE.test(line),
validate: basicValueDecodesToUserPass,
@@ -378,8 +406,9 @@ const CORE_PATTERNS: ReadonlyArray<CorePattern> = [
},
{
name: 'basic_auth',
- source: '((?:^|[^A-Za-z0-9_])basic\\s{1,16})([A-Za-z0-9+/=]{16,2048})(?![A-Za-z0-9+/=])',
+ source: `((?:^|[^A-Za-z0-9_])${BASIC_BARE}`,
prebuilt: true,
+ escaped: `(${ESCAPE_BOUNDARY}${BASIC_BARE}`,
flags: 'gi',
precheck: (line) => BASIC_PRECHECK_RE.test(line),
validate: basicValueDecodesToUserPass,
@@ -619,104 +648,7 @@ export function privateKeySpans(text: string): Array<{ start: number; end: numbe
return pemClaims(text).map((c) => ({ start: c.start, end: c.end }));
}
-// Opt-in: `secret|token|password|api key`-shaped assignment whose value has
-// high Shannon entropy. Keyword-anchored (compiled inline below so the group
-// layout matches the core patterns: group 1 = anchor, group 2 = value).
-//
-// The left boundary is [^A-Za-z0-9] rather than the core patterns'
-// [^A-Za-z0-9_]: `_` has to read as a SEPARATOR here, or `api_access_token`
-// and `SMTP_PASSWORD` cannot match their own keyword. The core patterns keep
-// `_` as a word character on purpose (a vendor prefix inside a longer
-// identifier must not fire); this rule is keyword-anchored, so the same
-// reasoning inverts. The keyword is also allowed trailing identifier
-// segments before the assignment, or `AWS_SECRET_ACCESS_KEY=` cannot match:
-// the keyword `SECRET` is not adjacent to the `=`, `_ACCESS_KEY` is. The
-// entropy gate still decides, so a wider anchor costs nothing on prose.
-//
-// The value floor is 12, not 20. A 16-character SMTP password sat under the
-// old floor and stayed plaintext in the receipt on disk even once the keyword
-// matched — the value length was doing gating the entropy check is there to
-// do. Real passwords are frequently 12-16 characters; secrets that long with
-// 3.5 bits/char of entropy are not prose.
-//
-// The value must ALSO contain at least one digit. Identifier-shaped values
-// (`DefaultAzureCredential`, `/usr/local/bin/aws-vault`, an env-var NAME
-// assigned to an `apiKeyEnvVar`) clear the entropy gate on mixed case and
-// separators alone and were being redacted out of ordinary code. Real
-// machine-minted secrets essentially always carry digits; a digitless
-// passphrase is the accepted miss. A 40-hex git sha assigned to a `token:`
-// key still redacts (digits + entropy) — documented, acceptable.
-//
-// Both quantifiers after the keyword are BOUNDED. The trailing identifier
-// segments used to be `[A-Za-z0-9_-]*`: because the left boundary admits `-`
-// and `_` and that class contains them too, every `-`/`_` in a run like
-// `-apikey-apikey…` or `_token_token…` was a fresh keyword start whose
-// suffix ran to end-of-line and backtracked — quadratic (210 KB of
-// `-apikey` took ~35 s, 180 KB of `_token` ~15 s). At `{0,64}` a start does
-// constant work; no real credential key carries 64 identifier characters
-// after its keyword. The value is capped at 4096 for the same
-// constant-work-per-occurrence guarantee (the jwt segments share the cap):
-// a value longer than that is redacted only through its first 4096 chars —
-// the accepted miss, well above any real token or base64 key blob.
-//
-// The 3.5 bits/char floor is judged on the bias-corrected (Miller-Madow)
-// estimate, not the raw per-character Shannon entropy. The raw figure of an
-// n-character string can never exceed log2(n) (3.58 at 12 characters, 3.91
-// at 15) and drops sharply with a single repeated character, so the raw
-// floor rejected 68% of uniformly random 12-character alphanumeric secrets,
-// 9% at 15 and 5% at 16: exactly the 12-16 character passwords the 12-char
-// value floor exists to catch (a 15-character value such as `!dcG4Gmw1qGRbR3`
-// after a password key stayed plaintext).
-// Miller-Madow adds (distinct - 1) / (2 n ln 2), which cuts those misses
-// below 1% at every length from 12 up and is negligible on long values.
-// Because the correction can lift an all-digit value past the floor, the
-// value must also carry a non-digit: counters, timestamps and numeric ids
-// (`token_count=1234567890123`) never reached 3.5 raw bits (log2(10) = 3.32)
-// and still never redact.
-const HIGH_ENTROPY_MIN_BITS_PER_CHAR = 3.5;
-const HIGH_ENTROPY_REQUIRES_DIGIT_RE = /[0-9]/;
-const HIGH_ENTROPY_REQUIRES_NON_DIGIT_RE = /[^0-9]/;
-
-// PUNCTUATED VALUES. The rule is two compiled patterns sharing the name and
-// the group-2-is-value contract. QUOTED: after an opening `"` or `'`, the
-// value is any run of non-quote, non-whitespace characters (a password like
-// `ab!#%…<…` is one value). UNQUOTED: the legacy class plus the password
-// punctuation `!#$%*@^~.?<>` — not `&` (it would swallow sibling query
-// parameters), not `()[]{}` (so `token = getToken(x)` never matches), not `:`,
-// and not `,` / `;` (the false-positive measurement found SQL `SET` lists
-// such as `execution_token=NULL,claim_expires_at=NULL,error_code=$3` read as
-// one value; a password carrying them is matched only when quoted) — and the
-// last character excludes `.?!`, so `TOKEN=<tok>.` claims the token without
-// the sentence's full stop and its bare echoes still scrub.
-// Both keep the 12-4096 bounds and the digit + entropy gates.
-const ASSIGNMENT_KEY =
- '(?:^|[^A-Za-z0-9])(?:secret|token|passwd|password|passphrase|credential|api[_-]?key|apikey)[A-Za-z0-9_-]{0,64}["\']?\\s*[:=]\\s*';
-const ASSIGNMENT_QUOTED_SOURCE = `(${ASSIGNMENT_KEY}["'])([^"'\\s]{12,4096})`;
-const ASSIGNMENT_UNQUOTED_SOURCE =
- `(${ASSIGNMENT_KEY})([A-Za-z0-9+/_=\\-!#$%*@^~.?<>]{11,4095}[A-Za-z0-9+/_=\\-#$%*@^~<>])`;
-/** The legacy (pre-punctuation) value class, whole-value form. */
-const LEGACY_ASSIGNMENT_VALUE_RE = /^[A-Za-z0-9+/_=-]+$/;
-/** A dotted member path in code (`process.env.DB_PASSWORD_2`): every segment an identifier. */
-const DOTTED_IDENTIFIER_RE = /^[A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)+$/;
-
-/**
- * Assignment-value validate: reject values that are references, not secrets
- * — a URL (`scheme://`), a relative or home path (`./`, `~/`), a shell
- * expansion (`${`, `$(`), an already-redacted token, an absolute path (`/`
- * plus a character outside the legacy class, such as the `.` of a file
- * name), and a dotted member path whose every segment is an identifier
- * (`process.env.X`, `config.secrets.key2`: the `.` admitted for passwords
- * would otherwise turn code into a value). A leading `/` on a pure base64
- * value still fires: one random base64 secret in 64 starts with `/`, and the
- * legacy rule redacted those. A dotted password whose every segment starts
- * with a letter is the accepted miss.
- */
-function assignmentValueIsSecretShaped(value: string): boolean {
- if (/^[A-Za-z][A-Za-z0-9+.-]{0,31}:\/\//.test(value)) return false;
- if (value.startsWith('./') || value.startsWith('~/') || value.startsWith('${') || value.startsWith('$(')) return false;
- if (value.startsWith('<REDACTED:') || DOTTED_IDENTIFIER_RE.test(value)) return false;
- return !(value.startsWith('/') && !LEGACY_ASSIGNMENT_VALUE_RE.test(value));
-}
+// The opt-in `high_entropy_assignment` shapes and gates live in secret-scan-assignment.ts.
/**
* One compiled set per option combination. scanInternal is synchronous and
@@ -734,24 +666,30 @@ function compilePatterns(opts: ScanOpts): CompiledPattern[] {
}
function buildPatterns(opts: ScanOpts): CompiledPattern[] {
- const out: CompiledPattern[] = CORE_PATTERNS.map((p) => ({
- name: p.name,
- // nosemgrep: javascript.lang.security.audit.detect-non-literal-regexp.detect-non-literal-regexp -- p.source is a compile-time literal from CORE_PATTERNS (never caller input); every pattern uses bounded quantifiers, pinned by test/secret-scan-perf.test.ts
- re: new RegExp(p.prebuilt ? p.source : `(^|[^A-Za-z0-9_])(${p.source})`, p.flags ?? 'g'),
- ...(p.precheck ? { precheck: p.precheck } : {}),
- ...(p.validate ? { validate: p.validate } : {}),
- }));
+ const out: CompiledPattern[] = CORE_PATTERNS.flatMap((p) => {
+ const shared = { name: p.name, ...(p.precheck ? { precheck: p.precheck } : {}), ...(p.validate ? { validate: p.validate } : {}) };
+ const twin = p.prebuilt ? p.escaped : `(${ESCAPE_BOUNDARY})(${p.source})`;
+ return [
+ // nosemgrep: javascript.lang.security.audit.detect-non-literal-regexp.detect-non-literal-regexp -- p.source is a compile-time literal from CORE_PATTERNS (never caller input); every pattern uses bounded quantifiers, pinned by test/secret-scan-perf.test.ts
+ { ...shared, re: new RegExp(p.prebuilt ? p.source : `(^|[^A-Za-z0-9_])(${p.source})`, p.flags ?? 'g') },
+ // nosemgrep: javascript.lang.security.audit.detect-non-literal-regexp.detect-non-literal-regexp -- compile-time literal (ESCAPE_BOUNDARY + p.source); bounded quantifiers
+ ...(twin ? [{ ...shared, re: new RegExp(twin, p.flags ?? 'g'), afterEscape: true }] : []),
+ ];
+ });
if (opts.highEntropy) {
// Group layout matches the core shape: group 1 = the keyword anchor
// through the `=`/`:` (and opening quote), group 2 = value.
for (const source of [ASSIGNMENT_QUOTED_SOURCE, ASSIGNMENT_UNQUOTED_SOURCE]) {
- out.push({
- name: 'high_entropy_assignment',
- // nosemgrep: javascript.lang.security.audit.detect-non-literal-regexp.detect-non-literal-regexp -- compile-time literal; bounded quantifiers, pinned by test/secret-scan-perf.test.ts
- re: new RegExp(source, 'gi'),
- entropyGated: true,
- validate: assignmentValueIsSecretShaped,
- });
+ for (const afterEscape of [false, true]) {
+ out.push({
+ name: 'high_entropy_assignment',
+ // nosemgrep: javascript.lang.security.audit.detect-non-literal-regexp.detect-non-literal-regexp -- compile-time literal; bounded quantifiers, pinned by test/secret-scan-perf.test.ts
+ re: new RegExp(source(assignmentKey(afterEscape ? ESCAPE_BOUNDARY : '(?:^|[^A-Za-z0-9])')), 'gi'),
+ entropyGated: true,
+ validate: assignmentValueIsSecretShaped,
+ ...(afterEscape ? { afterEscape } : {}),
+ });
+ }
}
}
if (opts.labeledCredentials) {
@@ -941,6 +879,7 @@ function scanInternal(text: string, opts: ScanOpts): RawHit[] {
if (pem.length > 0 && insideClaim(pem, lineStart, lineStart + line.length)) continue;
if (work) { work.linesScanned++; work.lineChars += line.length; }
const spans: LineSpans = { all: [] };
+ let escaped: boolean | undefined;
// Claimed-character bitmap behind the first-wins dedupe: allocated on the
// line's FIRST hit only (most lines have none), then O(value) to test and
// to mark. Same answer as scanning every prior span for an intersection.
@@ -957,6 +896,7 @@ function scanInternal(text: string, opts: ScanOpts): RawHit[] {
};
for (const p of patterns) {
if (p.continuation && p.continuation !== dangling) continue;
+ if (p.afterEscape && !(escaped ??= ESCAPE_IN_LINE_RE.test(line))) continue;
if (p.precheck && !p.precheck(line)) continue;
p.re.lastIndex = 0;
const subject = p.subject ? p.subject(line) : line;
@@ -1238,26 +1178,26 @@ export function scanFiles(paths: string[], opts: ScanOpts = {}): SecretFinding[]
// reports — a claimed value's bytes at a position the scanner did not claim
// (embedded past its left boundary inside a longer identifier) are left
// as-is, and the corpus write and the push gate see the same findings.
-// Vendor-prefixed values lose nothing by staying outside the echo pass: they
-// re-match on their own wherever they recur.
//
// THE ECHO PASS. A `bearer` token is claimed only where its `Bearer ` keyword
// anchors it and a `high_entropy_assignment` value only where its keyword
// does, yet transcripts routinely echo the same value bare (`Authorization:
// Bearer <tok>` in a tool call, then `<tok>` alone in the reply; a pasted
-// `PASSWORD=<v>` line, then `<v>` in prose). So the values those two patterns
+// `PASSWORD=<v>` line, then `<v>` in prose). A vendor or JWT value recurs
+// glued to a word (`token_<key>`, `prefix<key>`) where no boundary lets it
+// re-match (#7978). So the values those patterns (and `labeled_credential`)
// claim form an ECHO DICTIONARY (value → pattern) and every remaining
// occurrence — bare, or embedded inside a longer identifier — is redacted as
// that pattern's token too. Three bounds keep the pass cheap and
// collision-safe: a value must clear its pattern's floor
-// (ECHO_MIN_CHARS_BEARER = 20, ECHO_MIN_CHARS_ENTROPY = 12 — the rules' own
+// (ECHO_MIN_CHARS_BEARER = ECHO_MIN_CHARS_VENDOR = 20, ECHO_MIN_CHARS_ENTROPY = 12 — the rules' own
// floors, restated so a pattern edit alone can never widen the pass to
// short, collision-prone values; an opted-in `labeled_credential` value
// needs ECHO_MIN_CHARS_LABELED = 8 and must not be a stoplisted word, so
// `pwd: 1234` never scrubs every later "1234"), must be at most
// ECHO_MAX_VALUE_CHARS long,
// and only the first ECHO_MAX_UNIQUE unique values in claim order — ONE cap
-// across both patterns — join. A value past either cap is still redacted at
+// across every pattern — join. A value past either cap is still redacted at
// its claimed span; only its echoes are the accepted miss.
//
// The pass is a streaming merge over the ORIGINAL text: per value, the
@@ -1302,6 +1242,15 @@ export const ECHO_MIN_CHARS_BEARER = 20;
*/
export const ECHO_MIN_CHARS_ENTROPY = 12;
+/**
+ * Floor for a vendor-shape or `jwt` value (every core rule but the four
+ * catch-alls) to take part in the echo pass (#7978): bearer's floor, so a
+ * short, collision-prone value never scrubs its later lookalikes.
+ */
+export const ECHO_MIN_CHARS_VENDOR = 20;
+const NON_ECHO_CORE_PATTERNS = new Set(['bearer', 'db_url_credentials', 'url_credentials', 'basic_auth']);
+const VENDOR_ECHO_PATTERNS = new Set(CORE_PATTERNS.map((p) => p.name).filter((name) => !NON_ECHO_CORE_PATTERNS.has(name)));
+
/**
* A claimed value longer than this is redacted at its claimed span but does
* not join the echo dictionary. Bounds the per-candidate compare of the
@@ -1354,7 +1303,7 @@ function addEchoValue(into: EchoDictionary, pattern: string, value: string): voi
? ECHO_MIN_CHARS_ENTROPY
: pattern === 'labeled_credential'
? ECHO_MIN_CHARS_LABELED
- : Infinity;
+ : VENDOR_ECHO_PATTERNS.has(pattern) ? ECHO_MIN_CHARS_VENDOR : Infinity;
if (value.length < floor || value.length > ECHO_MAX_VALUE_CHARS) return;
if (into.has(value) || into.size >= ECHO_MAX_UNIQUE) return;
into.set(value, pattern);
diff --git a/test/secret-scan-escape-boundary-7978.test.ts b/test/secret-scan-escape-boundary-7978.test.ts
new file mode 100644
index 0000000000..699eb47004
--- /dev/null
+++ b/test/secret-scan-escape-boundary-7978.test.ts
@@ -0,0 +1,144 @@
+/**
+ * #7978: the vendor rules' left boundary `(^|[^A-Za-z0-9_])` read the last
+ * character of a JSON escape (`\\n` -> `n`) or a percent-escape (`%22` ->
+ * `2`) as part of a word, so a key at the start of an escaped JSON string or
+ * inside a URL-encoded value was never claimed, and a vendor-claimed value
+ * never joined the redaction echo dictionary, so a second copy glued to a word
+ * (`token_<key>`) stayed in the output. Both directions are pinned here: every
+ * escaped context now redacts, and a prefix embedded in an ordinary word, an
+ * escape-looking sequence that is not one, or an identifier-embedded key with
+ * no clean copy still does not fire.
+ *
+ * Every key is assembled at runtime from random parts (gitleaks, CEO-10).
+ */
+import { describe, expect, test } from 'bun:test';
+import { redactFindings, scanText } from '../src/core/secret-scan.ts';
+import { PROVIDER_KEY_SHAPES, alnum, hex } from './helpers/provider-keys.ts';
+
+const j = (sep: string, ...parts: string[]) => parts.join(sep);
+const SEG = () => alnum(24).replace(/[^A-Za-z0-9]/g, 'a');
+
+/** One synthetic key per default-boundary vendor rule, plus the prebuilt credential shapes. */
+const KEYS: ReadonlyArray<{ pattern: string; make(): string }> = [
+ { pattern: 'anthropic', make: () => j('-', 'sk', 'ant', 'api03', alnum(32)) },
+ { pattern: 'openai', make: () => j('-', 'sk', 'proj', alnum(32)) },
+ { pattern: 'openai', make: () => j('-', 'sk', alnum(40)) },
+ { pattern: 'voyage', make: () => j('-', 'pa', alnum(32)) },
+ { pattern: 'github_pat', make: () => j('_', 'github', 'pat', alnum(40)) },
+ { pattern: 'github_token', make: () => j('_', 'ghp', alnum(36)) },
+ { pattern: 'gitlab_pat', make: () => j('-', 'glpat', alnum(24)) },
+ { pattern: 'slack', make: () => j('-', 'xoxb', '1234567890', alnum(24)) },
+ { pattern: 'aws_access_key', make: () => 'AKIA' + hex(16).toUpperCase() },
+ { pattern: 'google_api_key', make: () => 'AIza' + alnum(35) },
+ { pattern: 'stripe', make: () => j('_', 'sk', 'live', alnum(32)) },
+ { pattern: 'sendgrid', make: () => j('.', 'SG', alnum(22), alnum(43)) },
+ { pattern: 'twilio', make: () => 'AC' + hex(32) },
+ { pattern: 'supabase_key', make: () => j('_', 'sb', 'secret', alnum(32)) },
+ { pattern: 'npm_token', make: () => j('_', 'npm', alnum(36)) },
+ { pattern: 'huggingface', make: () => j('_', 'hf', alnum(34)) },
+ { pattern: 'digitalocean', make: () => j('_', 'dop', 'v1', hex(64)) },
+ ...PROVIDER_KEY_SHAPES.map((s) => ({ pattern: s.pattern, make: s.make })),
+ { pattern: 'gbrain_token', make: () => j('_', 'gbrain', 'at', hex(64)) },
+ { pattern: 'jwt', make: () => j('.', 'eyJ' + SEG(), SEG(), SEG()) },
+];
+
+/** The escape that sits immediately before the key, as it appears in the raw text. */
+const ESCAPES = ['\\n', '\\r', '\\t', '\\\\n', '\\u0022', '%22', '%3D', '%3d', '%0A'];
+
+describe('#7978 a key right after a JSON escape or a percent-escape is claimed', () => {
+ for (const { pattern, make } of KEYS) {
+ test(`${pattern}: claimed after every escape, and redacted`, () => {
+ for (const esc of ESCAPES) {
+ const key = make();
+ const line = `{"role":"tool","out":"config loaded${esc}${key}${esc === '%22' ? '%22' : '\\n'}done"}`;
+ const findings = scanText(line);
+ expect({ esc, found: findings.map((f) => f.pattern) }).toEqual({ esc, found: [pattern] });
+ const out = redactFindings(line, { highEntropy: true }).text;
+ expect({ esc, leaked: out.includes(key) }).toEqual({ esc, leaked: false });
+ expect(out).toContain(`${esc}<REDACTED:${pattern}>`);
+ }
+ });
+ }
+
+ test('the #7978 reproduction: Bearer <openrouter key>, then the key after an escaped newline in tool output', () => {
+ const k = j('-', 'sk', 'or', 'v1', hex(64));
+ const out = redactFindings(`Authorization: Bearer ${k}\n{"out":"line\\n${k}"}`, { highEntropy: true }).text;
+ expect(out.includes(k)).toBe(false);
+ expect(out).toBe('Authorization: Bearer <REDACTED:openrouter>\n{"out":"line\\n<REDACTED:openrouter>"}');
+ });
+
+ test('a URL-encoded body: api_key%3D<key>%26next=1', () => {
+ const k = j('_', 'gsk', alnum(52));
+ expect(redactFindings(`body=api_key%3D${k}%26next%3D1`).text).toBe('body=api_key%3D<REDACTED:groq>%26next%3D1');
+ });
+
+ test('Bearer and Basic after an escaped newline claim their token', () => {
+ const tok = alnum(40);
+ expect(redactFindings(`{"h":"x\\nBearer ${tok}"}`).text).toBe('{"h":"x\\nBearer <REDACTED:bearer>"}');
+ const basic = Buffer.from(`alice-example:${alnum(20)}`).toString('base64');
+ expect(redactFindings(`{"h":"x\\nAuthorization: Basic ${basic}"}`).text).toBe('{"h":"x\\nAuthorization: Basic <REDACTED:basic_auth>"}');
+ });
+
+ test('a keyword-anchored high-entropy assignment after an escaped newline is claimed (transcript lane)', () => {
+ const v = alnum(24) + '7';
+ const out = redactFindings(`{"out":"ok\\npassword=${v}\\n"}`, { highEntropy: true }).text;
+ expect(out.includes(v)).toBe(false);
+ });
+});
+
+describe('#7978 a vendor-claimed value joins the echo dictionary', () => {
+ test('a second copy glued to a word (token_<key>, prefix<key>) is scrubbed as that vendor', () => {
+ const k = j('_', 'ghp', alnum(36));
+ const out = redactFindings(`GITHUB_TOKEN=${k}\nurl?token_${k}&x=1 id=prefix${k}\n`).text;
+ expect(out).toBe('GITHUB_TOKEN=<REDACTED:github_token>\nurl?token_<REDACTED:github_token>&x=1 id=prefix<REDACTED:github_token>\n');
+ });
+
+ test('a session-wide dictionary carries a vendor value claimed in one field to a glued copy in another', async () => {
+ const { planRedaction, applyRedaction } = await import('../src/core/secret-scan.ts');
+ const k = j('-', 'sk', 'or', 'v1', hex(64));
+ const echoValues = new Map<string, string>();
+ const glued = planRedaction(`cache_key=v1_${k}`, { echoValues });
+ planRedaction(`OPENROUTER_API_KEY=${k}`, { echoValues });
+ expect(applyRedaction(glued)).toBe('cache_key=v1_<REDACTED:openrouter>');
+ });
+
+ test('a vendor value under the 20-character echo floor (a short slack token) does not join', () => {
+ const k = j('-', 'xoxb', alnum(11));
+ expect(k.length).toBeLessThan(20);
+ expect(redactFindings(`t=${k}\nid=x${k}`).text).toBe(`t=<REDACTED:slack>\nid=x${k}`);
+ });
+});
+
+describe('#7978 no new false positives', () => {
+ test('a vendor prefix inside an ordinary word stays clean, escaped or not', () => {
+ const prose = [
+ '{"out":"line one\\nrisk-assessment-matrix-for-the-quarter and task-scheduler-for-the-cluster"}',
+ '{"out":"\\nskeleton-key-ceremony\\tdesk-assessment-overview-document"}',
+ 'q=%22risk-assessment-matrix-for-the-quarter%22&r=%20task-scheduler-configuration-for-cluster',
+ '{"out":"\\nwhsec-is-not-a-key\\nghp-is-not-a-key\\nnpm_install_dependencies_for_workspace_now"}',
+ ].join('\n');
+ expect(scanText(prose)).toEqual([]);
+ expect(redactFindings(prose, { highEntropy: true }).text).toBe(prose);
+ });
+
+ test('a backslash or percent sign that is not an escape is not a boundary', () => {
+ for (const before of ['\\q', '\\x', '%ZZ', '%2', 'n', '2', '_']) {
+ const key = j('_', 'ghp', alnum(36));
+ const line = `id=a${before}${key}`;
+ expect({ before, found: scanText(line).map((f) => f.pattern) }).toEqual({ before, found: [] });
+ }
+ });
+
+ test('a key embedded in an identifier with no clean copy anywhere stays unclaimed (the push gate and redaction agree)', () => {
+ const k = j('-', 'sk', alnum(40));
+ const text = `id=prefix${k} and token_${k}`;
+ expect(scanText(text)).toEqual([]);
+ expect(redactFindings(text, { highEntropy: true }).text).toBe(text);
+ });
+
+ test('the escape stays in the output; only the key is redacted', () => {
+ const k = j('_', 'npm', alnum(36));
+ expect(redactFindings(`"a\\n${k}"`).text).toBe('"a\\n<REDACTED:npm_token>"');
+ expect(redactFindings(`a%22${k}%22`).text).toBe('a%22<REDACTED:npm_token>%22');
+ });
+});
diff --git a/test/secret-scan.test.ts b/test/secret-scan.test.ts
index b7dd11bf44..bfed609245 100644
--- a/test/secret-scan.test.ts
+++ b/test/secret-scan.test.ts
@@ -717,21 +717,19 @@ describe('redactFindings — span-splice semantics and preview window', () => {
expect(redactions.map((r) => r.pattern)).toEqual(['bearer', 'db_url_credentials']);
});
- test('redacts exactly the claimed spans (every pattern outside the echo pass): the corpus write agrees with what scanText reports', () => {
- // Documented semantic delta from the replaceAll form: a claimed value's
- // bytes at a position the scanner did NOT claim (embedded past its left
- // boundary inside a longer identifier) are left as-is, so
- // `redactFindings(t).text` never redacts something `scanText(t)` would
- // not have reported. The exception is the bounded echo pass over the
- // bearer + high_entropy_assignment values (pinned in the next describe);
- // vendor shapes lose nothing by staying outside it — a repeat that clears
- // the boundary IS claimed on its own and both occurrences go.
+ test('redacts the claimed spans plus their echoes: the findings agree with scanText, a vendor value glued past its boundary is an echo (#7978)', () => {
+ // Documented semantic delta from the replaceAll form: the splice redacts
+ // exactly what `scanText(t)` reports, and the bounded echo pass (pinned in
+ // the next describe) adds the bytes of a claimed value at a position the
+ // scanner did NOT claim. Since #7978 vendor values join it, so a copy
+ // embedded past its left boundary inside a longer identifier goes too,
+ // without adding a finding.
const text = `k=${OPENAI} again ${OPENAI}\nid=prefix${OPENAI}\nsid ${TWILIO_SID} vs x${TWILIO_SID}\n`;
const findings = scanText(text);
expect(findings.map((f) => [f.pattern, f.line])).toEqual([['openai', 1], ['openai', 1], ['twilio', 3]]);
const { text: out, redactions } = redactFindings(text);
expect(redactions.length).toBe(findings.length);
- expect(out).toBe(`k=<REDACTED:openai> again <REDACTED:openai>\nid=prefix${OPENAI}\nsid <REDACTED:twilio> vs x${TWILIO_SID}\n`);
+ expect(out).toBe('k=<REDACTED:openai> again <REDACTED:openai>\nid=prefix<REDACTED:openai>\nsid <REDACTED:twilio> vs x<REDACTED:twilio>\n');
});
test('a per-line span running into a PEM block: both spans redact, nothing claimed survives', () => {
@@ -854,9 +852,12 @@ describe('redactFindings — bounded echo pass', () => {
expect(out).toBe('Bearer <REDACTED:anthropic>\nbare <REDACTED:anthropic>\n');
});
- test("other patterns' values are not echoed: a vendor key embedded past its boundary stays (it re-matches on its own wherever it clears the boundary)", () => {
- const { text: out } = redactFindings(`k=${OPENAI}\nid=prefix${OPENAI}\n`);
- expect(out).toBe(`k=<REDACTED:openai>\nid=prefix${OPENAI}\n`);
+ test('a vendor key embedded past its boundary is echoed once a clean copy is claimed (#7978); the catch-all basic_auth is not', () => {
+ const { text: out, redactions } = redactFindings(`k=${OPENAI}\nid=prefix${OPENAI}\n`);
+ expect(out).toBe('k=<REDACTED:openai>\nid=prefix<REDACTED:openai>\n');
+ expect(redactions.map((r) => r.pattern)).toEqual(['openai']);
+ const basic = Buffer.from('alice-example:' + 'Zq7'.repeat(8)).toString('base64');
+ expect(redactFindings(`Authorization: Basic ${basic}\nid=x${basic}\n`).text).toBe(`Authorization: Basic <REDACTED:basic_auth>\nid=x${basic}\n`);
});
test(`the cap is exactly ${ECHO_MAX_UNIQUE} unique values in claim order; the next value's echo is the accepted miss`, () => {