Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(secret-scan): claim keys after a JSON or percent escape; vendor values join the echo dictionary (#7978)
garrytan2bf98da2ac06
diff --git a/docs/architecture/key-files/secret-scan.md b/docs/architecture/key-files/secret-scan.mdindex a6997cd9d2..324cddf531 100644--- a/docs/architecture/key-files/secret-scan.md+++ b/docs/architecture/key-files/secret-scan.md@@ -6,6 +6,6 @@ The format-based secret detector shared by `sources push`, bootstrap verify, thecompiled-context sensitivity scan and the transcripts redactor, and the composedsensitivity detector built on it.-- `src/core/secret-scan.ts` — pattern-based secret detection shared by `sources push`, bootstrap verify, the compiled-context sensitivity scan and the transcripts redactor. Detection is FORMAT-based, not prefix-only: vendor shapes (anthropic, openai, voyage, github, gitlab, slack, aws AKIA/ASIA, google, stripe, sendgrid, twilio, supabase sb_secret_/sbp_, npm, huggingface, `digitalocean` `do[opr]_v1_` + 64 hex, the AI-provider keys `openrouter` `sk-or-v1-` + 64 lowercase hex, `groq` `gsk_` + 52, `xai` `xai-` + 80, `perplexity` `pplx-` + 48 and `replicate` `r8_` + 37, all fixed-length with a hard right edge, gbrain), JWTs, private keys (whole-text claims, below), plus four catch-alls appended LAST (`bearer` — prebuilt two-group layout so only the token is the value; `db_url_credentials` — the `scheme://user:pass@` span of database URLs, host/db kept; `url_credentials` — the same span for `http(s)` (scheme case-insensitive, `gi`), user 0–128 stopping at `:`/`/`, password 1–256 excluding whitespace, `@`, `"`, `'`, `/`, `?`, `#`, with a `validate` hook that rejects placeholder passwords (`<...>`, `${...}`, `$VAR`, all `*`, all `x`); `basic_auth` — two prebuilt case-insensitive forms, `Authorization: Basic` with a 4–2048 char value and bare `Basic` with 16–2048, both with a hard right edge and a `validate` that requires the value to base64-decode to printable ASCII containing `:`; it is NOT in the bearer family, so it joins neither the echo dictionary nor `secret:bearer`). The pattern contract: `CorePattern` carries optional `flags` (default `'g'`), `precheck`, `validate(value)` (O(value), run on the whole match before the claim and the union split; a rejected match is not re-tried from inside) and `since` (the gbrain version that added or last changed the rule; `SECURITY_WAVE_SINCE` = `0.60.31.0` on `digitalocean`, `url_credentials`, `basic_auth` and `private_key_pem`, `PROVIDER_KEYS_SINCE` on the five AI-provider rules (#7959), kept in step with the release that ships a rule change). Compiled patterns keep group 1 = anchor, group 2 = value. `patternSince(pattern)` reads the map; every finding of such a pattern carries `SecretFinding.since`, which the push and bootstrap refusals render. Fixed-length shapes carry a trailing negative lookahead. Per-line claimed-span dedupe is first-wins for ATTRIBUTION (order matters: `Bearer <vendor key>` keeps the vendor name, a fully covered later span is skipped) but coverage is the UNION: every unclaimed slice of a later overlapping span is claimed as a hit of the later pattern (own value, own fingerprint), so a database URL whose username is a vendor-shaped id still loses its password. The transcript-only `labeled_credential` rule (`ScanOpts.labeledCredentials`) lives in `secret-scan-labeled.ts` (Security). The opt-in `high_entropy_assignment` rule (`ScanOpts.highEntropy`) requires a 12+ char value that contains a digit and a non-digit AND clears 3.5 bits/char on the bias-corrected (Miller-Madow) estimate `correctedEntropy`, since raw per-character entropy is capped at log2(n) and rejected most random 12-16 char secrets. It is two compiled patterns sharing the name: QUOTED (after `"` or `'`, any run of non-quote, non-whitespace characters) and UNQUOTED (the legacy `[A-Za-z0-9+/_=-]` class plus `!#$%*@^~.?<>`; never `&`, `()[]{}`, `:`, `,` or `;`, so query strings, calls and SQL `SET` lists stay intact, and a password carrying `,`/`;` matches only quoted; the last character excludes `.?!`, so a sentence's full stop is not claimed). Its `validate` rejects values that start with `scheme://`, `./`, `~/`, `${`, `$(` or `<REDACTED:`, dotted identifier paths (`process.env.X`) and `/`-paths holding a character outside the legacy class. Consumers with it on: retrieval output, transcript ingest, hooks and the hook heartbeat, the atoms drain, `embed-facts.ts` error text and the context-engine relay re-scan. Off: the push gate, bootstrap verify/repo and the compiled-context sensitivity scan. Own minimal allowlist + `<ws>/.gbrain-scan-allow` per-finding overrides — deliberately NOT the repo's `.gitleaks.toml` (a public-repo CI fixture policy); the allowlist block (`loadWorkspaceAllowlist`, the glob dialect `globToRegExp`/`matchesGlob`, `valueAllowlisted`/`pathAllowlisted`, `SCAN_ALLOW_FILENAME`, the 16-hex fingerprint floor) lives in `src/core/secret-scan-allowlist.ts` since wave 16 (G4.1, the 1500-line module cap) and is re-exported here unchanged (`test/secret-scan-allowlist.test.ts`). `correctedEntropy` is one histogram pass (distinct count = histogram size; byte-identical to the former `shannonEntropy + Set` form, `test/secret-scan-entropy-equivalence.test.ts`; ~80 → ~25 ms per 250 × 4096-char segment). `ScanOpts.work` / `RedactOpts.work` takes a `ScanWork` accumulator (`src/core/secret-scan-work.ts`: lines, regex execs, candidates per pattern, chars handed to the entropy gate counted inside `correctedEntropy`, validate calls, claims, union splits, echo sweeps) pinned exactly on the perf fixture by `test/secret-scan-work-counters.test.ts`; the perf file's absolute bounds stay the only guard on per-character cost. Deliberately NOT patterns: `sb_publishable_`, Supabase project refs (identifiers, not credentials). Never spell a literal database or `http(s)` URL with userinfo, or any credential, in this file's comments or in fixtures (assemble test values at runtime; `scripts/check-pg-url-redaction.sh` greps `src/` for every `db_url_credentials` scheme). Long lines are scanned in full (no length cap — a secret on a 1 MB minified line must still redact); what keeps that linear is bounded quantifiers on every shape whose value class could otherwise run to end-of-line — `db_url_credentials` (user ≤128 stopping at `:`/`/`; password 1–256 stopping only at whitespace, `@` and the string delimiters `"`/`'`, so unescaped `/ { } < > | ^ \` and backtick passwords redact), `jwt` (prebuilt so its boundary excludes `-` as well as `_`; segments ≤4096), `high_entropy_assignment` (≤64 identifier chars after the keyword, value ≤4096), `url_credentials` (user ≤128, password ≤256), `basic_auth` (value ≤2048, so its decode is constant work) and the private-key walks (below) — plus a per-pattern `precheck` substring gate on the catch-alls (`@` for both URL patterns; `earer`/`EARER`, since an all-caps `BEARER` scheme is accepted; a case-insensitive `basic`), a carried next-newline position in `scanPemBlocks` (sticky -1, so the header-line count is O(text) even when thousands of hits sit ahead of a newline-free multi-megabyte tail), a lazily-allocated per-line claimed-char bitmap behind the first-wins dedupe (O(value) per hit), and a windowed `buildPreview` (whole line ≤160 chars; otherwise hit−40…+80, edges snapped outward to whole spans, every claimed span in the window spliced by position, so a preview never carries a sibling secret or a fragment). `redactFindings` (= `planRedaction` + `applyRedaction`) rebuilds the text by SPAN-SPLICE in one O(text + hits) pass over the claimed spans, and fills the UNCLAIMED gaps with a bounded echo pass: the values claimed by `bearer` and `high_entropy_assignment` form an echo dictionary (value → pattern; floors `ECHO_MIN_CHARS_BEARER` = 20 / `ECHO_MIN_CHARS_ENTROPY` = 12, values ≤ `ECHO_MAX_VALUE_CHARS` = 512, the first `ECHO_MAX_UNIQUE` = 64 unique values in claim order across BOTH patterns — a value past either cap is redacted at its claim, its echoes are the accepted miss), matched by a streaming merge over the ORIGINAL text — per value, the next occurrence at or beyond the emit cursor (`indexOf` resuming at the cursor, re-sought only once the cursor passes it), leftmost-longest at each step, emitted incrementally into the unclaimed gaps — so retained state is O(values) however long the text is (occurrences are never collected into a list) (never an alternation RegExp, never over emitted tokens). Echoes add no findings. `planRedaction` feeds a caller-supplied shared map (`RedactOpts.echoValues`) and `applyRedaction` reads it at apply time, so a multi-field document scrubs an echo in one field of a value claimed in another. Every other pattern redacts exactly what `scanText` reports (a value embedded past its boundary is left as-is; vendor shapes re-match on their own). PRIVATE KEYS: `pemClaims` finds `BEGIN`/`END … PRIVATE KEY` fences in two global passes and merges them in text order. BLOCK: a BEGIN whose next fence is an END within `PEM_BODY_MAX_CHARS` (16384) is claimed header through END; a block never swallows a later BEGIN. FORWARD (BEGIN with no END in view): the header plus the body lines that follow; a header followed by no body line (a marker quoted in prose) is claimed alone. BACKWARD (END with no BEGIN in view): the body lines before the fence plus the fence; an END with no body line before it is no claim. A body line is a whole-line base64 run (`A-Za-z0-9+/=`) indented by up to 16 spaces/tabs; line ends are `\n`, `\r\n` or the literal JSON escapes; interior lines are 16–128 chars, the line next to a fence and a partial line at a cut 1–128; optional RFC 1421 `Proc-Type:`/`DEK-Info:` lines and one blank line may follow BEGIN; a line with anything else, or a run over 128 chars, ends the body. Total claimed body stays within `PEM_BODY_MAX_CHARS`. Each walk stops at the neighbouring fence or the previous claim, so the pass is linear (20k unterminated BEGINs, 20k END-only fences and 1 MB of base64 lines are perf-pinned). Per-line hits that fall wholly inside a key claim are dropped (one key, one finding). A FORWARD claim's finding also carries `legacyFingerprint`, the header-only fingerprint earlier releases reported, so `workspace-push.ts` can name a stale allowlist entry; a truncated key's fingerprint covers its body and therefore depends on where the text was cut, so it is not a stable allowlist key. `privateKeySpans(text)` exports the same claims as sorted, disjoint `{start, end}` spans. Accepted misses: body on the header's own line, space-joined single-line keys, blockquoted (`> `) keys, base64 runs over 128 chars, body past 16384 chars, `http(s)` passwords containing a literal `/`, `?` or `#`, Basic values over 2048 chars, digitless assignment values, dotted passwords whose every segment starts with a letter, and unquoted passwords containing `,` or `;`. False-positive budget: `scripts/secret-scan-fp-budget.ts` diffs findings against a base ref's scanner (`--base`, default `origin/master`) (`--dir <brain>` reports per-pattern counts only, never content) and `test/secret-scan-fp-budget.test.ts` pins a synthetic brain-like fixture (zero findings on its non-secret part in both modes, every planted secret redacted). Shape tests: `test/secret-scan-credential-shapes.test.ts`, `test/secret-scan-private-key-spans.test.ts`. Perf pins: `test/secret-scan-perf.test.ts`. Retrieval output is another consumer of its canonical scanner; that output boundary does not sanitize internal evaluation/provider inputs.+- `src/core/secret-scan.ts` — pattern-based secret detection shared by `sources push`, bootstrap verify, the compiled-context sensitivity scan and the transcripts redactor. Detection is FORMAT-based, not prefix-only: vendor shapes (anthropic, openai, voyage, github, gitlab, slack, aws AKIA/ASIA, google, stripe, sendgrid, twilio, supabase sb_secret_/sbp_, npm, huggingface, `digitalocean` `do[opr]_v1_` + 64 hex, the AI-provider keys `openrouter` `sk-or-v1-` + 64 lowercase hex, `groq` `gsk_` + 52, `xai` `xai-` + 80, `perplexity` `pplx-` + 48 and `replicate` `r8_` + 37, all fixed-length with a hard right edge, gbrain), JWTs, private keys (whole-text claims, below), plus four catch-alls appended LAST (`bearer` — prebuilt two-group layout so only the token is the value; `db_url_credentials` — the `scheme://user:pass@` span of database URLs, host/db kept; `url_credentials` — the same span for `http(s)` (scheme case-insensitive, `gi`), user 0–128 stopping at `:`/`/`, password 1–256 excluding whitespace, `@`, `"`, `'`, `/`, `?`, `#`, with a `validate` hook that rejects placeholder passwords (`<...>`, `${...}`, `$VAR`, all `*`, all `x`); `basic_auth` — two prebuilt case-insensitive forms, `Authorization: Basic` with a 4–2048 char value and bare `Basic` with 16–2048, both with a hard right edge and a `validate` that requires the value to base64-decode to printable ASCII containing `:`; it is NOT in the bearer family, so it joins neither the echo dictionary nor `secret:bearer`). The pattern contract: `CorePattern` carries optional `flags` (default `'g'`), `precheck`, `validate(value)` (O(value), run on the whole match before the claim and the union split; a rejected match is not re-tried from inside) and `since` (the gbrain version that added or last changed the rule; `SECURITY_WAVE_SINCE` = `0.60.31.0` on `digitalocean`, `url_credentials`, `basic_auth` and `private_key_pem`, `PROVIDER_KEYS_SINCE` on the five AI-provider rules (#7959), kept in step with the release that ships a rule change). Compiled patterns keep group 1 = anchor, group 2 = value. Every core rule and both `high_entropy_assignment` forms also have an escape-boundary twin (#7978), compiled right after the rule under the same name and anchored on a JSON escape (`\n`, `\r`, `\t`, `\b`, `\f`, `\uXXXX`) or a percent-escape (`%XX`), whose last character the default boundary reads as a word character; a twin runs only on a line `ESCAPE_IN_LINE_RE` matches, so other lines run the old list, and a prebuilt rule carries its twin as `CorePattern.escaped`. `patternSince(pattern)` reads the map; every finding of such a pattern carries `SecretFinding.since`, which the push and bootstrap refusals render. Fixed-length shapes carry a trailing negative lookahead. Per-line claimed-span dedupe is first-wins for ATTRIBUTION (order matters: `Bearer <vendor key>` keeps the vendor name, a fully covered later span is skipped) but coverage is the UNION: every unclaimed slice of a later overlapping span is claimed as a hit of the later pattern (own value, own fingerprint), so a database URL whose username is a vendor-shaped id still loses its password. The transcript-only `labeled_credential` rule (`ScanOpts.labeledCredentials`) lives in `secret-scan-labeled.ts` (Security). The opt-in `high_entropy_assignment` rule (`ScanOpts.highEntropy`; its shapes and gates live in `src/core/secret-scan-assignment.ts`, split out at the 1500-line module cap) requires a 12+ char value that contains a digit and a non-digit AND clears 3.5 bits/char on the bias-corrected (Miller-Madow) estimate `correctedEntropy`, since raw per-character entropy is capped at log2(n) and rejected most random 12-16 char secrets. It is two compiled patterns sharing the name: QUOTED (after `"` or `'`, any run of non-quote, non-whitespace characters) and UNQUOTED (the legacy `[A-Za-z0-9+/_=-]` class plus `!#$%*@^~.?<>`; never `&`, `()[]{}`, `:`, `,` or `;`, so query strings, calls and SQL `SET` lists stay intact, and a password carrying `,`/`;` matches only quoted; the last character excludes `.?!`, so a sentence's full stop is not claimed). Its `validate` rejects values that start with `scheme://`, `./`, `~/`, `${`, `$(` or `<REDACTED:`, dotted identifier paths (`process.env.X`) and `/`-paths holding a character outside the legacy class. Consumers with it on: retrieval output, transcript ingest, hooks and the hook heartbeat, the atoms drain, `embed-facts.ts` error text and the context-engine relay re-scan. Off: the push gate, bootstrap verify/repo and the compiled-context sensitivity scan. Own minimal allowlist + `<ws>/.gbrain-scan-allow` per-finding overrides — deliberately NOT the repo's `.gitleaks.toml` (a public-repo CI fixture policy); the allowlist block (`loadWorkspaceAllowlist`, the glob dialect `globToRegExp`/`matchesGlob`, `valueAllowlisted`/`pathAllowlisted`, `SCAN_ALLOW_FILENAME`, the 16-hex fingerprint floor) lives in `src/core/secret-scan-allowlist.ts` since wave 16 (G4.1, the 1500-line module cap) and is re-exported here unchanged (`test/secret-scan-allowlist.test.ts`). `correctedEntropy` is one histogram pass (distinct count = histogram size; byte-identical to the former `shannonEntropy + Set` form, `test/secret-scan-entropy-equivalence.test.ts`; ~80 → ~25 ms per 250 × 4096-char segment). `ScanOpts.work` / `RedactOpts.work` takes a `ScanWork` accumulator (`src/core/secret-scan-work.ts`: lines, regex execs, candidates per pattern, chars handed to the entropy gate counted inside `correctedEntropy`, validate calls, claims, union splits, echo sweeps) pinned exactly on the perf fixture by `test/secret-scan-work-counters.test.ts`; the perf file's absolute bounds stay the only guard on per-character cost. Deliberately NOT patterns: `sb_publishable_`, Supabase project refs (identifiers, not credentials). Never spell a literal database or `http(s)` URL with userinfo, or any credential, in this file's comments or in fixtures (assemble test values at runtime; `scripts/check-pg-url-redaction.sh` greps `src/` for every `db_url_credentials` scheme). Long lines are scanned in full (no length cap — a secret on a 1 MB minified line must still redact); what keeps that linear is bounded quantifiers on every shape whose value class could otherwise run to end-of-line — `db_url_credentials` (user ≤128 stopping at `:`/`/`; password 1–256 stopping only at whitespace, `@` and the string delimiters `"`/`'`, so unescaped `/ { } < > | ^ \` and backtick passwords redact), `jwt` (prebuilt so its boundary excludes `-` as well as `_`; segments ≤4096), `high_entropy_assignment` (≤64 identifier chars after the keyword, value ≤4096), `url_credentials` (user ≤128, password ≤256), `basic_auth` (value ≤2048, so its decode is constant work) and the private-key walks (below) — plus a per-pattern `precheck` substring gate on the catch-alls (`@` for both URL patterns; `earer`/`EARER`, since an all-caps `BEARER` scheme is accepted; a case-insensitive `basic`), a carried next-newline position in `scanPemBlocks` (sticky -1, so the header-line count is O(text) even when thousands of hits sit ahead of a newline-free multi-megabyte tail), a lazily-allocated per-line claimed-char bitmap behind the first-wins dedupe (O(value) per hit), and a windowed `buildPreview` (whole line ≤160 chars; otherwise hit−40…+80, edges snapped outward to whole spans, every claimed span in the window spliced by position, so a preview never carries a sibling secret or a fragment). `redactFindings` (= `planRedaction` + `applyRedaction`) rebuilds the text by SPAN-SPLICE in one O(text + hits) pass over the claimed spans, and fills the UNCLAIMED gaps with a bounded echo pass: the values claimed by `bearer`, `high_entropy_assignment`, `labeled_credential` and every vendor or `jwt` rule (#7978; not `basic_auth` or the two URL userinfo catch-alls) form an echo dictionary (value → pattern; floors `ECHO_MIN_CHARS_BEARER` = `ECHO_MIN_CHARS_VENDOR` = 20 / `ECHO_MIN_CHARS_ENTROPY` = 12, values ≤ `ECHO_MAX_VALUE_CHARS` = 512, the first `ECHO_MAX_UNIQUE` = 64 unique values in claim order across every pattern — a value past either cap is redacted at its claim, its echoes are the accepted miss), matched by a streaming merge over the ORIGINAL text — per value, the next occurrence at or beyond the emit cursor (`indexOf` resuming at the cursor, re-sought only once the cursor passes it), leftmost-longest at each step, emitted incrementally into the unclaimed gaps — so retained state is O(values) however long the text is (occurrences are never collected into a list) (never an alternation RegExp, never over emitted tokens). Echoes add no findings. `planRedaction` feeds a caller-supplied shared map (`RedactOpts.echoValues`) and `applyRedaction` reads it at apply time, so a multi-field document scrubs an echo in one field of a value claimed in another. `basic_auth` and the URL userinfo catch-alls redact exactly what `scanText` reports; a vendor value embedded past its boundary (`token_<key>`) is scrubbed as an echo when the same value is claimed elsewhere in the document, and left as-is otherwise. PRIVATE KEYS: `pemClaims` finds `BEGIN`/`END … PRIVATE KEY` fences in two global passes and merges them in text order. BLOCK: a BEGIN whose next fence is an END within `PEM_BODY_MAX_CHARS` (16384) is claimed header through END; a block never swallows a later BEGIN. FORWARD (BEGIN with no END in view): the header plus the body lines that follow; a header followed by no body line (a marker quoted in prose) is claimed alone. BACKWARD (END with no BEGIN in view): the body lines before the fence plus the fence; an END with no body line before it is no claim. A body line is a whole-line base64 run (`A-Za-z0-9+/=`) indented by up to 16 spaces/tabs; line ends are `\n`, `\r\n` or the literal JSON escapes; interior lines are 16–128 chars, the line next to a fence and a partial line at a cut 1–128; optional RFC 1421 `Proc-Type:`/`DEK-Info:` lines and one blank line may follow BEGIN; a line with anything else, or a run over 128 chars, ends the body. Total claimed body stays within `PEM_BODY_MAX_CHARS`. Each walk stops at the neighbouring fence or the previous claim, so the pass is linear (20k unterminated BEGINs, 20k END-only fences and 1 MB of base64 lines are perf-pinned). Per-line hits that fall wholly inside a key claim are dropped (one key, one finding). A FORWARD claim's finding also carries `legacyFingerprint`, the header-only fingerprint earlier releases reported, so `workspace-push.ts` can name a stale allowlist entry; a truncated key's fingerprint covers its body and therefore depends on where the text was cut, so it is not a stable allowlist key. `privateKeySpans(text)` exports the same claims as sorted, disjoint `{start, end}` spans. Accepted misses: body on the header's own line, space-joined single-line keys, blockquoted (`> `) keys, base64 runs over 128 chars, body past 16384 chars, `http(s)` passwords containing a literal `/`, `?` or `#`, Basic values over 2048 chars, digitless assignment values, dotted passwords whose every segment starts with a letter, and unquoted passwords containing `,` or `;`. False-positive budget: `scripts/secret-scan-fp-budget.ts` diffs findings against a base ref's scanner (`--base`, default `origin/master`) (`--dir <brain>` reports per-pattern counts only, never content) and `test/secret-scan-fp-budget.test.ts` pins a synthetic brain-like fixture (zero findings on its non-secret part in both modes, every planted secret redacted). Shape tests: `test/secret-scan-credential-shapes.test.ts`, `test/secret-scan-private-key-spans.test.ts`. Perf pins: `test/secret-scan-perf.test.ts`. Retrieval output is another consumer of its canonical scanner; that output boundary does not sanitize internal evaluation/provider inputs.- `src/core/secret-scan-stream.ts` — `scanTextStream(chunks, opts)`, the windowed scan of a byte stream too large to hold as one string (the push gate's staged blobs over 25 MiB). Bytes are decoded by a streaming `TextDecoder` (`ignoreBOM`, so a split multi-byte character joins and invalid bytes become U+FFFD as `Buffer.toString` makes them) and handed to `scanTextSpans` (`secret-scan.ts`: `scanText` plus each claim's absolute span, allowlisted claims kept with `finding: null`) in windows of at most `STREAM_SCAN_WINDOW_CHARS` (8 Mi UTF-16 units). Each next window starts at a line start at least `STREAM_SCAN_OVERLAP_CHARS` (64 Ki, above the longest claim: a private key's 16384-char body plus fences, a jwt's 3 × 4096) before the previous end, or mid-line at exactly the overlap (never inside a surrogate pair) when the line is too long. A window keeps the claims that start before the next window's start, the last window keeps the rest, and a claim wholly inside a span an earlier window kept (a seam's partial view, or an exact duplicate) is dropped; lines are 1-based over the whole stream. `streamScanWork` (@internal) counts windows, bytes, scanned chars and the largest window. `test/secret-scan-stream.test.ts` pins equality with `scanText` across both seams of a window, mid-line splits, split multi-byte characters, invalid UTF-8 and a BOM.- `src/core/context/sensitivity-scan.ts` — composed detector for compiled-context entries (secret-scan → PII → path shapes → operator blocklist → operator pattern file). secret-scan OWNS credential-shaped families it also matches: a PII finding whose value (raw, or with the `Bearer ` keyword stripped) matches a secret finding's fingerprint is not reported twice; short bearer tokens under secret-scan's floor still surface as `pii:bearer`. Every family honors the same `.gbrain-scan-allow` fingerprint escape hatch. `formatSensitivityDrop(drop, workspaceRoot)` renders the one content-free stderr line `gbrain compile-context` prints per dropped entry: reason `sensitivity_scan`, the pattern family, the fingerprint and the absolute `.gbrain-scan-allow` path, never the value (`test/sensitivity-scan.test.ts`).diff --git a/src/core/secret-scan-assignment.ts b/src/core/secret-scan-assignment.tsnew file mode 100644index 0000000000..be950d69cb--- /dev/null+++ b/src/core/secret-scan-assignment.ts@@ -0,0 +1,107 @@+/**+ * secret-scan-assignment.ts: the opt-in `high_entropy_assignment` rule's+ * shapes and gates (`ScanOpts.highEntropy`), split out of secret-scan.ts at+ * its 1500-line module cap (#7978). secret-scan.ts compiles these sources,+ * plain and behind its ESCAPE_BOUNDARY, and runs the gates; nothing else+ * imports this module.+ */++// Opt-in: `secret|token|password|api key`-shaped assignment whose value has+// high Shannon entropy. Keyword-anchored (compiled inline below so the group+// layout matches the core patterns: group 1 = anchor, group 2 = value).+//+// The left boundary is [^A-Za-z0-9] rather than the core patterns'+// [^A-Za-z0-9_]: `_` has to read as a SEPARATOR here, or `api_access_token`+// and `SMTP_PASSWORD` cannot match their own keyword. The core patterns keep+// `_` as a word character on purpose (a vendor prefix inside a longer+// identifier must not fire); this rule is keyword-anchored, so the same+// reasoning inverts. The keyword is also allowed trailing identifier+// segments before the assignment, or `AWS_SECRET_ACCESS_KEY=` cannot match:+// the keyword `SECRET` is not adjacent to the `=`, `_ACCESS_KEY` is. The+// entropy gate still decides, so a wider anchor costs nothing on prose.+//+// The value floor is 12, not 20. A 16-character SMTP password sat under the+// old floor and stayed plaintext in the receipt on disk even once the keyword+// matched — the value length was doing gating the entropy check is there to+// do. Real passwords are frequently 12-16 characters; secrets that long with+// 3.5 bits/char of entropy are not prose.+//+// The value must ALSO contain at least one digit. Identifier-shaped values+// (`DefaultAzureCredential`, `/usr/local/bin/aws-vault`, an env-var NAME+// assigned to an `apiKeyEnvVar`) clear the entropy gate on mixed case and+// separators alone and were being redacted out of ordinary code. Real+// machine-minted secrets essentially always carry digits; a digitless+// passphrase is the accepted miss. A 40-hex git sha assigned to a `token:`+// key still redacts (digits + entropy) — documented, acceptable.+//+// Both quantifiers after the keyword are BOUNDED. The trailing identifier+// segments used to be `[A-Za-z0-9_-]*`: because the left boundary admits `-`+// and `_` and that class contains them too, every `-`/`_` in a run like+// `-apikey-apikey…` or `_token_token…` was a fresh keyword start whose+// suffix ran to end-of-line and backtracked — quadratic (210 KB of+// `-apikey` took ~35 s, 180 KB of `_token` ~15 s). At `{0,64}` a start does+// constant work; no real credential key carries 64 identifier characters+// after its keyword. The value is capped at 4096 for the same+// constant-work-per-occurrence guarantee (the jwt segments share the cap):+// a value longer than that is redacted only through its first 4096 chars —+// the accepted miss, well above any real token or base64 key blob.+//+// The 3.5 bits/char floor is judged on the bias-corrected (Miller-Madow)+// estimate, not the raw per-character Shannon entropy. The raw figure of an+// n-character string can never exceed log2(n) (3.58 at 12 characters, 3.91+// at 15) and drops sharply with a single repeated character, so the raw+// floor rejected 68% of uniformly random 12-character alphanumeric secrets,+// 9% at 15 and 5% at 16: exactly the 12-16 character passwords the 12-char+// value floor exists to catch (a 15-character value such as `!dcG4Gmw1qGRbR3`+// after a password key stayed plaintext).+// Miller-Madow adds (distinct - 1) / (2 n ln 2), which cuts those misses+// below 1% at every length from 12 up and is negligible on long values.+// Because the correction can lift an all-digit value past the floor, the+// value must also carry a non-digit: counters, timestamps and numeric ids+// (`token_count=1234567890123`) never reached 3.5 raw bits (log2(10) = 3.32)+// and still never redact.+export const HIGH_ENTROPY_MIN_BITS_PER_CHAR = 3.5;+export const HIGH_ENTROPY_REQUIRES_DIGIT_RE = /[0-9]/;+export const HIGH_ENTROPY_REQUIRES_NON_DIGIT_RE = /[^0-9]/;++// PUNCTUATED VALUES. The rule is two compiled patterns sharing the name and+// the group-2-is-value contract. QUOTED: after an opening `"` or `'`, the+// value is any run of non-quote, non-whitespace characters (a password like+// `ab!#%…<…` is one value). UNQUOTED: the legacy class plus the password+// punctuation `!#$%*@^~.?<>` — not `&` (it would swallow sibling query+// parameters), not `()[]{}` (so `token = getToken(x)` never matches), not `:`,+// and not `,` / `;` (the false-positive measurement found SQL `SET` lists+// such as `execution_token=NULL,claim_expires_at=NULL,error_code=$3` read as+// one value; a password carrying them is matched only when quoted) — and the+// last character excludes `.?!`, so `TOKEN=<tok>.` claims the token without+// the sentence's full stop and its bare echoes still scrub.+// Both keep the 12-4096 bounds and the digit + entropy gates. Each also has+// an ESCAPE_BOUNDARY twin (#7978): `\\npassword=…` in escaped tool output.+export const assignmentKey = (boundary: string) =>+ `${boundary}(?:secret|token|passwd|password|passphrase|credential|api[_-]?key|apikey)[A-Za-z0-9_-]{0,64}["']?\\s*[:=]\\s*`;+export const ASSIGNMENT_QUOTED_SOURCE = (key: string) => `(${key}["'])([^"'\\s]{12,4096})`;+export const ASSIGNMENT_UNQUOTED_SOURCE = (key: string) =>+ `(${key})([A-Za-z0-9+/_=\\-!#$%*@^~.?<>]{11,4095}[A-Za-z0-9+/_=\\-#$%*@^~<>])`;+/** The legacy (pre-punctuation) value class, whole-value form. */+const LEGACY_ASSIGNMENT_VALUE_RE = /^[A-Za-z0-9+/_=-]+$/;+/** A dotted member path in code (`process.env.DB_PASSWORD_2`): every segment an identifier. */+const DOTTED_IDENTIFIER_RE = /^[A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)+$/;++/**+ * Assignment-value validate: reject values that are references, not secrets+ * — a URL (`scheme://`), a relative or home path (`./`, `~/`), a shell+ * expansion (`${`, `$(`), an already-redacted token, an absolute path (`/`+ * plus a character outside the legacy class, such as the `.` of a file+ * name), and a dotted member path whose every segment is an identifier+ * (`process.env.X`, `config.secrets.key2`: the `.` admitted for passwords+ * would otherwise turn code into a value). A leading `/` on a pure base64+ * value still fires: one random base64 secret in 64 starts with `/`, and the+ * legacy rule redacted those. A dotted password whose every segment starts+ * with a letter is the accepted miss.+ */+export function assignmentValueIsSecretShaped(value: string): boolean {+ if (/^[A-Za-z][A-Za-z0-9+.-]{0,31}:\/\//.test(value)) return false;+ if (value.startsWith('./') || value.startsWith('~/') || value.startsWith('${') || value.startsWith('$(')) return false;+ if (value.startsWith('<REDACTED:') || DOTTED_IDENTIFIER_RE.test(value)) return false;+ return !(value.startsWith('/') && !LEGACY_ASSIGNMENT_VALUE_RE.test(value));+}diff --git a/src/core/secret-scan-stream.ts b/src/core/secret-scan-stream.tsindex a7ac59a49b..123c0f1afa 100644--- a/src/core/secret-scan-stream.ts+++ b/src/core/secret-scan-stream.ts@@ -16,7 +16,7 @@** OWNERSHIP. A claim belongs to the window holding the start of the match that* made it, anchor included (`from`: the `Bearer ` or `Authorization: Basic `- * keyword, an assignment key, or the boundary character before a value), not+ * keyword, an assignment key, or the boundary character or escape before a value), not* to the window holding the value's first character: a mid-line cut between* an anchor and its value leaves the next window a value with no anchor, which* no pattern claims. A window keeps a claim whose match starts before the nextdiff --git a/src/core/secret-scan.ts b/src/core/secret-scan.tsindex 9c21ea6c80..3bd0cb9bc1 100644--- a/src/core/secret-scan.ts+++ b/src/core/secret-scan.ts@@ -97,6 +97,10 @@ export {} from './secret-scan-allowlist.ts';import { valueAllowlisted, pathAllowlisted } from './secret-scan-allowlist.ts';import { PUSH_SCAN_MAX_FILE_BYTES } from './page-size-policy.ts';+import {+ ASSIGNMENT_QUOTED_SOURCE, ASSIGNMENT_UNQUOTED_SOURCE, HIGH_ENTROPY_MIN_BITS_PER_CHAR, HIGH_ENTROPY_REQUIRES_DIGIT_RE,+ HIGH_ENTROPY_REQUIRES_NON_DIGIT_RE, assignmentKey, assignmentValueIsSecretShaped,+} from './secret-scan-assignment.ts';import type { ScanWork } from './secret-scan-work.ts';export { createScanWork, type ScanWork } from './secret-scan-work.ts';@@ -104,7 +108,7 @@ export { createScanWork, type ScanWork } from './secret-scan-work.ts';//// Each core pattern is compiled with a left boundary group `(^|[^A-Za-z0-9_])`// so prefixes embedded inside longer identifiers ("risk-assessment…",-// "task-…") never fire. Order matters: 'anthropic' precedes 'openai' and the+// "task-…") never fire, plus an escape-boundary twin (ESCAPE_BOUNDARY, #7978). Order matters: 'anthropic' precedes 'openai' and the// per-line claimed-span set prevents a `sk-ant-…` key from double-reporting// as a generic `sk-` match.//@@ -133,6 +137,8 @@ interface CompiledPattern {continuation?: 'single' | 'pair';/** The text the regex reads instead of the line, the same length (offsets and reported values come from the line). */subject?: (line: string) => string;+ /** An ESCAPE_BOUNDARY twin: runs only on a line ESCAPE_IN_LINE_RE matches. */+ afterEscape?: boolean;}interface CorePattern {@@ -145,6 +151,11 @@ interface CorePattern {* literal keyword (e.g. `Bearer `) that must NOT be part of the value.*/prebuilt?: boolean;+ /**+ * Prebuilt rules only: the same two-group source anchored on ESCAPE_BOUNDARY+ * instead (#7978). A default-boundary rule gets that twin automatically.+ */+ escaped?: string;/** RegExp flags. Default `'g'`; a case-insensitive shape uses `'gi'`. */flags?: string;/**@@ -182,6 +193,22 @@ const SECURITY_WAVE_SINCE = '0.60.31.0';*/const PROVIDER_KEYS_SINCE = '0.60.502.0';+/**+ * #7978: a JSON escape (`\\n`, `\\t`, `\\u0022`) or a percent-escape (`%22`)+ * ends on a word character, so the default boundary never fired after one and+ * a key opening an escaped JSON string or a URL-encoded value went unclaimed+ * in every lane. Each rule gets a twin anchored here, compiled right after it+ * under the same name (attribution order unchanged) and run only on lines+ * holding such an escape, so every other line runs the old pattern list. A+ * backslash or `%` that does not start an escape is not a boundary.+ */+const ESCAPE_BOUNDARY = '(?:\\\\(?:[bfnrt]|u[0-9A-Fa-f]{4})|%[0-9A-Fa-f]{2})';+const ESCAPE_IN_LINE_RE = /\\[bfnrtu]|%[0-9A-Fa-f]{2}/;+const JWT_VALUE = '(eyJ[A-Za-z0-9_-]{8,4096}\\.[A-Za-z0-9_-]{8,4096}\\.[A-Za-z0-9_-]{8,4096})';+const BEARER_ANCHOR = '[Bb](?:earer|EARER)\\s+)([A-Za-z0-9._~+/=-]{20,})';+const BASIC_HEADER = 'authorization\\s{0,16}:\\s{0,16}basic\\s{1,16})([A-Za-z0-9+/=]{4,2048})(?![A-Za-z0-9+/=])';+const BASIC_BARE = 'basic\\s{1,16})([A-Za-z0-9+/=]{16,2048})(?![A-Za-z0-9+/=])';+/*** Placeholder passwords in documentation URLs: `<password>`, `${VAR}`,* `$VAR`, all-`*` and all-`x` masks. Never a live credential, so@@ -283,10 +310,9 @@ const CORE_PATTERNS: ReadonlyArray<CorePattern> = [// front of it); a segment over 4096 chars is the accepted miss.{name: 'jwt',- source:- '(^|[^A-Za-z0-9_-])' +- '(eyJ[A-Za-z0-9_-]{8,4096}\\.[A-Za-z0-9_-]{8,4096}\\.[A-Za-z0-9_-]{8,4096})',+ source: `(^|[^A-Za-z0-9_-])${JWT_VALUE}`,prebuilt: true,+ escaped: `(${ESCAPE_BOUNDARY})${JWT_VALUE}`,},// ── Catch-alls: LAST, so vendor/JWT attribution above claims the span first.// Bearer: prebuilt two-group layout — the `Bearer ` keyword is the anchor@@ -298,8 +324,9 @@ const CORE_PATTERNS: ReadonlyArray<CorePattern> = [// skipped before the regex runs.{name: 'bearer',- source: '((?:^|[^A-Za-z0-9_])[Bb](?:earer|EARER)\\s+)([A-Za-z0-9._~+/=-]{20,})',+ source: `((?:^|[^A-Za-z0-9_])${BEARER_ANCHOR}`,prebuilt: true,+ escaped: `(${ESCAPE_BOUNDARY}${BEARER_ANCHOR}`,precheck: (line) => line.includes('earer') || line.includes('EARER'),},// Connection strings with inline credentials: the value is exactly the@@ -369,8 +396,9 @@ const CORE_PATTERNS: ReadonlyArray<CorePattern> = [// family.{name: 'basic_auth',- source: '((?:^|[^A-Za-z0-9_])authorization\\s{0,16}:\\s{0,16}basic\\s{1,16})([A-Za-z0-9+/=]{4,2048})(?![A-Za-z0-9+/=])',+ source: `((?:^|[^A-Za-z0-9_])${BASIC_HEADER}`,prebuilt: true,+ escaped: `(${ESCAPE_BOUNDARY}${BASIC_HEADER}`,flags: 'gi',precheck: (line) => BASIC_PRECHECK_RE.test(line),validate: basicValueDecodesToUserPass,@@ -378,8 +406,9 @@ const CORE_PATTERNS: ReadonlyArray<CorePattern> = [},{name: 'basic_auth',- source: '((?:^|[^A-Za-z0-9_])basic\\s{1,16})([A-Za-z0-9+/=]{16,2048})(?![A-Za-z0-9+/=])',+ source: `((?:^|[^A-Za-z0-9_])${BASIC_BARE}`,prebuilt: true,+ escaped: `(${ESCAPE_BOUNDARY}${BASIC_BARE}`,flags: 'gi',precheck: (line) => BASIC_PRECHECK_RE.test(line),validate: basicValueDecodesToUserPass,@@ -619,104 +648,7 @@ export function privateKeySpans(text: string): Array<{ start: number; end: numbereturn pemClaims(text).map((c) => ({ start: c.start, end: c.end }));}-// Opt-in: `secret|token|password|api key`-shaped assignment whose value has-// high Shannon entropy. Keyword-anchored (compiled inline below so the group-// layout matches the core patterns: group 1 = anchor, group 2 = value).-//-// The left boundary is [^A-Za-z0-9] rather than the core patterns'-// [^A-Za-z0-9_]: `_` has to read as a SEPARATOR here, or `api_access_token`-// and `SMTP_PASSWORD` cannot match their own keyword. The core patterns keep-// `_` as a word character on purpose (a vendor prefix inside a longer-// identifier must not fire); this rule is keyword-anchored, so the same-// reasoning inverts. The keyword is also allowed trailing identifier-// segments before the assignment, or `AWS_SECRET_ACCESS_KEY=` cannot match:-// the keyword `SECRET` is not adjacent to the `=`, `_ACCESS_KEY` is. The-// entropy gate still decides, so a wider anchor costs nothing on prose.-//-// The value floor is 12, not 20. A 16-character SMTP password sat under the-// old floor and stayed plaintext in the receipt on disk even once the keyword-// matched — the value length was doing gating the entropy check is there to-// do. Real passwords are frequently 12-16 characters; secrets that long with-// 3.5 bits/char of entropy are not prose.-//-// The value must ALSO contain at least one digit. Identifier-shaped values-// (`DefaultAzureCredential`, `/usr/local/bin/aws-vault`, an env-var NAME-// assigned to an `apiKeyEnvVar`) clear the entropy gate on mixed case and-// separators alone and were being redacted out of ordinary code. Real-// machine-minted secrets essentially always carry digits; a digitless-// passphrase is the accepted miss. A 40-hex git sha assigned to a `token:`-// key still redacts (digits + entropy) — documented, acceptable.-//-// Both quantifiers after the keyword are BOUNDED. The trailing identifier-// segments used to be `[A-Za-z0-9_-]*`: because the left boundary admits `-`-// and `_` and that class contains them too, every `-`/`_` in a run like-// `-apikey-apikey…` or `_token_token…` was a fresh keyword start whose-// suffix ran to end-of-line and backtracked — quadratic (210 KB of-// `-apikey` took ~35 s, 180 KB of `_token` ~15 s). At `{0,64}` a start does-// constant work; no real credential key carries 64 identifier characters-// after its keyword. The value is capped at 4096 for the same-// constant-work-per-occurrence guarantee (the jwt segments share the cap):-// a value longer than that is redacted only through its first 4096 chars —-// the accepted miss, well above any real token or base64 key blob.-//-// The 3.5 bits/char floor is judged on the bias-corrected (Miller-Madow)-// estimate, not the raw per-character Shannon entropy. The raw figure of an-// n-character string can never exceed log2(n) (3.58 at 12 characters, 3.91-// at 15) and drops sharply with a single repeated character, so the raw-// floor rejected 68% of uniformly random 12-character alphanumeric secrets,-// 9% at 15 and 5% at 16: exactly the 12-16 character passwords the 12-char-// value floor exists to catch (a 15-character value such as `!dcG4Gmw1qGRbR3`-// after a password key stayed plaintext).-// Miller-Madow adds (distinct - 1) / (2 n ln 2), which cuts those misses-// below 1% at every length from 12 up and is negligible on long values.-// Because the correction can lift an all-digit value past the floor, the-// value must also carry a non-digit: counters, timestamps and numeric ids-// (`token_count=1234567890123`) never reached 3.5 raw bits (log2(10) = 3.32)-// and still never redact.-const HIGH_ENTROPY_MIN_BITS_PER_CHAR = 3.5;-const HIGH_ENTROPY_REQUIRES_DIGIT_RE = /[0-9]/;-const HIGH_ENTROPY_REQUIRES_NON_DIGIT_RE = /[^0-9]/;--// PUNCTUATED VALUES. The rule is two compiled patterns sharing the name and-// the group-2-is-value contract. QUOTED: after an opening `"` or `'`, the-// value is any run of non-quote, non-whitespace characters (a password like-// `ab!#%…<…` is one value). UNQUOTED: the legacy class plus the password-// punctuation `!#$%*@^~.?<>` — not `&` (it would swallow sibling query-// parameters), not `()[]{}` (so `token = getToken(x)` never matches), not `:`,-// and not `,` / `;` (the false-positive measurement found SQL `SET` lists-// such as `execution_token=NULL,claim_expires_at=NULL,error_code=$3` read as-// one value; a password carrying them is matched only when quoted) — and the-// last character excludes `.?!`, so `TOKEN=<tok>.` claims the token without-// the sentence's full stop and its bare echoes still scrub.-// Both keep the 12-4096 bounds and the digit + entropy gates.-const ASSIGNMENT_KEY =- '(?:^|[^A-Za-z0-9])(?:secret|token|passwd|password|passphrase|credential|api[_-]?key|apikey)[A-Za-z0-9_-]{0,64}["\']?\\s*[:=]\\s*';-const ASSIGNMENT_QUOTED_SOURCE = `(${ASSIGNMENT_KEY}["'])([^"'\\s]{12,4096})`;-const ASSIGNMENT_UNQUOTED_SOURCE =- `(${ASSIGNMENT_KEY})([A-Za-z0-9+/_=\\-!#$%*@^~.?<>]{11,4095}[A-Za-z0-9+/_=\\-#$%*@^~<>])`;-/** The legacy (pre-punctuation) value class, whole-value form. */-const LEGACY_ASSIGNMENT_VALUE_RE = /^[A-Za-z0-9+/_=-]+$/;-/** A dotted member path in code (`process.env.DB_PASSWORD_2`): every segment an identifier. */-const DOTTED_IDENTIFIER_RE = /^[A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)+$/;--/**- * Assignment-value validate: reject values that are references, not secrets- * — a URL (`scheme://`), a relative or home path (`./`, `~/`), a shell- * expansion (`${`, `$(`), an already-redacted token, an absolute path (`/`- * plus a character outside the legacy class, such as the `.` of a file- * name), and a dotted member path whose every segment is an identifier- * (`process.env.X`, `config.secrets.key2`: the `.` admitted for passwords- * would otherwise turn code into a value). A leading `/` on a pure base64- * value still fires: one random base64 secret in 64 starts with `/`, and the- * legacy rule redacted those. A dotted password whose every segment starts- * with a letter is the accepted miss.- */-function assignmentValueIsSecretShaped(value: string): boolean {- if (/^[A-Za-z][A-Za-z0-9+.-]{0,31}:\/\//.test(value)) return false;- if (value.startsWith('./') || value.startsWith('~/') || value.startsWith('${') || value.startsWith('$(')) return false;- if (value.startsWith('<REDACTED:') || DOTTED_IDENTIFIER_RE.test(value)) return false;- return !(value.startsWith('/') && !LEGACY_ASSIGNMENT_VALUE_RE.test(value));-}+// The opt-in `high_entropy_assignment` shapes and gates live in secret-scan-assignment.ts./*** One compiled set per option combination. scanInternal is synchronous and@@ -734,24 +666,30 @@ function compilePatterns(opts: ScanOpts): CompiledPattern[] {}function buildPatterns(opts: ScanOpts): CompiledPattern[] {- const out: CompiledPattern[] = CORE_PATTERNS.map((p) => ({- name: p.name,- // nosemgrep: javascript.lang.security.audit.detect-non-literal-regexp.detect-non-literal-regexp -- p.source is a compile-time literal from CORE_PATTERNS (never caller input); every pattern uses bounded quantifiers, pinned by test/secret-scan-perf.test.ts- re: new RegExp(p.prebuilt ? p.source : `(^|[^A-Za-z0-9_])(${p.source})`, p.flags ?? 'g'),- ...(p.precheck ? { precheck: p.precheck } : {}),- ...(p.validate ? { validate: p.validate } : {}),- }));+ const out: CompiledPattern[] = CORE_PATTERNS.flatMap((p) => {+ const shared = { name: p.name, ...(p.precheck ? { precheck: p.precheck } : {}), ...(p.validate ? { validate: p.validate } : {}) };+ const twin = p.prebuilt ? p.escaped : `(${ESCAPE_BOUNDARY})(${p.source})`;+ return [+ // nosemgrep: javascript.lang.security.audit.detect-non-literal-regexp.detect-non-literal-regexp -- p.source is a compile-time literal from CORE_PATTERNS (never caller input); every pattern uses bounded quantifiers, pinned by test/secret-scan-perf.test.ts+ { ...shared, re: new RegExp(p.prebuilt ? p.source : `(^|[^A-Za-z0-9_])(${p.source})`, p.flags ?? 'g') },+ // nosemgrep: javascript.lang.security.audit.detect-non-literal-regexp.detect-non-literal-regexp -- compile-time literal (ESCAPE_BOUNDARY + p.source); bounded quantifiers+ ...(twin ? [{ ...shared, re: new RegExp(twin, p.flags ?? 'g'), afterEscape: true }] : []),+ ];+ });if (opts.highEntropy) {// Group layout matches the core shape: group 1 = the keyword anchor// through the `=`/`:` (and opening quote), group 2 = value.for (const source of [ASSIGNMENT_QUOTED_SOURCE, ASSIGNMENT_UNQUOTED_SOURCE]) {- out.push({- name: 'high_entropy_assignment',- // nosemgrep: javascript.lang.security.audit.detect-non-literal-regexp.detect-non-literal-regexp -- compile-time literal; bounded quantifiers, pinned by test/secret-scan-perf.test.ts- re: new RegExp(source, 'gi'),- entropyGated: true,- validate: assignmentValueIsSecretShaped,- });+ for (const afterEscape of [false, true]) {+ out.push({+ name: 'high_entropy_assignment',+ // nosemgrep: javascript.lang.security.audit.detect-non-literal-regexp.detect-non-literal-regexp -- compile-time literal; bounded quantifiers, pinned by test/secret-scan-perf.test.ts+ re: new RegExp(source(assignmentKey(afterEscape ? ESCAPE_BOUNDARY : '(?:^|[^A-Za-z0-9])')), 'gi'),+ entropyGated: true,+ validate: assignmentValueIsSecretShaped,+ ...(afterEscape ? { afterEscape } : {}),+ });+ }}}if (opts.labeledCredentials) {@@ -941,6 +879,7 @@ function scanInternal(text: string, opts: ScanOpts): RawHit[] {if (pem.length > 0 && insideClaim(pem, lineStart, lineStart + line.length)) continue;if (work) { work.linesScanned++; work.lineChars += line.length; }const spans: LineSpans = { all: [] };+ let escaped: boolean | undefined;// Claimed-character bitmap behind the first-wins dedupe: allocated on the// line's FIRST hit only (most lines have none), then O(value) to test and// to mark. Same answer as scanning every prior span for an intersection.@@ -957,6 +896,7 @@ function scanInternal(text: string, opts: ScanOpts): RawHit[] {};for (const p of patterns) {if (p.continuation && p.continuation !== dangling) continue;+ if (p.afterEscape && !(escaped ??= ESCAPE_IN_LINE_RE.test(line))) continue;if (p.precheck && !p.precheck(line)) continue;p.re.lastIndex = 0;const subject = p.subject ? p.subject(line) : line;@@ -1238,26 +1178,26 @@ export function scanFiles(paths: string[], opts: ScanOpts = {}): SecretFinding[]// reports — a claimed value's bytes at a position the scanner did not claim// (embedded past its left boundary inside a longer identifier) are left// as-is, and the corpus write and the push gate see the same findings.-// Vendor-prefixed values lose nothing by staying outside the echo pass: they-// re-match on their own wherever they recur.//// THE ECHO PASS. A `bearer` token is claimed only where its `Bearer ` keyword// anchors it and a `high_entropy_assignment` value only where its keyword// does, yet transcripts routinely echo the same value bare (`Authorization:// Bearer <tok>` in a tool call, then `<tok>` alone in the reply; a pasted-// `PASSWORD=<v>` line, then `<v>` in prose). So the values those two patterns+// `PASSWORD=<v>` line, then `<v>` in prose). A vendor or JWT value recurs+// glued to a word (`token_<key>`, `prefix<key>`) where no boundary lets it+// re-match (#7978). So the values those patterns (and `labeled_credential`)// claim form an ECHO DICTIONARY (value → pattern) and every remaining// occurrence — bare, or embedded inside a longer identifier — is redacted as// that pattern's token too. Three bounds keep the pass cheap and// collision-safe: a value must clear its pattern's floor-// (ECHO_MIN_CHARS_BEARER = 20, ECHO_MIN_CHARS_ENTROPY = 12 — the rules' own+// (ECHO_MIN_CHARS_BEARER = ECHO_MIN_CHARS_VENDOR = 20, ECHO_MIN_CHARS_ENTROPY = 12 — the rules' own// floors, restated so a pattern edit alone can never widen the pass to// short, collision-prone values; an opted-in `labeled_credential` value// needs ECHO_MIN_CHARS_LABELED = 8 and must not be a stoplisted word, so// `pwd: 1234` never scrubs every later "1234"), must be at most// ECHO_MAX_VALUE_CHARS long,// and only the first ECHO_MAX_UNIQUE unique values in claim order — ONE cap-// across both patterns — join. A value past either cap is still redacted at+// across every pattern — join. A value past either cap is still redacted at// its claimed span; only its echoes are the accepted miss.//// The pass is a streaming merge over the ORIGINAL text: per value, the@@ -1302,6 +1242,15 @@ export const ECHO_MIN_CHARS_BEARER = 20;*/export const ECHO_MIN_CHARS_ENTROPY = 12;+/**+ * Floor for a vendor-shape or `jwt` value (every core rule but the four+ * catch-alls) to take part in the echo pass (#7978): bearer's floor, so a+ * short, collision-prone value never scrubs its later lookalikes.+ */+export const ECHO_MIN_CHARS_VENDOR = 20;+const NON_ECHO_CORE_PATTERNS = new Set(['bearer', 'db_url_credentials', 'url_credentials', 'basic_auth']);+const VENDOR_ECHO_PATTERNS = new Set(CORE_PATTERNS.map((p) => p.name).filter((name) => !NON_ECHO_CORE_PATTERNS.has(name)));+/*** A claimed value longer than this is redacted at its claimed span but does* not join the echo dictionary. Bounds the per-candidate compare of the@@ -1354,7 +1303,7 @@ function addEchoValue(into: EchoDictionary, pattern: string, value: string): voi? ECHO_MIN_CHARS_ENTROPY: pattern === 'labeled_credential'? ECHO_MIN_CHARS_LABELED- : Infinity;+ : VENDOR_ECHO_PATTERNS.has(pattern) ? ECHO_MIN_CHARS_VENDOR : Infinity;if (value.length < floor || value.length > ECHO_MAX_VALUE_CHARS) return;if (into.has(value) || into.size >= ECHO_MAX_UNIQUE) return;into.set(value, pattern);diff --git a/test/secret-scan-escape-boundary-7978.test.ts b/test/secret-scan-escape-boundary-7978.test.tsnew file mode 100644index 0000000000..699eb47004--- /dev/null+++ b/test/secret-scan-escape-boundary-7978.test.ts@@ -0,0 +1,144 @@+/**+ * #7978: the vendor rules' left boundary `(^|[^A-Za-z0-9_])` read the last+ * character of a JSON escape (`\\n` -> `n`) or a percent-escape (`%22` ->+ * `2`) as part of a word, so a key at the start of an escaped JSON string or+ * inside a URL-encoded value was never claimed, and a vendor-claimed value+ * never joined the redaction echo dictionary, so a second copy glued to a word+ * (`token_<key>`) stayed in the output. Both directions are pinned here: every+ * escaped context now redacts, and a prefix embedded in an ordinary word, an+ * escape-looking sequence that is not one, or an identifier-embedded key with+ * no clean copy still does not fire.+ *+ * Every key is assembled at runtime from random parts (gitleaks, CEO-10).+ */+import { describe, expect, test } from 'bun:test';+import { redactFindings, scanText } from '../src/core/secret-scan.ts';+import { PROVIDER_KEY_SHAPES, alnum, hex } from './helpers/provider-keys.ts';++const j = (sep: string, ...parts: string[]) => parts.join(sep);+const SEG = () => alnum(24).replace(/[^A-Za-z0-9]/g, 'a');++/** One synthetic key per default-boundary vendor rule, plus the prebuilt credential shapes. */+const KEYS: ReadonlyArray<{ pattern: string; make(): string }> = [+ { pattern: 'anthropic', make: () => j('-', 'sk', 'ant', 'api03', alnum(32)) },+ { pattern: 'openai', make: () => j('-', 'sk', 'proj', alnum(32)) },+ { pattern: 'openai', make: () => j('-', 'sk', alnum(40)) },+ { pattern: 'voyage', make: () => j('-', 'pa', alnum(32)) },+ { pattern: 'github_pat', make: () => j('_', 'github', 'pat', alnum(40)) },+ { pattern: 'github_token', make: () => j('_', 'ghp', alnum(36)) },+ { pattern: 'gitlab_pat', make: () => j('-', 'glpat', alnum(24)) },+ { pattern: 'slack', make: () => j('-', 'xoxb', '1234567890', alnum(24)) },+ { pattern: 'aws_access_key', make: () => 'AKIA' + hex(16).toUpperCase() },+ { pattern: 'google_api_key', make: () => 'AIza' + alnum(35) },+ { pattern: 'stripe', make: () => j('_', 'sk', 'live', alnum(32)) },+ { pattern: 'sendgrid', make: () => j('.', 'SG', alnum(22), alnum(43)) },+ { pattern: 'twilio', make: () => 'AC' + hex(32) },+ { pattern: 'supabase_key', make: () => j('_', 'sb', 'secret', alnum(32)) },+ { pattern: 'npm_token', make: () => j('_', 'npm', alnum(36)) },+ { pattern: 'huggingface', make: () => j('_', 'hf', alnum(34)) },+ { pattern: 'digitalocean', make: () => j('_', 'dop', 'v1', hex(64)) },+ ...PROVIDER_KEY_SHAPES.map((s) => ({ pattern: s.pattern, make: s.make })),+ { pattern: 'gbrain_token', make: () => j('_', 'gbrain', 'at', hex(64)) },+ { pattern: 'jwt', make: () => j('.', 'eyJ' + SEG(), SEG(), SEG()) },+];++/** The escape that sits immediately before the key, as it appears in the raw text. */+const ESCAPES = ['\\n', '\\r', '\\t', '\\\\n', '\\u0022', '%22', '%3D', '%3d', '%0A'];++describe('#7978 a key right after a JSON escape or a percent-escape is claimed', () => {+ for (const { pattern, make } of KEYS) {+ test(`${pattern}: claimed after every escape, and redacted`, () => {+ for (const esc of ESCAPES) {+ const key = make();+ const line = `{"role":"tool","out":"config loaded${esc}${key}${esc === '%22' ? '%22' : '\\n'}done"}`;+ const findings = scanText(line);+ expect({ esc, found: findings.map((f) => f.pattern) }).toEqual({ esc, found: [pattern] });+ const out = redactFindings(line, { highEntropy: true }).text;+ expect({ esc, leaked: out.includes(key) }).toEqual({ esc, leaked: false });+ expect(out).toContain(`${esc}<REDACTED:${pattern}>`);+ }+ });+ }++ test('the #7978 reproduction: Bearer <openrouter key>, then the key after an escaped newline in tool output', () => {+ const k = j('-', 'sk', 'or', 'v1', hex(64));+ const out = redactFindings(`Authorization: Bearer ${k}\n{"out":"line\\n${k}"}`, { highEntropy: true }).text;+ expect(out.includes(k)).toBe(false);+ expect(out).toBe('Authorization: Bearer <REDACTED:openrouter>\n{"out":"line\\n<REDACTED:openrouter>"}');+ });++ test('a URL-encoded body: api_key%3D<key>%26next=1', () => {+ const k = j('_', 'gsk', alnum(52));+ expect(redactFindings(`body=api_key%3D${k}%26next%3D1`).text).toBe('body=api_key%3D<REDACTED:groq>%26next%3D1');+ });++ test('Bearer and Basic after an escaped newline claim their token', () => {+ const tok = alnum(40);+ expect(redactFindings(`{"h":"x\\nBearer ${tok}"}`).text).toBe('{"h":"x\\nBearer <REDACTED:bearer>"}');+ const basic = Buffer.from(`alice-example:${alnum(20)}`).toString('base64');+ expect(redactFindings(`{"h":"x\\nAuthorization: Basic ${basic}"}`).text).toBe('{"h":"x\\nAuthorization: Basic <REDACTED:basic_auth>"}');+ });++ test('a keyword-anchored high-entropy assignment after an escaped newline is claimed (transcript lane)', () => {+ const v = alnum(24) + '7';+ const out = redactFindings(`{"out":"ok\\npassword=${v}\\n"}`, { highEntropy: true }).text;+ expect(out.includes(v)).toBe(false);+ });+});++describe('#7978 a vendor-claimed value joins the echo dictionary', () => {+ test('a second copy glued to a word (token_<key>, prefix<key>) is scrubbed as that vendor', () => {+ const k = j('_', 'ghp', alnum(36));+ const out = redactFindings(`GITHUB_TOKEN=${k}\nurl?token_${k}&x=1 id=prefix${k}\n`).text;+ expect(out).toBe('GITHUB_TOKEN=<REDACTED:github_token>\nurl?token_<REDACTED:github_token>&x=1 id=prefix<REDACTED:github_token>\n');+ });++ test('a session-wide dictionary carries a vendor value claimed in one field to a glued copy in another', async () => {+ const { planRedaction, applyRedaction } = await import('../src/core/secret-scan.ts');+ const k = j('-', 'sk', 'or', 'v1', hex(64));+ const echoValues = new Map<string, string>();+ const glued = planRedaction(`cache_key=v1_${k}`, { echoValues });+ planRedaction(`OPENROUTER_API_KEY=${k}`, { echoValues });+ expect(applyRedaction(glued)).toBe('cache_key=v1_<REDACTED:openrouter>');+ });++ test('a vendor value under the 20-character echo floor (a short slack token) does not join', () => {+ const k = j('-', 'xoxb', alnum(11));+ expect(k.length).toBeLessThan(20);+ expect(redactFindings(`t=${k}\nid=x${k}`).text).toBe(`t=<REDACTED:slack>\nid=x${k}`);+ });+});++describe('#7978 no new false positives', () => {+ test('a vendor prefix inside an ordinary word stays clean, escaped or not', () => {+ const prose = [+ '{"out":"line one\\nrisk-assessment-matrix-for-the-quarter and task-scheduler-for-the-cluster"}',+ '{"out":"\\nskeleton-key-ceremony\\tdesk-assessment-overview-document"}',+ 'q=%22risk-assessment-matrix-for-the-quarter%22&r=%20task-scheduler-configuration-for-cluster',+ '{"out":"\\nwhsec-is-not-a-key\\nghp-is-not-a-key\\nnpm_install_dependencies_for_workspace_now"}',+ ].join('\n');+ expect(scanText(prose)).toEqual([]);+ expect(redactFindings(prose, { highEntropy: true }).text).toBe(prose);+ });++ test('a backslash or percent sign that is not an escape is not a boundary', () => {+ for (const before of ['\\q', '\\x', '%ZZ', '%2', 'n', '2', '_']) {+ const key = j('_', 'ghp', alnum(36));+ const line = `id=a${before}${key}`;+ expect({ before, found: scanText(line).map((f) => f.pattern) }).toEqual({ before, found: [] });+ }+ });++ test('a key embedded in an identifier with no clean copy anywhere stays unclaimed (the push gate and redaction agree)', () => {+ const k = j('-', 'sk', alnum(40));+ const text = `id=prefix${k} and token_${k}`;+ expect(scanText(text)).toEqual([]);+ expect(redactFindings(text, { highEntropy: true }).text).toBe(text);+ });++ test('the escape stays in the output; only the key is redacted', () => {+ const k = j('_', 'npm', alnum(36));+ expect(redactFindings(`"a\\n${k}"`).text).toBe('"a\\n<REDACTED:npm_token>"');+ expect(redactFindings(`a%22${k}%22`).text).toBe('a%22<REDACTED:npm_token>%22');+ });+});diff --git a/test/secret-scan.test.ts b/test/secret-scan.test.tsindex b7dd11bf44..bfed609245 100644--- a/test/secret-scan.test.ts+++ b/test/secret-scan.test.ts@@ -717,21 +717,19 @@ describe('redactFindings — span-splice semantics and preview window', () => {expect(redactions.map((r) => r.pattern)).toEqual(['bearer', 'db_url_credentials']);});- test('redacts exactly the claimed spans (every pattern outside the echo pass): the corpus write agrees with what scanText reports', () => {- // Documented semantic delta from the replaceAll form: a claimed value's- // bytes at a position the scanner did NOT claim (embedded past its left- // boundary inside a longer identifier) are left as-is, so- // `redactFindings(t).text` never redacts something `scanText(t)` would- // not have reported. The exception is the bounded echo pass over the- // bearer + high_entropy_assignment values (pinned in the next describe);- // vendor shapes lose nothing by staying outside it — a repeat that clears- // the boundary IS claimed on its own and both occurrences go.+ test('redacts the claimed spans plus their echoes: the findings agree with scanText, a vendor value glued past its boundary is an echo (#7978)', () => {+ // Documented semantic delta from the replaceAll form: the splice redacts+ // exactly what `scanText(t)` reports, and the bounded echo pass (pinned in+ // the next describe) adds the bytes of a claimed value at a position the+ // scanner did NOT claim. Since #7978 vendor values join it, so a copy+ // embedded past its left boundary inside a longer identifier goes too,+ // without adding a finding.const text = `k=${OPENAI} again ${OPENAI}\nid=prefix${OPENAI}\nsid ${TWILIO_SID} vs x${TWILIO_SID}\n`;const findings = scanText(text);expect(findings.map((f) => [f.pattern, f.line])).toEqual([['openai', 1], ['openai', 1], ['twilio', 3]]);const { text: out, redactions } = redactFindings(text);expect(redactions.length).toBe(findings.length);- expect(out).toBe(`k=<REDACTED:openai> again <REDACTED:openai>\nid=prefix${OPENAI}\nsid <REDACTED:twilio> vs x${TWILIO_SID}\n`);+ expect(out).toBe('k=<REDACTED:openai> again <REDACTED:openai>\nid=prefix<REDACTED:openai>\nsid <REDACTED:twilio> vs x<REDACTED:twilio>\n');});test('a per-line span running into a PEM block: both spans redact, nothing claimed survives', () => {@@ -854,9 +852,12 @@ describe('redactFindings — bounded echo pass', () => {expect(out).toBe('Bearer <REDACTED:anthropic>\nbare <REDACTED:anthropic>\n');});- test("other patterns' values are not echoed: a vendor key embedded past its boundary stays (it re-matches on its own wherever it clears the boundary)", () => {- const { text: out } = redactFindings(`k=${OPENAI}\nid=prefix${OPENAI}\n`);- expect(out).toBe(`k=<REDACTED:openai>\nid=prefix${OPENAI}\n`);+ test('a vendor key embedded past its boundary is echoed once a clean copy is claimed (#7978); the catch-all basic_auth is not', () => {+ const { text: out, redactions } = redactFindings(`k=${OPENAI}\nid=prefix${OPENAI}\n`);+ expect(out).toBe('k=<REDACTED:openai>\nid=prefix<REDACTED:openai>\n');+ expect(redactions.map((r) => r.pattern)).toEqual(['openai']);+ const basic = Buffer.from('alice-example:' + 'Zq7'.repeat(8)).toString('base64');+ expect(redactFindings(`Authorization: Basic ${basic}\nid=x${basic}\n`).text).toBe(`Authorization: Basic <REDACTED:basic_auth>\nid=x${basic}\n`);});test(`the cap is exactly ${ECHO_MAX_UNIQUE} unique values in claim order; the next value's echo is the accepted miss`, () => {