Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(secret-scan): vendor and jwt echo values get their own cap so planted keys cannot crowd out a bearer echo (#7978 review)

garrytan10310b46696f
diff --git a/docs/architecture/key-files/secret-scan.md b/docs/architecture/key-files/secret-scan.md
index 324cddf531..f6a816170b 100644
--- a/docs/architecture/key-files/secret-scan.md
+++ b/docs/architecture/key-files/secret-scan.md
@@ -6,6 +6,6 @@ The format-based secret detector shared by `sources push`, bootstrap verify, the
compiled-context sensitivity scan and the transcripts redactor, and the composed
sensitivity detector built on it.
-- `src/core/secret-scan.ts` — pattern-based secret detection shared by `sources push`, bootstrap verify, the compiled-context sensitivity scan and the transcripts redactor. Detection is FORMAT-based, not prefix-only: vendor shapes (anthropic, openai, voyage, github, gitlab, slack, aws AKIA/ASIA, google, stripe, sendgrid, twilio, supabase sb_secret_/sbp_, npm, huggingface, `digitalocean` `do[opr]_v1_` + 64 hex, the AI-provider keys `openrouter` `sk-or-v1-` + 64 lowercase hex, `groq` `gsk_` + 52, `xai` `xai-` + 80, `perplexity` `pplx-` + 48 and `replicate` `r8_` + 37, all fixed-length with a hard right edge, gbrain), JWTs, private keys (whole-text claims, below), plus four catch-alls appended LAST (`bearer` — prebuilt two-group layout so only the token is the value; `db_url_credentials` — the `scheme://user:pass@` span of database URLs, host/db kept; `url_credentials` — the same span for `http(s)` (scheme case-insensitive, `gi`), user 0–128 stopping at `:`/`/`, password 1–256 excluding whitespace, `@`, `"`, `'`, `/`, `?`, `#`, with a `validate` hook that rejects placeholder passwords (`<...>`, `${...}`, `$VAR`, all `*`, all `x`); `basic_auth` — two prebuilt case-insensitive forms, `Authorization: Basic` with a 4–2048 char value and bare `Basic` with 16–2048, both with a hard right edge and a `validate` that requires the value to base64-decode to printable ASCII containing `:`; it is NOT in the bearer family, so it joins neither the echo dictionary nor `secret:bearer`). The pattern contract: `CorePattern` carries optional `flags` (default `'g'`), `precheck`, `validate(value)` (O(value), run on the whole match before the claim and the union split; a rejected match is not re-tried from inside) and `since` (the gbrain version that added or last changed the rule; `SECURITY_WAVE_SINCE` = `0.60.31.0` on `digitalocean`, `url_credentials`, `basic_auth` and `private_key_pem`, `PROVIDER_KEYS_SINCE` on the five AI-provider rules (#7959), kept in step with the release that ships a rule change). Compiled patterns keep group 1 = anchor, group 2 = value. Every core rule and both `high_entropy_assignment` forms also have an escape-boundary twin (#7978), compiled right after the rule under the same name and anchored on a JSON escape (`\n`, `\r`, `\t`, `\b`, `\f`, `\uXXXX`) or a percent-escape (`%XX`), whose last character the default boundary reads as a word character; a twin runs only on a line `ESCAPE_IN_LINE_RE` matches, so other lines run the old list, and a prebuilt rule carries its twin as `CorePattern.escaped`. `patternSince(pattern)` reads the map; every finding of such a pattern carries `SecretFinding.since`, which the push and bootstrap refusals render. Fixed-length shapes carry a trailing negative lookahead. Per-line claimed-span dedupe is first-wins for ATTRIBUTION (order matters: `Bearer <vendor key>` keeps the vendor name, a fully covered later span is skipped) but coverage is the UNION: every unclaimed slice of a later overlapping span is claimed as a hit of the later pattern (own value, own fingerprint), so a database URL whose username is a vendor-shaped id still loses its password. The transcript-only `labeled_credential` rule (`ScanOpts.labeledCredentials`) lives in `secret-scan-labeled.ts` (Security). The opt-in `high_entropy_assignment` rule (`ScanOpts.highEntropy`; its shapes and gates live in `src/core/secret-scan-assignment.ts`, split out at the 1500-line module cap) requires a 12+ char value that contains a digit and a non-digit AND clears 3.5 bits/char on the bias-corrected (Miller-Madow) estimate `correctedEntropy`, since raw per-character entropy is capped at log2(n) and rejected most random 12-16 char secrets. It is two compiled patterns sharing the name: QUOTED (after `"` or `'`, any run of non-quote, non-whitespace characters) and UNQUOTED (the legacy `[A-Za-z0-9+/_=-]` class plus `!#$%*@^~.?<>`; never `&`, `()[]{}`, `:`, `,` or `;`, so query strings, calls and SQL `SET` lists stay intact, and a password carrying `,`/`;` matches only quoted; the last character excludes `.?!`, so a sentence's full stop is not claimed). Its `validate` rejects values that start with `scheme://`, `./`, `~/`, `${`, `$(` or `<REDACTED:`, dotted identifier paths (`process.env.X`) and `/`-paths holding a character outside the legacy class. Consumers with it on: retrieval output, transcript ingest, hooks and the hook heartbeat, the atoms drain, `embed-facts.ts` error text and the context-engine relay re-scan. Off: the push gate, bootstrap verify/repo and the compiled-context sensitivity scan. Own minimal allowlist + `<ws>/.gbrain-scan-allow` per-finding overrides — deliberately NOT the repo's `.gitleaks.toml` (a public-repo CI fixture policy); the allowlist block (`loadWorkspaceAllowlist`, the glob dialect `globToRegExp`/`matchesGlob`, `valueAllowlisted`/`pathAllowlisted`, `SCAN_ALLOW_FILENAME`, the 16-hex fingerprint floor) lives in `src/core/secret-scan-allowlist.ts` since wave 16 (G4.1, the 1500-line module cap) and is re-exported here unchanged (`test/secret-scan-allowlist.test.ts`). `correctedEntropy` is one histogram pass (distinct count = histogram size; byte-identical to the former `shannonEntropy + Set` form, `test/secret-scan-entropy-equivalence.test.ts`; ~80 → ~25 ms per 250 × 4096-char segment). `ScanOpts.work` / `RedactOpts.work` takes a `ScanWork` accumulator (`src/core/secret-scan-work.ts`: lines, regex execs, candidates per pattern, chars handed to the entropy gate counted inside `correctedEntropy`, validate calls, claims, union splits, echo sweeps) pinned exactly on the perf fixture by `test/secret-scan-work-counters.test.ts`; the perf file's absolute bounds stay the only guard on per-character cost. Deliberately NOT patterns: `sb_publishable_`, Supabase project refs (identifiers, not credentials). Never spell a literal database or `http(s)` URL with userinfo, or any credential, in this file's comments or in fixtures (assemble test values at runtime; `scripts/check-pg-url-redaction.sh` greps `src/` for every `db_url_credentials` scheme). Long lines are scanned in full (no length cap — a secret on a 1 MB minified line must still redact); what keeps that linear is bounded quantifiers on every shape whose value class could otherwise run to end-of-line — `db_url_credentials` (user ≤128 stopping at `:`/`/`; password 1–256 stopping only at whitespace, `@` and the string delimiters `"`/`'`, so unescaped `/ { } < > | ^ \` and backtick passwords redact), `jwt` (prebuilt so its boundary excludes `-` as well as `_`; segments ≤4096), `high_entropy_assignment` (≤64 identifier chars after the keyword, value ≤4096), `url_credentials` (user ≤128, password ≤256), `basic_auth` (value ≤2048, so its decode is constant work) and the private-key walks (below) — plus a per-pattern `precheck` substring gate on the catch-alls (`@` for both URL patterns; `earer`/`EARER`, since an all-caps `BEARER` scheme is accepted; a case-insensitive `basic`), a carried next-newline position in `scanPemBlocks` (sticky -1, so the header-line count is O(text) even when thousands of hits sit ahead of a newline-free multi-megabyte tail), a lazily-allocated per-line claimed-char bitmap behind the first-wins dedupe (O(value) per hit), and a windowed `buildPreview` (whole line ≤160 chars; otherwise hit−40…+80, edges snapped outward to whole spans, every claimed span in the window spliced by position, so a preview never carries a sibling secret or a fragment). `redactFindings` (= `planRedaction` + `applyRedaction`) rebuilds the text by SPAN-SPLICE in one O(text + hits) pass over the claimed spans, and fills the UNCLAIMED gaps with a bounded echo pass: the values claimed by `bearer`, `high_entropy_assignment`, `labeled_credential` and every vendor or `jwt` rule (#7978; not `basic_auth` or the two URL userinfo catch-alls) form an echo dictionary (value → pattern; floors `ECHO_MIN_CHARS_BEARER` = `ECHO_MIN_CHARS_VENDOR` = 20 / `ECHO_MIN_CHARS_ENTROPY` = 12, values ≤ `ECHO_MAX_VALUE_CHARS` = 512, the first `ECHO_MAX_UNIQUE` = 64 unique values in claim order across every pattern — a value past either cap is redacted at its claim, its echoes are the accepted miss), matched by a streaming merge over the ORIGINAL text — per value, the next occurrence at or beyond the emit cursor (`indexOf` resuming at the cursor, re-sought only once the cursor passes it), leftmost-longest at each step, emitted incrementally into the unclaimed gaps — so retained state is O(values) however long the text is (occurrences are never collected into a list) (never an alternation RegExp, never over emitted tokens). Echoes add no findings. `planRedaction` feeds a caller-supplied shared map (`RedactOpts.echoValues`) and `applyRedaction` reads it at apply time, so a multi-field document scrubs an echo in one field of a value claimed in another. `basic_auth` and the URL userinfo catch-alls redact exactly what `scanText` reports; a vendor value embedded past its boundary (`token_<key>`) is scrubbed as an echo when the same value is claimed elsewhere in the document, and left as-is otherwise. PRIVATE KEYS: `pemClaims` finds `BEGIN`/`END … PRIVATE KEY` fences in two global passes and merges them in text order. BLOCK: a BEGIN whose next fence is an END within `PEM_BODY_MAX_CHARS` (16384) is claimed header through END; a block never swallows a later BEGIN. FORWARD (BEGIN with no END in view): the header plus the body lines that follow; a header followed by no body line (a marker quoted in prose) is claimed alone. BACKWARD (END with no BEGIN in view): the body lines before the fence plus the fence; an END with no body line before it is no claim. A body line is a whole-line base64 run (`A-Za-z0-9+/=`) indented by up to 16 spaces/tabs; line ends are `\n`, `\r\n` or the literal JSON escapes; interior lines are 16–128 chars, the line next to a fence and a partial line at a cut 1–128; optional RFC 1421 `Proc-Type:`/`DEK-Info:` lines and one blank line may follow BEGIN; a line with anything else, or a run over 128 chars, ends the body. Total claimed body stays within `PEM_BODY_MAX_CHARS`. Each walk stops at the neighbouring fence or the previous claim, so the pass is linear (20k unterminated BEGINs, 20k END-only fences and 1 MB of base64 lines are perf-pinned). Per-line hits that fall wholly inside a key claim are dropped (one key, one finding). A FORWARD claim's finding also carries `legacyFingerprint`, the header-only fingerprint earlier releases reported, so `workspace-push.ts` can name a stale allowlist entry; a truncated key's fingerprint covers its body and therefore depends on where the text was cut, so it is not a stable allowlist key. `privateKeySpans(text)` exports the same claims as sorted, disjoint `{start, end}` spans. Accepted misses: body on the header's own line, space-joined single-line keys, blockquoted (`> `) keys, base64 runs over 128 chars, body past 16384 chars, `http(s)` passwords containing a literal `/`, `?` or `#`, Basic values over 2048 chars, digitless assignment values, dotted passwords whose every segment starts with a letter, and unquoted passwords containing `,` or `;`. False-positive budget: `scripts/secret-scan-fp-budget.ts` diffs findings against a base ref's scanner (`--base`, default `origin/master`) (`--dir <brain>` reports per-pattern counts only, never content) and `test/secret-scan-fp-budget.test.ts` pins a synthetic brain-like fixture (zero findings on its non-secret part in both modes, every planted secret redacted). Shape tests: `test/secret-scan-credential-shapes.test.ts`, `test/secret-scan-private-key-spans.test.ts`. Perf pins: `test/secret-scan-perf.test.ts`. Retrieval output is another consumer of its canonical scanner; that output boundary does not sanitize internal evaluation/provider inputs.
+- `src/core/secret-scan.ts` — pattern-based secret detection shared by `sources push`, bootstrap verify, the compiled-context sensitivity scan and the transcripts redactor. Detection is FORMAT-based, not prefix-only: vendor shapes (anthropic, openai, voyage, github, gitlab, slack, aws AKIA/ASIA, google, stripe, sendgrid, twilio, supabase sb_secret_/sbp_, npm, huggingface, `digitalocean` `do[opr]_v1_` + 64 hex, the AI-provider keys `openrouter` `sk-or-v1-` + 64 lowercase hex, `groq` `gsk_` + 52, `xai` `xai-` + 80, `perplexity` `pplx-` + 48 and `replicate` `r8_` + 37, all fixed-length with a hard right edge, gbrain), JWTs, private keys (whole-text claims, below), plus four catch-alls appended LAST (`bearer` — prebuilt two-group layout so only the token is the value; `db_url_credentials` — the `scheme://user:pass@` span of database URLs, host/db kept; `url_credentials` — the same span for `http(s)` (scheme case-insensitive, `gi`), user 0–128 stopping at `:`/`/`, password 1–256 excluding whitespace, `@`, `"`, `'`, `/`, `?`, `#`, with a `validate` hook that rejects placeholder passwords (`<...>`, `${...}`, `$VAR`, all `*`, all `x`); `basic_auth` — two prebuilt case-insensitive forms, `Authorization: Basic` with a 4–2048 char value and bare `Basic` with 16–2048, both with a hard right edge and a `validate` that requires the value to base64-decode to printable ASCII containing `:`; it is NOT in the bearer family, so it joins neither the echo dictionary nor `secret:bearer`). The pattern contract: `CorePattern` carries optional `flags` (default `'g'`), `precheck`, `validate(value)` (O(value), run on the whole match before the claim and the union split; a rejected match is not re-tried from inside) and `since` (the gbrain version that added or last changed the rule; `SECURITY_WAVE_SINCE` = `0.60.31.0` on `digitalocean`, `url_credentials`, `basic_auth` and `private_key_pem`, `PROVIDER_KEYS_SINCE` on the five AI-provider rules (#7959), kept in step with the release that ships a rule change). Compiled patterns keep group 1 = anchor, group 2 = value. Every core rule and both `high_entropy_assignment` forms also have an escape-boundary twin (#7978), compiled right after the rule under the same name and anchored on a JSON escape (`\n`, `\r`, `\t`, `\b`, `\f`, `\uXXXX`) or a percent-escape (`%XX`), whose last character the default boundary reads as a word character; a twin runs only on a line `ESCAPE_IN_LINE_RE` matches, so other lines run the old list, and a prebuilt rule carries its twin as `CorePattern.escaped`. `patternSince(pattern)` reads the map; every finding of such a pattern carries `SecretFinding.since`, which the push and bootstrap refusals render. Fixed-length shapes carry a trailing negative lookahead. Per-line claimed-span dedupe is first-wins for ATTRIBUTION (order matters: `Bearer <vendor key>` keeps the vendor name, a fully covered later span is skipped) but coverage is the UNION: every unclaimed slice of a later overlapping span is claimed as a hit of the later pattern (own value, own fingerprint), so a database URL whose username is a vendor-shaped id still loses its password. The transcript-only `labeled_credential` rule (`ScanOpts.labeledCredentials`) lives in `secret-scan-labeled.ts` (Security). The opt-in `high_entropy_assignment` rule (`ScanOpts.highEntropy`; its shapes and gates live in `src/core/secret-scan-assignment.ts`, split out at the 1500-line module cap) requires a 12+ char value that contains a digit and a non-digit AND clears 3.5 bits/char on the bias-corrected (Miller-Madow) estimate `correctedEntropy`, since raw per-character entropy is capped at log2(n) and rejected most random 12-16 char secrets. It is two compiled patterns sharing the name: QUOTED (after `"` or `'`, any run of non-quote, non-whitespace characters) and UNQUOTED (the legacy `[A-Za-z0-9+/_=-]` class plus `!#$%*@^~.?<>`; never `&`, `()[]{}`, `:`, `,` or `;`, so query strings, calls and SQL `SET` lists stay intact, and a password carrying `,`/`;` matches only quoted; the last character excludes `.?!`, so a sentence's full stop is not claimed). Its `validate` rejects values that start with `scheme://`, `./`, `~/`, `${`, `$(` or `<REDACTED:`, dotted identifier paths (`process.env.X`) and `/`-paths holding a character outside the legacy class. Consumers with it on: retrieval output, transcript ingest, hooks and the hook heartbeat, the atoms drain, `embed-facts.ts` error text and the context-engine relay re-scan. Off: the push gate, bootstrap verify/repo and the compiled-context sensitivity scan. Own minimal allowlist + `<ws>/.gbrain-scan-allow` per-finding overrides — deliberately NOT the repo's `.gitleaks.toml` (a public-repo CI fixture policy); the allowlist block (`loadWorkspaceAllowlist`, the glob dialect `globToRegExp`/`matchesGlob`, `valueAllowlisted`/`pathAllowlisted`, `SCAN_ALLOW_FILENAME`, the 16-hex fingerprint floor) lives in `src/core/secret-scan-allowlist.ts` since wave 16 (G4.1, the 1500-line module cap) and is re-exported here unchanged (`test/secret-scan-allowlist.test.ts`). `correctedEntropy` is one histogram pass (distinct count = histogram size; byte-identical to the former `shannonEntropy + Set` form, `test/secret-scan-entropy-equivalence.test.ts`; ~80 → ~25 ms per 250 × 4096-char segment). `ScanOpts.work` / `RedactOpts.work` takes a `ScanWork` accumulator (`src/core/secret-scan-work.ts`: lines, regex execs, candidates per pattern, chars handed to the entropy gate counted inside `correctedEntropy`, validate calls, claims, union splits, echo sweeps) pinned exactly on the perf fixture by `test/secret-scan-work-counters.test.ts`; the perf file's absolute bounds stay the only guard on per-character cost. Deliberately NOT patterns: `sb_publishable_`, Supabase project refs (identifiers, not credentials). Never spell a literal database or `http(s)` URL with userinfo, or any credential, in this file's comments or in fixtures (assemble test values at runtime; `scripts/check-pg-url-redaction.sh` greps `src/` for every `db_url_credentials` scheme). Long lines are scanned in full (no length cap — a secret on a 1 MB minified line must still redact); what keeps that linear is bounded quantifiers on every shape whose value class could otherwise run to end-of-line — `db_url_credentials` (user ≤128 stopping at `:`/`/`; password 1–256 stopping only at whitespace, `@` and the string delimiters `"`/`'`, so unescaped `/ { } < > | ^ \` and backtick passwords redact), `jwt` (prebuilt so its boundary excludes `-` as well as `_`; segments ≤4096), `high_entropy_assignment` (≤64 identifier chars after the keyword, value ≤4096), `url_credentials` (user ≤128, password ≤256), `basic_auth` (value ≤2048, so its decode is constant work) and the private-key walks (below) — plus a per-pattern `precheck` substring gate on the catch-alls (`@` for both URL patterns; `earer`/`EARER`, since an all-caps `BEARER` scheme is accepted; a case-insensitive `basic`), a carried next-newline position in `scanPemBlocks` (sticky -1, so the header-line count is O(text) even when thousands of hits sit ahead of a newline-free multi-megabyte tail), a lazily-allocated per-line claimed-char bitmap behind the first-wins dedupe (O(value) per hit), and a windowed `buildPreview` (whole line ≤160 chars; otherwise hit−40…+80, edges snapped outward to whole spans, every claimed span in the window spliced by position, so a preview never carries a sibling secret or a fragment). `redactFindings` (= `planRedaction` + `applyRedaction`) rebuilds the text by SPAN-SPLICE in one O(text + hits) pass over the claimed spans, and fills the UNCLAIMED gaps with a bounded echo pass: the values claimed by `bearer`, `high_entropy_assignment`, `labeled_credential` and every vendor or `jwt` rule (#7978; not `basic_auth` or the two URL userinfo catch-alls) form an echo dictionary (value → pattern; floors `ECHO_MIN_CHARS_BEARER` = `ECHO_MIN_CHARS_VENDOR` = 20 / `ECHO_MIN_CHARS_ENTROPY` = 12, values ≤ `ECHO_MAX_VALUE_CHARS` = 512, the first `ECHO_MAX_UNIQUE` = 64 unique bearer, assignment and labeled values plus, under their own cap, the first `ECHO_MAX_UNIQUE_VENDOR` = 32 unique vendor or `jwt` values, in claim order, so planted vendor strings never crowd out a bearer token's echo — a value past a cap is redacted at its claim, its echoes are the accepted miss), matched by a streaming merge over the ORIGINAL text — per value, the next occurrence at or beyond the emit cursor (`indexOf` resuming at the cursor, re-sought only once the cursor passes it), leftmost-longest at each step, emitted incrementally into the unclaimed gaps — so retained state is O(values) however long the text is (occurrences are never collected into a list) (never an alternation RegExp, never over emitted tokens). Echoes add no findings. `planRedaction` feeds a caller-supplied shared map (`RedactOpts.echoValues`) and `applyRedaction` reads it at apply time, so a multi-field document scrubs an echo in one field of a value claimed in another. `basic_auth` and the URL userinfo catch-alls redact exactly what `scanText` reports; a vendor value embedded past its boundary (`token_<key>`) is scrubbed as an echo when the same value is claimed elsewhere in the document, and left as-is otherwise. PRIVATE KEYS: `pemClaims` finds `BEGIN`/`END … PRIVATE KEY` fences in two global passes and merges them in text order. BLOCK: a BEGIN whose next fence is an END within `PEM_BODY_MAX_CHARS` (16384) is claimed header through END; a block never swallows a later BEGIN. FORWARD (BEGIN with no END in view): the header plus the body lines that follow; a header followed by no body line (a marker quoted in prose) is claimed alone. BACKWARD (END with no BEGIN in view): the body lines before the fence plus the fence; an END with no body line before it is no claim. A body line is a whole-line base64 run (`A-Za-z0-9+/=`) indented by up to 16 spaces/tabs; line ends are `\n`, `\r\n` or the literal JSON escapes; interior lines are 16–128 chars, the line next to a fence and a partial line at a cut 1–128; optional RFC 1421 `Proc-Type:`/`DEK-Info:` lines and one blank line may follow BEGIN; a line with anything else, or a run over 128 chars, ends the body. Total claimed body stays within `PEM_BODY_MAX_CHARS`. Each walk stops at the neighbouring fence or the previous claim, so the pass is linear (20k unterminated BEGINs, 20k END-only fences and 1 MB of base64 lines are perf-pinned). Per-line hits that fall wholly inside a key claim are dropped (one key, one finding). A FORWARD claim's finding also carries `legacyFingerprint`, the header-only fingerprint earlier releases reported, so `workspace-push.ts` can name a stale allowlist entry; a truncated key's fingerprint covers its body and therefore depends on where the text was cut, so it is not a stable allowlist key. `privateKeySpans(text)` exports the same claims as sorted, disjoint `{start, end}` spans. Accepted misses: body on the header's own line, space-joined single-line keys, blockquoted (`> `) keys, base64 runs over 128 chars, body past 16384 chars, `http(s)` passwords containing a literal `/`, `?` or `#`, Basic values over 2048 chars, digitless assignment values, dotted passwords whose every segment starts with a letter, and unquoted passwords containing `,` or `;`. False-positive budget: `scripts/secret-scan-fp-budget.ts` diffs findings against a base ref's scanner (`--base`, default `origin/master`) (`--dir <brain>` reports per-pattern counts only, never content) and `test/secret-scan-fp-budget.test.ts` pins a synthetic brain-like fixture (zero findings on its non-secret part in both modes, every planted secret redacted). Shape tests: `test/secret-scan-credential-shapes.test.ts`, `test/secret-scan-private-key-spans.test.ts`. Perf pins: `test/secret-scan-perf.test.ts`. Retrieval output is another consumer of its canonical scanner; that output boundary does not sanitize internal evaluation/provider inputs.
- `src/core/secret-scan-stream.ts` — `scanTextStream(chunks, opts)`, the windowed scan of a byte stream too large to hold as one string (the push gate's staged blobs over 25 MiB). Bytes are decoded by a streaming `TextDecoder` (`ignoreBOM`, so a split multi-byte character joins and invalid bytes become U+FFFD as `Buffer.toString` makes them) and handed to `scanTextSpans` (`secret-scan.ts`: `scanText` plus each claim's absolute span, allowlisted claims kept with `finding: null`) in windows of at most `STREAM_SCAN_WINDOW_CHARS` (8 Mi UTF-16 units). Each next window starts at a line start at least `STREAM_SCAN_OVERLAP_CHARS` (64 Ki, above the longest claim: a private key's 16384-char body plus fences, a jwt's 3 × 4096) before the previous end, or mid-line at exactly the overlap (never inside a surrogate pair) when the line is too long. A window keeps the claims that start before the next window's start, the last window keeps the rest, and a claim wholly inside a span an earlier window kept (a seam's partial view, or an exact duplicate) is dropped; lines are 1-based over the whole stream. `streamScanWork` (@internal) counts windows, bytes, scanned chars and the largest window. `test/secret-scan-stream.test.ts` pins equality with `scanText` across both seams of a window, mid-line splits, split multi-byte characters, invalid UTF-8 and a BOM.
- `src/core/context/sensitivity-scan.ts` — composed detector for compiled-context entries (secret-scan → PII → path shapes → operator blocklist → operator pattern file). secret-scan OWNS credential-shaped families it also matches: a PII finding whose value (raw, or with the `Bearer ` keyword stripped) matches a secret finding's fingerprint is not reported twice; short bearer tokens under secret-scan's floor still surface as `pii:bearer`. Every family honors the same `.gbrain-scan-allow` fingerprint escape hatch. `formatSensitivityDrop(drop, workspaceRoot)` renders the one content-free stderr line `gbrain compile-context` prints per dropped entry: reason `sensitivity_scan`, the pattern family, the fingerprint and the absolute `.gbrain-scan-allow` path, never the value (`test/sensitivity-scan.test.ts`).
diff --git a/src/core/secret-scan.ts b/src/core/secret-scan.ts
index 3bd0cb9bc1..df3e61f017 100644
--- a/src/core/secret-scan.ts
+++ b/src/core/secret-scan.ts
@@ -1196,9 +1196,11 @@ export function scanFiles(paths: string[], opts: ScanOpts = {}): SecretFinding[]
// needs ECHO_MIN_CHARS_LABELED = 8 and must not be a stoplisted word, so
// `pwd: 1234` never scrubs every later "1234"), must be at most
// ECHO_MAX_VALUE_CHARS long,
-// and only the first ECHO_MAX_UNIQUE unique values in claim order — ONE cap
-// across every pattern — join. A value past either cap is still redacted at
-// its claimed span; only its echoes are the accepted miss.
+// and only the first ECHO_MAX_UNIQUE unique bearer, assignment and labeled
+// values plus the first ECHO_MAX_UNIQUE_VENDOR unique vendor or jwt values, in
+// claim order, join (two caps, so planted vendor strings never crowd out a
+// bearer token's echo). A value past a cap is still redacted at its claimed
+// span; only its echoes are the accepted miss.
//
// The pass is a streaming merge over the ORIGINAL text: per value, the
// position of its next occurrence at or beyond the emit cursor (found by
@@ -1225,7 +1227,7 @@ export function scanFiles(paths: string[], opts: ScanOpts = {}): SecretFinding[]
// message bodies, speaker labels, title, metadata values) must scrub an echo
// in one field of a value claimed in ANOTHER. `planRedaction` scans one
// field and adds its eligible values to a caller-supplied shared map under
-// the same floors and the same single cap; `applyRedaction` splices later and
+// the same floors and the same caps; `applyRedaction` splices later and
// reads the map as it stands THEN — so a caller plans every field first and
// applies afterwards, and a value claimed in the last field is scrubbed from
// the first. `redactFindings` is the one-shot composition for a single text.
@@ -1260,20 +1262,28 @@ const VENDOR_ECHO_PATTERNS = new Set(CORE_PATTERNS.map((p) => p.name).filter((na
export const ECHO_MAX_VALUE_CHARS = 512;
/**
- * Cap on the unique values (bearer + high_entropy_assignment together, claim
- * order, first N) the echo pass will look for. Keeps the pass a bounded
- * number of linear sweeps; an echo of a value past the cap is the
- * documented, accepted miss.
+ * Cap on the unique bearer, high_entropy_assignment and labeled_credential
+ * values (together, claim order, first N) the echo pass will look for. Keeps
+ * the pass a bounded number of linear sweeps; an echo of a value past the cap
+ * is the documented, accepted miss.
*/
export const ECHO_MAX_UNIQUE = 64;
+/**
+ * Separate cap on the unique vendor-shape and `jwt` values (claim order, first
+ * N) in the echo dictionary. Planted vendor-shaped strings fill only this cap,
+ * never ECHO_MAX_UNIQUE, so they cannot crowd out a later bearer token or
+ * assignment value whose bare echo only the dictionary scrubs (#7978).
+ */
+export const ECHO_MAX_UNIQUE_VENDOR = 32;
+
/** Echo dictionary: claimed value → the pattern that claimed it first. */
export type EchoDictionary = Map<string, string>;
export interface RedactOpts extends ScanOpts {
/**
* Shared echo dictionary. `planRedaction` ADDS the eligible values it
- * claims (shared floors, one cap, claim order); `applyRedaction` reads the
+ * claims (shared floors and caps, claim order); `applyRedaction` reads the
* map at apply time. Pass one map across every field of a document for
* session-wide echo coverage. Default: a fresh map per `planRedaction`.
*/
@@ -1304,8 +1314,11 @@ function addEchoValue(into: EchoDictionary, pattern: string, value: string): voi
: pattern === 'labeled_credential'
? ECHO_MIN_CHARS_LABELED
: VENDOR_ECHO_PATTERNS.has(pattern) ? ECHO_MIN_CHARS_VENDOR : Infinity;
- if (value.length < floor || value.length > ECHO_MAX_VALUE_CHARS) return;
- if (into.has(value) || into.size >= ECHO_MAX_UNIQUE) return;
+ if (value.length < floor || value.length > ECHO_MAX_VALUE_CHARS || into.has(value)) return;
+ const vendor = VENDOR_ECHO_PATTERNS.has(pattern);
+ let used = 0;
+ for (const claimedBy of into.values()) if (VENDOR_ECHO_PATTERNS.has(claimedBy) === vendor) used++;
+ if (used >= (vendor ? ECHO_MAX_UNIQUE_VENDOR : ECHO_MAX_UNIQUE)) return;
into.set(value, pattern);
}
diff --git a/test/secret-scan-escape-boundary-7978.test.ts b/test/secret-scan-escape-boundary-7978.test.ts
index 699eb47004..8ca4633fc2 100644
--- a/test/secret-scan-escape-boundary-7978.test.ts
+++ b/test/secret-scan-escape-boundary-7978.test.ts
@@ -12,7 +12,7 @@
* Every key is assembled at runtime from random parts (gitleaks, CEO-10).
*/
import { describe, expect, test } from 'bun:test';
-import { redactFindings, scanText } from '../src/core/secret-scan.ts';
+import { ECHO_MAX_UNIQUE, ECHO_MAX_UNIQUE_VENDOR, redactFindings, scanText } from '../src/core/secret-scan.ts';
import { PROVIDER_KEY_SHAPES, alnum, hex } from './helpers/provider-keys.ts';
const j = (sep: string, ...parts: string[]) => parts.join(sep);
@@ -142,3 +142,26 @@ describe('#7978 no new false positives', () => {
expect(redactFindings(`a%22${k}%22`).text).toBe('a%22<REDACTED:npm_token>%22');
});
});
+
+describe('#7978 vendor values cannot crowd the echo dictionary', () => {
+ test('64 planted vendor keys ahead of a bearer token and an assignment still leave room for their bare echoes', () => {
+ const planted = Array.from({ length: 64 }, () => `k=${j('_', 'ghp', alnum(36))}`);
+ const tok = alnum(40);
+ const pw = `Xk9${alnum(16)}7q`;
+ const text = [...planted, `Authorization: Bearer ${tok}`, `PASSWORD=${pw}`, `your token is ${tok} and the password is ${pw}`].join('\n');
+ const out = redactFindings(text, { highEntropy: true });
+ expect(out.text.includes(tok)).toBe(false);
+ expect(out.text.includes(pw)).toBe(false);
+ expect(out.text.split('\n').at(-1)).toBe('your token is <REDACTED:bearer> and the password is <REDACTED:high_entropy_assignment>');
+ });
+
+ test('vendor values stop joining at ECHO_MAX_UNIQUE_VENDOR; the other families keep all ECHO_MAX_UNIQUE slots', () => {
+ const vendor = Array.from({ length: ECHO_MAX_UNIQUE_VENDOR + 8 }, () => j('_', 'ghp', alnum(36)));
+ const bearer = Array.from({ length: ECHO_MAX_UNIQUE }, () => alnum(40));
+ const text = [...vendor.map((k) => `k=${k}`), ...bearer.map((t) => `Authorization: Bearer ${t}`)].join('\n');
+ const echo = redactFindings(text).echoValues;
+ expect([...echo.values()].filter((p) => p === 'github_token').length).toBe(ECHO_MAX_UNIQUE_VENDOR);
+ expect([...echo.values()].filter((p) => p === 'bearer').length).toBe(ECHO_MAX_UNIQUE);
+ });
+});
+