Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Allow one secret-scan rule per file glob in .gbrain-scan-allow (#9589)

Garry Tana2c5c6cab121
diff --git a/SECURITY.md b/SECURITY.md
index 41cd61c122..500092c894 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -641,7 +641,9 @@ complete. If you find a live credential in a page:
value; rewrite/re-push or regenerate those as needed.
4. Add a value's fingerprint (printed with every `gbrain sources push` finding)
to `<workspace>/.gbrain-scan-allow` ONLY for values you have confirmed are
- not credentials.
+ not credentials. A `path:<glob> pattern:<rule>` line allows one rule in the
+ files the glob matches and keeps every other rule active there; a malformed
+ line or an unknown rule name allows nothing and is reported.
**Say to your agent:** *"a secret leaked into a brain page — rotate it and
purge the page"* (your agent rotates the credential, then runs
diff --git a/behavior-changes.d/scan_allow_rule_entry.md b/behavior-changes.d/scan_allow_rule_entry.md
new file mode 100644
index 0000000000..1f538d9741
--- /dev/null
+++ b/behavior-changes.d/scan_allow_rule_entry.md
@@ -0,0 +1,4 @@
+---
+kind: add
+---
+A .gbrain-scan-allow line of the form path:<glob> pattern:<rule> now allows only that secret-scan rule in the files the glob matches, so a rule that keeps flagging harmless values (for example jwt on newsletter unsubscribe links) can be quieted without turning off every other rule for those files. Existing fingerprint and path-glob lines work as before. A line that starts with pattern:, or with path: and contains a space, is read only as a rule entry: if it is malformed or names an unknown rule it allows nothing, and the push refusal, gbrain git scan and bootstrap verify name it. A bare path: line with no space is still a path glob, and gbrain git scan now warns about it.
diff --git a/changelog.d/gbra587-scan-allow-rule.md b/changelog.d/gbra587-scan-allow-rule.md
new file mode 100644
index 0000000000..dc165539e4
--- /dev/null
+++ b/changelog.d/gbra587-scan-allow-rule.md
@@ -0,0 +1,6 @@
+**A `.gbrain-scan-allow` line can now allow one secret-scan rule in matching files, instead of skipping every rule there.**
+
+<!-- behavior-change: scan_allow_rule_entry -->
+
+### Added
+- `.gbrain-scan-allow` accepts `path:<glob> pattern:<rule>`, which allows only the named rule (for example `jwt` on newsletter unsubscribe links) in the files the glob matches; every other rule still blocks a push there. Path globs and fingerprints keep their meaning. A malformed rule line or an unknown rule name allows nothing and is named by the push refusal, `gbrain git scan` and bootstrap verify, and `gbrain git scan` lists each hit a rule line allowed with the line that allowed it (`allowed[]`, `allowlist_problems[]` in `--json`).
diff --git a/docs/architecture/key-files/secret-scan.md b/docs/architecture/key-files/secret-scan.md
index eb759b9d06..6619007527 100644
--- a/docs/architecture/key-files/secret-scan.md
+++ b/docs/architecture/key-files/secret-scan.md
@@ -6,7 +6,7 @@ The format-based secret detector shared by `sources push`, bootstrap verify, the
compiled-context sensitivity scan and the transcripts redactor, and the composed
sensitivity detector built on it.
-- `src/core/secret-scan.ts` — pattern-based secret detection shared by `sources push`, bootstrap verify, the compiled-context sensitivity scan and the transcripts redactor. Detection is FORMAT-based, not prefix-only: vendor shapes (anthropic, openai, voyage, github, gitlab, slack, aws AKIA/ASIA, google, stripe, sendgrid, twilio, supabase sb_secret_/sbp_, npm, huggingface, `digitalocean` `do[opr]_v1_` + 64 hex, the AI-provider keys `openrouter` `sk-or-v1-` + 64 lowercase hex, `groq` `gsk_` + 52, `xai` `xai-` + 80, `perplexity` `pplx-` + 48 and `replicate` `r8_` + 37, all fixed-length with a hard right edge, gbrain), JWTs, private keys (whole-text claims, below), plus four catch-alls appended LAST (`bearer` — prebuilt two-group layout so only the token is the value, which may hold a JSON-escaped `\/` (#8266); `db_url_credentials` — the `scheme://user:pass@` span of database URLs, host/db kept; `url_credentials` — the same span for `http(s)` (scheme case-insensitive, `gi`), user 0–128 stopping at `:`/`/`, password 1–256 excluding whitespace, `@`, `"`, `'`, `/`, `?`, `#`, with a `validate` hook that rejects placeholder passwords (`<...>`, `${...}`, `$VAR`, all `*`, all `x`); `basic_auth` — two prebuilt case-insensitive forms, `Authorization: Basic` with a 4–2048 char value and bare `Basic` with 16–2048, both with a hard right edge and a `validate` that requires the value to base64-decode to printable ASCII containing `:`; it is NOT in the bearer family, so it joins neither the echo dictionary nor `secret:bearer`). The pattern contract: `CorePattern` carries optional `flags` (default `'g'`), `precheck`, `validate(value)` (O(value), run on the whole match before the claim and the union split; a rejected match is not re-tried from inside) and `since` (the gbrain version that added or last changed the rule; `SECURITY_WAVE_SINCE` = `0.60.31.0` on `digitalocean`, `url_credentials`, `basic_auth` and `private_key_pem`, `PROVIDER_KEYS_SINCE` on the five AI-provider rules (#7959), kept in step with the release that ships a rule change). Compiled patterns keep group 1 = anchor, group 2 = value. Every core rule and both `high_entropy_assignment` forms also have an escape-boundary twin (#7978), compiled right after the rule under the same name and anchored on a JSON escape (`\n`, `\r`, `\t`, `\b`, `\f`, `\uXXXX`, and since #8266 `\xNN`, `\UXXXXXXXX`), a percent-escape (`%XX`), a quoted-printable `=3D` or a terminal color code (`ESC[…m`, raw or written `\u001b[`/`\x1b[`/`\033[`/`\e[`), whose last character the default boundary reads as a word character; a twin runs only on a line `ESCAPE_IN_LINE_RE` matches, so other lines run the old list, and a prebuilt rule carries its twin as `CorePattern.escaped`. `patternSince(pattern)` reads the map; every finding of such a pattern carries `SecretFinding.since`, which the push and bootstrap refusals render. Fixed-length shapes carry a trailing negative lookahead. Per-line claimed-span dedupe is first-wins for ATTRIBUTION (order matters: `Bearer <vendor key>` keeps the vendor name, a fully covered later span is skipped) but coverage is the UNION: every unclaimed slice of a later overlapping span is claimed as a hit of the later pattern (own value, own fingerprint), so a database URL whose username is a vendor-shaped id still loses its password. The transcript-only `labeled_credential` rule (`ScanOpts.labeledCredentials`) lives in `secret-scan-labeled.ts` (Security). The opt-in `high_entropy_assignment` rule (`ScanOpts.highEntropy`; its shapes and gates live in `src/core/secret-scan-assignment.ts`, split out at the 1500-line module cap) is anchored on a credential keyword or on an identifier whose last word is `key` (`STRIPE_KEY=`, never `MONKEY=` or a bare `KEY=`) and requires a 12+ char value that contains a digit and a non-digit AND clears 3.5 bits/char on the bias-corrected (Miller-Madow) estimate `correctedEntropy`, since raw per-character entropy is capped at log2(n) and rejected most random 12-16 char secrets. It is two compiled patterns sharing the name: QUOTED (after `"` or `'`, any run of non-quote, non-whitespace characters) and UNQUOTED (the legacy `[A-Za-z0-9+/_=-]` class plus `!#$%*@^~.?<>`; never `&`, `()[]{}`, `:`, `,` or `;`, so query strings, calls and SQL `SET` lists stay intact, and a password carrying `,`/`;` matches only quoted; the last character excludes `.?!`, so a sentence's full stop is not claimed). Its `validate` rejects values that start with `scheme://`, `./`, `~/`, `${`, `$(` or `<REDACTED:`, dotted identifier paths (`process.env.X`) and `/`-paths holding a character outside the legacy class. Consumers with it on: retrieval output, transcript ingest, hooks and the hook heartbeat, the atoms drain, `embed-facts.ts` error text and the context-engine relay re-scan. Off: the push gate, bootstrap verify/repo and the compiled-context sensitivity scan. Own minimal allowlist + `<ws>/.gbrain-scan-allow` per-finding overrides — deliberately NOT the repo's `.gitleaks.toml` (a public-repo CI fixture policy); the allowlist block (`loadWorkspaceAllowlist`, the glob dialect `globToRegExp`/`matchesGlob`, `valueAllowlisted`/`pathAllowlisted`, `SCAN_ALLOW_FILENAME`, the 16-hex fingerprint floor) lives in `src/core/secret-scan-allowlist.ts` since wave 16 (G4.1, the 1500-line module cap) and is re-exported here unchanged (`test/secret-scan-allowlist.test.ts`). `correctedEntropy` is one histogram pass (distinct count = histogram size; byte-identical to the former `shannonEntropy + Set` form, `test/secret-scan-entropy-equivalence.test.ts`; ~80 → ~25 ms per 250 × 4096-char segment). `ScanOpts.work` / `RedactOpts.work` takes a `ScanWork` accumulator (`src/core/secret-scan-work.ts`: lines, regex execs, candidates per pattern, chars handed to the entropy gate counted inside `correctedEntropy`, validate calls, claims, union splits, echo sweeps) pinned exactly on the perf fixture by `test/secret-scan-work-counters.test.ts`; the perf file's absolute bounds stay the only guard on per-character cost. Deliberately NOT patterns: `sb_publishable_`, Supabase project refs (identifiers, not credentials). Never spell a literal database or `http(s)` URL with userinfo, or any credential, in this file's comments or in fixtures (assemble test values at runtime; `scripts/check-pg-url-redaction.sh` greps `src/` for every `db_url_credentials` scheme). Long lines are scanned in full (no length cap — a secret on a 1 MB minified line must still redact); what keeps that linear is bounded quantifiers on every shape whose value class could otherwise run to end-of-line — `db_url_credentials` (user ≤128 stopping at `:`/`/`; password 1–256 stopping only at whitespace, `@` and the string delimiters `"`/`'`, so unescaped `/ { } < > | ^ \` and backtick passwords redact), `jwt` (prebuilt so its boundary excludes `-` as well as `_`; segments ≤4096), `high_entropy_assignment` (≤64 identifier chars after the keyword, value ≤4096), `url_credentials` (user ≤128, password ≤256), `basic_auth` (value ≤2048, so its decode is constant work) and the private-key walks (below) — plus a per-pattern `precheck` substring gate on the catch-alls (`@` for both URL patterns; `earer`/`EARER`, since an all-caps `BEARER` scheme is accepted; a case-insensitive `basic`), a carried next-newline position in `scanPemBlocks` (sticky -1, so the header-line count is O(text) even when thousands of hits sit ahead of a newline-free multi-megabyte tail), a lazily-allocated per-line claimed-char bitmap behind the first-wins dedupe (O(value) per hit), and a windowed `buildPreview` (whole line ≤160 chars; otherwise hit−40…+80, edges snapped outward to whole spans, every claimed span in the window spliced by position, so a preview never carries a sibling secret or a fragment). `redactFindings` (= `planRedaction` + `applyRedaction`) rebuilds the text by SPAN-SPLICE in one O(text + hits) pass over the claimed spans, and fills the UNCLAIMED gaps with a bounded echo pass: the values claimed by `bearer`, `high_entropy_assignment`, `labeled_credential` and every vendor or `jwt` rule (#7978; not `basic_auth` or the two URL userinfo catch-alls) form an echo dictionary (value → pattern; floors `ECHO_MIN_CHARS_BEARER` = `ECHO_MIN_CHARS_VENDOR` = 20 / `ECHO_MIN_CHARS_ENTROPY` = 12, values ≤ `ECHO_MAX_VALUE_CHARS` = 512, the first `ECHO_MAX_UNIQUE` = 64 unique bearer, assignment and labeled values plus, under their own cap, the first `ECHO_MAX_UNIQUE_VENDOR` = 32 unique vendor or `jwt` values, in claim order, so planted vendor strings never crowd out a bearer token's echo — a value past a cap is redacted at its claim, its echoes are the accepted miss), matched by a streaming merge over the ORIGINAL text — per value, the next occurrence at or beyond the emit cursor (`indexOf` resuming at the cursor, re-sought only once the cursor passes it), leftmost-longest at each step, emitted incrementally into the unclaimed gaps — so retained state is O(values) however long the text is (occurrences are never collected into a list) (never an alternation RegExp, never over emitted tokens). Echoes add no findings. `planRedaction` feeds a caller-supplied shared map (`RedactOpts.echoValues`) and `applyRedaction` reads it at apply time, so a multi-field document scrubs an echo in one field of a value claimed in another. `basic_auth` and the URL userinfo catch-alls redact exactly what `scanText` reports; a vendor value embedded past its boundary (`token_<key>`) is scrubbed as an echo when the same value is claimed elsewhere in the document, and left as-is otherwise. PRIVATE KEYS: `pemClaims` finds `BEGIN`/`END … PRIVATE KEY` fences in two global passes and merges them in text order. BLOCK: a BEGIN whose next fence is an END within `PEM_BODY_MAX_CHARS` (16384) is claimed header through END; a block never swallows a later BEGIN. FORWARD (BEGIN with no END in view): the header plus the body lines that follow; a header followed by no body line (a marker quoted in prose) is claimed alone. BACKWARD (END with no BEGIN in view): the body lines before the fence plus the fence; an END with no body line before it is no claim. A body line is a whole-line base64 run (`A-Za-z0-9+/=`) indented by up to 16 spaces/tabs; line ends are `\n`, `\r\n` or the literal JSON escapes; interior lines are 16–128 chars, the line next to a fence and a partial line at a cut 1–128; optional RFC 1421 `Proc-Type:`/`DEK-Info:` lines and one blank line may follow BEGIN; a line with anything else, or a run over 128 chars, ends the body. Total claimed body stays within `PEM_BODY_MAX_CHARS`. Each walk stops at the neighbouring fence or the previous claim, so the pass is linear (20k unterminated BEGINs, 20k END-only fences and 1 MB of base64 lines are perf-pinned). Per-line hits that fall wholly inside a key claim are dropped (one key, one finding). A FORWARD claim's finding also carries `legacyFingerprint`, the header-only fingerprint earlier releases reported, so `workspace-push.ts` can name a stale allowlist entry; a truncated key's fingerprint covers its body and therefore depends on where the text was cut, so it is not a stable allowlist key. `privateKeySpans(text)` exports the same claims as sorted, disjoint `{start, end}` spans. Accepted misses: body on the header's own line, space-joined single-line keys, blockquoted (`> `) keys, base64 runs over 128 chars, body past 16384 chars, `http(s)` passwords containing a literal `/`, `?` or `#`, Basic values over 2048 chars, digitless assignment values, dotted passwords whose every segment starts with a letter, and unquoted passwords containing `,` or `;`. False-positive budget: `scripts/secret-scan-fp-budget.ts` diffs findings against a base ref's scanner (`--base`, default `origin/master`) (`--dir <brain>` reports per-pattern counts only, never content) and `test/secret-scan-fp-budget.test.ts` pins a synthetic brain-like fixture (zero findings on its non-secret part in both modes, every planted secret redacted). Shape tests: `test/secret-scan-credential-shapes.test.ts`, `test/secret-scan-private-key-spans.test.ts`. Perf pins: `test/secret-scan-perf.test.ts`. Retrieval output is another consumer of its canonical scanner; that output boundary does not sanitize internal evaluation/provider inputs.
+- `src/core/secret-scan.ts` — pattern-based secret detection shared by `sources push`, bootstrap verify, the compiled-context sensitivity scan and the transcripts redactor. Detection is FORMAT-based, not prefix-only: vendor shapes (anthropic, openai, voyage, github, gitlab, slack, aws AKIA/ASIA, google, stripe, sendgrid, twilio, supabase sb_secret_/sbp_, npm, huggingface, `digitalocean` `do[opr]_v1_` + 64 hex, the AI-provider keys `openrouter` `sk-or-v1-` + 64 lowercase hex, `groq` `gsk_` + 52, `xai` `xai-` + 80, `perplexity` `pplx-` + 48 and `replicate` `r8_` + 37, all fixed-length with a hard right edge, gbrain), JWTs, private keys (whole-text claims, below), plus four catch-alls appended LAST (`bearer` — prebuilt two-group layout so only the token is the value, which may hold a JSON-escaped `\/` (#8266); `db_url_credentials` — the `scheme://user:pass@` span of database URLs, host/db kept; `url_credentials` — the same span for `http(s)` (scheme case-insensitive, `gi`), user 0–128 stopping at `:`/`/`, password 1–256 excluding whitespace, `@`, `"`, `'`, `/`, `?`, `#`, with a `validate` hook that rejects placeholder passwords (`<...>`, `${...}`, `$VAR`, all `*`, all `x`); `basic_auth` — two prebuilt case-insensitive forms, `Authorization: Basic` with a 4–2048 char value and bare `Basic` with 16–2048, both with a hard right edge and a `validate` that requires the value to base64-decode to printable ASCII containing `:`; it is NOT in the bearer family, so it joins neither the echo dictionary nor `secret:bearer`). The pattern contract: `CorePattern` carries optional `flags` (default `'g'`), `precheck`, `validate(value)` (O(value), run on the whole match before the claim and the union split; a rejected match is not re-tried from inside) and `since` (the gbrain version that added or last changed the rule; `SECURITY_WAVE_SINCE` = `0.60.31.0` on `digitalocean`, `url_credentials`, `basic_auth` and `private_key_pem`, `PROVIDER_KEYS_SINCE` on the five AI-provider rules (#7959), kept in step with the release that ships a rule change). Compiled patterns keep group 1 = anchor, group 2 = value. Every core rule and both `high_entropy_assignment` forms also have an escape-boundary twin (#7978), compiled right after the rule under the same name and anchored on a JSON escape (`\n`, `\r`, `\t`, `\b`, `\f`, `\uXXXX`, and since #8266 `\xNN`, `\UXXXXXXXX`), a percent-escape (`%XX`), a quoted-printable `=3D` or a terminal color code (`ESC[…m`, raw or written `\u001b[`/`\x1b[`/`\033[`/`\e[`), whose last character the default boundary reads as a word character; a twin runs only on a line `ESCAPE_IN_LINE_RE` matches, so other lines run the old list, and a prebuilt rule carries its twin as `CorePattern.escaped`. `patternSince(pattern)` reads the map; every finding of such a pattern carries `SecretFinding.since`, which the push and bootstrap refusals render. Fixed-length shapes carry a trailing negative lookahead. Per-line claimed-span dedupe is first-wins for ATTRIBUTION (order matters: `Bearer <vendor key>` keeps the vendor name, a fully covered later span is skipped) but coverage is the UNION: every unclaimed slice of a later overlapping span is claimed as a hit of the later pattern (own value, own fingerprint), so a database URL whose username is a vendor-shaped id still loses its password. The transcript-only `labeled_credential` rule (`ScanOpts.labeledCredentials`) lives in `secret-scan-labeled.ts` (Security). The opt-in `high_entropy_assignment` rule (`ScanOpts.highEntropy`; its shapes and gates live in `src/core/secret-scan-assignment.ts`, split out at the 1500-line module cap) is anchored on a credential keyword or on an identifier whose last word is `key` (`STRIPE_KEY=`, never `MONKEY=` or a bare `KEY=`) and requires a 12+ char value that contains a digit and a non-digit AND clears 3.5 bits/char on the bias-corrected (Miller-Madow) estimate `correctedEntropy`, since raw per-character entropy is capped at log2(n) and rejected most random 12-16 char secrets. It is two compiled patterns sharing the name: QUOTED (after `"` or `'`, any run of non-quote, non-whitespace characters) and UNQUOTED (the legacy `[A-Za-z0-9+/_=-]` class plus `!#$%*@^~.?<>`; never `&`, `()[]{}`, `:`, `,` or `;`, so query strings, calls and SQL `SET` lists stay intact, and a password carrying `,`/`;` matches only quoted; the last character excludes `.?!`, so a sentence's full stop is not claimed). Its `validate` rejects values that start with `scheme://`, `./`, `~/`, `${`, `$(` or `<REDACTED:`, dotted identifier paths (`process.env.X`) and `/`-paths holding a character outside the legacy class. Consumers with it on: retrieval output, transcript ingest, hooks and the hook heartbeat, the atoms drain, `embed-facts.ts` error text and the context-engine relay re-scan. Off: the push gate, bootstrap verify/repo and the compiled-context sensitivity scan. Own minimal allowlist + `<ws>/.gbrain-scan-allow` per-finding overrides — deliberately NOT the repo's `.gitleaks.toml` (a public-repo CI fixture policy); the allowlist block (`loadWorkspaceAllowlist`, the glob dialect `globToRegExp`/`matchesGlob`, `valueAllowlisted`/`pathAllowlisted`, `SCAN_ALLOW_FILENAME`, the 16-hex fingerprint floor, and rule entries `path:<glob> pattern:<rule>`: `ruleAllowEntry`/`withoutRuleAllowed` drop only that rule's findings in matching files and report the entry, `allowlistProblems` names malformed rule entries and unknown rule names, which allow nothing, and a bare `path:` glob; `pathAllowlisted` never reads a rule entry as a glob; `scanFiles` (`ScanOpts.onAllowed`), the range scan (`RangeScan.allowed`/`allowlist_problems`, `scanAllowProblems`), the push scan and bootstrap repo/verify apply and report them, `test/secret-scan-rule-allow-2901.test.ts`) lives in `src/core/secret-scan-allowlist.ts` since wave 16 (G4.1, the 1500-line module cap) and is re-exported here unchanged (`test/secret-scan-allowlist.test.ts`). `correctedEntropy` is one histogram pass (distinct count = histogram size; byte-identical to the former `shannonEntropy + Set` form, `test/secret-scan-entropy-equivalence.test.ts`; ~80 → ~25 ms per 250 × 4096-char segment). `ScanOpts.work` / `RedactOpts.work` takes a `ScanWork` accumulator (`src/core/secret-scan-work.ts`: lines, regex execs, candidates per pattern, chars handed to the entropy gate counted inside `correctedEntropy`, validate calls, claims, union splits, echo sweeps) pinned exactly on the perf fixture by `test/secret-scan-work-counters.test.ts`; the perf file's absolute bounds stay the only guard on per-character cost. Deliberately NOT patterns: `sb_publishable_`, Supabase project refs (identifiers, not credentials). Never spell a literal database or `http(s)` URL with userinfo, or any credential, in this file's comments or in fixtures (assemble test values at runtime; `scripts/check-pg-url-redaction.sh` greps `src/` for every `db_url_credentials` scheme). Long lines are scanned in full (no length cap — a secret on a 1 MB minified line must still redact); what keeps that linear is bounded quantifiers on every shape whose value class could otherwise run to end-of-line — `db_url_credentials` (user ≤128 stopping at `:`/`/`; password 1–256 stopping only at whitespace, `@` and the string delimiters `"`/`'`, so unescaped `/ { } < > | ^ \` and backtick passwords redact), `jwt` (prebuilt so its boundary excludes `-` as well as `_`; segments ≤4096), `high_entropy_assignment` (≤64 identifier chars after the keyword, value ≤4096), `url_credentials` (user ≤128, password ≤256), `basic_auth` (value ≤2048, so its decode is constant work) and the private-key walks (below) — plus a per-pattern `precheck` substring gate on the catch-alls (`@` for both URL patterns; `earer`/`EARER`, since an all-caps `BEARER` scheme is accepted; a case-insensitive `basic`), a carried next-newline position in `scanPemBlocks` (sticky -1, so the header-line count is O(text) even when thousands of hits sit ahead of a newline-free multi-megabyte tail), a lazily-allocated per-line claimed-char bitmap behind the first-wins dedupe (O(value) per hit), and a windowed `buildPreview` (whole line ≤160 chars; otherwise hit−40…+80, edges snapped outward to whole spans, every claimed span in the window spliced by position, so a preview never carries a sibling secret or a fragment). `redactFindings` (= `planRedaction` + `applyRedaction`) rebuilds the text by SPAN-SPLICE in one O(text + hits) pass over the claimed spans, and fills the UNCLAIMED gaps with a bounded echo pass: the values claimed by `bearer`, `high_entropy_assignment`, `labeled_credential` and every vendor or `jwt` rule (#7978; not `basic_auth` or the two URL userinfo catch-alls) form an echo dictionary (value → pattern; floors `ECHO_MIN_CHARS_BEARER` = `ECHO_MIN_CHARS_VENDOR` = 20 / `ECHO_MIN_CHARS_ENTROPY` = 12, values ≤ `ECHO_MAX_VALUE_CHARS` = 512, the first `ECHO_MAX_UNIQUE` = 64 unique bearer, assignment and labeled values plus, under their own cap, the first `ECHO_MAX_UNIQUE_VENDOR` = 32 unique vendor or `jwt` values, in claim order, so planted vendor strings never crowd out a bearer token's echo — a value past a cap is redacted at its claim, its echoes are the accepted miss), matched by a streaming merge over the ORIGINAL text — per value, the next occurrence at or beyond the emit cursor (`indexOf` resuming at the cursor, re-sought only once the cursor passes it), leftmost-longest at each step, emitted incrementally into the unclaimed gaps — so retained state is O(values) however long the text is (occurrences are never collected into a list) (never an alternation RegExp, never over emitted tokens). Echoes add no findings. `planRedaction` feeds a caller-supplied shared map (`RedactOpts.echoValues`) and `applyRedaction` reads it at apply time, so a multi-field document scrubs an echo in one field of a value claimed in another. `basic_auth` and the URL userinfo catch-alls redact exactly what `scanText` reports; a vendor value embedded past its boundary (`token_<key>`) is scrubbed as an echo when the same value is claimed elsewhere in the document, and left as-is otherwise. PRIVATE KEYS: `pemClaims` finds `BEGIN`/`END … PRIVATE KEY` fences in two global passes and merges them in text order. BLOCK: a BEGIN whose next fence is an END within `PEM_BODY_MAX_CHARS` (16384) is claimed header through END; a block never swallows a later BEGIN. FORWARD (BEGIN with no END in view): the header plus the body lines that follow; a header followed by no body line (a marker quoted in prose) is claimed alone. BACKWARD (END with no BEGIN in view): the body lines before the fence plus the fence; an END with no body line before it is no claim. A body line is a whole-line base64 run (`A-Za-z0-9+/=`) indented by up to 16 spaces/tabs; line ends are `\n`, `\r\n` or the literal JSON escapes; interior lines are 16–128 chars, the line next to a fence and a partial line at a cut 1–128; optional RFC 1421 `Proc-Type:`/`DEK-Info:` lines and one blank line may follow BEGIN; a line with anything else, or a run over 128 chars, ends the body. Total claimed body stays within `PEM_BODY_MAX_CHARS`. Each walk stops at the neighbouring fence or the previous claim, so the pass is linear (20k unterminated BEGINs, 20k END-only fences and 1 MB of base64 lines are perf-pinned). Per-line hits that fall wholly inside a key claim are dropped (one key, one finding). A FORWARD claim's finding also carries `legacyFingerprint`, the header-only fingerprint earlier releases reported, so `workspace-push.ts` can name a stale allowlist entry; a truncated key's fingerprint covers its body and therefore depends on where the text was cut, so it is not a stable allowlist key. `privateKeySpans(text)` exports the same claims as sorted, disjoint `{start, end}` spans. Accepted misses: body on the header's own line, space-joined single-line keys, blockquoted (`> `) keys, base64 runs over 128 chars, body past 16384 chars, `http(s)` passwords containing a literal `/`, `?` or `#`, Basic values over 2048 chars, digitless assignment values, dotted passwords whose every segment starts with a letter, and unquoted passwords containing `,` or `;`. False-positive budget: `scripts/secret-scan-fp-budget.ts` diffs findings against a base ref's scanner (`--base`, default `origin/master`) (`--dir <brain>` reports per-pattern counts only, never content) and `test/secret-scan-fp-budget.test.ts` pins a synthetic brain-like fixture (zero findings on its non-secret part in both modes, every planted secret redacted). Shape tests: `test/secret-scan-credential-shapes.test.ts`, `test/secret-scan-private-key-spans.test.ts`. Perf pins: `test/secret-scan-perf.test.ts`. Retrieval output is another consumer of its canonical scanner; that output boundary does not sanitize internal evaluation/provider inputs.
- `src/core/secret-scan-cookie.ts` — the transcript-lane `cookie_header` rule (GST-2768), run by `scanInternal` only under `ScanOpts.transcriptRules` (`TRANSCRIPT_LANE_SCAN_OPTS` in `secret-scan.ts`: redactSession, the recent-transcripts read and `transcripts audit-secrets`). `cookieHeaderValues(line)` reads the `name=value` pairs after a `Cookie:` / `Set-Cookie:` header (any case, quoted or JSON-escaped) or a curl `-b`/`--cookie` argument and returns the values only (8+ chars, no placeholder or mask); `Set-Cookie` stops after its first pair, and parsing stops at the first segment that is not `name=value`. A quoted value may be JSON-escaped (`sid=\"…\"`, up to three backslashes before each quote), and `curlContinues` carries a curl command across lines ending in `\`, so a `-b` alone on a continued line (Chrome's "Copy as cURL") is curl's (`scanInternal` passes `inCurl`). Values join the echo dictionary at 12 chars under their own cap, `ECHO_MAX_UNIQUE_COOKIE` = 64, apart from `ECHO_MAX_UNIQUE`. `test/secret-scan-push-gate-unchanged.test.ts` pins every non-transcript option set byte-identical on the synthetic corpus `test/fixtures/transcripts/synthetic-credentials.ts`.
- `src/core/secret-scan-transcript.ts` — `compileTranscriptPatterns()`, the transcript-lane-only labeled rules (`TRANSCRIPT_LABELED_PATTERNS` from `secret-scan-labeled.ts`: the `.netrc` forms and the `key`/`netrc` continuations) compiled as `labeled_credential` for `buildPatterns` in `secret-scan.ts` under `ScanOpts.transcriptRules`, after every other rule. Moved out of `secret-scan.ts` at its 1500-line cap with no logic change.
- `src/core/secret-scan-stream.ts` — `scanTextStream(chunks, opts)`, the windowed scan of a byte stream too large to hold as one string (the push gate's staged blobs over 25 MiB). Bytes are decoded by a streaming `TextDecoder` (`ignoreBOM`, so a split multi-byte character joins and invalid bytes become U+FFFD as `Buffer.toString` makes them) and handed to `scanTextSpans` (`secret-scan.ts`: `scanText` plus each claim's absolute span, allowlisted claims kept with `finding: null`) in windows of at most `STREAM_SCAN_WINDOW_CHARS` (8 Mi UTF-16 units). Each next window starts at a line start at least `STREAM_SCAN_OVERLAP_CHARS` (64 Ki, above the longest claim: a private key's 16384-char body plus fences, a jwt's 3 × 4096) before the previous end, or mid-line at exactly the overlap (never inside a surrogate pair) when the line is too long. A window keeps the claims that start before the next window's start, the last window keeps the rest, and a claim wholly inside a span an earlier window kept (a seam's partial view, or an exact duplicate) is dropped; lines are 1-based over the whole stream. `streamScanWork` (@internal) counts windows, bytes, scanned chars and the largest window. `test/secret-scan-stream.test.ts` pins equality with `scanText` across both seams of a window, mid-line splits, split multi-byte characters, invalid UTF-8 and a BOM.
diff --git a/docs/guides/shared-checkout-git.md b/docs/guides/shared-checkout-git.md
index 71b2a3c15e..a5369dc5e7 100644
--- a/docs/guides/shared-checkout-git.md
+++ b/docs/guides/shared-checkout-git.md
@@ -188,7 +188,12 @@ mode scans either way.
A finding that is not a secret: add its fingerprint (`sha256:` and 16 hex
characters, as the preview prints it) as a line of `.gbrain-scan-allow` at the
-checkout root (`#` starts a comment). The scan reads the file from the working
+checkout root (`#` starts a comment). A rule that fires on many harmless values
+in one kind of file takes a rule entry instead, `path:<glob> pattern:<rule>`,
+which allows only that rule in the matching files; the preview lists each hit it
+allowed with the line that allowed it, and names any line that allows nothing
+because it is malformed or names an unknown rule
+([details](write-refusals.md#allowlisting-a-reviewed-false-positive)). The scan reads the file from the working
tree, so the next push uses it without a commit. That is deliberate: the file is
the local user's answer to the scan, and writing it takes local write access to
the checkout, which could push directly anyway. In a managed root below the
diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.md
index 04154dee52..da27bcfabc 100644
--- a/docs/guides/write-refusals.md
+++ b/docs/guides/write-refusals.md
@@ -943,14 +943,32 @@ real credential publishes it.
If it is not (a sample value in documentation, a test fixture), run the
"allow this finding" command from the refusal, review the change to
`.gbrain-scan-allow`, then run the retry command. The file holds one entry
-per line: a fingerprint (`sha256:` and at least 16 hex characters) or a path
+per line: a fingerprint (`sha256:` and at least 16 hex characters), a path
glob (a glob without `/` matches a file name at any depth; one with `/` is
-anchored at the repository root), with `#` comments. Commit the file so
-other machines push the same tree.
+anchored at the repository root) or a rule entry (below), with `#` comments.
+Commit the file so other machines push the same tree.
Prefer a fingerprint over a glob. A glob skips the scan for every file it
matches, including credentials added later.
+When one rule keeps firing on many harmless values in the same kind of file
+(for example `jwt` on the unsubscribe links in saved newsletter HTML), a rule
+entry allows that rule alone in the files a glob matches:
+
+```
+path:mail/*.html pattern:jwt
+```
+
+Every other rule still runs on those files, so a cloud key pasted into one of
+them is still refused. Write one rule per line; the glob is the same as a path
+glob's, and the rule name is the one the refusal prints in brackets. A line that
+starts with `pattern:`, or with `path:` and has a space in it, is read only as a
+rule entry: if it is malformed (two rules, a missing glob) or names a rule that
+does not exist, it allows nothing, and the refusal, `gbrain git scan` and
+bootstrap verify name the line and what is wrong with it. `gbrain git scan`
+also lists each hit a rule entry allowed and the line that allowed it
+(`allowed[]` in `--json`), and `gbrain sources push` logs them.
+
### A stale allowlist entry for a private key
The fingerprint of a private key whose `END` line is missing (a cut-off
diff --git a/src/commands/git.ts b/src/commands/git.ts
index 8694cf4a53..21a197177e 100644
--- a/src/commands/git.ts
+++ b/src/commands/git.ts
@@ -217,9 +217,11 @@ function runScan(p: Parsed): number {
const commits = Number(execFileSync('git', ['-C', repo, 'rev-list', '--count', `${from}..${to}`], { env: gitChildEnv({ ...GIT_ENV }) }).toString().trim());
const seconds = Math.round((Date.now() - started) / 100) / 10;
const items = [...scan.findings.map(f => ` ${f.commit.slice(0, 7)} ${f.path}:${f.line} ${f.pattern} ${f.fingerprint}`),
- ...scan.unscanned.map(u => ` ${u.commit.slice(0, 7)} ${u.path} too large to scan`), ...scan.gate_changes.map(g => ` ${g.commit.slice(0, 7)} ${g.path} changes what the scan sees`)];
+ ...scan.unscanned.map(u => ` ${u.commit.slice(0, 7)} ${u.path} too large to scan`), ...scan.gate_changes.map(g => ` ${g.commit.slice(0, 7)} ${g.path} changes what the scan sees`),
+ ...(scan.allowed ?? []).map(a => ` ${a.commit.slice(0, 7)} ${a.path}:${a.line} ${a.pattern} allowed by .gbrain-scan-allow line: ${a.entry}`),
+ ...(scan.allowlist_problems ?? []).map(p => ` .gbrain-scan-allow line ${p.entry}: ${p.problem}`)];
emit(p.json, { repo, from, to, commits, seconds, would_refuse: refusal?.code ?? null, findings: scan.findings, unscanned: scan.unscanned, gate_changes: scan.gate_changes,
- unreadable: scan.unreadable ?? null }, [`scanned ${commits} commit(s) ${from.slice(0, 12)}..${to.slice(0, 12)} in ${seconds} s: ${refusal ? `a push would refuse (${refusal.code})` : 'a push would not refuse'}`,
+ allowed: scan.allowed ?? [], allowlist_problems: scan.allowlist_problems ?? [], unreadable: scan.unreadable ?? null }, [`scanned ${commits} commit(s) ${from.slice(0, 12)}..${to.slice(0, 12)} in ${seconds} s: ${refusal ? `a push would refuse (${refusal.code})` : 'a push would not refuse'}`,
...(scan.unreadable !== undefined ? [` not read in full: ${scan.unreadable}`] : []), ...items].join('\n'));
return 0;
}
diff --git a/src/core/bootstrap/repo.ts b/src/core/bootstrap/repo.ts
index 960b0ea7fb..af56fdf073 100644
--- a/src/core/bootstrap/repo.ts
+++ b/src/core/bootstrap/repo.ts
@@ -38,7 +38,7 @@ import { invalidateBackupStatus } from '../backup/status-file.ts';
import { realpathOrResolve } from '../path-confine.ts';
import { defaultRunner, isProxyBlocked403, parseGithubOwnerRepo, type ExecRunner } from '../repo-visibility.ts';
import { detectExecutionEnvironment } from '../execution-env.ts';
-import { stagedScanPublishRefusal } from '../git/range-secret-scan.ts';
+import { scanAllowProblems, stagedScanPublishRefusal } from '../git/range-secret-scan.ts';
import type { OperationError } from '../ops/contract.ts';
import { loadWorkspaceAllowlist, scanFiles, SCAN_ALLOW_FILENAME, type SecretFinding } from '../secret-scan.ts';
import { SECRET_SCAN_REFUSAL_DOCS } from '../workspace-push.ts';
@@ -429,7 +429,8 @@ function secretScanOrThrow(workspaceDir: string, relFiles: string[]): void {
`secret scan found ${findings.length} finding(s) before the first push: ${sample}` +
`${findings.length > 5 ? '; …' : ''} — nothing was committed or pushed. ` +
'Remove a real credential first; allowlist only a reviewed false positive by appending its fingerprint to ' +
- `${shellQuote(resolve(workspaceDir, SCAN_ALLOW_FILENAME))} (${SECRET_SCAN_REFUSAL_DOCS}), then re-run \`gbrain bootstrap repo\`.`,
+ `${shellQuote(resolve(workspaceDir, SCAN_ALLOW_FILENAME))} (${SECRET_SCAN_REFUSAL_DOCS}), then re-run \`gbrain bootstrap repo\`.` +
+ scanAllowProblems(allowlist).map((p) => ` ${SCAN_ALLOW_FILENAME} line ${p.entry}: ${p.problem}.`).join(''),
{ details: { findings: findings.length } },
);
}
diff --git a/src/core/bootstrap/verify.ts b/src/core/bootstrap/verify.ts
index 0bfe3143ae..8217a8daac 100644
--- a/src/core/bootstrap/verify.ts
+++ b/src/core/bootstrap/verify.ts
@@ -37,7 +37,7 @@ import { maintenanceAttribution } from '../persistence/attribution.ts';
import { existsSync, mkdirSync, readdirSync, readFileSync, renameSync, rmSync, statSync, writeFileSync } from 'node:fs';
import { createHash, randomUUID } from 'node:crypto';
import { execFileSync } from '../spawn.ts';
-import { join, resolve } from 'node:path';
+import { join, relative, resolve } from 'node:path';
import type { BrainEngine } from '../engine.ts';
import { operations, type Operation, type OperationContext } from '../operations.ts';
@@ -48,6 +48,7 @@ import { realpathOrResolve } from '../path-confine.ts';
import { runMaintenanceSweep } from '../sweep.ts';
import { detectCapabilities, renderCapabilityReport, type CapabilityReport } from '../capability.ts';
import { loadWorkspaceAllowlist, matchesGlob, SCAN_ALLOW_FILENAME, scanFiles, type SecretFinding } from '../secret-scan.ts';
+import { scanAllowProblems } from '../git/range-secret-scan.ts';
import {
PUSH_DENY_GLOBS, SECRET_SCAN_REFUSAL_DOCS, verifyRemotePrivacy, readPushStatuses, summarizePushStatuses,
} from '../workspace-push.ts';
@@ -374,10 +375,13 @@ function checkSecretScan(ws: string): VerifyCheck {
try {
const { files, via } = trackedWorkspaceFiles(ws);
const allowlist = loadWorkspaceAllowlist(ws);
+ const allowed: string[] = [];
const findings: SecretFinding[] = scanFiles(files.map((f) => join(ws, f)), {
allowlist,
workspaceRoot: ws,
+ onAllowed: (hit) => allowed.push(`${relative(ws, hit.file)}:${hit.line} [${hit.pattern}] by ${hit.entry}`),
});
+ const allowedNote = allowed.length ? `; ${allowed.length} hit(s) allowed by ${SCAN_ALLOW_FILENAME} rule lines: ${allowed.slice(0, 5).join('; ')}${allowed.length > 5 ? '; …' : ''}` : '';
if (findings.length > 0) {
const sample = findings.slice(0, 5).map((f) => {
const { since } = f as SecretFinding & { since?: string };
@@ -387,10 +391,11 @@ function checkSecretScan(ws: string): VerifyCheck {
id, ok: false,
detail: `${findings.length} secret-scan finding(s): ${sample}${findings.length > 5 ? '; …' : ''} — ` +
'remove a real credential first; allowlist only a reviewed false positive by appending its fingerprint to ' +
- `${shellQuote(resolve(ws, SCAN_ALLOW_FILENAME))} (${SECRET_SCAN_REFUSAL_DOCS})`,
+ `${shellQuote(resolve(ws, SCAN_ALLOW_FILENAME))} (${SECRET_SCAN_REFUSAL_DOCS})` +
+ scanAllowProblems(allowlist).map((p) => `; ${SCAN_ALLOW_FILENAME} line ${p.entry}: ${p.problem}`).join(''),
};
}
- return { id, ok: true, detail: `no secrets in ${files.length} ${via === 'git' ? 'tracked' : 'workspace'} file(s)` };
+ return { id, ok: true, detail: `no secrets in ${files.length} ${via === 'git' ? 'tracked' : 'workspace'} file(s)${allowedNote}` };
} catch (e) {
return { id, ok: false, detail: `secret scan failed: ${(e as Error).message}` };
}
diff --git a/src/core/git/range-secret-scan.ts b/src/core/git/range-secret-scan.ts
index 671cdaf74a..edee44ecbe 100644
--- a/src/core/git/range-secret-scan.ts
+++ b/src/core/git/range-secret-scan.ts
@@ -29,8 +29,8 @@ import { readLedger } from './commit-ledger.ts';
import { GIT_ENV } from '../git-transport.ts';
import { opError, type OperationError } from '../ops/contract.ts';
import { shellQuote } from '../agent-output.ts';
-import { loadWorkspaceAllowlist, pathAllowlisted, SCAN_ALLOW_FILENAME } from '../secret-scan-allowlist.ts';
-import { scanText, SCAN_MAX_FILE_BYTES } from '../secret-scan.ts';
+import { allowlistProblems, loadWorkspaceAllowlist, pathAllowlisted, SCAN_ALLOW_FILENAME, withoutRuleAllowed, type AllowlistProblem } from '../secret-scan-allowlist.ts';
+import { NAMED_SECRET_RULES, scanText, SCAN_MAX_FILE_BYTES } from '../secret-scan.ts';
import { spawnSync } from '../spawn.ts';
export interface RangeSecretFinding { path: string; commit: string; line: number; pattern: string; fingerprint: string }
@@ -40,6 +40,15 @@ export interface RangeScan {
unreadable?: string;
/** Set with `unreadable` when the reason is the time budget, so a caller with a short budget can scan again elsewhere. */
timed_out?: true;
+ /** Hits a `path:<glob> pattern:<rule>` entry allowed, each with that entry (never the value). */
+ allowed?: Array<{ path: string; commit: string; line: number; pattern: string; entry: string }>;
+ /** `.gbrain-scan-allow` entries that allow nothing, or other than they seem to (scanAllowProblems). */
+ allowlist_problems?: AllowlistProblem[];
+}
+
+/** Malformed rule entries and unknown rule names in a loaded `.gbrain-scan-allow`. */
+export function scanAllowProblems(allowlist: string[]): AllowlistProblem[] {
+ return allowlistProblems(allowlist, [...NAMED_SECRET_RULES, 'high_entropy_assignment', 'labeled_credential'].sort());
}
/** The commit message as `path` in a finding. */
@@ -149,12 +158,18 @@ export function scanRange(root: string, base: string, head: string, timeoutMs =
};
const scan: RangeScan = { findings: [], unscanned: [], gate_changes: [] };
const allowlist = loadWorkspaceAllowlist(root);
+ const problems = scanAllowProblems(allowlist);
+ if (problems.length) scan.allowlist_problems = problems;
let allowFile = SCAN_ALLOW_FILENAME, prefix = '';
const allowed = (path: string) => path.startsWith(prefix) && pathAllowlisted(path.slice(prefix.length), allowlist);
const scanned = (path: string, commit: string, text: string, lineAt: (line: number) => number = line => line) => {
if (Date.now() > deadline) throw new Unreadable('the scan ran past its time budget', true);
if (Buffer.byteLength(text) > SCAN_MAX_FILE_BYTES) { scan.unscanned.push({ path, commit }); return; }
- for (const f of scanText(text, { allowlist })) scan.findings.push({ path, commit, line: lineAt(f.line), pattern: f.pattern, fingerprint: f.fingerprint });
+ const found = scanText(text, { allowlist });
+ const kept = path !== MESSAGE_PATH && path.startsWith(prefix)
+ ? withoutRuleAllowed(found, path.slice(prefix.length), allowlist, hit => (scan.allowed ??= []).push({ path, commit, line: lineAt(hit.line), pattern: hit.pattern, entry: hit.entry }))
+ : found;
+ for (const f of kept) scan.findings.push({ path, commit, line: lineAt(f.line), pattern: f.pattern, fingerprint: f.fingerprint });
};
const dir = mkdtempSync(join(tmpdir(), 'gbrain-range-scan-'));
try {
@@ -384,8 +399,8 @@ export function rangeSecretRefusal(root: string, base: string, target: string, s
...scan.gate_changes.map(g => `${g.path} (changes what the scan sees, ${g.commit.slice(0, 7)})`)];
return opError('publish_secret_found',
`${named.length} item(s) in the local commits ${range} look like secrets or could not be scanned: ${named.slice(0, 10).join(', ')}${named.length > 10 ? ', ...' : ''}. Nothing was pushed.`,
- `Remove the value from those commits (rewriting unpushed local commits is the user's call), or add a false positive's fingerprint to .gbrain-scan-allow, then retry. Never publish them with a command that skips this scan.${base ? '' : ' The remote has no such branch, so this covered the branch\'s whole history, which may include commits published elsewhere before: review it, and push the branch yourself once it is safe.'}${scan.gate_changes.length ? ` A commit that changes .gbrain-scan-allow or a .gitattributes is never published automatically, because it changes what the scan sees: review it and push it yourself (git -C ${root} push), then retry.` : ''}`,
- { detail: JSON.stringify(scan),
+ `Remove the value from those commits (rewriting unpushed local commits is the user's call), or add a false positive's fingerprint to .gbrain-scan-allow (or, for a rule that misfires on a whole kind of file, a path:<glob> pattern:<rule> line), then retry. Never publish them with a command that skips this scan.${scan.allowlist_problems?.length ? ` Check these .gbrain-scan-allow lines: ${scan.allowlist_problems.map(p => `${p.entry} (${p.problem})`).join('; ')}.` : ''}${base ? '' : ' The remote has no such branch, so this covered the branch\'s whole history, which may include commits published elsewhere before: review it, and push the branch yourself once it is safe.'}${scan.gate_changes.length ? ` A commit that changes .gbrain-scan-allow or a .gitattributes is never published automatically, because it changes what the scan sees: review it and push it yourself (git -C ${root} push), then retry.` : ''}`,
+ { detail: JSON.stringify(scan.allowed && scan.allowed.length > 10 ? { ...scan, allowed: scan.allowed.slice(0, 10), allowed_total: scan.allowed.length } : scan),
fix: { ask_user: true, consent: [], actor: 'user', requires_exclusive: false,
argv: ['git', '-C', root, 'log', '--stat', base ? `${base}..${target}` : target],
why: 'Lists the unpublished commits the scan refused; read-only. A secret pushed to the remote stays in its history.',
diff --git a/src/core/secret-scan-allowlist.ts b/src/core/secret-scan-allowlist.ts
index 37d40787bb..8f56f83db0 100644
--- a/src/core/secret-scan-allowlist.ts
+++ b/src/core/secret-scan-allowlist.ts
@@ -112,7 +112,60 @@ export function valueAllowlisted(fullHex: string, allowlist: string[]): boolean
return false;
}
+/**
+ * A rule entry, `path:<glob> pattern:<rule>`, allows one named rule in the files the glob matches (the same glob
+ * dialect as a path entry); every other rule still applies there. One rule per line. A line that starts with
+ * `pattern:`, or with `path:` and holds whitespace, is read only as a rule entry: when it is malformed or names an
+ * unknown rule it allows nothing and `allowlistProblems` reports it.
+ */
+const RULE_ENTRY_RE = /^path:(\S+)\s+pattern:([a-z0-9_]+)$/;
+
+function isRuleEntry(entry: string): boolean {
+ return entry.startsWith('pattern:') || (entry.startsWith('path:') && /\s/.test(entry));
+}
+
/** True when a file path is allowlisted by any glob entry. */
export function pathAllowlisted(relPath: string, allowlist: string[]): boolean {
- return allowlist.some((entry) => !isFingerprintEntry(entry) && matchesGlob(entry, relPath));
+ return allowlist.some((entry) => !isFingerprintEntry(entry) && !isRuleEntry(entry) && matchesGlob(entry, relPath));
+}
+
+/** The rule entry that allows `pattern` in `relPath`, or null. */
+export function ruleAllowEntry(relPath: string, pattern: string, allowlist: string[]): string | null {
+ for (const entry of allowlist) {
+ const m = RULE_ENTRY_RE.exec(entry);
+ if (m && m[2] === pattern && matchesGlob(m[1]!, relPath)) return entry;
+ }
+ return null;
+}
+
+/** A finding a rule entry allowed, with the entry that allowed it; never the matched value. */
+export interface AllowedHit { pattern: string; line: number; fingerprint: string; entry: string }
+
+/** `findings` of the file at `relPath` without those a rule entry allows; each allowed one goes to `onAllowed`. */
+export function withoutRuleAllowed<F extends { pattern: string; line: number; fingerprint: string }>(
+ findings: F[], relPath: string, allowlist: string[], onAllowed?: (hit: AllowedHit) => void,
+): F[] {
+ return findings.filter((f) => {
+ const entry = ruleAllowEntry(relPath, f.pattern, allowlist);
+ if (entry === null) return true;
+ onAllowed?.({ pattern: f.pattern, line: f.line, fingerprint: f.fingerprint, entry });
+ return false;
+ });
+}
+
+export interface AllowlistProblem { entry: string; problem: string }
+
+/** Entries that allow nothing, or something other than they seem to: malformed rule entries and unknown rule names. */
+export function allowlistProblems(allowlist: string[], knownRules: readonly string[]): AllowlistProblem[] {
+ const problems: AllowlistProblem[] = [];
+ for (const entry of allowlist) {
+ if (isRuleEntry(entry)) {
+ const m = RULE_ENTRY_RE.exec(entry);
+ if (!m) problems.push({ entry, problem: 'malformed rule entry, so it allows nothing: write path:<glob> pattern:<rule>, one rule per line' });
+ else if (!knownRules.includes(m[2]!)) problems.push({ entry, problem: `unknown rule ${m[2]}, so it allows nothing; the rules are ${knownRules.join(', ')}` });
+ } else if (entry.startsWith('path:')) {
+ problems.push({ entry, problem: 'read as a path glob for files whose name starts with path:, which skips every rule for them; to allow one rule, add a space and pattern:<rule>' });
+ }
+ }
+ return problems;
}
diff --git a/src/core/secret-scan.ts b/src/core/secret-scan.ts
index f1fb2aabe1..da6890ef57 100644
--- a/src/core/secret-scan.ts
+++ b/src/core/secret-scan.ts
@@ -82,6 +82,8 @@ export interface ScanOpts {
allowlist?: string[];
/** Workspace root: allowlist path-globs match paths relative to this. */
workspaceRoot?: string;
+ /** scanFiles: each finding a `path:<glob> pattern:<rule>` entry allowed, with that entry and its file. */
+ onAllowed?: (hit: AllowedHit & { file: string }) => void;
/** Opt-in generic high-entropy assignment heuristic. Default false. */
highEntropy?: boolean;
/**
@@ -104,7 +106,7 @@ export interface ScanOpts {
export {
SCAN_ALLOW_FILENAME, ALLOWLIST_FINGERPRINT_MIN_HEX, loadWorkspaceAllowlist, globToRegExp, matchesGlob, valueAllowlisted, pathAllowlisted,
} from './secret-scan-allowlist.ts';
-import { valueAllowlisted, pathAllowlisted } from './secret-scan-allowlist.ts';
+import { valueAllowlisted, pathAllowlisted, withoutRuleAllowed, type AllowedHit } from './secret-scan-allowlist.ts';
import { PUSH_SCAN_MAX_FILE_BYTES } from './page-size-policy.ts';
import {
ASSIGNMENT_QUOTED_SOURCE, ASSIGNMENT_UNQUOTED_SOURCE, HIGH_ENTROPY_MIN_BITS_PER_CHAR, HIGH_ENTROPY_REQUIRES_DIGIT_RE,
@@ -1186,7 +1188,7 @@ export function scanFiles(paths: string[], opts: ScanOpts = {}): SecretFinding[]
} catch {
continue;
}
- for (const f of scanText(text, opts)) {
+ for (const f of withoutRuleAllowed(scanText(text, opts), rel, allowlist, (hit) => opts.onAllowed?.({ ...hit, file: p }))) {
out.push({ ...f, file: p });
}
}
diff --git a/src/core/workspace-push.ts b/src/core/workspace-push.ts
index d886e5b6d7..f8d613756d 100644
--- a/src/core/workspace-push.ts
+++ b/src/core/workspace-push.ts
@@ -76,10 +76,11 @@ import {
type SecretFinding,
} from './secret-scan.ts';
import { scanTextStream } from './secret-scan-stream.ts';
+import { withoutRuleAllowed, type AllowlistProblem } from './secret-scan-allowlist.ts';
import { shellQuote } from './mcp-registration.ts';
import { gitChildEnv } from './git-env.ts';
import { recordCommit, recordIntent } from './git/commit-ledger.ts';
-import { stagedScanPublishRefusal } from './git/range-secret-scan.ts';
+import { scanAllowProblems, stagedScanPublishRefusal } from './git/range-secret-scan.ts';
// ── Types ───────────────────────────────────────────────────────────────────
@@ -113,6 +114,8 @@ export interface WorkspacePushResult {
findings?: PushSecretFinding[];
/** Staged paths the secret scan could not read (fail-closed block reason). */
unscannable?: string[];
+ /** blocked_secrets: `.gbrain-scan-allow` lines that allow nothing (a malformed rule entry, an unknown rule name). */
+ allowlistProblems?: AllowlistProblem[];
/** blocked_history: the refusal's code, what to do (a longer `--scan-timeout` after a timeout) and its findings JSON (never a value). */
code?: string;
suggestion?: string;
@@ -1081,6 +1084,8 @@ export async function workspacePush(opts: WorkspacePushOpts): Promise<WorkspaceP
const allowlist = loadWorkspaceAllowlist(root);
// nosemgrep: javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal -- fixed filename under the resolved workspace root (same path loadWorkspaceAllowlist reads); used only in printed guidance.
const allowlistPath = join(root, SCAN_ALLOW_FILENAME);
+ const allowlistProblems = scanAllowProblems(allowlist);
+ for (const p of allowlistProblems) log(`${SCAN_ALLOW_FILENAME} line ${p.entry}: ${p.problem}`);
const retryCommand = pushRetryCommand(opts);
const findings: PushSecretFinding[] = [];
const unscannable: string[] = [];
@@ -1093,8 +1098,9 @@ export async function workspacePush(opts: WorkspacePushOpts): Promise<WorkspaceP
unscannable.push(fileFindings);
continue;
}
- if (fileFindings.length === 0) continue;
- for (const f of buildPushFindings({ file: rel, findings: fileFindings, allowlist, allowlistPath, retryCommand })) {
+ const blocking = withoutRuleAllowed(fileFindings, rel, allowlist, hit => log(`${rel}:${hit.line} ${hit.pattern} allowed by ${SCAN_ALLOW_FILENAME} line: ${hit.entry}`));
+ if (blocking.length === 0) continue;
+ for (const f of buildPushFindings({ file: rel, findings: blocking, allowlist, allowlistPath, retryCommand })) {
findings.push(f);
}
}
@@ -1106,6 +1112,7 @@ export async function workspacePush(opts: WorkspacePushOpts): Promise<WorkspaceP
log(`PUSH BLOCKED: ${reason}`);
return finish({
ok: false, status: 'blocked_secrets', repoRoot: root, branch, findings, reason,
+ ...(allowlistProblems.length ? { allowlistProblems } : {}),
});
}
if (unscannable.length > 0) {
diff --git a/test/secret-scan-rule-allow-2901.test.ts b/test/secret-scan-rule-allow-2901.test.ts
new file mode 100644
index 0000000000..5f0aea4df1
--- /dev/null
+++ b/test/secret-scan-rule-allow-2901.test.ts
@@ -0,0 +1,142 @@
+/**
+ * GST-2901: a `.gbrain-scan-allow` line `path:<glob> pattern:<rule>` allows one rule in the files the glob matches and
+ * leaves every other rule active there. Path globs and fingerprints keep their meaning; a malformed line or an unknown
+ * rule name allows nothing and is reported; the scan output names the line that allowed a hit. Synthetic values only,
+ * assembled at runtime.
+ */
+import { afterEach, describe, expect, test } from 'bun:test';
+import { execFileSync } from 'node:child_process';
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { allowlistProblems } from '../src/core/secret-scan-allowlist.ts';
+import { loadWorkspaceAllowlist, NAMED_SECRET_RULES, SCAN_ALLOW_FILENAME, scanFiles, type SecretFinding } from '../src/core/secret-scan.ts';
+import { rangeSecretRefusal, scanRange } from '../src/core/git/range-secret-scan.ts';
+
+const JWT = ['eyJhbGciOiJIUzI1NiJ9', 'eyJyb2xlIjoic2VydmljZV9yb2xlIiwiaWF0IjoxNzAwMDAwMDAwfQ', 'c2lnbmF0dXJlLXBsYWNlaG9sZGVyLTAwMDA'].join('.');
+const AWS = 'AKIA' + 'IOSFODNN7EXAMPLE';
+const MAILER = `<a href="https://mail.example.invalid/unsubscribe?t=${JWT}">Unsubscribe</a>\n`;
+const KNOWN = [...NAMED_SECRET_RULES, 'high_entropy_assignment', 'labeled_credential'];
+
+const roots: string[] = [];
+afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
+
+function workspace(allow: string, files: Record<string, string>): string {
+ const root = mkdtempSync(join(tmpdir(), 'gbrain-scan-rule-'));
+ roots.push(root);
+ writeFileSync(join(root, SCAN_ALLOW_FILENAME), allow);
+ for (const [rel, text] of Object.entries(files)) {
+ mkdirSync(join(root, rel, '..'), { recursive: true });
+ writeFileSync(join(root, rel), text);
+ }
+ return root;
+}
+function scan(root: string, rels: string[]) {
+ const allowed: Array<{ file: string; pattern: string; entry: string }> = [];
+ const findings = scanFiles(rels.map(rel => join(root, rel)), { allowlist: loadWorkspaceAllowlist(root), workspaceRoot: root,
+ onAllowed: hit => allowed.push({ file: hit.file.slice(root.length + 1), pattern: hit.pattern, entry: hit.entry }) });
+ return { found: findings.map((f: SecretFinding) => `${f.file!.slice(root.length + 1)}:${f.pattern}`).sort(), allowed };
+}
+
+describe('path:<glob> pattern:<rule>', () => {
+ test('allows only the named rule, only in matching files, and names the line that allowed it', () => {
+ const root = workspace('path:mail/*.html pattern:jwt\n', {
+ 'mail/digest.html': MAILER + `aws ${AWS}\n`,
+ 'mail/sub/receipt.html': MAILER,
+ 'notes/plan.md': MAILER,
+ });
+ const r = scan(root, ['mail/digest.html', 'mail/sub/receipt.html', 'notes/plan.md']);
+ expect(r.found).toEqual(['mail/digest.html:aws_access_key', 'mail/sub/receipt.html:jwt', 'notes/plan.md:jwt']);
+ expect(r.allowed).toEqual([{ file: 'mail/digest.html', pattern: 'jwt', entry: 'path:mail/*.html pattern:jwt' }]);
+ });
+
+ test('a glob with no slash matches the file name at any depth, like a path entry', () => {
+ const root = workspace('path:*.html pattern:jwt\n', { 'a/b/one.html': MAILER, 'two.md': MAILER });
+ expect(scan(root, ['a/b/one.html', 'two.md']).found).toEqual(['two.md:jwt']);
+ });
+});
+
+describe('existing entries keep their meaning', () => {
+ test('a path glob still skips every rule in the file and a fingerprint still allows one value anywhere', () => {
+ const glob = workspace('mail/*.html\n', { 'mail/digest.html': MAILER + `aws ${AWS}\n` });
+ expect(scan(glob, ['mail/digest.html']).found).toEqual([]);
+ const first = scan(workspace('', { 'x.md': MAILER }), ['x.md']);
+ expect(first.found).toEqual(['x.md:jwt']);
+ const fingerprintRoot = workspace('', { 'x.md': MAILER, 'y.md': MAILER });
+ const [finding] = scanFiles([join(fingerprintRoot, 'x.md')], { allowlist: [], workspaceRoot: fingerprintRoot });
+ writeFileSync(join(fingerprintRoot, SCAN_ALLOW_FILENAME), `${finding!.fingerprint}\n`);
+ expect(scan(fingerprintRoot, ['x.md', 'y.md']).found).toEqual([]);
+ expect(allowlistProblems(['mail/*.html', finding!.fingerprint, '*.pem', 'fixtures/**'], KNOWN)).toEqual([]);
+ });
+});
+
+describe('a malformed line or an unknown rule allows nothing and is reported', () => {
+ const cases = [
+ ['path:mail/*.html pattern:jwtt', /unknown rule jwtt/],
+ ['path:mail/*.html pattern:jwt,bearer', /malformed rule entry/],
+ ['path:mail/*.html pattern:jwt bearer', /malformed rule entry/],
+ ['path:mail/*.html pattern:', /malformed rule entry/],
+ ['pattern:jwt', /malformed rule entry/],
+ ['path:mail/*.html', /read as a path glob for files whose name starts with path:/],
+ ] as const;
+ for (const [entry, problem] of cases) {
+ test(entry, () => {
+ const root = workspace(`${entry}\n`, { 'mail/digest.html': MAILER + `aws ${AWS}\n` });
+ expect(scan(root, ['mail/digest.html']).found).toEqual(['mail/digest.html:aws_access_key', 'mail/digest.html:jwt']);
+ const problems = allowlistProblems([entry], KNOWN);
+ expect(problems).toHaveLength(1);
+ expect(problems[0]!.problem).toMatch(problem);
+ });
+ }
+});
+
+describe('the publish scan', () => {
+ const git = (cwd: string, ...args: string[]) => execFileSync('git', ['-C', cwd, ...args], { env: { ...process.env, GIT_AUTHOR_NAME: 'Example', GIT_AUTHOR_EMAIL: '[email protected]',
+ GIT_COMMITTER_NAME: 'Example', GIT_COMMITTER_EMAIL: '[email protected]' } }).toString().trim();
+ function repo(allow: string) {
+ const root = mkdtempSync(join(tmpdir(), 'gbrain-scan-rule-repo-'));
+ roots.push(root);
+ git(root, 'init', '-q', '-b', 'main');
+ writeFileSync(join(root, SCAN_ALLOW_FILENAME), allow);
+ git(root, 'add', '.');
+ git(root, 'commit', '-q', '-m', 'allowlist');
+ const base = git(root, 'rev-parse', 'HEAD');
+ mkdirSync(join(root, 'mail'));
+ writeFileSync(join(root, 'mail/digest.html'), MAILER);
+ writeFileSync(join(root, 'notes.md'), MAILER);
+ git(root, 'add', '.');
+ git(root, 'commit', '-q', '-m', 'mailers');
+ return { root, base, head: git(root, 'rev-parse', 'HEAD') };
+ }
+
+ test('a rule entry allows the hit in matching files, records the line that allowed it, and still refuses elsewhere', () => {
+ const { root, base, head } = repo('path:mail/*.html pattern:jwt\n');
+ const result = scanRange(root, base, head);
+ expect(result.findings.map(f => `${f.path}:${f.pattern}`)).toEqual(['notes.md:jwt']);
+ expect(result.allowed).toEqual([{ path: 'mail/digest.html', commit: head, line: 1, pattern: 'jwt', entry: 'path:mail/*.html pattern:jwt' }]);
+ expect(result.allowlist_problems).toBeUndefined();
+ });
+
+ test('an unknown rule name allows nothing, and the refusal names the line', () => {
+ const { root, base, head } = repo('path:mail/*.html pattern:jwtt\n');
+ const result = scanRange(root, base, head);
+ expect(result.findings.map(f => `${f.path}:${f.pattern}`).sort()).toEqual(['mail/digest.html:jwt', 'notes.md:jwt']);
+ expect(result.allowed).toBeUndefined();
+ expect(result.allowlist_problems?.map(p => p.entry)).toEqual(['path:mail/*.html pattern:jwtt']);
+ expect(rangeSecretRefusal(root, base, head, result)!.suggestion).toContain('Check these .gbrain-scan-allow lines: path:mail/*.html pattern:jwtt (unknown rule jwtt, so it allows nothing');
+ });
+
+ test('the refusal detail keeps the first 10 allowed hits and their total, however many a rule entry allowed', () => {
+ const { root, base } = repo('path:mail/*.html pattern:jwt\n');
+ writeFileSync(join(root, 'mail/archive.html'), MAILER.repeat(30));
+ git(root, 'add', '.');
+ git(root, 'commit', '-q', '-m', 'archive');
+ const head = git(root, 'rev-parse', 'HEAD');
+ const result = scanRange(root, base, head);
+ expect(result.allowed).toHaveLength(31);
+ const detail = JSON.parse(rangeSecretRefusal(root, base, head, result)!.detail!);
+ expect(detail.findings.map((f: { path: string }) => f.path)).toEqual(['notes.md']);
+ expect(detail.allowed).toHaveLength(10);
+ expect(detail.allowed_total).toBe(31);
+ });
+});
diff --git a/test/workspace-push.serial.test.ts b/test/workspace-push.serial.test.ts
index 3815e81a83..70f1157924 100644
--- a/test/workspace-push.serial.test.ts
+++ b/test/workspace-push.serial.test.ts
@@ -304,6 +304,35 @@ describe('secret-scan gate', () => {
writeFileSync(join(work, SCAN_ALLOW_FILENAME), `${r.findings![0]!.fingerprint}\n`);
expect((await push()).status).toBe('pushed');
}, T);
+
+ const JWT = ['eyJhbGciOiJIUzI1NiJ9', 'eyJyb2xlIjoic2VydmljZV9yb2xlIiwiaWF0IjoxNzAwMDAwMDAwfQ', 'c2lnbmF0dXJlLXBsYWNlaG9sZGVyLTAwMDA'].join('.');
+ const MAILER = `<a href="https://mail.example.invalid/unsubscribe?t=${JWT}">Unsubscribe</a>\n`;
+
+ test('GST-2901: a path:<glob> pattern:<rule> line allows that rule in matching files only, and the log names the line', async () => {
+ writeFileSync(join(work, SCAN_ALLOW_FILENAME), 'path:mail/*.html pattern:jwt\n');
+ mkdirSync(join(work, 'mail'));
+ writeFileSync(join(work, 'mail/digest.html'), MAILER);
+ writeFileSync(join(work, 'notes.md'), MAILER);
+ const lines: string[] = [];
+ const r = await push({ logger: (l: string) => lines.push(l) });
+ expect(r.status).toBe('blocked_secrets');
+ expect(r.findings?.map(f => `${f.file}:${f.pattern}`)).toEqual(['notes.md:jwt']);
+ expect(r.allowlistProblems).toBeUndefined();
+ expect(lines).toContain(`mail/digest.html:1 jwt allowed by ${SCAN_ALLOW_FILENAME} line: path:mail/*.html pattern:jwt`);
+ rmSync(join(work, 'notes.md'));
+ expect((await push()).status).toBe('pushed');
+ }, T);
+
+ test('GST-2901: a malformed rule line allows nothing, and the blocked_secrets result names it', async () => {
+ writeFileSync(join(work, SCAN_ALLOW_FILENAME), 'path:mail/*.html pattern:JWT\n');
+ mkdirSync(join(work, 'mail'));
+ writeFileSync(join(work, 'mail/digest.html'), MAILER);
+ const r = await push();
+ expect(r.status).toBe('blocked_secrets');
+ expect(r.findings?.map(f => `${f.file}:${f.pattern}`)).toEqual(['mail/digest.html:jwt']);
+ expect(r.allowlistProblems?.map(p => p.entry)).toEqual(['path:mail/*.html pattern:JWT']);
+ expect(r.allowlistProblems?.[0]?.problem).toMatch(/malformed rule entry, so it allows nothing/);
+ }, T);
});
describe('secret-scan refusal guidance (DX-3/ENG-11)', () => {