Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
Allow one secret-scan rule per file glob in .gbrain-scan-allow (#9589)
Garry Tana2c5c6cab121
diff --git a/SECURITY.md b/SECURITY.mdindex 41cd61c122..500092c894 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -641,7 +641,9 @@ complete. If you find a live credential in a page:value; rewrite/re-push or regenerate those as needed.4. Add a value's fingerprint (printed with every `gbrain sources push` finding)to `<workspace>/.gbrain-scan-allow` ONLY for values you have confirmed are- not credentials.+ not credentials. A `path:<glob> pattern:<rule>` line allows one rule in the+ files the glob matches and keeps every other rule active there; a malformed+ line or an unknown rule name allows nothing and is reported.**Say to your agent:** *"a secret leaked into a brain page — rotate it andpurge the page"* (your agent rotates the credential, then runsdiff --git a/behavior-changes.d/scan_allow_rule_entry.md b/behavior-changes.d/scan_allow_rule_entry.mdnew file mode 100644index 0000000000..1f538d9741--- /dev/null+++ b/behavior-changes.d/scan_allow_rule_entry.md@@ -0,0 +1,4 @@+---+kind: add+---+A .gbrain-scan-allow line of the form path:<glob> pattern:<rule> now allows only that secret-scan rule in the files the glob matches, so a rule that keeps flagging harmless values (for example jwt on newsletter unsubscribe links) can be quieted without turning off every other rule for those files. Existing fingerprint and path-glob lines work as before. A line that starts with pattern:, or with path: and contains a space, is read only as a rule entry: if it is malformed or names an unknown rule it allows nothing, and the push refusal, gbrain git scan and bootstrap verify name it. A bare path: line with no space is still a path glob, and gbrain git scan now warns about it.diff --git a/changelog.d/gbra587-scan-allow-rule.md b/changelog.d/gbra587-scan-allow-rule.mdnew file mode 100644index 0000000000..dc165539e4--- /dev/null+++ b/changelog.d/gbra587-scan-allow-rule.md@@ -0,0 +1,6 @@+**A `.gbrain-scan-allow` line can now allow one secret-scan rule in matching files, instead of skipping every rule there.**++<!-- behavior-change: scan_allow_rule_entry -->++### Added+- `.gbrain-scan-allow` accepts `path:<glob> pattern:<rule>`, which allows only the named rule (for example `jwt` on newsletter unsubscribe links) in the files the glob matches; every other rule still blocks a push there. Path globs and fingerprints keep their meaning. A malformed rule line or an unknown rule name allows nothing and is named by the push refusal, `gbrain git scan` and bootstrap verify, and `gbrain git scan` lists each hit a rule line allowed with the line that allowed it (`allowed[]`, `allowlist_problems[]` in `--json`).diff --git a/docs/architecture/key-files/secret-scan.md b/docs/architecture/key-files/secret-scan.mdindex eb759b9d06..6619007527 100644--- a/docs/architecture/key-files/secret-scan.md+++ b/docs/architecture/key-files/secret-scan.md@@ -6,7 +6,7 @@ The format-based secret detector shared by `sources push`, bootstrap verify, thecompiled-context sensitivity scan and the transcripts redactor, and the composedsensitivity detector built on it.-- `src/core/secret-scan.ts` — pattern-based secret detection shared by `sources push`, bootstrap verify, the compiled-context sensitivity scan and the transcripts redactor. Detection is FORMAT-based, not prefix-only: vendor shapes (anthropic, openai, voyage, github, gitlab, slack, aws AKIA/ASIA, google, stripe, sendgrid, twilio, supabase sb_secret_/sbp_, npm, huggingface, `digitalocean` `do[opr]_v1_` + 64 hex, the AI-provider keys `openrouter` `sk-or-v1-` + 64 lowercase hex, `groq` `gsk_` + 52, `xai` `xai-` + 80, `perplexity` `pplx-` + 48 and `replicate` `r8_` + 37, all fixed-length with a hard right edge, gbrain), JWTs, private keys (whole-text claims, below), plus four catch-alls appended LAST (`bearer` — prebuilt two-group layout so only the token is the value, which may hold a JSON-escaped `\/` (#8266); `db_url_credentials` — the `scheme://user:pass@` span of database URLs, host/db kept; `url_credentials` — the same span for `http(s)` (scheme case-insensitive, `gi`), user 0–128 stopping at `:`/`/`, password 1–256 excluding whitespace, `@`, `"`, `'`, `/`, `?`, `#`, with a `validate` hook that rejects placeholder passwords (`<...>`, `${...}`, `$VAR`, all `*`, all `x`); `basic_auth` — two prebuilt case-insensitive forms, `Authorization: Basic` with a 4–2048 char value and bare `Basic` with 16–2048, both with a hard right edge and a `validate` that requires the value to base64-decode to printable ASCII containing `:`; it is NOT in the bearer family, so it joins neither the echo dictionary nor `secret:bearer`). The pattern contract: `CorePattern` carries optional `flags` (default `'g'`), `precheck`, `validate(value)` (O(value), run on the whole match before the claim and the union split; a rejected match is not re-tried from inside) and `since` (the gbrain version that added or last changed the rule; `SECURITY_WAVE_SINCE` = `0.60.31.0` on `digitalocean`, `url_credentials`, `basic_auth` and `private_key_pem`, `PROVIDER_KEYS_SINCE` on the five AI-provider rules (#7959), kept in step with the release that ships a rule change). Compiled patterns keep group 1 = anchor, group 2 = value. Every core rule and both `high_entropy_assignment` forms also have an escape-boundary twin (#7978), compiled right after the rule under the same name and anchored on a JSON escape (`\n`, `\r`, `\t`, `\b`, `\f`, `\uXXXX`, and since #8266 `\xNN`, `\UXXXXXXXX`), a percent-escape (`%XX`), a quoted-printable `=3D` or a terminal color code (`ESC[…m`, raw or written `\u001b[`/`\x1b[`/`\033[`/`\e[`), whose last character the default boundary reads as a word character; a twin runs only on a line `ESCAPE_IN_LINE_RE` matches, so other lines run the old list, and a prebuilt rule carries its twin as `CorePattern.escaped`. `patternSince(pattern)` reads the map; every finding of such a pattern carries `SecretFinding.since`, which the push and bootstrap refusals render. Fixed-length shapes carry a trailing negative lookahead. Per-line claimed-span dedupe is first-wins for ATTRIBUTION (order matters: `Bearer <vendor key>` keeps the vendor name, a fully covered later span is skipped) but coverage is the UNION: every unclaimed slice of a later overlapping span is claimed as a hit of the later pattern (own value, own fingerprint), so a database URL whose username is a vendor-shaped id still loses its password. The transcript-only `labeled_credential` rule (`ScanOpts.labeledCredentials`) lives in `secret-scan-labeled.ts` (Security). The opt-in `high_entropy_assignment` rule (`ScanOpts.highEntropy`; its shapes and gates live in `src/core/secret-scan-assignment.ts`, split out at the 1500-line module cap) is anchored on a credential keyword or on an identifier whose last word is `key` (`STRIPE_KEY=`, never `MONKEY=` or a bare `KEY=`) and requires a 12+ char value that contains a digit and a non-digit AND clears 3.5 bits/char on the bias-corrected (Miller-Madow) estimate `correctedEntropy`, since raw per-character entropy is capped at log2(n) and rejected most random 12-16 char secrets. It is two compiled patterns sharing the name: QUOTED (after `"` or `'`, any run of non-quote, non-whitespace characters) and UNQUOTED (the legacy `[A-Za-z0-9+/_=-]` class plus `!#$%*@^~.?<>`; never `&`, `()[]{}`, `:`, `,` or `;`, so query strings, calls and SQL `SET` lists stay intact, and a password carrying `,`/`;` matches only quoted; the last character excludes `.?!`, so a sentence's full stop is not claimed). Its `validate` rejects values that start with `scheme://`, `./`, `~/`, `${`, `$(` or `<REDACTED:`, dotted identifier paths (`process.env.X`) and `/`-paths holding a character outside the legacy class. Consumers with it on: retrieval output, transcript ingest, hooks and the hook heartbeat, the atoms drain, `embed-facts.ts` error text and the context-engine relay re-scan. Off: the push gate, bootstrap verify/repo and the compiled-context sensitivity scan. Own minimal allowlist + `<ws>/.gbrain-scan-allow` per-finding overrides — deliberately NOT the repo's `.gitleaks.toml` (a public-repo CI fixture policy); the allowlist block (`loadWorkspaceAllowlist`, the glob dialect `globToRegExp`/`matchesGlob`, `valueAllowlisted`/`pathAllowlisted`, `SCAN_ALLOW_FILENAME`, the 16-hex fingerprint floor) lives in `src/core/secret-scan-allowlist.ts` since wave 16 (G4.1, the 1500-line module cap) and is re-exported here unchanged (`test/secret-scan-allowlist.test.ts`). `correctedEntropy` is one histogram pass (distinct count = histogram size; byte-identical to the former `shannonEntropy + Set` form, `test/secret-scan-entropy-equivalence.test.ts`; ~80 → ~25 ms per 250 × 4096-char segment). `ScanOpts.work` / `RedactOpts.work` takes a `ScanWork` accumulator (`src/core/secret-scan-work.ts`: lines, regex execs, candidates per pattern, chars handed to the entropy gate counted inside `correctedEntropy`, validate calls, claims, union splits, echo sweeps) pinned exactly on the perf fixture by `test/secret-scan-work-counters.test.ts`; the perf file's absolute bounds stay the only guard on per-character cost. Deliberately NOT patterns: `sb_publishable_`, Supabase project refs (identifiers, not credentials). Never spell a literal database or `http(s)` URL with userinfo, or any credential, in this file's comments or in fixtures (assemble test values at runtime; `scripts/check-pg-url-redaction.sh` greps `src/` for every `db_url_credentials` scheme). Long lines are scanned in full (no length cap — a secret on a 1 MB minified line must still redact); what keeps that linear is bounded quantifiers on every shape whose value class could otherwise run to end-of-line — `db_url_credentials` (user ≤128 stopping at `:`/`/`; password 1–256 stopping only at whitespace, `@` and the string delimiters `"`/`'`, so unescaped `/ { } < > | ^ \` and backtick passwords redact), `jwt` (prebuilt so its boundary excludes `-` as well as `_`; segments ≤4096), `high_entropy_assignment` (≤64 identifier chars after the keyword, value ≤4096), `url_credentials` (user ≤128, password ≤256), `basic_auth` (value ≤2048, so its decode is constant work) and the private-key walks (below) — plus a per-pattern `precheck` substring gate on the catch-alls (`@` for both URL patterns; `earer`/`EARER`, since an all-caps `BEARER` scheme is accepted; a case-insensitive `basic`), a carried next-newline position in `scanPemBlocks` (sticky -1, so the header-line count is O(text) even when thousands of hits sit ahead of a newline-free multi-megabyte tail), a lazily-allocated per-line claimed-char bitmap behind the first-wins dedupe (O(value) per hit), and a windowed `buildPreview` (whole line ≤160 chars; otherwise hit−40…+80, edges snapped outward to whole spans, every claimed span in the window spliced by position, so a preview never carries a sibling secret or a fragment). `redactFindings` (= `planRedaction` + `applyRedaction`) rebuilds the text by SPAN-SPLICE in one O(text + hits) pass over the claimed spans, and fills the UNCLAIMED gaps with a bounded echo pass: the values claimed by `bearer`, `high_entropy_assignment`, `labeled_credential` and every vendor or `jwt` rule (#7978; not `basic_auth` or the two URL userinfo catch-alls) form an echo dictionary (value → pattern; floors `ECHO_MIN_CHARS_BEARER` = `ECHO_MIN_CHARS_VENDOR` = 20 / `ECHO_MIN_CHARS_ENTROPY` = 12, values ≤ `ECHO_MAX_VALUE_CHARS` = 512, the first `ECHO_MAX_UNIQUE` = 64 unique bearer, assignment and labeled values plus, under their own cap, the first `ECHO_MAX_UNIQUE_VENDOR` = 32 unique vendor or `jwt` values, in claim order, so planted vendor strings never crowd out a bearer token's echo — a value past a cap is redacted at its claim, its echoes are the accepted miss), matched by a streaming merge over the ORIGINAL text — per value, the next occurrence at or beyond the emit cursor (`indexOf` resuming at the cursor, re-sought only once the cursor passes it), leftmost-longest at each step, emitted incrementally into the unclaimed gaps — so retained state is O(values) however long the text is (occurrences are never collected into a list) (never an alternation RegExp, never over emitted tokens). Echoes add no findings. `planRedaction` feeds a caller-supplied shared map (`RedactOpts.echoValues`) and `applyRedaction` reads it at apply time, so a multi-field document scrubs an echo in one field of a value claimed in another. `basic_auth` and the URL userinfo catch-alls redact exactly what `scanText` reports; a vendor value embedded past its boundary (`token_<key>`) is scrubbed as an echo when the same value is claimed elsewhere in the document, and left as-is otherwise. PRIVATE KEYS: `pemClaims` finds `BEGIN`/`END … PRIVATE KEY` fences in two global passes and merges them in text order. BLOCK: a BEGIN whose next fence is an END within `PEM_BODY_MAX_CHARS` (16384) is claimed header through END; a block never swallows a later BEGIN. FORWARD (BEGIN with no END in view): the header plus the body lines that follow; a header followed by no body line (a marker quoted in prose) is claimed alone. BACKWARD (END with no BEGIN in view): the body lines before the fence plus the fence; an END with no body line before it is no claim. A body line is a whole-line base64 run (`A-Za-z0-9+/=`) indented by up to 16 spaces/tabs; line ends are `\n`, `\r\n` or the literal JSON escapes; interior lines are 16–128 chars, the line next to a fence and a partial line at a cut 1–128; optional RFC 1421 `Proc-Type:`/`DEK-Info:` lines and one blank line may follow BEGIN; a line with anything else, or a run over 128 chars, ends the body. Total claimed body stays within `PEM_BODY_MAX_CHARS`. Each walk stops at the neighbouring fence or the previous claim, so the pass is linear (20k unterminated BEGINs, 20k END-only fences and 1 MB of base64 lines are perf-pinned). Per-line hits that fall wholly inside a key claim are dropped (one key, one finding). A FORWARD claim's finding also carries `legacyFingerprint`, the header-only fingerprint earlier releases reported, so `workspace-push.ts` can name a stale allowlist entry; a truncated key's fingerprint covers its body and therefore depends on where the text was cut, so it is not a stable allowlist key. `privateKeySpans(text)` exports the same claims as sorted, disjoint `{start, end}` spans. Accepted misses: body on the header's own line, space-joined single-line keys, blockquoted (`> `) keys, base64 runs over 128 chars, body past 16384 chars, `http(s)` passwords containing a literal `/`, `?` or `#`, Basic values over 2048 chars, digitless assignment values, dotted passwords whose every segment starts with a letter, and unquoted passwords containing `,` or `;`. False-positive budget: `scripts/secret-scan-fp-budget.ts` diffs findings against a base ref's scanner (`--base`, default `origin/master`) (`--dir <brain>` reports per-pattern counts only, never content) and `test/secret-scan-fp-budget.test.ts` pins a synthetic brain-like fixture (zero findings on its non-secret part in both modes, every planted secret redacted). Shape tests: `test/secret-scan-credential-shapes.test.ts`, `test/secret-scan-private-key-spans.test.ts`. Perf pins: `test/secret-scan-perf.test.ts`. Retrieval output is another consumer of its canonical scanner; that output boundary does not sanitize internal evaluation/provider inputs.+- `src/core/secret-scan.ts` — pattern-based secret detection shared by `sources push`, bootstrap verify, the compiled-context sensitivity scan and the transcripts redactor. Detection is FORMAT-based, not prefix-only: vendor shapes (anthropic, openai, voyage, github, gitlab, slack, aws AKIA/ASIA, google, stripe, sendgrid, twilio, supabase sb_secret_/sbp_, npm, huggingface, `digitalocean` `do[opr]_v1_` + 64 hex, the AI-provider keys `openrouter` `sk-or-v1-` + 64 lowercase hex, `groq` `gsk_` + 52, `xai` `xai-` + 80, `perplexity` `pplx-` + 48 and `replicate` `r8_` + 37, all fixed-length with a hard right edge, gbrain), JWTs, private keys (whole-text claims, below), plus four catch-alls appended LAST (`bearer` — prebuilt two-group layout so only the token is the value, which may hold a JSON-escaped `\/` (#8266); `db_url_credentials` — the `scheme://user:pass@` span of database URLs, host/db kept; `url_credentials` — the same span for `http(s)` (scheme case-insensitive, `gi`), user 0–128 stopping at `:`/`/`, password 1–256 excluding whitespace, `@`, `"`, `'`, `/`, `?`, `#`, with a `validate` hook that rejects placeholder passwords (`<...>`, `${...}`, `$VAR`, all `*`, all `x`); `basic_auth` — two prebuilt case-insensitive forms, `Authorization: Basic` with a 4–2048 char value and bare `Basic` with 16–2048, both with a hard right edge and a `validate` that requires the value to base64-decode to printable ASCII containing `:`; it is NOT in the bearer family, so it joins neither the echo dictionary nor `secret:bearer`). The pattern contract: `CorePattern` carries optional `flags` (default `'g'`), `precheck`, `validate(value)` (O(value), run on the whole match before the claim and the union split; a rejected match is not re-tried from inside) and `since` (the gbrain version that added or last changed the rule; `SECURITY_WAVE_SINCE` = `0.60.31.0` on `digitalocean`, `url_credentials`, `basic_auth` and `private_key_pem`, `PROVIDER_KEYS_SINCE` on the five AI-provider rules (#7959), kept in step with the release that ships a rule change). Compiled patterns keep group 1 = anchor, group 2 = value. Every core rule and both `high_entropy_assignment` forms also have an escape-boundary twin (#7978), compiled right after the rule under the same name and anchored on a JSON escape (`\n`, `\r`, `\t`, `\b`, `\f`, `\uXXXX`, and since #8266 `\xNN`, `\UXXXXXXXX`), a percent-escape (`%XX`), a quoted-printable `=3D` or a terminal color code (`ESC[…m`, raw or written `\u001b[`/`\x1b[`/`\033[`/`\e[`), whose last character the default boundary reads as a word character; a twin runs only on a line `ESCAPE_IN_LINE_RE` matches, so other lines run the old list, and a prebuilt rule carries its twin as `CorePattern.escaped`. `patternSince(pattern)` reads the map; every finding of such a pattern carries `SecretFinding.since`, which the push and bootstrap refusals render. Fixed-length shapes carry a trailing negative lookahead. Per-line claimed-span dedupe is first-wins for ATTRIBUTION (order matters: `Bearer <vendor key>` keeps the vendor name, a fully covered later span is skipped) but coverage is the UNION: every unclaimed slice of a later overlapping span is claimed as a hit of the later pattern (own value, own fingerprint), so a database URL whose username is a vendor-shaped id still loses its password. The transcript-only `labeled_credential` rule (`ScanOpts.labeledCredentials`) lives in `secret-scan-labeled.ts` (Security). The opt-in `high_entropy_assignment` rule (`ScanOpts.highEntropy`; its shapes and gates live in `src/core/secret-scan-assignment.ts`, split out at the 1500-line module cap) is anchored on a credential keyword or on an identifier whose last word is `key` (`STRIPE_KEY=`, never `MONKEY=` or a bare `KEY=`) and requires a 12+ char value that contains a digit and a non-digit AND clears 3.5 bits/char on the bias-corrected (Miller-Madow) estimate `correctedEntropy`, since raw per-character entropy is capped at log2(n) and rejected most random 12-16 char secrets. It is two compiled patterns sharing the name: QUOTED (after `"` or `'`, any run of non-quote, non-whitespace characters) and UNQUOTED (the legacy `[A-Za-z0-9+/_=-]` class plus `!#$%*@^~.?<>`; never `&`, `()[]{}`, `:`, `,` or `;`, so query strings, calls and SQL `SET` lists stay intact, and a password carrying `,`/`;` matches only quoted; the last character excludes `.?!`, so a sentence's full stop is not claimed). Its `validate` rejects values that start with `scheme://`, `./`, `~/`, `${`, `$(` or `<REDACTED:`, dotted identifier paths (`process.env.X`) and `/`-paths holding a character outside the legacy class. Consumers with it on: retrieval output, transcript ingest, hooks and the hook heartbeat, the atoms drain, `embed-facts.ts` error text and the context-engine relay re-scan. Off: the push gate, bootstrap verify/repo and the compiled-context sensitivity scan. Own minimal allowlist + `<ws>/.gbrain-scan-allow` per-finding overrides — deliberately NOT the repo's `.gitleaks.toml` (a public-repo CI fixture policy); the allowlist block (`loadWorkspaceAllowlist`, the glob dialect `globToRegExp`/`matchesGlob`, `valueAllowlisted`/`pathAllowlisted`, `SCAN_ALLOW_FILENAME`, the 16-hex fingerprint floor, and rule entries `path:<glob> pattern:<rule>`: `ruleAllowEntry`/`withoutRuleAllowed` drop only that rule's findings in matching files and report the entry, `allowlistProblems` names malformed rule entries and unknown rule names, which allow nothing, and a bare `path:` glob; `pathAllowlisted` never reads a rule entry as a glob; `scanFiles` (`ScanOpts.onAllowed`), the range scan (`RangeScan.allowed`/`allowlist_problems`, `scanAllowProblems`), the push scan and bootstrap repo/verify apply and report them, `test/secret-scan-rule-allow-2901.test.ts`) lives in `src/core/secret-scan-allowlist.ts` since wave 16 (G4.1, the 1500-line module cap) and is re-exported here unchanged (`test/secret-scan-allowlist.test.ts`). `correctedEntropy` is one histogram pass (distinct count = histogram size; byte-identical to the former `shannonEntropy + Set` form, `test/secret-scan-entropy-equivalence.test.ts`; ~80 → ~25 ms per 250 × 4096-char segment). `ScanOpts.work` / `RedactOpts.work` takes a `ScanWork` accumulator (`src/core/secret-scan-work.ts`: lines, regex execs, candidates per pattern, chars handed to the entropy gate counted inside `correctedEntropy`, validate calls, claims, union splits, echo sweeps) pinned exactly on the perf fixture by `test/secret-scan-work-counters.test.ts`; the perf file's absolute bounds stay the only guard on per-character cost. Deliberately NOT patterns: `sb_publishable_`, Supabase project refs (identifiers, not credentials). Never spell a literal database or `http(s)` URL with userinfo, or any credential, in this file's comments or in fixtures (assemble test values at runtime; `scripts/check-pg-url-redaction.sh` greps `src/` for every `db_url_credentials` scheme). Long lines are scanned in full (no length cap — a secret on a 1 MB minified line must still redact); what keeps that linear is bounded quantifiers on every shape whose value class could otherwise run to end-of-line — `db_url_credentials` (user ≤128 stopping at `:`/`/`; password 1–256 stopping only at whitespace, `@` and the string delimiters `"`/`'`, so unescaped `/ { } < > | ^ \` and backtick passwords redact), `jwt` (prebuilt so its boundary excludes `-` as well as `_`; segments ≤4096), `high_entropy_assignment` (≤64 identifier chars after the keyword, value ≤4096), `url_credentials` (user ≤128, password ≤256), `basic_auth` (value ≤2048, so its decode is constant work) and the private-key walks (below) — plus a per-pattern `precheck` substring gate on the catch-alls (`@` for both URL patterns; `earer`/`EARER`, since an all-caps `BEARER` scheme is accepted; a case-insensitive `basic`), a carried next-newline position in `scanPemBlocks` (sticky -1, so the header-line count is O(text) even when thousands of hits sit ahead of a newline-free multi-megabyte tail), a lazily-allocated per-line claimed-char bitmap behind the first-wins dedupe (O(value) per hit), and a windowed `buildPreview` (whole line ≤160 chars; otherwise hit−40…+80, edges snapped outward to whole spans, every claimed span in the window spliced by position, so a preview never carries a sibling secret or a fragment). `redactFindings` (= `planRedaction` + `applyRedaction`) rebuilds the text by SPAN-SPLICE in one O(text + hits) pass over the claimed spans, and fills the UNCLAIMED gaps with a bounded echo pass: the values claimed by `bearer`, `high_entropy_assignment`, `labeled_credential` and every vendor or `jwt` rule (#7978; not `basic_auth` or the two URL userinfo catch-alls) form an echo dictionary (value → pattern; floors `ECHO_MIN_CHARS_BEARER` = `ECHO_MIN_CHARS_VENDOR` = 20 / `ECHO_MIN_CHARS_ENTROPY` = 12, values ≤ `ECHO_MAX_VALUE_CHARS` = 512, the first `ECHO_MAX_UNIQUE` = 64 unique bearer, assignment and labeled values plus, under their own cap, the first `ECHO_MAX_UNIQUE_VENDOR` = 32 unique vendor or `jwt` values, in claim order, so planted vendor strings never crowd out a bearer token's echo — a value past a cap is redacted at its claim, its echoes are the accepted miss), matched by a streaming merge over the ORIGINAL text — per value, the next occurrence at or beyond the emit cursor (`indexOf` resuming at the cursor, re-sought only once the cursor passes it), leftmost-longest at each step, emitted incrementally into the unclaimed gaps — so retained state is O(values) however long the text is (occurrences are never collected into a list) (never an alternation RegExp, never over emitted tokens). Echoes add no findings. `planRedaction` feeds a caller-supplied shared map (`RedactOpts.echoValues`) and `applyRedaction` reads it at apply time, so a multi-field document scrubs an echo in one field of a value claimed in another. `basic_auth` and the URL userinfo catch-alls redact exactly what `scanText` reports; a vendor value embedded past its boundary (`token_<key>`) is scrubbed as an echo when the same value is claimed elsewhere in the document, and left as-is otherwise. PRIVATE KEYS: `pemClaims` finds `BEGIN`/`END … PRIVATE KEY` fences in two global passes and merges them in text order. BLOCK: a BEGIN whose next fence is an END within `PEM_BODY_MAX_CHARS` (16384) is claimed header through END; a block never swallows a later BEGIN. FORWARD (BEGIN with no END in view): the header plus the body lines that follow; a header followed by no body line (a marker quoted in prose) is claimed alone. BACKWARD (END with no BEGIN in view): the body lines before the fence plus the fence; an END with no body line before it is no claim. A body line is a whole-line base64 run (`A-Za-z0-9+/=`) indented by up to 16 spaces/tabs; line ends are `\n`, `\r\n` or the literal JSON escapes; interior lines are 16–128 chars, the line next to a fence and a partial line at a cut 1–128; optional RFC 1421 `Proc-Type:`/`DEK-Info:` lines and one blank line may follow BEGIN; a line with anything else, or a run over 128 chars, ends the body. Total claimed body stays within `PEM_BODY_MAX_CHARS`. Each walk stops at the neighbouring fence or the previous claim, so the pass is linear (20k unterminated BEGINs, 20k END-only fences and 1 MB of base64 lines are perf-pinned). Per-line hits that fall wholly inside a key claim are dropped (one key, one finding). A FORWARD claim's finding also carries `legacyFingerprint`, the header-only fingerprint earlier releases reported, so `workspace-push.ts` can name a stale allowlist entry; a truncated key's fingerprint covers its body and therefore depends on where the text was cut, so it is not a stable allowlist key. `privateKeySpans(text)` exports the same claims as sorted, disjoint `{start, end}` spans. Accepted misses: body on the header's own line, space-joined single-line keys, blockquoted (`> `) keys, base64 runs over 128 chars, body past 16384 chars, `http(s)` passwords containing a literal `/`, `?` or `#`, Basic values over 2048 chars, digitless assignment values, dotted passwords whose every segment starts with a letter, and unquoted passwords containing `,` or `;`. False-positive budget: `scripts/secret-scan-fp-budget.ts` diffs findings against a base ref's scanner (`--base`, default `origin/master`) (`--dir <brain>` reports per-pattern counts only, never content) and `test/secret-scan-fp-budget.test.ts` pins a synthetic brain-like fixture (zero findings on its non-secret part in both modes, every planted secret redacted). Shape tests: `test/secret-scan-credential-shapes.test.ts`, `test/secret-scan-private-key-spans.test.ts`. Perf pins: `test/secret-scan-perf.test.ts`. Retrieval output is another consumer of its canonical scanner; that output boundary does not sanitize internal evaluation/provider inputs.- `src/core/secret-scan-cookie.ts` — the transcript-lane `cookie_header` rule (GST-2768), run by `scanInternal` only under `ScanOpts.transcriptRules` (`TRANSCRIPT_LANE_SCAN_OPTS` in `secret-scan.ts`: redactSession, the recent-transcripts read and `transcripts audit-secrets`). `cookieHeaderValues(line)` reads the `name=value` pairs after a `Cookie:` / `Set-Cookie:` header (any case, quoted or JSON-escaped) or a curl `-b`/`--cookie` argument and returns the values only (8+ chars, no placeholder or mask); `Set-Cookie` stops after its first pair, and parsing stops at the first segment that is not `name=value`. A quoted value may be JSON-escaped (`sid=\"…\"`, up to three backslashes before each quote), and `curlContinues` carries a curl command across lines ending in `\`, so a `-b` alone on a continued line (Chrome's "Copy as cURL") is curl's (`scanInternal` passes `inCurl`). Values join the echo dictionary at 12 chars under their own cap, `ECHO_MAX_UNIQUE_COOKIE` = 64, apart from `ECHO_MAX_UNIQUE`. `test/secret-scan-push-gate-unchanged.test.ts` pins every non-transcript option set byte-identical on the synthetic corpus `test/fixtures/transcripts/synthetic-credentials.ts`.- `src/core/secret-scan-transcript.ts` — `compileTranscriptPatterns()`, the transcript-lane-only labeled rules (`TRANSCRIPT_LABELED_PATTERNS` from `secret-scan-labeled.ts`: the `.netrc` forms and the `key`/`netrc` continuations) compiled as `labeled_credential` for `buildPatterns` in `secret-scan.ts` under `ScanOpts.transcriptRules`, after every other rule. Moved out of `secret-scan.ts` at its 1500-line cap with no logic change.- `src/core/secret-scan-stream.ts` — `scanTextStream(chunks, opts)`, the windowed scan of a byte stream too large to hold as one string (the push gate's staged blobs over 25 MiB). Bytes are decoded by a streaming `TextDecoder` (`ignoreBOM`, so a split multi-byte character joins and invalid bytes become U+FFFD as `Buffer.toString` makes them) and handed to `scanTextSpans` (`secret-scan.ts`: `scanText` plus each claim's absolute span, allowlisted claims kept with `finding: null`) in windows of at most `STREAM_SCAN_WINDOW_CHARS` (8 Mi UTF-16 units). Each next window starts at a line start at least `STREAM_SCAN_OVERLAP_CHARS` (64 Ki, above the longest claim: a private key's 16384-char body plus fences, a jwt's 3 × 4096) before the previous end, or mid-line at exactly the overlap (never inside a surrogate pair) when the line is too long. A window keeps the claims that start before the next window's start, the last window keeps the rest, and a claim wholly inside a span an earlier window kept (a seam's partial view, or an exact duplicate) is dropped; lines are 1-based over the whole stream. `streamScanWork` (@internal) counts windows, bytes, scanned chars and the largest window. `test/secret-scan-stream.test.ts` pins equality with `scanText` across both seams of a window, mid-line splits, split multi-byte characters, invalid UTF-8 and a BOM.diff --git a/docs/guides/shared-checkout-git.md b/docs/guides/shared-checkout-git.mdindex 71b2a3c15e..a5369dc5e7 100644--- a/docs/guides/shared-checkout-git.md+++ b/docs/guides/shared-checkout-git.md@@ -188,7 +188,12 @@ mode scans either way.A finding that is not a secret: add its fingerprint (`sha256:` and 16 hexcharacters, as the preview prints it) as a line of `.gbrain-scan-allow` at the-checkout root (`#` starts a comment). The scan reads the file from the working+checkout root (`#` starts a comment). A rule that fires on many harmless values+in one kind of file takes a rule entry instead, `path:<glob> pattern:<rule>`,+which allows only that rule in the matching files; the preview lists each hit it+allowed with the line that allowed it, and names any line that allows nothing+because it is malformed or names an unknown rule+([details](write-refusals.md#allowlisting-a-reviewed-false-positive)). The scan reads the file from the workingtree, so the next push uses it without a commit. That is deliberate: the file isthe local user's answer to the scan, and writing it takes local write access tothe checkout, which could push directly anyway. In a managed root below thediff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.mdindex 04154dee52..da27bcfabc 100644--- a/docs/guides/write-refusals.md+++ b/docs/guides/write-refusals.md@@ -943,14 +943,32 @@ real credential publishes it.If it is not (a sample value in documentation, a test fixture), run the"allow this finding" command from the refusal, review the change to`.gbrain-scan-allow`, then run the retry command. The file holds one entry-per line: a fingerprint (`sha256:` and at least 16 hex characters) or a path+per line: a fingerprint (`sha256:` and at least 16 hex characters), a pathglob (a glob without `/` matches a file name at any depth; one with `/` is-anchored at the repository root), with `#` comments. Commit the file so-other machines push the same tree.+anchored at the repository root) or a rule entry (below), with `#` comments.+Commit the file so other machines push the same tree.Prefer a fingerprint over a glob. A glob skips the scan for every file itmatches, including credentials added later.+When one rule keeps firing on many harmless values in the same kind of file+(for example `jwt` on the unsubscribe links in saved newsletter HTML), a rule+entry allows that rule alone in the files a glob matches:++```+path:mail/*.html pattern:jwt+```++Every other rule still runs on those files, so a cloud key pasted into one of+them is still refused. Write one rule per line; the glob is the same as a path+glob's, and the rule name is the one the refusal prints in brackets. A line that+starts with `pattern:`, or with `path:` and has a space in it, is read only as a+rule entry: if it is malformed (two rules, a missing glob) or names a rule that+does not exist, it allows nothing, and the refusal, `gbrain git scan` and+bootstrap verify name the line and what is wrong with it. `gbrain git scan`+also lists each hit a rule entry allowed and the line that allowed it+(`allowed[]` in `--json`), and `gbrain sources push` logs them.+### A stale allowlist entry for a private keyThe fingerprint of a private key whose `END` line is missing (a cut-offdiff --git a/src/commands/git.ts b/src/commands/git.tsindex 8694cf4a53..21a197177e 100644--- a/src/commands/git.ts+++ b/src/commands/git.ts@@ -217,9 +217,11 @@ function runScan(p: Parsed): number {const commits = Number(execFileSync('git', ['-C', repo, 'rev-list', '--count', `${from}..${to}`], { env: gitChildEnv({ ...GIT_ENV }) }).toString().trim());const seconds = Math.round((Date.now() - started) / 100) / 10;const items = [...scan.findings.map(f => ` ${f.commit.slice(0, 7)} ${f.path}:${f.line} ${f.pattern} ${f.fingerprint}`),- ...scan.unscanned.map(u => ` ${u.commit.slice(0, 7)} ${u.path} too large to scan`), ...scan.gate_changes.map(g => ` ${g.commit.slice(0, 7)} ${g.path} changes what the scan sees`)];+ ...scan.unscanned.map(u => ` ${u.commit.slice(0, 7)} ${u.path} too large to scan`), ...scan.gate_changes.map(g => ` ${g.commit.slice(0, 7)} ${g.path} changes what the scan sees`),+ ...(scan.allowed ?? []).map(a => ` ${a.commit.slice(0, 7)} ${a.path}:${a.line} ${a.pattern} allowed by .gbrain-scan-allow line: ${a.entry}`),+ ...(scan.allowlist_problems ?? []).map(p => ` .gbrain-scan-allow line ${p.entry}: ${p.problem}`)];emit(p.json, { repo, from, to, commits, seconds, would_refuse: refusal?.code ?? null, findings: scan.findings, unscanned: scan.unscanned, gate_changes: scan.gate_changes,- unreadable: scan.unreadable ?? null }, [`scanned ${commits} commit(s) ${from.slice(0, 12)}..${to.slice(0, 12)} in ${seconds} s: ${refusal ? `a push would refuse (${refusal.code})` : 'a push would not refuse'}`,+ allowed: scan.allowed ?? [], allowlist_problems: scan.allowlist_problems ?? [], unreadable: scan.unreadable ?? null }, [`scanned ${commits} commit(s) ${from.slice(0, 12)}..${to.slice(0, 12)} in ${seconds} s: ${refusal ? `a push would refuse (${refusal.code})` : 'a push would not refuse'}`,...(scan.unreadable !== undefined ? [` not read in full: ${scan.unreadable}`] : []), ...items].join('\n'));return 0;}diff --git a/src/core/bootstrap/repo.ts b/src/core/bootstrap/repo.tsindex 960b0ea7fb..af56fdf073 100644--- a/src/core/bootstrap/repo.ts+++ b/src/core/bootstrap/repo.ts@@ -38,7 +38,7 @@ import { invalidateBackupStatus } from '../backup/status-file.ts';import { realpathOrResolve } from '../path-confine.ts';import { defaultRunner, isProxyBlocked403, parseGithubOwnerRepo, type ExecRunner } from '../repo-visibility.ts';import { detectExecutionEnvironment } from '../execution-env.ts';-import { stagedScanPublishRefusal } from '../git/range-secret-scan.ts';+import { scanAllowProblems, stagedScanPublishRefusal } from '../git/range-secret-scan.ts';import type { OperationError } from '../ops/contract.ts';import { loadWorkspaceAllowlist, scanFiles, SCAN_ALLOW_FILENAME, type SecretFinding } from '../secret-scan.ts';import { SECRET_SCAN_REFUSAL_DOCS } from '../workspace-push.ts';@@ -429,7 +429,8 @@ function secretScanOrThrow(workspaceDir: string, relFiles: string[]): void {`secret scan found ${findings.length} finding(s) before the first push: ${sample}` +`${findings.length > 5 ? '; …' : ''} — nothing was committed or pushed. ` +'Remove a real credential first; allowlist only a reviewed false positive by appending its fingerprint to ' +- `${shellQuote(resolve(workspaceDir, SCAN_ALLOW_FILENAME))} (${SECRET_SCAN_REFUSAL_DOCS}), then re-run \`gbrain bootstrap repo\`.`,+ `${shellQuote(resolve(workspaceDir, SCAN_ALLOW_FILENAME))} (${SECRET_SCAN_REFUSAL_DOCS}), then re-run \`gbrain bootstrap repo\`.` ++ scanAllowProblems(allowlist).map((p) => ` ${SCAN_ALLOW_FILENAME} line ${p.entry}: ${p.problem}.`).join(''),{ details: { findings: findings.length } },);}diff --git a/src/core/bootstrap/verify.ts b/src/core/bootstrap/verify.tsindex 0bfe3143ae..8217a8daac 100644--- a/src/core/bootstrap/verify.ts+++ b/src/core/bootstrap/verify.ts@@ -37,7 +37,7 @@ import { maintenanceAttribution } from '../persistence/attribution.ts';import { existsSync, mkdirSync, readdirSync, readFileSync, renameSync, rmSync, statSync, writeFileSync } from 'node:fs';import { createHash, randomUUID } from 'node:crypto';import { execFileSync } from '../spawn.ts';-import { join, resolve } from 'node:path';+import { join, relative, resolve } from 'node:path';import type { BrainEngine } from '../engine.ts';import { operations, type Operation, type OperationContext } from '../operations.ts';@@ -48,6 +48,7 @@ import { realpathOrResolve } from '../path-confine.ts';import { runMaintenanceSweep } from '../sweep.ts';import { detectCapabilities, renderCapabilityReport, type CapabilityReport } from '../capability.ts';import { loadWorkspaceAllowlist, matchesGlob, SCAN_ALLOW_FILENAME, scanFiles, type SecretFinding } from '../secret-scan.ts';+import { scanAllowProblems } from '../git/range-secret-scan.ts';import {PUSH_DENY_GLOBS, SECRET_SCAN_REFUSAL_DOCS, verifyRemotePrivacy, readPushStatuses, summarizePushStatuses,} from '../workspace-push.ts';@@ -374,10 +375,13 @@ function checkSecretScan(ws: string): VerifyCheck {try {const { files, via } = trackedWorkspaceFiles(ws);const allowlist = loadWorkspaceAllowlist(ws);+ const allowed: string[] = [];const findings: SecretFinding[] = scanFiles(files.map((f) => join(ws, f)), {allowlist,workspaceRoot: ws,+ onAllowed: (hit) => allowed.push(`${relative(ws, hit.file)}:${hit.line} [${hit.pattern}] by ${hit.entry}`),});+ const allowedNote = allowed.length ? `; ${allowed.length} hit(s) allowed by ${SCAN_ALLOW_FILENAME} rule lines: ${allowed.slice(0, 5).join('; ')}${allowed.length > 5 ? '; …' : ''}` : '';if (findings.length > 0) {const sample = findings.slice(0, 5).map((f) => {const { since } = f as SecretFinding & { since?: string };@@ -387,10 +391,11 @@ function checkSecretScan(ws: string): VerifyCheck {id, ok: false,detail: `${findings.length} secret-scan finding(s): ${sample}${findings.length > 5 ? '; …' : ''} — ` +'remove a real credential first; allowlist only a reviewed false positive by appending its fingerprint to ' +- `${shellQuote(resolve(ws, SCAN_ALLOW_FILENAME))} (${SECRET_SCAN_REFUSAL_DOCS})`,+ `${shellQuote(resolve(ws, SCAN_ALLOW_FILENAME))} (${SECRET_SCAN_REFUSAL_DOCS})` ++ scanAllowProblems(allowlist).map((p) => `; ${SCAN_ALLOW_FILENAME} line ${p.entry}: ${p.problem}`).join(''),};}- return { id, ok: true, detail: `no secrets in ${files.length} ${via === 'git' ? 'tracked' : 'workspace'} file(s)` };+ return { id, ok: true, detail: `no secrets in ${files.length} ${via === 'git' ? 'tracked' : 'workspace'} file(s)${allowedNote}` };} catch (e) {return { id, ok: false, detail: `secret scan failed: ${(e as Error).message}` };}diff --git a/src/core/git/range-secret-scan.ts b/src/core/git/range-secret-scan.tsindex 671cdaf74a..edee44ecbe 100644--- a/src/core/git/range-secret-scan.ts+++ b/src/core/git/range-secret-scan.ts@@ -29,8 +29,8 @@ import { readLedger } from './commit-ledger.ts';import { GIT_ENV } from '../git-transport.ts';import { opError, type OperationError } from '../ops/contract.ts';import { shellQuote } from '../agent-output.ts';-import { loadWorkspaceAllowlist, pathAllowlisted, SCAN_ALLOW_FILENAME } from '../secret-scan-allowlist.ts';-import { scanText, SCAN_MAX_FILE_BYTES } from '../secret-scan.ts';+import { allowlistProblems, loadWorkspaceAllowlist, pathAllowlisted, SCAN_ALLOW_FILENAME, withoutRuleAllowed, type AllowlistProblem } from '../secret-scan-allowlist.ts';+import { NAMED_SECRET_RULES, scanText, SCAN_MAX_FILE_BYTES } from '../secret-scan.ts';import { spawnSync } from '../spawn.ts';export interface RangeSecretFinding { path: string; commit: string; line: number; pattern: string; fingerprint: string }@@ -40,6 +40,15 @@ export interface RangeScan {unreadable?: string;/** Set with `unreadable` when the reason is the time budget, so a caller with a short budget can scan again elsewhere. */timed_out?: true;+ /** Hits a `path:<glob> pattern:<rule>` entry allowed, each with that entry (never the value). */+ allowed?: Array<{ path: string; commit: string; line: number; pattern: string; entry: string }>;+ /** `.gbrain-scan-allow` entries that allow nothing, or other than they seem to (scanAllowProblems). */+ allowlist_problems?: AllowlistProblem[];+}++/** Malformed rule entries and unknown rule names in a loaded `.gbrain-scan-allow`. */+export function scanAllowProblems(allowlist: string[]): AllowlistProblem[] {+ return allowlistProblems(allowlist, [...NAMED_SECRET_RULES, 'high_entropy_assignment', 'labeled_credential'].sort());}/** The commit message as `path` in a finding. */@@ -149,12 +158,18 @@ export function scanRange(root: string, base: string, head: string, timeoutMs =};const scan: RangeScan = { findings: [], unscanned: [], gate_changes: [] };const allowlist = loadWorkspaceAllowlist(root);+ const problems = scanAllowProblems(allowlist);+ if (problems.length) scan.allowlist_problems = problems;let allowFile = SCAN_ALLOW_FILENAME, prefix = '';const allowed = (path: string) => path.startsWith(prefix) && pathAllowlisted(path.slice(prefix.length), allowlist);const scanned = (path: string, commit: string, text: string, lineAt: (line: number) => number = line => line) => {if (Date.now() > deadline) throw new Unreadable('the scan ran past its time budget', true);if (Buffer.byteLength(text) > SCAN_MAX_FILE_BYTES) { scan.unscanned.push({ path, commit }); return; }- for (const f of scanText(text, { allowlist })) scan.findings.push({ path, commit, line: lineAt(f.line), pattern: f.pattern, fingerprint: f.fingerprint });+ const found = scanText(text, { allowlist });+ const kept = path !== MESSAGE_PATH && path.startsWith(prefix)+ ? withoutRuleAllowed(found, path.slice(prefix.length), allowlist, hit => (scan.allowed ??= []).push({ path, commit, line: lineAt(hit.line), pattern: hit.pattern, entry: hit.entry }))+ : found;+ for (const f of kept) scan.findings.push({ path, commit, line: lineAt(f.line), pattern: f.pattern, fingerprint: f.fingerprint });};const dir = mkdtempSync(join(tmpdir(), 'gbrain-range-scan-'));try {@@ -384,8 +399,8 @@ export function rangeSecretRefusal(root: string, base: string, target: string, s...scan.gate_changes.map(g => `${g.path} (changes what the scan sees, ${g.commit.slice(0, 7)})`)];return opError('publish_secret_found',`${named.length} item(s) in the local commits ${range} look like secrets or could not be scanned: ${named.slice(0, 10).join(', ')}${named.length > 10 ? ', ...' : ''}. Nothing was pushed.`,- `Remove the value from those commits (rewriting unpushed local commits is the user's call), or add a false positive's fingerprint to .gbrain-scan-allow, then retry. Never publish them with a command that skips this scan.${base ? '' : ' The remote has no such branch, so this covered the branch\'s whole history, which may include commits published elsewhere before: review it, and push the branch yourself once it is safe.'}${scan.gate_changes.length ? ` A commit that changes .gbrain-scan-allow or a .gitattributes is never published automatically, because it changes what the scan sees: review it and push it yourself (git -C ${root} push), then retry.` : ''}`,- { detail: JSON.stringify(scan),+ `Remove the value from those commits (rewriting unpushed local commits is the user's call), or add a false positive's fingerprint to .gbrain-scan-allow (or, for a rule that misfires on a whole kind of file, a path:<glob> pattern:<rule> line), then retry. Never publish them with a command that skips this scan.${scan.allowlist_problems?.length ? ` Check these .gbrain-scan-allow lines: ${scan.allowlist_problems.map(p => `${p.entry} (${p.problem})`).join('; ')}.` : ''}${base ? '' : ' The remote has no such branch, so this covered the branch\'s whole history, which may include commits published elsewhere before: review it, and push the branch yourself once it is safe.'}${scan.gate_changes.length ? ` A commit that changes .gbrain-scan-allow or a .gitattributes is never published automatically, because it changes what the scan sees: review it and push it yourself (git -C ${root} push), then retry.` : ''}`,+ { detail: JSON.stringify(scan.allowed && scan.allowed.length > 10 ? { ...scan, allowed: scan.allowed.slice(0, 10), allowed_total: scan.allowed.length } : scan),fix: { ask_user: true, consent: [], actor: 'user', requires_exclusive: false,argv: ['git', '-C', root, 'log', '--stat', base ? `${base}..${target}` : target],why: 'Lists the unpublished commits the scan refused; read-only. A secret pushed to the remote stays in its history.',diff --git a/src/core/secret-scan-allowlist.ts b/src/core/secret-scan-allowlist.tsindex 37d40787bb..8f56f83db0 100644--- a/src/core/secret-scan-allowlist.ts+++ b/src/core/secret-scan-allowlist.ts@@ -112,7 +112,60 @@ export function valueAllowlisted(fullHex: string, allowlist: string[]): booleanreturn false;}+/**+ * A rule entry, `path:<glob> pattern:<rule>`, allows one named rule in the files the glob matches (the same glob+ * dialect as a path entry); every other rule still applies there. One rule per line. A line that starts with+ * `pattern:`, or with `path:` and holds whitespace, is read only as a rule entry: when it is malformed or names an+ * unknown rule it allows nothing and `allowlistProblems` reports it.+ */+const RULE_ENTRY_RE = /^path:(\S+)\s+pattern:([a-z0-9_]+)$/;++function isRuleEntry(entry: string): boolean {+ return entry.startsWith('pattern:') || (entry.startsWith('path:') && /\s/.test(entry));+}+/** True when a file path is allowlisted by any glob entry. */export function pathAllowlisted(relPath: string, allowlist: string[]): boolean {- return allowlist.some((entry) => !isFingerprintEntry(entry) && matchesGlob(entry, relPath));+ return allowlist.some((entry) => !isFingerprintEntry(entry) && !isRuleEntry(entry) && matchesGlob(entry, relPath));+}++/** The rule entry that allows `pattern` in `relPath`, or null. */+export function ruleAllowEntry(relPath: string, pattern: string, allowlist: string[]): string | null {+ for (const entry of allowlist) {+ const m = RULE_ENTRY_RE.exec(entry);+ if (m && m[2] === pattern && matchesGlob(m[1]!, relPath)) return entry;+ }+ return null;+}++/** A finding a rule entry allowed, with the entry that allowed it; never the matched value. */+export interface AllowedHit { pattern: string; line: number; fingerprint: string; entry: string }++/** `findings` of the file at `relPath` without those a rule entry allows; each allowed one goes to `onAllowed`. */+export function withoutRuleAllowed<F extends { pattern: string; line: number; fingerprint: string }>(+ findings: F[], relPath: string, allowlist: string[], onAllowed?: (hit: AllowedHit) => void,+): F[] {+ return findings.filter((f) => {+ const entry = ruleAllowEntry(relPath, f.pattern, allowlist);+ if (entry === null) return true;+ onAllowed?.({ pattern: f.pattern, line: f.line, fingerprint: f.fingerprint, entry });+ return false;+ });+}++export interface AllowlistProblem { entry: string; problem: string }++/** Entries that allow nothing, or something other than they seem to: malformed rule entries and unknown rule names. */+export function allowlistProblems(allowlist: string[], knownRules: readonly string[]): AllowlistProblem[] {+ const problems: AllowlistProblem[] = [];+ for (const entry of allowlist) {+ if (isRuleEntry(entry)) {+ const m = RULE_ENTRY_RE.exec(entry);+ if (!m) problems.push({ entry, problem: 'malformed rule entry, so it allows nothing: write path:<glob> pattern:<rule>, one rule per line' });+ else if (!knownRules.includes(m[2]!)) problems.push({ entry, problem: `unknown rule ${m[2]}, so it allows nothing; the rules are ${knownRules.join(', ')}` });+ } else if (entry.startsWith('path:')) {+ problems.push({ entry, problem: 'read as a path glob for files whose name starts with path:, which skips every rule for them; to allow one rule, add a space and pattern:<rule>' });+ }+ }+ return problems;}diff --git a/src/core/secret-scan.ts b/src/core/secret-scan.tsindex f1fb2aabe1..da6890ef57 100644--- a/src/core/secret-scan.ts+++ b/src/core/secret-scan.ts@@ -82,6 +82,8 @@ export interface ScanOpts {allowlist?: string[];/** Workspace root: allowlist path-globs match paths relative to this. */workspaceRoot?: string;+ /** scanFiles: each finding a `path:<glob> pattern:<rule>` entry allowed, with that entry and its file. */+ onAllowed?: (hit: AllowedHit & { file: string }) => void;/** Opt-in generic high-entropy assignment heuristic. Default false. */highEntropy?: boolean;/**@@ -104,7 +106,7 @@ export interface ScanOpts {export {SCAN_ALLOW_FILENAME, ALLOWLIST_FINGERPRINT_MIN_HEX, loadWorkspaceAllowlist, globToRegExp, matchesGlob, valueAllowlisted, pathAllowlisted,} from './secret-scan-allowlist.ts';-import { valueAllowlisted, pathAllowlisted } from './secret-scan-allowlist.ts';+import { valueAllowlisted, pathAllowlisted, withoutRuleAllowed, type AllowedHit } from './secret-scan-allowlist.ts';import { PUSH_SCAN_MAX_FILE_BYTES } from './page-size-policy.ts';import {ASSIGNMENT_QUOTED_SOURCE, ASSIGNMENT_UNQUOTED_SOURCE, HIGH_ENTROPY_MIN_BITS_PER_CHAR, HIGH_ENTROPY_REQUIRES_DIGIT_RE,@@ -1186,7 +1188,7 @@ export function scanFiles(paths: string[], opts: ScanOpts = {}): SecretFinding[]} catch {continue;}- for (const f of scanText(text, opts)) {+ for (const f of withoutRuleAllowed(scanText(text, opts), rel, allowlist, (hit) => opts.onAllowed?.({ ...hit, file: p }))) {out.push({ ...f, file: p });}}diff --git a/src/core/workspace-push.ts b/src/core/workspace-push.tsindex d886e5b6d7..f8d613756d 100644--- a/src/core/workspace-push.ts+++ b/src/core/workspace-push.ts@@ -76,10 +76,11 @@ import {type SecretFinding,} from './secret-scan.ts';import { scanTextStream } from './secret-scan-stream.ts';+import { withoutRuleAllowed, type AllowlistProblem } from './secret-scan-allowlist.ts';import { shellQuote } from './mcp-registration.ts';import { gitChildEnv } from './git-env.ts';import { recordCommit, recordIntent } from './git/commit-ledger.ts';-import { stagedScanPublishRefusal } from './git/range-secret-scan.ts';+import { scanAllowProblems, stagedScanPublishRefusal } from './git/range-secret-scan.ts';// ── Types ───────────────────────────────────────────────────────────────────@@ -113,6 +114,8 @@ export interface WorkspacePushResult {findings?: PushSecretFinding[];/** Staged paths the secret scan could not read (fail-closed block reason). */unscannable?: string[];+ /** blocked_secrets: `.gbrain-scan-allow` lines that allow nothing (a malformed rule entry, an unknown rule name). */+ allowlistProblems?: AllowlistProblem[];/** blocked_history: the refusal's code, what to do (a longer `--scan-timeout` after a timeout) and its findings JSON (never a value). */code?: string;suggestion?: string;@@ -1081,6 +1084,8 @@ export async function workspacePush(opts: WorkspacePushOpts): Promise<WorkspacePconst allowlist = loadWorkspaceAllowlist(root);// nosemgrep: javascript.lang.security.audit.path-traversal.path-join-resolve-traversal.path-join-resolve-traversal -- fixed filename under the resolved workspace root (same path loadWorkspaceAllowlist reads); used only in printed guidance.const allowlistPath = join(root, SCAN_ALLOW_FILENAME);+ const allowlistProblems = scanAllowProblems(allowlist);+ for (const p of allowlistProblems) log(`${SCAN_ALLOW_FILENAME} line ${p.entry}: ${p.problem}`);const retryCommand = pushRetryCommand(opts);const findings: PushSecretFinding[] = [];const unscannable: string[] = [];@@ -1093,8 +1098,9 @@ export async function workspacePush(opts: WorkspacePushOpts): Promise<WorkspacePunscannable.push(fileFindings);continue;}- if (fileFindings.length === 0) continue;- for (const f of buildPushFindings({ file: rel, findings: fileFindings, allowlist, allowlistPath, retryCommand })) {+ const blocking = withoutRuleAllowed(fileFindings, rel, allowlist, hit => log(`${rel}:${hit.line} ${hit.pattern} allowed by ${SCAN_ALLOW_FILENAME} line: ${hit.entry}`));+ if (blocking.length === 0) continue;+ for (const f of buildPushFindings({ file: rel, findings: blocking, allowlist, allowlistPath, retryCommand })) {findings.push(f);}}@@ -1106,6 +1112,7 @@ export async function workspacePush(opts: WorkspacePushOpts): Promise<WorkspacePlog(`PUSH BLOCKED: ${reason}`);return finish({ok: false, status: 'blocked_secrets', repoRoot: root, branch, findings, reason,+ ...(allowlistProblems.length ? { allowlistProblems } : {}),});}if (unscannable.length > 0) {diff --git a/test/secret-scan-rule-allow-2901.test.ts b/test/secret-scan-rule-allow-2901.test.tsnew file mode 100644index 0000000000..5f0aea4df1--- /dev/null+++ b/test/secret-scan-rule-allow-2901.test.ts@@ -0,0 +1,142 @@+/**+ * GST-2901: a `.gbrain-scan-allow` line `path:<glob> pattern:<rule>` allows one rule in the files the glob matches and+ * leaves every other rule active there. Path globs and fingerprints keep their meaning; a malformed line or an unknown+ * rule name allows nothing and is reported; the scan output names the line that allowed a hit. Synthetic values only,+ * assembled at runtime.+ */+import { afterEach, describe, expect, test } from 'bun:test';+import { execFileSync } from 'node:child_process';+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import { allowlistProblems } from '../src/core/secret-scan-allowlist.ts';+import { loadWorkspaceAllowlist, NAMED_SECRET_RULES, SCAN_ALLOW_FILENAME, scanFiles, type SecretFinding } from '../src/core/secret-scan.ts';+import { rangeSecretRefusal, scanRange } from '../src/core/git/range-secret-scan.ts';++const JWT = ['eyJhbGciOiJIUzI1NiJ9', 'eyJyb2xlIjoic2VydmljZV9yb2xlIiwiaWF0IjoxNzAwMDAwMDAwfQ', 'c2lnbmF0dXJlLXBsYWNlaG9sZGVyLTAwMDA'].join('.');+const AWS = 'AKIA' + 'IOSFODNN7EXAMPLE';+const MAILER = `<a href="https://mail.example.invalid/unsubscribe?t=${JWT}">Unsubscribe</a>\n`;+const KNOWN = [...NAMED_SECRET_RULES, 'high_entropy_assignment', 'labeled_credential'];++const roots: string[] = [];+afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });++function workspace(allow: string, files: Record<string, string>): string {+ const root = mkdtempSync(join(tmpdir(), 'gbrain-scan-rule-'));+ roots.push(root);+ writeFileSync(join(root, SCAN_ALLOW_FILENAME), allow);+ for (const [rel, text] of Object.entries(files)) {+ mkdirSync(join(root, rel, '..'), { recursive: true });+ writeFileSync(join(root, rel), text);+ }+ return root;+}+function scan(root: string, rels: string[]) {+ const allowed: Array<{ file: string; pattern: string; entry: string }> = [];+ const findings = scanFiles(rels.map(rel => join(root, rel)), { allowlist: loadWorkspaceAllowlist(root), workspaceRoot: root,+ onAllowed: hit => allowed.push({ file: hit.file.slice(root.length + 1), pattern: hit.pattern, entry: hit.entry }) });+ return { found: findings.map((f: SecretFinding) => `${f.file!.slice(root.length + 1)}:${f.pattern}`).sort(), allowed };+}++describe('path:<glob> pattern:<rule>', () => {+ test('allows only the named rule, only in matching files, and names the line that allowed it', () => {+ const root = workspace('path:mail/*.html pattern:jwt\n', {+ 'mail/digest.html': MAILER + `aws ${AWS}\n`,+ 'mail/sub/receipt.html': MAILER,+ 'notes/plan.md': MAILER,+ });+ const r = scan(root, ['mail/digest.html', 'mail/sub/receipt.html', 'notes/plan.md']);+ expect(r.found).toEqual(['mail/digest.html:aws_access_key', 'mail/sub/receipt.html:jwt', 'notes/plan.md:jwt']);+ expect(r.allowed).toEqual([{ file: 'mail/digest.html', pattern: 'jwt', entry: 'path:mail/*.html pattern:jwt' }]);+ });++ test('a glob with no slash matches the file name at any depth, like a path entry', () => {+ const root = workspace('path:*.html pattern:jwt\n', { 'a/b/one.html': MAILER, 'two.md': MAILER });+ expect(scan(root, ['a/b/one.html', 'two.md']).found).toEqual(['two.md:jwt']);+ });+});++describe('existing entries keep their meaning', () => {+ test('a path glob still skips every rule in the file and a fingerprint still allows one value anywhere', () => {+ const glob = workspace('mail/*.html\n', { 'mail/digest.html': MAILER + `aws ${AWS}\n` });+ expect(scan(glob, ['mail/digest.html']).found).toEqual([]);+ const first = scan(workspace('', { 'x.md': MAILER }), ['x.md']);+ expect(first.found).toEqual(['x.md:jwt']);+ const fingerprintRoot = workspace('', { 'x.md': MAILER, 'y.md': MAILER });+ const [finding] = scanFiles([join(fingerprintRoot, 'x.md')], { allowlist: [], workspaceRoot: fingerprintRoot });+ writeFileSync(join(fingerprintRoot, SCAN_ALLOW_FILENAME), `${finding!.fingerprint}\n`);+ expect(scan(fingerprintRoot, ['x.md', 'y.md']).found).toEqual([]);+ expect(allowlistProblems(['mail/*.html', finding!.fingerprint, '*.pem', 'fixtures/**'], KNOWN)).toEqual([]);+ });+});++describe('a malformed line or an unknown rule allows nothing and is reported', () => {+ const cases = [+ ['path:mail/*.html pattern:jwtt', /unknown rule jwtt/],+ ['path:mail/*.html pattern:jwt,bearer', /malformed rule entry/],+ ['path:mail/*.html pattern:jwt bearer', /malformed rule entry/],+ ['path:mail/*.html pattern:', /malformed rule entry/],+ ['pattern:jwt', /malformed rule entry/],+ ['path:mail/*.html', /read as a path glob for files whose name starts with path:/],+ ] as const;+ for (const [entry, problem] of cases) {+ test(entry, () => {+ const root = workspace(`${entry}\n`, { 'mail/digest.html': MAILER + `aws ${AWS}\n` });+ expect(scan(root, ['mail/digest.html']).found).toEqual(['mail/digest.html:aws_access_key', 'mail/digest.html:jwt']);+ const problems = allowlistProblems([entry], KNOWN);+ expect(problems).toHaveLength(1);+ expect(problems[0]!.problem).toMatch(problem);+ });+ }+});++describe('the publish scan', () => {+ const git = (cwd: string, ...args: string[]) => execFileSync('git', ['-C', cwd, ...args], { env: { ...process.env, GIT_AUTHOR_NAME: 'Example', GIT_AUTHOR_EMAIL: '[email protected]',+ GIT_COMMITTER_NAME: 'Example', GIT_COMMITTER_EMAIL: '[email protected]' } }).toString().trim();+ function repo(allow: string) {+ const root = mkdtempSync(join(tmpdir(), 'gbrain-scan-rule-repo-'));+ roots.push(root);+ git(root, 'init', '-q', '-b', 'main');+ writeFileSync(join(root, SCAN_ALLOW_FILENAME), allow);+ git(root, 'add', '.');+ git(root, 'commit', '-q', '-m', 'allowlist');+ const base = git(root, 'rev-parse', 'HEAD');+ mkdirSync(join(root, 'mail'));+ writeFileSync(join(root, 'mail/digest.html'), MAILER);+ writeFileSync(join(root, 'notes.md'), MAILER);+ git(root, 'add', '.');+ git(root, 'commit', '-q', '-m', 'mailers');+ return { root, base, head: git(root, 'rev-parse', 'HEAD') };+ }++ test('a rule entry allows the hit in matching files, records the line that allowed it, and still refuses elsewhere', () => {+ const { root, base, head } = repo('path:mail/*.html pattern:jwt\n');+ const result = scanRange(root, base, head);+ expect(result.findings.map(f => `${f.path}:${f.pattern}`)).toEqual(['notes.md:jwt']);+ expect(result.allowed).toEqual([{ path: 'mail/digest.html', commit: head, line: 1, pattern: 'jwt', entry: 'path:mail/*.html pattern:jwt' }]);+ expect(result.allowlist_problems).toBeUndefined();+ });++ test('an unknown rule name allows nothing, and the refusal names the line', () => {+ const { root, base, head } = repo('path:mail/*.html pattern:jwtt\n');+ const result = scanRange(root, base, head);+ expect(result.findings.map(f => `${f.path}:${f.pattern}`).sort()).toEqual(['mail/digest.html:jwt', 'notes.md:jwt']);+ expect(result.allowed).toBeUndefined();+ expect(result.allowlist_problems?.map(p => p.entry)).toEqual(['path:mail/*.html pattern:jwtt']);+ expect(rangeSecretRefusal(root, base, head, result)!.suggestion).toContain('Check these .gbrain-scan-allow lines: path:mail/*.html pattern:jwtt (unknown rule jwtt, so it allows nothing');+ });++ test('the refusal detail keeps the first 10 allowed hits and their total, however many a rule entry allowed', () => {+ const { root, base } = repo('path:mail/*.html pattern:jwt\n');+ writeFileSync(join(root, 'mail/archive.html'), MAILER.repeat(30));+ git(root, 'add', '.');+ git(root, 'commit', '-q', '-m', 'archive');+ const head = git(root, 'rev-parse', 'HEAD');+ const result = scanRange(root, base, head);+ expect(result.allowed).toHaveLength(31);+ const detail = JSON.parse(rangeSecretRefusal(root, base, head, result)!.detail!);+ expect(detail.findings.map((f: { path: string }) => f.path)).toEqual(['notes.md']);+ expect(detail.allowed).toHaveLength(10);+ expect(detail.allowed_total).toBe(31);+ });+});diff --git a/test/workspace-push.serial.test.ts b/test/workspace-push.serial.test.tsindex 3815e81a83..70f1157924 100644--- a/test/workspace-push.serial.test.ts+++ b/test/workspace-push.serial.test.ts@@ -304,6 +304,35 @@ describe('secret-scan gate', () => {writeFileSync(join(work, SCAN_ALLOW_FILENAME), `${r.findings![0]!.fingerprint}\n`);expect((await push()).status).toBe('pushed');}, T);++ const JWT = ['eyJhbGciOiJIUzI1NiJ9', 'eyJyb2xlIjoic2VydmljZV9yb2xlIiwiaWF0IjoxNzAwMDAwMDAwfQ', 'c2lnbmF0dXJlLXBsYWNlaG9sZGVyLTAwMDA'].join('.');+ const MAILER = `<a href="https://mail.example.invalid/unsubscribe?t=${JWT}">Unsubscribe</a>\n`;++ test('GST-2901: a path:<glob> pattern:<rule> line allows that rule in matching files only, and the log names the line', async () => {+ writeFileSync(join(work, SCAN_ALLOW_FILENAME), 'path:mail/*.html pattern:jwt\n');+ mkdirSync(join(work, 'mail'));+ writeFileSync(join(work, 'mail/digest.html'), MAILER);+ writeFileSync(join(work, 'notes.md'), MAILER);+ const lines: string[] = [];+ const r = await push({ logger: (l: string) => lines.push(l) });+ expect(r.status).toBe('blocked_secrets');+ expect(r.findings?.map(f => `${f.file}:${f.pattern}`)).toEqual(['notes.md:jwt']);+ expect(r.allowlistProblems).toBeUndefined();+ expect(lines).toContain(`mail/digest.html:1 jwt allowed by ${SCAN_ALLOW_FILENAME} line: path:mail/*.html pattern:jwt`);+ rmSync(join(work, 'notes.md'));+ expect((await push()).status).toBe('pushed');+ }, T);++ test('GST-2901: a malformed rule line allows nothing, and the blocked_secrets result names it', async () => {+ writeFileSync(join(work, SCAN_ALLOW_FILENAME), 'path:mail/*.html pattern:JWT\n');+ mkdirSync(join(work, 'mail'));+ writeFileSync(join(work, 'mail/digest.html'), MAILER);+ const r = await push();+ expect(r.status).toBe('blocked_secrets');+ expect(r.findings?.map(f => `${f.file}:${f.pattern}`)).toEqual(['mail/digest.html:jwt']);+ expect(r.allowlistProblems?.map(p => p.entry)).toEqual(['path:mail/*.html pattern:JWT']);+ expect(r.allowlistProblems?.[0]?.problem).toMatch(/malformed rule entry, so it allows nothing/);+ }, T);});describe('secret-scan refusal guidance (DX-3/ENG-11)', () => {