Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Redact multi-line .netrc passwords, continued curl -b cookies and escaped cookie quotes in transcripts (GST-2768) (#9418)

Garry Tanc442f944ad43
diff --git a/changelog.d/transcript-redaction-gaps.md b/changelog.d/transcript-redaction-gaps.md
new file mode 100644
index 0000000000..15715ffd5e
--- /dev/null
+++ b/changelog.d/transcript-redaction-gaps.md
@@ -0,0 +1,17 @@
+---
+important: security
+---
+**Imported conversations no longer keep three kinds of secret that the last cookie and `.netrc` fix still let through.** `gbrain transcripts ingest` now also redacts a `.netrc` password when the entry is written over several lines (`machine`, `login` and `password` each on its own line), a curl cookie passed with `-b` on its own line of a command split with trailing backslashes (what Chrome's "Copy as cURL" produces), and a quoted cookie value inside escaped JSON (`sid=\"…\"`). Before this change all three were copied into the searchable page as plain text. Cookie names, curl's other arguments and ordinary prose near these shapes stay as they were.
+
+What to do: conversations imported earlier keep what they stored. `gbrain doctor` asks for a fresh audit; run `gbrain transcripts audit-secrets --json` to list affected pages (slugs and hit counts, never values). The `sources push` gate and the compiled-context scan are unchanged.
+
+Still not caught: a curl cookie written with bash's `$'…'` quoting (`-b $'sid=…'`, which Chrome's "Copy as cURL (bash)" uses for a cookie holding a quote or non-ASCII text); a `.netrc` entry or curl command split across two messages; and a cookie header whose opening and closing escaped quotes differ in depth (`sid=\\\"…\"`), where that value and the pairs after it stay. In the other direction, a line that is only `default` or `machine <word>` followed by a line that is only `password <word>` has that word redacted even in prose.
+
+### Itemized changes
+
+- `src/core/secret-scan-labeled.ts`: a `password` line inside a multi-line `.netrc` stanza (opened by a `machine` or `default` line, kept open by `login`, `account` and `password` lines, blank lines and `#` comments) is redacted. The line must be only `password <value>`, so prose such as "machine learning" followed by "password managers are useful" stays.
+- `src/core/secret-scan-cookie.ts`: a `-b` on a line that continues a curl command (each earlier line ends in `\`) counts as curl's cookie argument; a cookie value quoted with escaped quotes (`\"`, or `\\\"` one level deeper) is read. A `-b` after a continued line that is not curl (`git checkout -b`) stays.
+- `src/core/secret-scan.ts`: the transcript-lane scan carries the `.netrc` stanza and the curl continuation from line to line.
+- `src/core/transcripts/secret-audit.ts`: `TRANSCRIPT_SECRET_AUDIT_VERSION` 6 → 7, because the audit now finds these shapes on stored pages.
+- Tests: `test/transcripts-secret-cookie-header.test.ts`, `test/transcripts-secret-corpus.test.ts`.
+- Docs: `docs/guides/data-ingestion.md#credential-redaction` and the key-files entries.
diff --git a/docs/architecture/key-files/secret-scan.md b/docs/architecture/key-files/secret-scan.md
index bafc148335..eb759b9d06 100644
--- a/docs/architecture/key-files/secret-scan.md
+++ b/docs/architecture/key-files/secret-scan.md
@@ -7,6 +7,7 @@ compiled-context sensitivity scan and the transcripts redactor, and the composed
sensitivity detector built on it.
- `src/core/secret-scan.ts` — pattern-based secret detection shared by `sources push`, bootstrap verify, the compiled-context sensitivity scan and the transcripts redactor. Detection is FORMAT-based, not prefix-only: vendor shapes (anthropic, openai, voyage, github, gitlab, slack, aws AKIA/ASIA, google, stripe, sendgrid, twilio, supabase sb_secret_/sbp_, npm, huggingface, `digitalocean` `do[opr]_v1_` + 64 hex, the AI-provider keys `openrouter` `sk-or-v1-` + 64 lowercase hex, `groq` `gsk_` + 52, `xai` `xai-` + 80, `perplexity` `pplx-` + 48 and `replicate` `r8_` + 37, all fixed-length with a hard right edge, gbrain), JWTs, private keys (whole-text claims, below), plus four catch-alls appended LAST (`bearer` — prebuilt two-group layout so only the token is the value, which may hold a JSON-escaped `\/` (#8266); `db_url_credentials` — the `scheme://user:pass@` span of database URLs, host/db kept; `url_credentials` — the same span for `http(s)` (scheme case-insensitive, `gi`), user 0–128 stopping at `:`/`/`, password 1–256 excluding whitespace, `@`, `"`, `'`, `/`, `?`, `#`, with a `validate` hook that rejects placeholder passwords (`<...>`, `${...}`, `$VAR`, all `*`, all `x`); `basic_auth` — two prebuilt case-insensitive forms, `Authorization: Basic` with a 4–2048 char value and bare `Basic` with 16–2048, both with a hard right edge and a `validate` that requires the value to base64-decode to printable ASCII containing `:`; it is NOT in the bearer family, so it joins neither the echo dictionary nor `secret:bearer`). The pattern contract: `CorePattern` carries optional `flags` (default `'g'`), `precheck`, `validate(value)` (O(value), run on the whole match before the claim and the union split; a rejected match is not re-tried from inside) and `since` (the gbrain version that added or last changed the rule; `SECURITY_WAVE_SINCE` = `0.60.31.0` on `digitalocean`, `url_credentials`, `basic_auth` and `private_key_pem`, `PROVIDER_KEYS_SINCE` on the five AI-provider rules (#7959), kept in step with the release that ships a rule change). Compiled patterns keep group 1 = anchor, group 2 = value. Every core rule and both `high_entropy_assignment` forms also have an escape-boundary twin (#7978), compiled right after the rule under the same name and anchored on a JSON escape (`\n`, `\r`, `\t`, `\b`, `\f`, `\uXXXX`, and since #8266 `\xNN`, `\UXXXXXXXX`), a percent-escape (`%XX`), a quoted-printable `=3D` or a terminal color code (`ESC[…m`, raw or written `\u001b[`/`\x1b[`/`\033[`/`\e[`), whose last character the default boundary reads as a word character; a twin runs only on a line `ESCAPE_IN_LINE_RE` matches, so other lines run the old list, and a prebuilt rule carries its twin as `CorePattern.escaped`. `patternSince(pattern)` reads the map; every finding of such a pattern carries `SecretFinding.since`, which the push and bootstrap refusals render. Fixed-length shapes carry a trailing negative lookahead. Per-line claimed-span dedupe is first-wins for ATTRIBUTION (order matters: `Bearer <vendor key>` keeps the vendor name, a fully covered later span is skipped) but coverage is the UNION: every unclaimed slice of a later overlapping span is claimed as a hit of the later pattern (own value, own fingerprint), so a database URL whose username is a vendor-shaped id still loses its password. The transcript-only `labeled_credential` rule (`ScanOpts.labeledCredentials`) lives in `secret-scan-labeled.ts` (Security). The opt-in `high_entropy_assignment` rule (`ScanOpts.highEntropy`; its shapes and gates live in `src/core/secret-scan-assignment.ts`, split out at the 1500-line module cap) is anchored on a credential keyword or on an identifier whose last word is `key` (`STRIPE_KEY=`, never `MONKEY=` or a bare `KEY=`) and requires a 12+ char value that contains a digit and a non-digit AND clears 3.5 bits/char on the bias-corrected (Miller-Madow) estimate `correctedEntropy`, since raw per-character entropy is capped at log2(n) and rejected most random 12-16 char secrets. It is two compiled patterns sharing the name: QUOTED (after `"` or `'`, any run of non-quote, non-whitespace characters) and UNQUOTED (the legacy `[A-Za-z0-9+/_=-]` class plus `!#$%*@^~.?<>`; never `&`, `()[]{}`, `:`, `,` or `;`, so query strings, calls and SQL `SET` lists stay intact, and a password carrying `,`/`;` matches only quoted; the last character excludes `.?!`, so a sentence's full stop is not claimed). Its `validate` rejects values that start with `scheme://`, `./`, `~/`, `${`, `$(` or `<REDACTED:`, dotted identifier paths (`process.env.X`) and `/`-paths holding a character outside the legacy class. Consumers with it on: retrieval output, transcript ingest, hooks and the hook heartbeat, the atoms drain, `embed-facts.ts` error text and the context-engine relay re-scan. Off: the push gate, bootstrap verify/repo and the compiled-context sensitivity scan. Own minimal allowlist + `<ws>/.gbrain-scan-allow` per-finding overrides — deliberately NOT the repo's `.gitleaks.toml` (a public-repo CI fixture policy); the allowlist block (`loadWorkspaceAllowlist`, the glob dialect `globToRegExp`/`matchesGlob`, `valueAllowlisted`/`pathAllowlisted`, `SCAN_ALLOW_FILENAME`, the 16-hex fingerprint floor) lives in `src/core/secret-scan-allowlist.ts` since wave 16 (G4.1, the 1500-line module cap) and is re-exported here unchanged (`test/secret-scan-allowlist.test.ts`). `correctedEntropy` is one histogram pass (distinct count = histogram size; byte-identical to the former `shannonEntropy + Set` form, `test/secret-scan-entropy-equivalence.test.ts`; ~80 → ~25 ms per 250 × 4096-char segment). `ScanOpts.work` / `RedactOpts.work` takes a `ScanWork` accumulator (`src/core/secret-scan-work.ts`: lines, regex execs, candidates per pattern, chars handed to the entropy gate counted inside `correctedEntropy`, validate calls, claims, union splits, echo sweeps) pinned exactly on the perf fixture by `test/secret-scan-work-counters.test.ts`; the perf file's absolute bounds stay the only guard on per-character cost. Deliberately NOT patterns: `sb_publishable_`, Supabase project refs (identifiers, not credentials). Never spell a literal database or `http(s)` URL with userinfo, or any credential, in this file's comments or in fixtures (assemble test values at runtime; `scripts/check-pg-url-redaction.sh` greps `src/` for every `db_url_credentials` scheme). Long lines are scanned in full (no length cap — a secret on a 1 MB minified line must still redact); what keeps that linear is bounded quantifiers on every shape whose value class could otherwise run to end-of-line — `db_url_credentials` (user ≤128 stopping at `:`/`/`; password 1–256 stopping only at whitespace, `@` and the string delimiters `"`/`'`, so unescaped `/ { } < > | ^ \` and backtick passwords redact), `jwt` (prebuilt so its boundary excludes `-` as well as `_`; segments ≤4096), `high_entropy_assignment` (≤64 identifier chars after the keyword, value ≤4096), `url_credentials` (user ≤128, password ≤256), `basic_auth` (value ≤2048, so its decode is constant work) and the private-key walks (below) — plus a per-pattern `precheck` substring gate on the catch-alls (`@` for both URL patterns; `earer`/`EARER`, since an all-caps `BEARER` scheme is accepted; a case-insensitive `basic`), a carried next-newline position in `scanPemBlocks` (sticky -1, so the header-line count is O(text) even when thousands of hits sit ahead of a newline-free multi-megabyte tail), a lazily-allocated per-line claimed-char bitmap behind the first-wins dedupe (O(value) per hit), and a windowed `buildPreview` (whole line ≤160 chars; otherwise hit−40…+80, edges snapped outward to whole spans, every claimed span in the window spliced by position, so a preview never carries a sibling secret or a fragment). `redactFindings` (= `planRedaction` + `applyRedaction`) rebuilds the text by SPAN-SPLICE in one O(text + hits) pass over the claimed spans, and fills the UNCLAIMED gaps with a bounded echo pass: the values claimed by `bearer`, `high_entropy_assignment`, `labeled_credential` and every vendor or `jwt` rule (#7978; not `basic_auth` or the two URL userinfo catch-alls) form an echo dictionary (value → pattern; floors `ECHO_MIN_CHARS_BEARER` = `ECHO_MIN_CHARS_VENDOR` = 20 / `ECHO_MIN_CHARS_ENTROPY` = 12, values ≤ `ECHO_MAX_VALUE_CHARS` = 512, the first `ECHO_MAX_UNIQUE` = 64 unique bearer, assignment and labeled values plus, under their own cap, the first `ECHO_MAX_UNIQUE_VENDOR` = 32 unique vendor or `jwt` values, in claim order, so planted vendor strings never crowd out a bearer token's echo — a value past a cap is redacted at its claim, its echoes are the accepted miss), matched by a streaming merge over the ORIGINAL text — per value, the next occurrence at or beyond the emit cursor (`indexOf` resuming at the cursor, re-sought only once the cursor passes it), leftmost-longest at each step, emitted incrementally into the unclaimed gaps — so retained state is O(values) however long the text is (occurrences are never collected into a list) (never an alternation RegExp, never over emitted tokens). Echoes add no findings. `planRedaction` feeds a caller-supplied shared map (`RedactOpts.echoValues`) and `applyRedaction` reads it at apply time, so a multi-field document scrubs an echo in one field of a value claimed in another. `basic_auth` and the URL userinfo catch-alls redact exactly what `scanText` reports; a vendor value embedded past its boundary (`token_<key>`) is scrubbed as an echo when the same value is claimed elsewhere in the document, and left as-is otherwise. PRIVATE KEYS: `pemClaims` finds `BEGIN`/`END … PRIVATE KEY` fences in two global passes and merges them in text order. BLOCK: a BEGIN whose next fence is an END within `PEM_BODY_MAX_CHARS` (16384) is claimed header through END; a block never swallows a later BEGIN. FORWARD (BEGIN with no END in view): the header plus the body lines that follow; a header followed by no body line (a marker quoted in prose) is claimed alone. BACKWARD (END with no BEGIN in view): the body lines before the fence plus the fence; an END with no body line before it is no claim. A body line is a whole-line base64 run (`A-Za-z0-9+/=`) indented by up to 16 spaces/tabs; line ends are `\n`, `\r\n` or the literal JSON escapes; interior lines are 16–128 chars, the line next to a fence and a partial line at a cut 1–128; optional RFC 1421 `Proc-Type:`/`DEK-Info:` lines and one blank line may follow BEGIN; a line with anything else, or a run over 128 chars, ends the body. Total claimed body stays within `PEM_BODY_MAX_CHARS`. Each walk stops at the neighbouring fence or the previous claim, so the pass is linear (20k unterminated BEGINs, 20k END-only fences and 1 MB of base64 lines are perf-pinned). Per-line hits that fall wholly inside a key claim are dropped (one key, one finding). A FORWARD claim's finding also carries `legacyFingerprint`, the header-only fingerprint earlier releases reported, so `workspace-push.ts` can name a stale allowlist entry; a truncated key's fingerprint covers its body and therefore depends on where the text was cut, so it is not a stable allowlist key. `privateKeySpans(text)` exports the same claims as sorted, disjoint `{start, end}` spans. Accepted misses: body on the header's own line, space-joined single-line keys, blockquoted (`> `) keys, base64 runs over 128 chars, body past 16384 chars, `http(s)` passwords containing a literal `/`, `?` or `#`, Basic values over 2048 chars, digitless assignment values, dotted passwords whose every segment starts with a letter, and unquoted passwords containing `,` or `;`. False-positive budget: `scripts/secret-scan-fp-budget.ts` diffs findings against a base ref's scanner (`--base`, default `origin/master`) (`--dir <brain>` reports per-pattern counts only, never content) and `test/secret-scan-fp-budget.test.ts` pins a synthetic brain-like fixture (zero findings on its non-secret part in both modes, every planted secret redacted). Shape tests: `test/secret-scan-credential-shapes.test.ts`, `test/secret-scan-private-key-spans.test.ts`. Perf pins: `test/secret-scan-perf.test.ts`. Retrieval output is another consumer of its canonical scanner; that output boundary does not sanitize internal evaluation/provider inputs.
-- `src/core/secret-scan-cookie.ts` — the transcript-lane `cookie_header` rule (GST-2768), run by `scanInternal` only under `ScanOpts.transcriptRules` (`TRANSCRIPT_LANE_SCAN_OPTS` in `secret-scan.ts`: redactSession, the recent-transcripts read and `transcripts audit-secrets`). `cookieHeaderValues(line)` reads the `name=value` pairs after a `Cookie:` / `Set-Cookie:` header (any case, quoted or JSON-escaped) or a curl `-b`/`--cookie` argument and returns the values only (8+ chars, no placeholder or mask); `Set-Cookie` stops after its first pair, and parsing stops at the first segment that is not `name=value`. Values join the echo dictionary at 12 chars under their own cap, `ECHO_MAX_UNIQUE_COOKIE` = 64, apart from `ECHO_MAX_UNIQUE`. Known misses: a multi-line `.netrc` stanza, a `-b` on a continued line of a multi-line curl command (Chrome's "Copy as cURL") and a JSON-escaped quoted value (`sid=\"…\"`). `test/secret-scan-push-gate-unchanged.test.ts` pins every non-transcript option set byte-identical on the synthetic corpus `test/fixtures/transcripts/synthetic-credentials.ts`.
+- `src/core/secret-scan-cookie.ts` — the transcript-lane `cookie_header` rule (GST-2768), run by `scanInternal` only under `ScanOpts.transcriptRules` (`TRANSCRIPT_LANE_SCAN_OPTS` in `secret-scan.ts`: redactSession, the recent-transcripts read and `transcripts audit-secrets`). `cookieHeaderValues(line)` reads the `name=value` pairs after a `Cookie:` / `Set-Cookie:` header (any case, quoted or JSON-escaped) or a curl `-b`/`--cookie` argument and returns the values only (8+ chars, no placeholder or mask); `Set-Cookie` stops after its first pair, and parsing stops at the first segment that is not `name=value`. A quoted value may be JSON-escaped (`sid=\"…\"`, up to three backslashes before each quote), and `curlContinues` carries a curl command across lines ending in `\`, so a `-b` alone on a continued line (Chrome's "Copy as cURL") is curl's (`scanInternal` passes `inCurl`). Values join the echo dictionary at 12 chars under their own cap, `ECHO_MAX_UNIQUE_COOKIE` = 64, apart from `ECHO_MAX_UNIQUE`. `test/secret-scan-push-gate-unchanged.test.ts` pins every non-transcript option set byte-identical on the synthetic corpus `test/fixtures/transcripts/synthetic-credentials.ts`.
+- `src/core/secret-scan-transcript.ts` — `compileTranscriptPatterns()`, the transcript-lane-only labeled rules (`TRANSCRIPT_LABELED_PATTERNS` from `secret-scan-labeled.ts`: the `.netrc` forms and the `key`/`netrc` continuations) compiled as `labeled_credential` for `buildPatterns` in `secret-scan.ts` under `ScanOpts.transcriptRules`, after every other rule. Moved out of `secret-scan.ts` at its 1500-line cap with no logic change.
- `src/core/secret-scan-stream.ts` — `scanTextStream(chunks, opts)`, the windowed scan of a byte stream too large to hold as one string (the push gate's staged blobs over 25 MiB). Bytes are decoded by a streaming `TextDecoder` (`ignoreBOM`, so a split multi-byte character joins and invalid bytes become U+FFFD as `Buffer.toString` makes them) and handed to `scanTextSpans` (`secret-scan.ts`: `scanText` plus each claim's absolute span, allowlisted claims kept with `finding: null`) in windows of at most `STREAM_SCAN_WINDOW_CHARS` (8 Mi UTF-16 units). Each next window starts at a line start at least `STREAM_SCAN_OVERLAP_CHARS` (64 Ki, above the longest claim: a private key's 16384-char body plus fences, a jwt's 3 × 4096) before the previous end, or mid-line at exactly the overlap (never inside a surrogate pair) when the line is too long. A window keeps the claims that start before the next window's start, the last window keeps the rest, and a claim wholly inside a span an earlier window kept (a seam's partial view, or an exact duplicate) is dropped; lines are 1-based over the whole stream. `streamScanWork` (@internal) counts windows, bytes, scanned chars and the largest window. `test/secret-scan-stream.test.ts` pins equality with `scanText` across both seams of a window, mid-line splits, split multi-byte characters, invalid UTF-8 and a BOM.
- `src/core/context/sensitivity-scan.ts` — composed detector for compiled-context entries (secret-scan → PII → path shapes → operator blocklist → operator pattern file). secret-scan OWNS credential-shaped families it also matches: a PII finding whose value (raw, or with the `Bearer ` keyword stripped) matches a secret finding's fingerprint is not reported twice; short bearer tokens under secret-scan's floor still surface as `pii:bearer`. Every family honors the same `.gbrain-scan-allow` fingerprint escape hatch. `formatSensitivityDrop(drop, workspaceRoot)` renders the one content-free stderr line `gbrain compile-context` prints per dropped entry: reason `sensitivity_scan`, the pattern family, the fingerprint and the absolute `.gbrain-scan-allow` path, never the value (`test/sensitivity-scan.test.ts`).
diff --git a/docs/architecture/key-files/security.md b/docs/architecture/key-files/security.md
index 690b767ae1..6553e008d4 100644
--- a/docs/architecture/key-files/security.md
+++ b/docs/architecture/key-files/security.md
@@ -27,7 +27,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/scope.ts` — OAuth scope hierarchy + allowlist (the 6 scope strings, `hasScope`, `assertAllowedScopes`, `filterAllowedScopes`, `parseScopeString`, `normalizeScopesInput`) AND the anonymous-registration policy: `DCR_REGISTRABLE_SCOPES` (`read`, `write`) and `dcrScopeViolation(requested, grantTypes)` — returns `null` when the requested scopes fit the Dynamic Client Registration ceiling, otherwise the `invalid_client_metadata` message naming the offending scopes and the operator remedy (`gbrain auth register-client` / admin API / `gbrain auth rescope-client <client_id> --scopes ...`). `agent` keeps its own "operator-approved grant with explicit delegation bindings" wording; when `grantTypes` includes `client_credentials` (only reachable under `--enable-dcr-insecure`) the ceiling is `read` alone; unknown scope strings are ignored here (the unknown-filter drops them). `scopesSupportedForDiscovery({ enableDcr })` is the `scopes_supported` list for OAuth discovery (consumed by `serve-http.ts`'s `authRouterOptions`): the DCR-registrable subset (`read write`, canonical order) while DCR is enabled — so a client that copies discovery into its registration passes `dcrScopeViolation` — and every scope except `agent` otherwise; it never narrows an operator-registered client, because a client that omits `scope` at `/authorize` receives its full registered set via `grantScopes` (`oauth-grants.ts`) and an explicit request is intersected with the registration. The DCR constants and the helper sit BELOW `ALLOWED_SCOPES_LIST`, and no comment line pairs that identifier with `[`, because `scripts/check-admin-scope-drift.sh` extracts the allowlist block with an awk match on exactly that pair. Pinned by `test/scope.test.ts` + `test/oauth-dcr-scope-ceiling.test.ts` + the discovery test in `test/e2e/serve-http-consent.test.ts`.
-- `src/core/secret-scan-labeled.ts` — the transcript-lane `labeled_credential` detector (#6147 idea, widened), compiled by `secret-scan.ts` only under `ScanOpts.labeledCredentials` (transcript pages, the recent-transcripts read and `transcripts audit-secrets`), after every other rule so a vendor or entropy claim keeps attribution. SINGLE labels (`password|passwd|passcode|passphrase|pwd`, optional identifier prefix, JSON-key quote, markdown bold) before `:`/`=`/`:`, and PAIR labels (`login|log-in|credentials|creds|user/pass|username/password|…`) before `:`/`=`/`:`/` - `/nothing then `user / pass` or `user:pass`, each in a bare and a quoted/backticked form; only the password half is claimed. `labeledValueIsCredential(value, form, head)` (the scanner passes the match head to `validate`) rejects placeholders, masks, templating, URLs, paths, calls, dotted references, digitless code identifiers, a backtick that closes a code span holding the label, a meeting link's `?pwd=`, and `LABELED_STOPLIST` words; a pair's password also needs 4+ chars with a letter and a digit or symbol, and its user half must not be a stopword. W4.1 widens it: CLI flags with a separate value (`--password X`, `--pass "a b"`; suffixed flags such as `--password-file` are not labels), an env-name suffix after the label (`PASSWORD_DB=`; a setting suffix such as `_FILE`/`_MIN_LENGTH`/`_HINT` rejects via `settingSuffix`), a `cli` form (prefixed flags such as `--db-password`, `--pw`, openssl `-passin pass:`) whose value may not start with `-` (other forms keep `-`-leading passwords), the setting-suffix exemption only for single labels, and Markdown tables (`tableCredentialCells`, `pw`/`Password (prod)` headers, `|-|` separators, a per-row column Set so a row is linear in cells with per-scan `TableScanState`: cells under a password-label header column and the value cell of a non-header `| password | X |` row; escaped pipes, no outer pipes, several columns). `danglingLabel` gates the next-line continuation patterns per form: `pair` after a dangling pair, `single` (the whole next line is the value, validated as a single label) after a line ending in a single label and delimiter; such a line, and a table row with a claimed cell, is scanned even when shorter than 8 chars. Claimed values join the echo dictionary at `ECHO_MIN_CHARS_LABELED` = 8 when `labeledEchoEligible`. Every quantifier is bounded (perf-pinned in `test/secret-scan-perf.test.ts`); acceptance table in `test/transcript-labeled-credential.test.ts`; user doc `docs/guides/data-ingestion.md#credential-redaction`. GST-2768: `TRANSCRIPT_LABELED_PATTERNS` (compiled only under `ScanOpts.transcriptRules`) adds the `.netrc` `password` form and the `key` continuation after `danglingKeyLabel` (a key or token label ending its line), validated by `keyLabelValueIsCredential` (12+ chars, one token, a letter and a digit).
+- `src/core/secret-scan-labeled.ts` — the transcript-lane `labeled_credential` detector (#6147 idea, widened), compiled by `secret-scan.ts` only under `ScanOpts.labeledCredentials` (transcript pages, the recent-transcripts read and `transcripts audit-secrets`), after every other rule so a vendor or entropy claim keeps attribution. SINGLE labels (`password|passwd|passcode|passphrase|pwd`, optional identifier prefix, JSON-key quote, markdown bold) before `:`/`=`/`:`, and PAIR labels (`login|log-in|credentials|creds|user/pass|username/password|…`) before `:`/`=`/`:`/` - `/nothing then `user / pass` or `user:pass`, each in a bare and a quoted/backticked form; only the password half is claimed. `labeledValueIsCredential(value, form, head)` (the scanner passes the match head to `validate`) rejects placeholders, masks, templating, URLs, paths, calls, dotted references, digitless code identifiers, a backtick that closes a code span holding the label, a meeting link's `?pwd=`, and `LABELED_STOPLIST` words; a pair's password also needs 4+ chars with a letter and a digit or symbol, and its user half must not be a stopword. W4.1 widens it: CLI flags with a separate value (`--password X`, `--pass "a b"`; suffixed flags such as `--password-file` are not labels), an env-name suffix after the label (`PASSWORD_DB=`; a setting suffix such as `_FILE`/`_MIN_LENGTH`/`_HINT` rejects via `settingSuffix`), a `cli` form (prefixed flags such as `--db-password`, `--pw`, openssl `-passin pass:`) whose value may not start with `-` (other forms keep `-`-leading passwords), the setting-suffix exemption only for single labels, and Markdown tables (`tableCredentialCells`, `pw`/`Password (prod)` headers, `|-|` separators, a per-row column Set so a row is linear in cells with per-scan `TableScanState`: cells under a password-label header column and the value cell of a non-header `| password | X |` row; escaped pipes, no outer pipes, several columns). `danglingLabel` gates the next-line continuation patterns per form: `pair` after a dangling pair, `single` (the whole next line is the value, validated as a single label) after a line ending in a single label and delimiter; such a line, and a table row with a claimed cell, is scanned even when shorter than 8 chars. Claimed values join the echo dictionary at `ECHO_MIN_CHARS_LABELED` = 8 when `labeledEchoEligible`. Every quantifier is bounded (perf-pinned in `test/secret-scan-perf.test.ts`); acceptance table in `test/transcript-labeled-credential.test.ts`; user doc `docs/guides/data-ingestion.md#credential-redaction`. GST-2768: `TRANSCRIPT_LABELED_PATTERNS` (compiled only under `ScanOpts.transcriptRules`) adds the `.netrc` `password` form (one line, or a `password` line inside a multi-line stanza that `netrcStanzaOpen` tracks from a `machine`/`default` line through `login`/`account`/`password`, blank and `#` comment lines, the `netrc` continuation form) and the `key` continuation after `danglingKeyLabel` (a key or token label ending its line), validated by `keyLabelValueIsCredential` (12+ chars, one token, a letter and a digit).
- `src/core/ssrf-validate.ts` — DNS-rebinding-defended URL validation. `validateAndResolveUrl(url)` resolves the hostname via `dns.lookup({all: true, family: 0})`, checks EVERY A and AAAA record against the internal-IP deny list, and returns the resolved IP so callers fetch by IP (validation IP === fetch IP defeats DNS rebinding). `fetchWithSSRFGuard(url, opts)` does redirect-aware fetching with per-hop re-validation (max 3 hops by default). Reusable across all URL-fetching features. Test seam `__setDnsLookupForTests` for hermetic tests.
diff --git a/docs/guides/data-ingestion.md b/docs/guides/data-ingestion.md
index 0cae7d70eb..6649137f81 100644
--- a/docs/guides/data-ingestion.md
+++ b/docs/guides/data-ingestion.md
@@ -122,7 +122,8 @@ forms are recognized, with bare, quoted or backticked values:
`DB password` and the like, and the value in a `| password | … |` row,
including rows without outer pipes, escaped `\|` pipes and tables with
several credential columns;
-- a one-line `.netrc` entry, `machine host login alice password …`;
+- a `.netrc` entry, on one line (`machine host login alice password …`) or
+ over several (`machine`, `login` and `password` each on its own line);
- a key or token label that ends its line (`here is my API key:`,
`ACCESS_TOKEN=`), with one value of 12+ characters holding a letter and a
digit alone on the next line;
@@ -131,14 +132,13 @@ forms are recognized, with bare, quoted or backticked values:
Cookie values are redacted too (`cookie_header`): after a `Cookie:` or
`Set-Cookie:` header (raw headers, `curl -H`, a copy-as-fetch JSON blob) and
-after `-b`/`--cookie` on the same line as `curl`, every value of 8+
-characters is replaced and the cookie names and `Set-Cookie` attributes
+after `-b`/`--cookie` in a `curl` command, including on a continued line
+of a multi-line command (what Chrome's "Copy as cURL" produces), every
+value of 8+ characters is replaced, quoted values inside escaped JSON
+(`sid=\"…\"`) too, and the cookie names and `Set-Cookie` attributes
stay. These transcript-only rules never run in the `sources push` gate or
the compiled-context scan.
-Known gaps: three cases still reach the page verbatim, as they did before
-these rules: a `.netrc` entry written over several lines (`machine`, `login` and `password` each on its own line), a `-b` cookie on its own continued line of a multi-line curl command (what Chrome's "Copy as cURL" produces), and a quoted cookie value inside escaped JSON (`sid=\"…\"`).
-
To keep prose and code intact, a value is never redacted when it is a
placeholder (`<…>`, `${…}`, `$VAR`), a mask (`****`), a URL, a path, a
function call, a dotted reference (`req.body.password`), a code identifier
diff --git a/src/core/secret-scan-cookie.ts b/src/core/secret-scan-cookie.ts
index 78a9f90117..90e4757239 100644
--- a/src/core/secret-scan-cookie.ts
+++ b/src/core/secret-scan-cookie.ts
@@ -16,15 +16,16 @@
* (`Path=/`, `Max-Age=3600`) are kept. Parsing stops at the first segment
* that is not `name=value`, so prose (`Cookie: chocolate chip`) is never
* claimed. A `'` or `,` inside a value and a JSON-escaped `\/` are part of it.
+ * A quoted value may be JSON-escaped (`sid=\"…\"`, `sid=\\\"…\\\"`), and a `-b` on a continued
+ * line of a multi-line curl command (Chrome's "Copy as cURL": every line before it ends in `\`)
+ * counts as a curl argument (`curlContinues`).
* Values shorter than 8 characters (preference flags such as
* `theme=dark`), placeholders (`<…>`, `$VAR`, `{…}`, `%VAR%`) and masks (`...`, `***`)
* are left alone. Every quantifier is bounded; the parse is linear in the line.
- * Not caught: a `-b` on a continued line of a multi-line curl command
- * (Chrome's "Copy as cURL") and a JSON-escaped quoted value (`sid=\"…\"`).
*/
const COOKIE_HEAD_RE = /(?:(?:^|[^A-Za-z0-9_-])(set-)?cookie\\?["'`]?\s{0,4}:|(?:^|\s)(-b|--cookie)(?=[\s=]))[\s=]{0,4}\\?["'`]?/gi;
-const COOKIE_PAIR_RE = /([!#$%&*+.^_|~0-9A-Za-z-]{1,128})=("?)((?:[^\s;"',\\]|[',](?=[^\s;"',\\])|\\\/){0,4096})/y;
+const COOKIE_PAIR_RE = /([!#$%&*+.^_|~0-9A-Za-z-]{1,128})=((?:\\{1,3})?"|)((?:[^\s;"',\\]|[',](?=[^\s;"',\\])|\\\/){0,4096})/y;
const COOKIE_SEP_RE = /\s{0,4};\s{0,4}/y;
const PLACEHOLDER_RE = /^(?:[<${[]|%[A-Za-z_][A-Za-z0-9_]{0,64}%$)/;
const CURL_RE = /(?:^|[^A-Za-z0-9_-])curl(?:\.exe)?(?:\s|$)/i;
@@ -43,27 +44,35 @@ export const COOKIE_VALUE_MIN_CHARS = 8;
/** Cheap gate before the header regex runs on a line. */
export const COOKIE_PRECHECK_RE = /cookie|-b\s/i;
-/** The cookie values on `line`: `start` of the value, the value, and `from`, where its header starts. */
-export function cookieHeaderValues(line: string): Array<{ start: number; value: string; from: number }> {
+/** True when the line after `line` still belongs to a curl command: `line` ends in `\` and is a curl line or one continued from it (`inCurl`). */
+export function curlContinues(line: string, inCurl: boolean): boolean {
+ return /\\\s{0,4}$/.test(line) && (inCurl || CURL_RE.test(line));
+}
+
+/**
+ * The cookie values on `line`: `start` of the value, the value, and `from`, where its header starts.
+ * `inCurl`: the line continues a curl command (`curlContinues`), so a `-b` on it is curl's.
+ */
+export function cookieHeaderValues(line: string, inCurl = false): Array<{ start: number; value: string; from: number }> {
const out: Array<{ start: number; value: string; from: number }> = [];
let curlAt: number | undefined;
COOKIE_HEAD_RE.lastIndex = 0;
for (let head = COOKIE_HEAD_RE.exec(line); head !== null; head = COOKIE_HEAD_RE.exec(line)) {
const setCookie = head[1] !== undefined;
let at = COOKIE_HEAD_RE.lastIndex;
- if (head[2] !== undefined && !((curlAt ??= line.search(CURL_RE)) >= 0 && curlAt < head.index)) continue;
+ if (head[2] !== undefined && !inCurl && !((curlAt ??= line.search(CURL_RE)) >= 0 && curlAt < head.index)) continue;
for (;;) {
COOKIE_PAIR_RE.lastIndex = at;
const m = COOKIE_PAIR_RE.exec(line);
if (m === null) break;
- const quoted = m[2] === '"';
- if (quoted && line[COOKIE_PAIR_RE.lastIndex] !== '"') break;
+ const quote = m[2]!;
+ if (quote !== '' && !line.startsWith(quote, COOKIE_PAIR_RE.lastIndex)) break;
const value = m[3]!;
if (value.length >= COOKIE_VALUE_MIN_CHARS && !PLACEHOLDER_RE.test(value) && !MASK_RE.test(value)) {
- out.push({ start: m.index + m[1]!.length + 1 + m[2]!.length, value, from: head.index });
+ out.push({ start: m.index + m[1]!.length + 1 + quote.length, value, from: head.index });
}
if (setCookie) break;
- COOKIE_SEP_RE.lastIndex = COOKIE_PAIR_RE.lastIndex + (quoted ? 1 : 0);
+ COOKIE_SEP_RE.lastIndex = COOKIE_PAIR_RE.lastIndex + quote.length;
if (COOKIE_SEP_RE.exec(line) === null) break;
at = COOKIE_SEP_RE.lastIndex;
}
diff --git a/src/core/secret-scan-labeled.ts b/src/core/secret-scan-labeled.ts
index 36197b72d9..db453ca621 100644
--- a/src/core/secret-scan-labeled.ts
+++ b/src/core/secret-scan-labeled.ts
@@ -43,7 +43,11 @@
* `password <value>`, and a key or token label left dangling at a line end
* (`here is my API key:`, `ACCESS_TOKEN=`) with its value alone on the next
* line. The transcript lane also carries a dangling label of any form into
- * the first line of the NEXT message (`ScanOpts.leadingDangling`).
+ * the first line of the NEXT message (`ScanOpts.leadingDangling`). A `.netrc`
+ * entry written over several lines (`machine host`, then `login alice` and
+ * `password …` on lines of their own) is read as one stanza
+ * (`netrcStanzaOpen`): a line that is only `password <value>` (optionally
+ * after `login`/`account` tokens) inside it is claimed.
*
* Accepted misses: an unquoted multi-word passphrase (`login: alice / pass
* word`), a delimiter-free single label (`the password is hunter2`), a pair
@@ -76,7 +80,7 @@ const CLI_HEAD = '(?:^|\\s)--?(?!(?:no|skip|ask|prompt|reset|show|print|change|c
const OPENSSL_PASS_HEAD = '(?:^|\\s)-pass(?:in|out)\\s{1,4}pass:';
/** The label form a line leaves dangling for the next line: `key` is the transcript-lane key or token label. */
-export type DanglingForm = 'single' | 'pair' | 'key';
+export type DanglingForm = 'single' | 'pair' | 'key' | 'netrc';
export interface LabeledPattern {
source: string;
@@ -90,7 +94,7 @@ export interface LabeledPattern {
export interface TranscriptLabeledPattern {
source: string;
form: 'single' | 'key';
- continuation?: 'key';
+ continuation?: 'key' | 'netrc';
}
/** Group 1 = everything before the value, group 2 = the value (the scanner's shape). */
@@ -111,14 +115,28 @@ export const LABELED_CREDENTIAL_PATTERNS: readonly LabeledPattern[] = [
/** A key or token label (`API key`, `access_token`, `client secret`), transcript lane only. */
const KEY_LABEL = '(?:api[ _-]?key|access[ _-]?key|secret[ _-]?key|client[ _-]?secret|secret|token)s?';
const NETRC_HEAD = '(?:^|\\s)(?:machine\\s{1,4}[^\\s]{1,253}|default)(?:\\s{1,4}(?:login|account)\\s{1,4}[^\\s]{1,128}){0,2}\\s{1,4}password\\s{1,4}';
+/** A `password` line inside a multi-line `.netrc` stanza, optionally after its `login`/`account` tokens. */
+const NETRC_LINE_HEAD = '^\\s{0,8}(?:(?:login|account)\\s{1,4}[^\\s]{1,128}\\s{1,4}){0,2}password\\s{1,4}';
+/** A line that opens a multi-line `.netrc` stanza (`machine host`, `default`, optionally with login/account but no password). */
+const NETRC_STANZA_OPEN_RE = /^\s{0,8}(?:machine\s{1,4}\S{1,253}|default)(?:\s{1,4}(?:login|account)\s{1,4}\S{1,128}){0,2}\s{0,4}$/;
+/** A line that keeps a `.netrc` stanza open: only `login`, `account` and `password` tokens, a blank line or a `#` comment. */
+const NETRC_STANZA_LINE_RE = /^\s{0,8}(?:(?:(?:login|account|password)\s{1,4}\S{1,255}\s{0,4}){1,3}|#.*|\s*)$/;
+
+/** True when the line after `line` is inside a multi-line `.netrc` stanza; `open` says whether `line` itself was. */
+export function netrcStanzaOpen(line: string, open: boolean): boolean {
+ return NETRC_STANZA_OPEN_RE.test(line) || (open && NETRC_STANZA_LINE_RE.test(line));
+}
/**
* GST-2768: the transcript-lane-only labeled shapes (`ScanOpts.transcriptRules`), claimed as
- * `labeled_credential`: a `.netrc` password, and the value on the line after a dangling key label.
+ * `labeled_credential`: a `.netrc` password (one line, or a `password` line inside a multi-line
+ * stanza), and the value on the line after a dangling key label.
*/
export const TRANSCRIPT_LABELED_PATTERNS: readonly TranscriptLabeledPattern[] = [
{ form: 'single', source: `(${NETRC_HEAD}${QUOTE})(${QUOTED_VALUE})(?=${QUOTE})` },
{ form: 'single', source: `(${NETRC_HEAD})(${BARE_VALUE})${BARE_END}` },
+ { form: 'single', continuation: 'netrc', source: `(${NETRC_LINE_HEAD}${QUOTE})(${QUOTED_VALUE})(?=${QUOTE}\\s{0,8}$)` },
+ { form: 'single', continuation: 'netrc', source: `(${NETRC_LINE_HEAD})([^\\s"'\`]{1,256})(?=\\s{0,8}$)` },
{ form: 'key', continuation: 'key', source: `(^\\s{0,8}${QUOTE})(${QUOTED_VALUE})(?=${QUOTE}[,;]?\\s*$)` },
{ form: 'key', continuation: 'key', source: `(^\\s{0,8})(${BARE_VALUE})(?=[.,;]?\\s*$)` },
];
diff --git a/src/core/secret-scan-transcript.ts b/src/core/secret-scan-transcript.ts
new file mode 100644
index 0000000000..09f17633fb
--- /dev/null
+++ b/src/core/secret-scan-transcript.ts
@@ -0,0 +1,39 @@
+/**
+ * secret-scan-transcript.ts: the transcript-lane-only (`ScanOpts.transcriptRules`, GST-2768) labeled patterns, compiled
+ * for `buildPatterns` in secret-scan.ts. Moved out of secret-scan.ts at its 1500-line module cap; the shapes and their
+ * checks live in secret-scan-labeled.ts.
+ */
+import {
+ TRANSCRIPT_LABELED_PATTERNS,
+ NETRC_PRECHECK_RE,
+ keyLabelValueIsCredential,
+ labeledValueIsCredential,
+ neutralizeRedactionMarkers,
+ type DanglingForm,
+} from './secret-scan-labeled.ts';
+
+/** A compiled transcript-lane pattern, in the shape of secret-scan.ts's `CompiledPattern`. */
+export interface CompiledTranscriptPattern {
+ name: 'labeled_credential';
+ re: RegExp;
+ continuation?: DanglingForm;
+ precheck?: (line: string) => boolean;
+ validate: (value: string, head: string) => boolean;
+ subject: (line: string) => string;
+}
+
+/** `TRANSCRIPT_LABELED_PATTERNS` compiled as `labeled_credential` rules, appended after every other rule. */
+export function compileTranscriptPatterns(): CompiledTranscriptPattern[] {
+ const out: CompiledTranscriptPattern[] = [];
+ for (const p of TRANSCRIPT_LABELED_PATTERNS) {
+ out.push({
+ name: 'labeled_credential',
+ // nosemgrep: javascript.lang.security.audit.detect-non-literal-regexp.detect-non-literal-regexp -- compile-time literal; bounded quantifiers, pinned by test/secret-scan-perf.test.ts
+ re: new RegExp(p.source, 'gi'),
+ ...(p.continuation ? { continuation: p.continuation } : { precheck: (line: string) => NETRC_PRECHECK_RE.test(line) }),
+ validate: p.form === 'key' ? keyLabelValueIsCredential : (value: string, head: string) => labeledValueIsCredential(value, 'single', head),
+ subject: neutralizeRedactionMarkers,
+ });
+ }
+ return out;
+}
diff --git a/src/core/secret-scan.ts b/src/core/secret-scan.ts
index 7b786e6e6d..f1fb2aabe1 100644
--- a/src/core/secret-scan.ts
+++ b/src/core/secret-scan.ts
@@ -35,10 +35,8 @@ import {
LABELED_CREDENTIAL_PATTERNS,
ECHO_MIN_CHARS_LABELED,
LABELED_PRECHECK_RE,
- TRANSCRIPT_LABELED_PATTERNS,
- NETRC_PRECHECK_RE,
danglingFor,
- keyLabelValueIsCredential,
+ netrcStanzaOpen,
type DanglingForm,
labeledEchoEligible,
labeledValueIsCredential,
@@ -93,8 +91,8 @@ export interface ScanOpts {
*/
labeledCredentials?: boolean;
/**
- * Opt-in transcript-lane rules (GST-2768): `cookie_header` (secret-scan-cookie.ts), the `.netrc` password and the
- * key-label next-line value (secret-scan-labeled.ts). Only via `TRANSCRIPT_LANE_SCAN_OPTS`. Default false.
+ * Opt-in transcript-lane rules (GST-2768): `cookie_header` (secret-scan-cookie.ts), the `.netrc` password (one line or a
+ * multi-line stanza) and the key-label next-line value (secret-scan-labeled.ts). Only via `TRANSCRIPT_LANE_SCAN_OPTS`. Default false.
*/
transcriptRules?: boolean;
/** Label form the previous message left dangling (`trailingDanglingLabel`): line 1 here is read as its value. Default none. */
@@ -114,7 +112,8 @@ import {
} from './secret-scan-assignment.ts';
import type { ScanWork } from './secret-scan-work.ts';
export { createScanWork, type ScanWork } from './secret-scan-work.ts';
-import { COOKIE_PRECHECK_RE, ECHO_MAX_UNIQUE_COOKIE, cookieHeaderValues } from './secret-scan-cookie.ts';
+import { COOKIE_PRECHECK_RE, ECHO_MAX_UNIQUE_COOKIE, cookieHeaderValues, curlContinues } from './secret-scan-cookie.ts';
+import { compileTranscriptPatterns } from './secret-scan-transcript.ts';
export { TRANSCRIPT_LANE_SCAN_OPTS, trailingDanglingLabel } from './secret-scan-labeled.ts';
// ── Patterns ────────────────────────────────────────────────────────────────
@@ -719,18 +718,7 @@ function buildPatterns(opts: ScanOpts): CompiledPattern[] {
});
}
}
- if (opts.transcriptRules) {
- for (const p of TRANSCRIPT_LABELED_PATTERNS) {
- out.push({
- name: 'labeled_credential',
- // nosemgrep: javascript.lang.security.audit.detect-non-literal-regexp.detect-non-literal-regexp -- compile-time literal; bounded quantifiers, pinned by test/secret-scan-perf.test.ts
- re: new RegExp(p.source, 'gi'),
- ...(p.continuation ? { continuation: p.continuation } : { precheck: (line: string) => NETRC_PRECHECK_RE.test(line) }),
- validate: p.form === 'key' ? keyLabelValueIsCredential : (value: string, head: string) => labeledValueIsCredential(value, 'single', head),
- subject: neutralizeRedactionMarkers,
- });
- }
- }
+ if (opts.transcriptRules) out.push(...compileTranscriptPatterns());
return out;
}
@@ -895,13 +883,16 @@ function scanInternal(text: string, opts: ScanOpts): RawHit[] {
// W4.1: Markdown table state for the labeled lane (credential columns of the table being scanned).
const table = opts.labeledCredentials === true ? newTableScanState() : null;
let leadingLine = 0;
+ // GST-2768 follow-up: a multi-line .netrc stanza and a curl command continued with `\` carry to later lines.
+ let netrc = false, inCurl = false;
for (let i = 0; i < lines.length; i++) {
const line = lines[i] ?? '';
const lineStart = offset;
offset += line.length + 1;
+ if (opts.transcriptRules === true && i > 0) { netrc = netrcStanzaOpen(lines[i - 1] ?? '', netrc); inCurl = curlContinues(lines[i - 1] ?? '', inCurl); }
// The previous message's dangling label reads the first non-blank line here.
if (i === leadingLine && opts.leadingDangling && line.trim() === '') leadingLine = i + 1;
- const dangling = i === leadingLine ? (opts.leadingDangling ?? null) : i > 0 ? danglingFor(lines[i - 1] ?? '', opts) : null;
+ const dangling = (i === leadingLine ? (opts.leadingDangling ?? null) : i > 0 ? danglingFor(lines[i - 1] ?? '', opts) : null) ?? (netrc ? 'netrc' : null);
const cells = table ? tableCredentialCells(line, lines[i + 1], table) : [];
if (line.length < 8 && !dangling && cells.length === 0) continue;
// ENG-12: a line wholly inside a private-key claim is already redacted by
@@ -984,7 +975,7 @@ function scanInternal(text: string, opts: ScanOpts): RawHit[] {
for (const c of cells) if (!taken || !anyTaken(taken, c.start, c.start + c.value.length)) { taken ??= new Uint8Array(line.length); claim('labeled_credential', c.value, c.start, c.start); }
// GST-2768: cookie values (transcript lane), union coverage around anything already claimed.
if (opts.transcriptRules === true && COOKIE_PRECHECK_RE.test(line)) {
- for (const c of cookieHeaderValues(line)) {
+ for (const c of cookieHeaderValues(line, inCurl)) {
taken ??= new Uint8Array(line.length);
for (let a = c.start, k = c.start, end = c.start + c.value.length; k <= end; k++) {
if (k < end && !taken[k]) continue;
diff --git a/src/core/transcripts/secret-audit.ts b/src/core/transcripts/secret-audit.ts
index 7feff2d5d3..6707ebda8d 100644
--- a/src/core/transcripts/secret-audit.ts
+++ b/src/core/transcripts/secret-audit.ts
@@ -16,8 +16,11 @@
import type { BrainEngine } from '../engine.ts';
import { TRANSCRIPT_LANE_SCAN_OPTS, scanText } from '../secret-scan.ts';
-/** Bump when the transcript-lane detectors change, so doctor asks for a fresh audit. 6: cookie_header, .netrc and the key-label join (GST-2768). */
-export const TRANSCRIPT_SECRET_AUDIT_VERSION = 6;
+/**
+ * Bump when the transcript-lane detectors change, so doctor asks for a fresh audit. 6: cookie_header, .netrc and the
+ * key-label join (GST-2768). 7: a multi-line .netrc stanza, a curl -b on a continued line and a JSON-escaped quoted cookie.
+ */
+export const TRANSCRIPT_SECRET_AUDIT_VERSION = 7;
export const TRANSCRIPT_SECRET_AUDIT_KEY = 'transcripts.secret_audit';
const BATCH = 25;
const LINES_PER_PAGE = 20;
diff --git a/test/transcripts-secret-cookie-header.test.ts b/test/transcripts-secret-cookie-header.test.ts
index 9f5b27efbb..381c9f57fa 100644
--- a/test/transcripts-secret-cookie-header.test.ts
+++ b/test/transcripts-secret-cookie-header.test.ts
@@ -116,7 +116,7 @@ describe('audit-secrets uses the transcript-lane rules', () => {
await resetPgliteState(engine);
});
- test('a stored page with a cookie header is affected; the audit version moved to 6', async () => {
+ test('a stored page with a cookie header is affected; the audit version moved to 7', async () => {
await engine.putPage('conversations/sessions/2026-01-02-codex-cccccccccccc', {
type: 'conversation', title: 'session', compiled_truth: `intro\ncurl -H 'Cookie: sid=${SID}' https://app.acme-example.test/`,
frontmatter: { transcript_import: { harness: 'codex', session_id: 'cookie' } },
@@ -125,6 +125,48 @@ describe('audit-secrets uses the transcript-lane rules', () => {
expect(result.by_pattern).toEqual({ cookie_header: 1 });
expect(result.pages[0]!.lines).toEqual([2]);
expect(JSON.stringify(result)).not.toContain(SID);
- expect(TRANSCRIPT_SECRET_AUDIT_VERSION).toBe(6);
+ expect(TRANSCRIPT_SECRET_AUDIT_VERSION).toBe(7);
+ });
+
+ test('the audit finds the follow-up shapes on a stored page', async () => {
+ const pw = ['Synth', 'NetrcMulti4321'].join('');
+ await engine.putPage('conversations/sessions/2026-01-03-codex-dddddddddddd', {
+ type: 'conversation', title: 'session',
+ compiled_truth: [
+ 'machine api.acme-example.test', ' login alice-example', ` password ${pw}`,
+ "curl 'https://app.acme-example.test/' \\", ` -b 'sid=${SID}'`,
+ `{"headers": "Cookie: csrf=\\"${CSRF}\\""}`,
+ ].join('\n'),
+ frontmatter: { transcript_import: { harness: 'codex', session_id: 'follow-up' } },
+ });
+ const result = await auditTranscriptSecrets(engine);
+ expect(result.by_pattern).toEqual({ labeled_credential: 1, cookie_header: 2 });
+ expect(result.pages[0]!.lines).toEqual([3, 5, 6]);
+ for (const s of [pw, SID, CSRF]) expect(JSON.stringify(result)).not.toContain(s);
});
});
+
+describe('GST-2768 follow-up: continued curl lines and JSON-escaped quoted values', () => {
+ test('a -b cookie on its own continued line of a multi-line curl command is redacted', () => {
+ const input = [`curl 'https://app.acme-example.test/api' \\`, ` -H 'accept: */*' \\`, ` -b 'sid=${SID}; csrf=${CSRF}' \\`, ' --compressed'].join('\n');
+ const out = one(input);
+ expect(out).not.toContain(SID);
+ expect(out).not.toContain(CSRF);
+ expect(out).toContain(" -b 'sid=<REDACTED:cookie_header>; csrf=<REDACTED:cookie_header>' \\");
+ });
+ test('a JSON-escaped quoted cookie value is redacted', () => {
+ expect(one(`{"headers": "Cookie: sid=\\"${SID}\\"; csrf=\\"${CSRF}\\""}`)).toBe('{"headers": "Cookie: sid=\\"<REDACTED:cookie_header>\\"; csrf=\\"<REDACTED:cookie_header>\\""}');
+ expect(one(`Cookie: sid=\\\\\\"${SID}\\\\\\"`)).toBe('Cookie: sid=\\\\\\"<REDACTED:cookie_header>\\\\\\"');
+ });
+ test('ordinary text near these shapes stays', () => {
+ const notCurl = ['make build \\', ' && git checkout -b feature=abcdefgh12345'].join('\n');
+ expect(one(notCurl)).toBe(notCurl);
+ const ended = ['curl https://app.acme-example.test/ \\', ' --compressed', 'git checkout -b feature=abcdefgh12345'].join('\n');
+ expect(one(ended)).toBe(ended);
+ const prefs = '{"headers": "Cookie: theme=\\"dark\\"; lang=\\"en\\""}';
+ expect(one(prefs)).toBe(prefs);
+ const prose = '{"msg": "tick the cookie: \\"remember me\\" box"}';
+ expect(one(prose)).toBe(prose);
+ });
+});
+
diff --git a/test/transcripts-secret-corpus.test.ts b/test/transcripts-secret-corpus.test.ts
index 3514e94133..eb724400d0 100644
--- a/test/transcripts-secret-corpus.test.ts
+++ b/test/transcripts-secret-corpus.test.ts
@@ -6,7 +6,7 @@
* (test/fixtures/transcripts/synthetic-credentials.ts).
*/
import { describe, expect, test } from 'bun:test';
-import { NAMED_SECRET_RULES } from '../src/core/secret-scan.ts';
+import { NAMED_SECRET_RULES, TRANSCRIPT_LANE_SCAN_OPTS, scanText } from '../src/core/secret-scan.ts';
import { redactSession } from '../src/core/transcripts/render.ts';
import type { ParsedSession } from '../src/core/transcripts/types.ts';
import { SYNTHETIC_CREDENTIAL_CORPUS } from './fixtures/transcripts/synthetic-credentials.ts';
@@ -31,6 +31,44 @@ describe('synthetic credential corpus', () => {
expect(out).toContain('password \\"<REDACTED:labeled_credential>\\"');
});
+ test('a .netrc entry written over several lines: the password is redacted', () => {
+ const pw = ['Synth', 'NetrcMulti4321'].join('');
+ for (const text of [
+ `machine api.acme-example.test\n login alice-example\n password ${pw}`,
+ `machine api.acme-example.test\nlogin alice-example\npassword "${pw}"`,
+ `machine api.acme-example.test login alice-example\n password ${pw}`,
+ `default\n password ${pw}\n login alice-example`,
+ ]) {
+ const out = run({ meta: { harness: 'claude-code', sessionId: 'netrc-multi' }, messages: [{ role: 'user', timestamp: TS(0), text }] });
+ expect([text, out.includes(pw)]).toEqual([text, false]);
+ expect(out).toContain('<REDACTED:labeled_credential>');
+ }
+ });
+
+ test('a blank or comment line inside a multi-line .netrc stanza does not end it', () => {
+ const pw = ['Synth', 'NetrcGap7788'].join('');
+ for (const text of [
+ `machine api.acme-example.test\nlogin alice-example\n\npassword ${pw}`,
+ `machine api.acme-example.test\n# service account\nlogin alice-example\npassword ${pw}`,
+ `machine api.acme-example.test\n login alice-example\n # rotated monthly\n\n password "${pw}"`,
+ ]) {
+ const out = run({ meta: { harness: 'claude-code', sessionId: 'netrc-gap' }, messages: [{ role: 'user', timestamp: TS(0), text }] });
+ expect([text, out.includes(pw)]).toEqual([text, false]);
+ expect(scanText(text, TRANSCRIPT_LANE_SCAN_OPTS).map((f) => f.pattern)).toEqual(['labeled_credential']);
+ }
+ });
+
+ test('prose near a .netrc shape is not redacted', () => {
+ for (const text of [
+ 'machine learning\npassword managers are useful',
+ 'machine api.acme-example.test\nthen we talked\npassword hunter2x',
+ 'we used the machine api.acme-example.test\npassword Synth4242pw',
+ ]) {
+ const out = redactSession({ meta: { harness: 'claude-code', sessionId: 'netrc-prose' }, messages: [{ role: 'user', timestamp: TS(0), text }] }, { patterns: [] }).session.messages[0]!.text;
+ expect(out).toBe(text);
+ }
+ });
+
for (const e of SYNTHETIC_CREDENTIAL_CORPUS) {
test(`${e.id} (${e.shape}) in message bodies`, () => {
const out = run({