Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
Redact multi-line .netrc passwords, continued curl -b cookies and escaped cookie quotes in transcripts (GST-2768) (#9418)
Garry Tanc442f944ad43
diff --git a/changelog.d/transcript-redaction-gaps.md b/changelog.d/transcript-redaction-gaps.mdnew file mode 100644index 0000000000..15715ffd5e--- /dev/null+++ b/changelog.d/transcript-redaction-gaps.md@@ -0,0 +1,17 @@+---+important: security+---+**Imported conversations no longer keep three kinds of secret that the last cookie and `.netrc` fix still let through.** `gbrain transcripts ingest` now also redacts a `.netrc` password when the entry is written over several lines (`machine`, `login` and `password` each on its own line), a curl cookie passed with `-b` on its own line of a command split with trailing backslashes (what Chrome's "Copy as cURL" produces), and a quoted cookie value inside escaped JSON (`sid=\"…\"`). Before this change all three were copied into the searchable page as plain text. Cookie names, curl's other arguments and ordinary prose near these shapes stay as they were.++What to do: conversations imported earlier keep what they stored. `gbrain doctor` asks for a fresh audit; run `gbrain transcripts audit-secrets --json` to list affected pages (slugs and hit counts, never values). The `sources push` gate and the compiled-context scan are unchanged.++Still not caught: a curl cookie written with bash's `$'…'` quoting (`-b $'sid=…'`, which Chrome's "Copy as cURL (bash)" uses for a cookie holding a quote or non-ASCII text); a `.netrc` entry or curl command split across two messages; and a cookie header whose opening and closing escaped quotes differ in depth (`sid=\\\"…\"`), where that value and the pairs after it stay. In the other direction, a line that is only `default` or `machine <word>` followed by a line that is only `password <word>` has that word redacted even in prose.++### Itemized changes++- `src/core/secret-scan-labeled.ts`: a `password` line inside a multi-line `.netrc` stanza (opened by a `machine` or `default` line, kept open by `login`, `account` and `password` lines, blank lines and `#` comments) is redacted. The line must be only `password <value>`, so prose such as "machine learning" followed by "password managers are useful" stays.+- `src/core/secret-scan-cookie.ts`: a `-b` on a line that continues a curl command (each earlier line ends in `\`) counts as curl's cookie argument; a cookie value quoted with escaped quotes (`\"`, or `\\\"` one level deeper) is read. A `-b` after a continued line that is not curl (`git checkout -b`) stays.+- `src/core/secret-scan.ts`: the transcript-lane scan carries the `.netrc` stanza and the curl continuation from line to line.+- `src/core/transcripts/secret-audit.ts`: `TRANSCRIPT_SECRET_AUDIT_VERSION` 6 → 7, because the audit now finds these shapes on stored pages.+- Tests: `test/transcripts-secret-cookie-header.test.ts`, `test/transcripts-secret-corpus.test.ts`.+- Docs: `docs/guides/data-ingestion.md#credential-redaction` and the key-files entries.diff --git a/docs/architecture/key-files/secret-scan.md b/docs/architecture/key-files/secret-scan.mdindex bafc148335..eb759b9d06 100644--- a/docs/architecture/key-files/secret-scan.md+++ b/docs/architecture/key-files/secret-scan.md@@ -7,6 +7,7 @@ compiled-context sensitivity scan and the transcripts redactor, and the composedsensitivity detector built on it.- `src/core/secret-scan.ts` — pattern-based secret detection shared by `sources push`, bootstrap verify, the compiled-context sensitivity scan and the transcripts redactor. Detection is FORMAT-based, not prefix-only: vendor shapes (anthropic, openai, voyage, github, gitlab, slack, aws AKIA/ASIA, google, stripe, sendgrid, twilio, supabase sb_secret_/sbp_, npm, huggingface, `digitalocean` `do[opr]_v1_` + 64 hex, the AI-provider keys `openrouter` `sk-or-v1-` + 64 lowercase hex, `groq` `gsk_` + 52, `xai` `xai-` + 80, `perplexity` `pplx-` + 48 and `replicate` `r8_` + 37, all fixed-length with a hard right edge, gbrain), JWTs, private keys (whole-text claims, below), plus four catch-alls appended LAST (`bearer` — prebuilt two-group layout so only the token is the value, which may hold a JSON-escaped `\/` (#8266); `db_url_credentials` — the `scheme://user:pass@` span of database URLs, host/db kept; `url_credentials` — the same span for `http(s)` (scheme case-insensitive, `gi`), user 0–128 stopping at `:`/`/`, password 1–256 excluding whitespace, `@`, `"`, `'`, `/`, `?`, `#`, with a `validate` hook that rejects placeholder passwords (`<...>`, `${...}`, `$VAR`, all `*`, all `x`); `basic_auth` — two prebuilt case-insensitive forms, `Authorization: Basic` with a 4–2048 char value and bare `Basic` with 16–2048, both with a hard right edge and a `validate` that requires the value to base64-decode to printable ASCII containing `:`; it is NOT in the bearer family, so it joins neither the echo dictionary nor `secret:bearer`). The pattern contract: `CorePattern` carries optional `flags` (default `'g'`), `precheck`, `validate(value)` (O(value), run on the whole match before the claim and the union split; a rejected match is not re-tried from inside) and `since` (the gbrain version that added or last changed the rule; `SECURITY_WAVE_SINCE` = `0.60.31.0` on `digitalocean`, `url_credentials`, `basic_auth` and `private_key_pem`, `PROVIDER_KEYS_SINCE` on the five AI-provider rules (#7959), kept in step with the release that ships a rule change). Compiled patterns keep group 1 = anchor, group 2 = value. Every core rule and both `high_entropy_assignment` forms also have an escape-boundary twin (#7978), compiled right after the rule under the same name and anchored on a JSON escape (`\n`, `\r`, `\t`, `\b`, `\f`, `\uXXXX`, and since #8266 `\xNN`, `\UXXXXXXXX`), a percent-escape (`%XX`), a quoted-printable `=3D` or a terminal color code (`ESC[…m`, raw or written `\u001b[`/`\x1b[`/`\033[`/`\e[`), whose last character the default boundary reads as a word character; a twin runs only on a line `ESCAPE_IN_LINE_RE` matches, so other lines run the old list, and a prebuilt rule carries its twin as `CorePattern.escaped`. `patternSince(pattern)` reads the map; every finding of such a pattern carries `SecretFinding.since`, which the push and bootstrap refusals render. Fixed-length shapes carry a trailing negative lookahead. Per-line claimed-span dedupe is first-wins for ATTRIBUTION (order matters: `Bearer <vendor key>` keeps the vendor name, a fully covered later span is skipped) but coverage is the UNION: every unclaimed slice of a later overlapping span is claimed as a hit of the later pattern (own value, own fingerprint), so a database URL whose username is a vendor-shaped id still loses its password. The transcript-only `labeled_credential` rule (`ScanOpts.labeledCredentials`) lives in `secret-scan-labeled.ts` (Security). The opt-in `high_entropy_assignment` rule (`ScanOpts.highEntropy`; its shapes and gates live in `src/core/secret-scan-assignment.ts`, split out at the 1500-line module cap) is anchored on a credential keyword or on an identifier whose last word is `key` (`STRIPE_KEY=`, never `MONKEY=` or a bare `KEY=`) and requires a 12+ char value that contains a digit and a non-digit AND clears 3.5 bits/char on the bias-corrected (Miller-Madow) estimate `correctedEntropy`, since raw per-character entropy is capped at log2(n) and rejected most random 12-16 char secrets. It is two compiled patterns sharing the name: QUOTED (after `"` or `'`, any run of non-quote, non-whitespace characters) and UNQUOTED (the legacy `[A-Za-z0-9+/_=-]` class plus `!#$%*@^~.?<>`; never `&`, `()[]{}`, `:`, `,` or `;`, so query strings, calls and SQL `SET` lists stay intact, and a password carrying `,`/`;` matches only quoted; the last character excludes `.?!`, so a sentence's full stop is not claimed). Its `validate` rejects values that start with `scheme://`, `./`, `~/`, `${`, `$(` or `<REDACTED:`, dotted identifier paths (`process.env.X`) and `/`-paths holding a character outside the legacy class. Consumers with it on: retrieval output, transcript ingest, hooks and the hook heartbeat, the atoms drain, `embed-facts.ts` error text and the context-engine relay re-scan. Off: the push gate, bootstrap verify/repo and the compiled-context sensitivity scan. Own minimal allowlist + `<ws>/.gbrain-scan-allow` per-finding overrides — deliberately NOT the repo's `.gitleaks.toml` (a public-repo CI fixture policy); the allowlist block (`loadWorkspaceAllowlist`, the glob dialect `globToRegExp`/`matchesGlob`, `valueAllowlisted`/`pathAllowlisted`, `SCAN_ALLOW_FILENAME`, the 16-hex fingerprint floor) lives in `src/core/secret-scan-allowlist.ts` since wave 16 (G4.1, the 1500-line module cap) and is re-exported here unchanged (`test/secret-scan-allowlist.test.ts`). `correctedEntropy` is one histogram pass (distinct count = histogram size; byte-identical to the former `shannonEntropy + Set` form, `test/secret-scan-entropy-equivalence.test.ts`; ~80 → ~25 ms per 250 × 4096-char segment). `ScanOpts.work` / `RedactOpts.work` takes a `ScanWork` accumulator (`src/core/secret-scan-work.ts`: lines, regex execs, candidates per pattern, chars handed to the entropy gate counted inside `correctedEntropy`, validate calls, claims, union splits, echo sweeps) pinned exactly on the perf fixture by `test/secret-scan-work-counters.test.ts`; the perf file's absolute bounds stay the only guard on per-character cost. Deliberately NOT patterns: `sb_publishable_`, Supabase project refs (identifiers, not credentials). Never spell a literal database or `http(s)` URL with userinfo, or any credential, in this file's comments or in fixtures (assemble test values at runtime; `scripts/check-pg-url-redaction.sh` greps `src/` for every `db_url_credentials` scheme). Long lines are scanned in full (no length cap — a secret on a 1 MB minified line must still redact); what keeps that linear is bounded quantifiers on every shape whose value class could otherwise run to end-of-line — `db_url_credentials` (user ≤128 stopping at `:`/`/`; password 1–256 stopping only at whitespace, `@` and the string delimiters `"`/`'`, so unescaped `/ { } < > | ^ \` and backtick passwords redact), `jwt` (prebuilt so its boundary excludes `-` as well as `_`; segments ≤4096), `high_entropy_assignment` (≤64 identifier chars after the keyword, value ≤4096), `url_credentials` (user ≤128, password ≤256), `basic_auth` (value ≤2048, so its decode is constant work) and the private-key walks (below) — plus a per-pattern `precheck` substring gate on the catch-alls (`@` for both URL patterns; `earer`/`EARER`, since an all-caps `BEARER` scheme is accepted; a case-insensitive `basic`), a carried next-newline position in `scanPemBlocks` (sticky -1, so the header-line count is O(text) even when thousands of hits sit ahead of a newline-free multi-megabyte tail), a lazily-allocated per-line claimed-char bitmap behind the first-wins dedupe (O(value) per hit), and a windowed `buildPreview` (whole line ≤160 chars; otherwise hit−40…+80, edges snapped outward to whole spans, every claimed span in the window spliced by position, so a preview never carries a sibling secret or a fragment). `redactFindings` (= `planRedaction` + `applyRedaction`) rebuilds the text by SPAN-SPLICE in one O(text + hits) pass over the claimed spans, and fills the UNCLAIMED gaps with a bounded echo pass: the values claimed by `bearer`, `high_entropy_assignment`, `labeled_credential` and every vendor or `jwt` rule (#7978; not `basic_auth` or the two URL userinfo catch-alls) form an echo dictionary (value → pattern; floors `ECHO_MIN_CHARS_BEARER` = `ECHO_MIN_CHARS_VENDOR` = 20 / `ECHO_MIN_CHARS_ENTROPY` = 12, values ≤ `ECHO_MAX_VALUE_CHARS` = 512, the first `ECHO_MAX_UNIQUE` = 64 unique bearer, assignment and labeled values plus, under their own cap, the first `ECHO_MAX_UNIQUE_VENDOR` = 32 unique vendor or `jwt` values, in claim order, so planted vendor strings never crowd out a bearer token's echo — a value past a cap is redacted at its claim, its echoes are the accepted miss), matched by a streaming merge over the ORIGINAL text — per value, the next occurrence at or beyond the emit cursor (`indexOf` resuming at the cursor, re-sought only once the cursor passes it), leftmost-longest at each step, emitted incrementally into the unclaimed gaps — so retained state is O(values) however long the text is (occurrences are never collected into a list) (never an alternation RegExp, never over emitted tokens). Echoes add no findings. `planRedaction` feeds a caller-supplied shared map (`RedactOpts.echoValues`) and `applyRedaction` reads it at apply time, so a multi-field document scrubs an echo in one field of a value claimed in another. `basic_auth` and the URL userinfo catch-alls redact exactly what `scanText` reports; a vendor value embedded past its boundary (`token_<key>`) is scrubbed as an echo when the same value is claimed elsewhere in the document, and left as-is otherwise. PRIVATE KEYS: `pemClaims` finds `BEGIN`/`END … PRIVATE KEY` fences in two global passes and merges them in text order. BLOCK: a BEGIN whose next fence is an END within `PEM_BODY_MAX_CHARS` (16384) is claimed header through END; a block never swallows a later BEGIN. FORWARD (BEGIN with no END in view): the header plus the body lines that follow; a header followed by no body line (a marker quoted in prose) is claimed alone. BACKWARD (END with no BEGIN in view): the body lines before the fence plus the fence; an END with no body line before it is no claim. A body line is a whole-line base64 run (`A-Za-z0-9+/=`) indented by up to 16 spaces/tabs; line ends are `\n`, `\r\n` or the literal JSON escapes; interior lines are 16–128 chars, the line next to a fence and a partial line at a cut 1–128; optional RFC 1421 `Proc-Type:`/`DEK-Info:` lines and one blank line may follow BEGIN; a line with anything else, or a run over 128 chars, ends the body. Total claimed body stays within `PEM_BODY_MAX_CHARS`. Each walk stops at the neighbouring fence or the previous claim, so the pass is linear (20k unterminated BEGINs, 20k END-only fences and 1 MB of base64 lines are perf-pinned). Per-line hits that fall wholly inside a key claim are dropped (one key, one finding). A FORWARD claim's finding also carries `legacyFingerprint`, the header-only fingerprint earlier releases reported, so `workspace-push.ts` can name a stale allowlist entry; a truncated key's fingerprint covers its body and therefore depends on where the text was cut, so it is not a stable allowlist key. `privateKeySpans(text)` exports the same claims as sorted, disjoint `{start, end}` spans. Accepted misses: body on the header's own line, space-joined single-line keys, blockquoted (`> `) keys, base64 runs over 128 chars, body past 16384 chars, `http(s)` passwords containing a literal `/`, `?` or `#`, Basic values over 2048 chars, digitless assignment values, dotted passwords whose every segment starts with a letter, and unquoted passwords containing `,` or `;`. False-positive budget: `scripts/secret-scan-fp-budget.ts` diffs findings against a base ref's scanner (`--base`, default `origin/master`) (`--dir <brain>` reports per-pattern counts only, never content) and `test/secret-scan-fp-budget.test.ts` pins a synthetic brain-like fixture (zero findings on its non-secret part in both modes, every planted secret redacted). Shape tests: `test/secret-scan-credential-shapes.test.ts`, `test/secret-scan-private-key-spans.test.ts`. Perf pins: `test/secret-scan-perf.test.ts`. Retrieval output is another consumer of its canonical scanner; that output boundary does not sanitize internal evaluation/provider inputs.-- `src/core/secret-scan-cookie.ts` — the transcript-lane `cookie_header` rule (GST-2768), run by `scanInternal` only under `ScanOpts.transcriptRules` (`TRANSCRIPT_LANE_SCAN_OPTS` in `secret-scan.ts`: redactSession, the recent-transcripts read and `transcripts audit-secrets`). `cookieHeaderValues(line)` reads the `name=value` pairs after a `Cookie:` / `Set-Cookie:` header (any case, quoted or JSON-escaped) or a curl `-b`/`--cookie` argument and returns the values only (8+ chars, no placeholder or mask); `Set-Cookie` stops after its first pair, and parsing stops at the first segment that is not `name=value`. Values join the echo dictionary at 12 chars under their own cap, `ECHO_MAX_UNIQUE_COOKIE` = 64, apart from `ECHO_MAX_UNIQUE`. Known misses: a multi-line `.netrc` stanza, a `-b` on a continued line of a multi-line curl command (Chrome's "Copy as cURL") and a JSON-escaped quoted value (`sid=\"…\"`). `test/secret-scan-push-gate-unchanged.test.ts` pins every non-transcript option set byte-identical on the synthetic corpus `test/fixtures/transcripts/synthetic-credentials.ts`.+- `src/core/secret-scan-cookie.ts` — the transcript-lane `cookie_header` rule (GST-2768), run by `scanInternal` only under `ScanOpts.transcriptRules` (`TRANSCRIPT_LANE_SCAN_OPTS` in `secret-scan.ts`: redactSession, the recent-transcripts read and `transcripts audit-secrets`). `cookieHeaderValues(line)` reads the `name=value` pairs after a `Cookie:` / `Set-Cookie:` header (any case, quoted or JSON-escaped) or a curl `-b`/`--cookie` argument and returns the values only (8+ chars, no placeholder or mask); `Set-Cookie` stops after its first pair, and parsing stops at the first segment that is not `name=value`. A quoted value may be JSON-escaped (`sid=\"…\"`, up to three backslashes before each quote), and `curlContinues` carries a curl command across lines ending in `\`, so a `-b` alone on a continued line (Chrome's "Copy as cURL") is curl's (`scanInternal` passes `inCurl`). Values join the echo dictionary at 12 chars under their own cap, `ECHO_MAX_UNIQUE_COOKIE` = 64, apart from `ECHO_MAX_UNIQUE`. `test/secret-scan-push-gate-unchanged.test.ts` pins every non-transcript option set byte-identical on the synthetic corpus `test/fixtures/transcripts/synthetic-credentials.ts`.+- `src/core/secret-scan-transcript.ts` — `compileTranscriptPatterns()`, the transcript-lane-only labeled rules (`TRANSCRIPT_LABELED_PATTERNS` from `secret-scan-labeled.ts`: the `.netrc` forms and the `key`/`netrc` continuations) compiled as `labeled_credential` for `buildPatterns` in `secret-scan.ts` under `ScanOpts.transcriptRules`, after every other rule. Moved out of `secret-scan.ts` at its 1500-line cap with no logic change.- `src/core/secret-scan-stream.ts` — `scanTextStream(chunks, opts)`, the windowed scan of a byte stream too large to hold as one string (the push gate's staged blobs over 25 MiB). Bytes are decoded by a streaming `TextDecoder` (`ignoreBOM`, so a split multi-byte character joins and invalid bytes become U+FFFD as `Buffer.toString` makes them) and handed to `scanTextSpans` (`secret-scan.ts`: `scanText` plus each claim's absolute span, allowlisted claims kept with `finding: null`) in windows of at most `STREAM_SCAN_WINDOW_CHARS` (8 Mi UTF-16 units). Each next window starts at a line start at least `STREAM_SCAN_OVERLAP_CHARS` (64 Ki, above the longest claim: a private key's 16384-char body plus fences, a jwt's 3 × 4096) before the previous end, or mid-line at exactly the overlap (never inside a surrogate pair) when the line is too long. A window keeps the claims that start before the next window's start, the last window keeps the rest, and a claim wholly inside a span an earlier window kept (a seam's partial view, or an exact duplicate) is dropped; lines are 1-based over the whole stream. `streamScanWork` (@internal) counts windows, bytes, scanned chars and the largest window. `test/secret-scan-stream.test.ts` pins equality with `scanText` across both seams of a window, mid-line splits, split multi-byte characters, invalid UTF-8 and a BOM.- `src/core/context/sensitivity-scan.ts` — composed detector for compiled-context entries (secret-scan → PII → path shapes → operator blocklist → operator pattern file). secret-scan OWNS credential-shaped families it also matches: a PII finding whose value (raw, or with the `Bearer ` keyword stripped) matches a secret finding's fingerprint is not reported twice; short bearer tokens under secret-scan's floor still surface as `pii:bearer`. Every family honors the same `.gbrain-scan-allow` fingerprint escape hatch. `formatSensitivityDrop(drop, workspaceRoot)` renders the one content-free stderr line `gbrain compile-context` prints per dropped entry: reason `sensitivity_scan`, the pattern family, the fingerprint and the absolute `.gbrain-scan-allow` path, never the value (`test/sensitivity-scan.test.ts`).diff --git a/docs/architecture/key-files/security.md b/docs/architecture/key-files/security.mdindex 690b767ae1..6553e008d4 100644--- a/docs/architecture/key-files/security.md+++ b/docs/architecture/key-files/security.md@@ -27,7 +27,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/core/scope.ts` — OAuth scope hierarchy + allowlist (the 6 scope strings, `hasScope`, `assertAllowedScopes`, `filterAllowedScopes`, `parseScopeString`, `normalizeScopesInput`) AND the anonymous-registration policy: `DCR_REGISTRABLE_SCOPES` (`read`, `write`) and `dcrScopeViolation(requested, grantTypes)` — returns `null` when the requested scopes fit the Dynamic Client Registration ceiling, otherwise the `invalid_client_metadata` message naming the offending scopes and the operator remedy (`gbrain auth register-client` / admin API / `gbrain auth rescope-client <client_id> --scopes ...`). `agent` keeps its own "operator-approved grant with explicit delegation bindings" wording; when `grantTypes` includes `client_credentials` (only reachable under `--enable-dcr-insecure`) the ceiling is `read` alone; unknown scope strings are ignored here (the unknown-filter drops them). `scopesSupportedForDiscovery({ enableDcr })` is the `scopes_supported` list for OAuth discovery (consumed by `serve-http.ts`'s `authRouterOptions`): the DCR-registrable subset (`read write`, canonical order) while DCR is enabled — so a client that copies discovery into its registration passes `dcrScopeViolation` — and every scope except `agent` otherwise; it never narrows an operator-registered client, because a client that omits `scope` at `/authorize` receives its full registered set via `grantScopes` (`oauth-grants.ts`) and an explicit request is intersected with the registration. The DCR constants and the helper sit BELOW `ALLOWED_SCOPES_LIST`, and no comment line pairs that identifier with `[`, because `scripts/check-admin-scope-drift.sh` extracts the allowlist block with an awk match on exactly that pair. Pinned by `test/scope.test.ts` + `test/oauth-dcr-scope-ceiling.test.ts` + the discovery test in `test/e2e/serve-http-consent.test.ts`.-- `src/core/secret-scan-labeled.ts` — the transcript-lane `labeled_credential` detector (#6147 idea, widened), compiled by `secret-scan.ts` only under `ScanOpts.labeledCredentials` (transcript pages, the recent-transcripts read and `transcripts audit-secrets`), after every other rule so a vendor or entropy claim keeps attribution. SINGLE labels (`password|passwd|passcode|passphrase|pwd`, optional identifier prefix, JSON-key quote, markdown bold) before `:`/`=`/`:`, and PAIR labels (`login|log-in|credentials|creds|user/pass|username/password|…`) before `:`/`=`/`:`/` - `/nothing then `user / pass` or `user:pass`, each in a bare and a quoted/backticked form; only the password half is claimed. `labeledValueIsCredential(value, form, head)` (the scanner passes the match head to `validate`) rejects placeholders, masks, templating, URLs, paths, calls, dotted references, digitless code identifiers, a backtick that closes a code span holding the label, a meeting link's `?pwd=`, and `LABELED_STOPLIST` words; a pair's password also needs 4+ chars with a letter and a digit or symbol, and its user half must not be a stopword. W4.1 widens it: CLI flags with a separate value (`--password X`, `--pass "a b"`; suffixed flags such as `--password-file` are not labels), an env-name suffix after the label (`PASSWORD_DB=`; a setting suffix such as `_FILE`/`_MIN_LENGTH`/`_HINT` rejects via `settingSuffix`), a `cli` form (prefixed flags such as `--db-password`, `--pw`, openssl `-passin pass:`) whose value may not start with `-` (other forms keep `-`-leading passwords), the setting-suffix exemption only for single labels, and Markdown tables (`tableCredentialCells`, `pw`/`Password (prod)` headers, `|-|` separators, a per-row column Set so a row is linear in cells with per-scan `TableScanState`: cells under a password-label header column and the value cell of a non-header `| password | X |` row; escaped pipes, no outer pipes, several columns). `danglingLabel` gates the next-line continuation patterns per form: `pair` after a dangling pair, `single` (the whole next line is the value, validated as a single label) after a line ending in a single label and delimiter; such a line, and a table row with a claimed cell, is scanned even when shorter than 8 chars. Claimed values join the echo dictionary at `ECHO_MIN_CHARS_LABELED` = 8 when `labeledEchoEligible`. Every quantifier is bounded (perf-pinned in `test/secret-scan-perf.test.ts`); acceptance table in `test/transcript-labeled-credential.test.ts`; user doc `docs/guides/data-ingestion.md#credential-redaction`. GST-2768: `TRANSCRIPT_LABELED_PATTERNS` (compiled only under `ScanOpts.transcriptRules`) adds the `.netrc` `password` form and the `key` continuation after `danglingKeyLabel` (a key or token label ending its line), validated by `keyLabelValueIsCredential` (12+ chars, one token, a letter and a digit).+- `src/core/secret-scan-labeled.ts` — the transcript-lane `labeled_credential` detector (#6147 idea, widened), compiled by `secret-scan.ts` only under `ScanOpts.labeledCredentials` (transcript pages, the recent-transcripts read and `transcripts audit-secrets`), after every other rule so a vendor or entropy claim keeps attribution. SINGLE labels (`password|passwd|passcode|passphrase|pwd`, optional identifier prefix, JSON-key quote, markdown bold) before `:`/`=`/`:`, and PAIR labels (`login|log-in|credentials|creds|user/pass|username/password|…`) before `:`/`=`/`:`/` - `/nothing then `user / pass` or `user:pass`, each in a bare and a quoted/backticked form; only the password half is claimed. `labeledValueIsCredential(value, form, head)` (the scanner passes the match head to `validate`) rejects placeholders, masks, templating, URLs, paths, calls, dotted references, digitless code identifiers, a backtick that closes a code span holding the label, a meeting link's `?pwd=`, and `LABELED_STOPLIST` words; a pair's password also needs 4+ chars with a letter and a digit or symbol, and its user half must not be a stopword. W4.1 widens it: CLI flags with a separate value (`--password X`, `--pass "a b"`; suffixed flags such as `--password-file` are not labels), an env-name suffix after the label (`PASSWORD_DB=`; a setting suffix such as `_FILE`/`_MIN_LENGTH`/`_HINT` rejects via `settingSuffix`), a `cli` form (prefixed flags such as `--db-password`, `--pw`, openssl `-passin pass:`) whose value may not start with `-` (other forms keep `-`-leading passwords), the setting-suffix exemption only for single labels, and Markdown tables (`tableCredentialCells`, `pw`/`Password (prod)` headers, `|-|` separators, a per-row column Set so a row is linear in cells with per-scan `TableScanState`: cells under a password-label header column and the value cell of a non-header `| password | X |` row; escaped pipes, no outer pipes, several columns). `danglingLabel` gates the next-line continuation patterns per form: `pair` after a dangling pair, `single` (the whole next line is the value, validated as a single label) after a line ending in a single label and delimiter; such a line, and a table row with a claimed cell, is scanned even when shorter than 8 chars. Claimed values join the echo dictionary at `ECHO_MIN_CHARS_LABELED` = 8 when `labeledEchoEligible`. Every quantifier is bounded (perf-pinned in `test/secret-scan-perf.test.ts`); acceptance table in `test/transcript-labeled-credential.test.ts`; user doc `docs/guides/data-ingestion.md#credential-redaction`. GST-2768: `TRANSCRIPT_LABELED_PATTERNS` (compiled only under `ScanOpts.transcriptRules`) adds the `.netrc` `password` form (one line, or a `password` line inside a multi-line stanza that `netrcStanzaOpen` tracks from a `machine`/`default` line through `login`/`account`/`password`, blank and `#` comment lines, the `netrc` continuation form) and the `key` continuation after `danglingKeyLabel` (a key or token label ending its line), validated by `keyLabelValueIsCredential` (12+ chars, one token, a letter and a digit).- `src/core/ssrf-validate.ts` — DNS-rebinding-defended URL validation. `validateAndResolveUrl(url)` resolves the hostname via `dns.lookup({all: true, family: 0})`, checks EVERY A and AAAA record against the internal-IP deny list, and returns the resolved IP so callers fetch by IP (validation IP === fetch IP defeats DNS rebinding). `fetchWithSSRFGuard(url, opts)` does redirect-aware fetching with per-hop re-validation (max 3 hops by default). Reusable across all URL-fetching features. Test seam `__setDnsLookupForTests` for hermetic tests.diff --git a/docs/guides/data-ingestion.md b/docs/guides/data-ingestion.mdindex 0cae7d70eb..6649137f81 100644--- a/docs/guides/data-ingestion.md+++ b/docs/guides/data-ingestion.md@@ -122,7 +122,8 @@ forms are recognized, with bare, quoted or backticked values:`DB password` and the like, and the value in a `| password | … |` row,including rows without outer pipes, escaped `\|` pipes and tables withseveral credential columns;-- a one-line `.netrc` entry, `machine host login alice password …`;+- a `.netrc` entry, on one line (`machine host login alice password …`) or+ over several (`machine`, `login` and `password` each on its own line);- a key or token label that ends its line (`here is my API key:`,`ACCESS_TOKEN=`), with one value of 12+ characters holding a letter and adigit alone on the next line;@@ -131,14 +132,13 @@ forms are recognized, with bare, quoted or backticked values:Cookie values are redacted too (`cookie_header`): after a `Cookie:` or`Set-Cookie:` header (raw headers, `curl -H`, a copy-as-fetch JSON blob) and-after `-b`/`--cookie` on the same line as `curl`, every value of 8+-characters is replaced and the cookie names and `Set-Cookie` attributes+after `-b`/`--cookie` in a `curl` command, including on a continued line+of a multi-line command (what Chrome's "Copy as cURL" produces), every+value of 8+ characters is replaced, quoted values inside escaped JSON+(`sid=\"…\"`) too, and the cookie names and `Set-Cookie` attributesstay. These transcript-only rules never run in the `sources push` gate orthe compiled-context scan.-Known gaps: three cases still reach the page verbatim, as they did before-these rules: a `.netrc` entry written over several lines (`machine`, `login` and `password` each on its own line), a `-b` cookie on its own continued line of a multi-line curl command (what Chrome's "Copy as cURL" produces), and a quoted cookie value inside escaped JSON (`sid=\"…\"`).-To keep prose and code intact, a value is never redacted when it is aplaceholder (`<…>`, `${…}`, `$VAR`), a mask (`****`), a URL, a path, afunction call, a dotted reference (`req.body.password`), a code identifierdiff --git a/src/core/secret-scan-cookie.ts b/src/core/secret-scan-cookie.tsindex 78a9f90117..90e4757239 100644--- a/src/core/secret-scan-cookie.ts+++ b/src/core/secret-scan-cookie.ts@@ -16,15 +16,16 @@* (`Path=/`, `Max-Age=3600`) are kept. Parsing stops at the first segment* that is not `name=value`, so prose (`Cookie: chocolate chip`) is never* claimed. A `'` or `,` inside a value and a JSON-escaped `\/` are part of it.+ * A quoted value may be JSON-escaped (`sid=\"…\"`, `sid=\\\"…\\\"`), and a `-b` on a continued+ * line of a multi-line curl command (Chrome's "Copy as cURL": every line before it ends in `\`)+ * counts as a curl argument (`curlContinues`).* Values shorter than 8 characters (preference flags such as* `theme=dark`), placeholders (`<…>`, `$VAR`, `{…}`, `%VAR%`) and masks (`...`, `***`)* are left alone. Every quantifier is bounded; the parse is linear in the line.- * Not caught: a `-b` on a continued line of a multi-line curl command- * (Chrome's "Copy as cURL") and a JSON-escaped quoted value (`sid=\"…\"`).*/const COOKIE_HEAD_RE = /(?:(?:^|[^A-Za-z0-9_-])(set-)?cookie\\?["'`]?\s{0,4}:|(?:^|\s)(-b|--cookie)(?=[\s=]))[\s=]{0,4}\\?["'`]?/gi;-const COOKIE_PAIR_RE = /([!#$%&*+.^_|~0-9A-Za-z-]{1,128})=("?)((?:[^\s;"',\\]|[',](?=[^\s;"',\\])|\\\/){0,4096})/y;+const COOKIE_PAIR_RE = /([!#$%&*+.^_|~0-9A-Za-z-]{1,128})=((?:\\{1,3})?"|)((?:[^\s;"',\\]|[',](?=[^\s;"',\\])|\\\/){0,4096})/y;const COOKIE_SEP_RE = /\s{0,4};\s{0,4}/y;const PLACEHOLDER_RE = /^(?:[<${[]|%[A-Za-z_][A-Za-z0-9_]{0,64}%$)/;const CURL_RE = /(?:^|[^A-Za-z0-9_-])curl(?:\.exe)?(?:\s|$)/i;@@ -43,27 +44,35 @@ export const COOKIE_VALUE_MIN_CHARS = 8;/** Cheap gate before the header regex runs on a line. */export const COOKIE_PRECHECK_RE = /cookie|-b\s/i;-/** The cookie values on `line`: `start` of the value, the value, and `from`, where its header starts. */-export function cookieHeaderValues(line: string): Array<{ start: number; value: string; from: number }> {+/** True when the line after `line` still belongs to a curl command: `line` ends in `\` and is a curl line or one continued from it (`inCurl`). */+export function curlContinues(line: string, inCurl: boolean): boolean {+ return /\\\s{0,4}$/.test(line) && (inCurl || CURL_RE.test(line));+}++/**+ * The cookie values on `line`: `start` of the value, the value, and `from`, where its header starts.+ * `inCurl`: the line continues a curl command (`curlContinues`), so a `-b` on it is curl's.+ */+export function cookieHeaderValues(line: string, inCurl = false): Array<{ start: number; value: string; from: number }> {const out: Array<{ start: number; value: string; from: number }> = [];let curlAt: number | undefined;COOKIE_HEAD_RE.lastIndex = 0;for (let head = COOKIE_HEAD_RE.exec(line); head !== null; head = COOKIE_HEAD_RE.exec(line)) {const setCookie = head[1] !== undefined;let at = COOKIE_HEAD_RE.lastIndex;- if (head[2] !== undefined && !((curlAt ??= line.search(CURL_RE)) >= 0 && curlAt < head.index)) continue;+ if (head[2] !== undefined && !inCurl && !((curlAt ??= line.search(CURL_RE)) >= 0 && curlAt < head.index)) continue;for (;;) {COOKIE_PAIR_RE.lastIndex = at;const m = COOKIE_PAIR_RE.exec(line);if (m === null) break;- const quoted = m[2] === '"';- if (quoted && line[COOKIE_PAIR_RE.lastIndex] !== '"') break;+ const quote = m[2]!;+ if (quote !== '' && !line.startsWith(quote, COOKIE_PAIR_RE.lastIndex)) break;const value = m[3]!;if (value.length >= COOKIE_VALUE_MIN_CHARS && !PLACEHOLDER_RE.test(value) && !MASK_RE.test(value)) {- out.push({ start: m.index + m[1]!.length + 1 + m[2]!.length, value, from: head.index });+ out.push({ start: m.index + m[1]!.length + 1 + quote.length, value, from: head.index });}if (setCookie) break;- COOKIE_SEP_RE.lastIndex = COOKIE_PAIR_RE.lastIndex + (quoted ? 1 : 0);+ COOKIE_SEP_RE.lastIndex = COOKIE_PAIR_RE.lastIndex + quote.length;if (COOKIE_SEP_RE.exec(line) === null) break;at = COOKIE_SEP_RE.lastIndex;}diff --git a/src/core/secret-scan-labeled.ts b/src/core/secret-scan-labeled.tsindex 36197b72d9..db453ca621 100644--- a/src/core/secret-scan-labeled.ts+++ b/src/core/secret-scan-labeled.ts@@ -43,7 +43,11 @@* `password <value>`, and a key or token label left dangling at a line end* (`here is my API key:`, `ACCESS_TOKEN=`) with its value alone on the next* line. The transcript lane also carries a dangling label of any form into- * the first line of the NEXT message (`ScanOpts.leadingDangling`).+ * the first line of the NEXT message (`ScanOpts.leadingDangling`). A `.netrc`+ * entry written over several lines (`machine host`, then `login alice` and+ * `password …` on lines of their own) is read as one stanza+ * (`netrcStanzaOpen`): a line that is only `password <value>` (optionally+ * after `login`/`account` tokens) inside it is claimed.** Accepted misses: an unquoted multi-word passphrase (`login: alice / pass* word`), a delimiter-free single label (`the password is hunter2`), a pair@@ -76,7 +80,7 @@ const CLI_HEAD = '(?:^|\\s)--?(?!(?:no|skip|ask|prompt|reset|show|print|change|cconst OPENSSL_PASS_HEAD = '(?:^|\\s)-pass(?:in|out)\\s{1,4}pass:';/** The label form a line leaves dangling for the next line: `key` is the transcript-lane key or token label. */-export type DanglingForm = 'single' | 'pair' | 'key';+export type DanglingForm = 'single' | 'pair' | 'key' | 'netrc';export interface LabeledPattern {source: string;@@ -90,7 +94,7 @@ export interface LabeledPattern {export interface TranscriptLabeledPattern {source: string;form: 'single' | 'key';- continuation?: 'key';+ continuation?: 'key' | 'netrc';}/** Group 1 = everything before the value, group 2 = the value (the scanner's shape). */@@ -111,14 +115,28 @@ export const LABELED_CREDENTIAL_PATTERNS: readonly LabeledPattern[] = [/** A key or token label (`API key`, `access_token`, `client secret`), transcript lane only. */const KEY_LABEL = '(?:api[ _-]?key|access[ _-]?key|secret[ _-]?key|client[ _-]?secret|secret|token)s?';const NETRC_HEAD = '(?:^|\\s)(?:machine\\s{1,4}[^\\s]{1,253}|default)(?:\\s{1,4}(?:login|account)\\s{1,4}[^\\s]{1,128}){0,2}\\s{1,4}password\\s{1,4}';+/** A `password` line inside a multi-line `.netrc` stanza, optionally after its `login`/`account` tokens. */+const NETRC_LINE_HEAD = '^\\s{0,8}(?:(?:login|account)\\s{1,4}[^\\s]{1,128}\\s{1,4}){0,2}password\\s{1,4}';+/** A line that opens a multi-line `.netrc` stanza (`machine host`, `default`, optionally with login/account but no password). */+const NETRC_STANZA_OPEN_RE = /^\s{0,8}(?:machine\s{1,4}\S{1,253}|default)(?:\s{1,4}(?:login|account)\s{1,4}\S{1,128}){0,2}\s{0,4}$/;+/** A line that keeps a `.netrc` stanza open: only `login`, `account` and `password` tokens, a blank line or a `#` comment. */+const NETRC_STANZA_LINE_RE = /^\s{0,8}(?:(?:(?:login|account|password)\s{1,4}\S{1,255}\s{0,4}){1,3}|#.*|\s*)$/;++/** True when the line after `line` is inside a multi-line `.netrc` stanza; `open` says whether `line` itself was. */+export function netrcStanzaOpen(line: string, open: boolean): boolean {+ return NETRC_STANZA_OPEN_RE.test(line) || (open && NETRC_STANZA_LINE_RE.test(line));+}/*** GST-2768: the transcript-lane-only labeled shapes (`ScanOpts.transcriptRules`), claimed as- * `labeled_credential`: a `.netrc` password, and the value on the line after a dangling key label.+ * `labeled_credential`: a `.netrc` password (one line, or a `password` line inside a multi-line+ * stanza), and the value on the line after a dangling key label.*/export const TRANSCRIPT_LABELED_PATTERNS: readonly TranscriptLabeledPattern[] = [{ form: 'single', source: `(${NETRC_HEAD}${QUOTE})(${QUOTED_VALUE})(?=${QUOTE})` },{ form: 'single', source: `(${NETRC_HEAD})(${BARE_VALUE})${BARE_END}` },+ { form: 'single', continuation: 'netrc', source: `(${NETRC_LINE_HEAD}${QUOTE})(${QUOTED_VALUE})(?=${QUOTE}\\s{0,8}$)` },+ { form: 'single', continuation: 'netrc', source: `(${NETRC_LINE_HEAD})([^\\s"'\`]{1,256})(?=\\s{0,8}$)` },{ form: 'key', continuation: 'key', source: `(^\\s{0,8}${QUOTE})(${QUOTED_VALUE})(?=${QUOTE}[,;]?\\s*$)` },{ form: 'key', continuation: 'key', source: `(^\\s{0,8})(${BARE_VALUE})(?=[.,;]?\\s*$)` },];diff --git a/src/core/secret-scan-transcript.ts b/src/core/secret-scan-transcript.tsnew file mode 100644index 0000000000..09f17633fb--- /dev/null+++ b/src/core/secret-scan-transcript.ts@@ -0,0 +1,39 @@+/**+ * secret-scan-transcript.ts: the transcript-lane-only (`ScanOpts.transcriptRules`, GST-2768) labeled patterns, compiled+ * for `buildPatterns` in secret-scan.ts. Moved out of secret-scan.ts at its 1500-line module cap; the shapes and their+ * checks live in secret-scan-labeled.ts.+ */+import {+ TRANSCRIPT_LABELED_PATTERNS,+ NETRC_PRECHECK_RE,+ keyLabelValueIsCredential,+ labeledValueIsCredential,+ neutralizeRedactionMarkers,+ type DanglingForm,+} from './secret-scan-labeled.ts';++/** A compiled transcript-lane pattern, in the shape of secret-scan.ts's `CompiledPattern`. */+export interface CompiledTranscriptPattern {+ name: 'labeled_credential';+ re: RegExp;+ continuation?: DanglingForm;+ precheck?: (line: string) => boolean;+ validate: (value: string, head: string) => boolean;+ subject: (line: string) => string;+}++/** `TRANSCRIPT_LABELED_PATTERNS` compiled as `labeled_credential` rules, appended after every other rule. */+export function compileTranscriptPatterns(): CompiledTranscriptPattern[] {+ const out: CompiledTranscriptPattern[] = [];+ for (const p of TRANSCRIPT_LABELED_PATTERNS) {+ out.push({+ name: 'labeled_credential',+ // nosemgrep: javascript.lang.security.audit.detect-non-literal-regexp.detect-non-literal-regexp -- compile-time literal; bounded quantifiers, pinned by test/secret-scan-perf.test.ts+ re: new RegExp(p.source, 'gi'),+ ...(p.continuation ? { continuation: p.continuation } : { precheck: (line: string) => NETRC_PRECHECK_RE.test(line) }),+ validate: p.form === 'key' ? keyLabelValueIsCredential : (value: string, head: string) => labeledValueIsCredential(value, 'single', head),+ subject: neutralizeRedactionMarkers,+ });+ }+ return out;+}diff --git a/src/core/secret-scan.ts b/src/core/secret-scan.tsindex 7b786e6e6d..f1fb2aabe1 100644--- a/src/core/secret-scan.ts+++ b/src/core/secret-scan.ts@@ -35,10 +35,8 @@ import {LABELED_CREDENTIAL_PATTERNS,ECHO_MIN_CHARS_LABELED,LABELED_PRECHECK_RE,- TRANSCRIPT_LABELED_PATTERNS,- NETRC_PRECHECK_RE,danglingFor,- keyLabelValueIsCredential,+ netrcStanzaOpen,type DanglingForm,labeledEchoEligible,labeledValueIsCredential,@@ -93,8 +91,8 @@ export interface ScanOpts {*/labeledCredentials?: boolean;/**- * Opt-in transcript-lane rules (GST-2768): `cookie_header` (secret-scan-cookie.ts), the `.netrc` password and the- * key-label next-line value (secret-scan-labeled.ts). Only via `TRANSCRIPT_LANE_SCAN_OPTS`. Default false.+ * Opt-in transcript-lane rules (GST-2768): `cookie_header` (secret-scan-cookie.ts), the `.netrc` password (one line or a+ * multi-line stanza) and the key-label next-line value (secret-scan-labeled.ts). Only via `TRANSCRIPT_LANE_SCAN_OPTS`. Default false.*/transcriptRules?: boolean;/** Label form the previous message left dangling (`trailingDanglingLabel`): line 1 here is read as its value. Default none. */@@ -114,7 +112,8 @@ import {} from './secret-scan-assignment.ts';import type { ScanWork } from './secret-scan-work.ts';export { createScanWork, type ScanWork } from './secret-scan-work.ts';-import { COOKIE_PRECHECK_RE, ECHO_MAX_UNIQUE_COOKIE, cookieHeaderValues } from './secret-scan-cookie.ts';+import { COOKIE_PRECHECK_RE, ECHO_MAX_UNIQUE_COOKIE, cookieHeaderValues, curlContinues } from './secret-scan-cookie.ts';+import { compileTranscriptPatterns } from './secret-scan-transcript.ts';export { TRANSCRIPT_LANE_SCAN_OPTS, trailingDanglingLabel } from './secret-scan-labeled.ts';// ── Patterns ────────────────────────────────────────────────────────────────@@ -719,18 +718,7 @@ function buildPatterns(opts: ScanOpts): CompiledPattern[] {});}}- if (opts.transcriptRules) {- for (const p of TRANSCRIPT_LABELED_PATTERNS) {- out.push({- name: 'labeled_credential',- // nosemgrep: javascript.lang.security.audit.detect-non-literal-regexp.detect-non-literal-regexp -- compile-time literal; bounded quantifiers, pinned by test/secret-scan-perf.test.ts- re: new RegExp(p.source, 'gi'),- ...(p.continuation ? { continuation: p.continuation } : { precheck: (line: string) => NETRC_PRECHECK_RE.test(line) }),- validate: p.form === 'key' ? keyLabelValueIsCredential : (value: string, head: string) => labeledValueIsCredential(value, 'single', head),- subject: neutralizeRedactionMarkers,- });- }- }+ if (opts.transcriptRules) out.push(...compileTranscriptPatterns());return out;}@@ -895,13 +883,16 @@ function scanInternal(text: string, opts: ScanOpts): RawHit[] {// W4.1: Markdown table state for the labeled lane (credential columns of the table being scanned).const table = opts.labeledCredentials === true ? newTableScanState() : null;let leadingLine = 0;+ // GST-2768 follow-up: a multi-line .netrc stanza and a curl command continued with `\` carry to later lines.+ let netrc = false, inCurl = false;for (let i = 0; i < lines.length; i++) {const line = lines[i] ?? '';const lineStart = offset;offset += line.length + 1;+ if (opts.transcriptRules === true && i > 0) { netrc = netrcStanzaOpen(lines[i - 1] ?? '', netrc); inCurl = curlContinues(lines[i - 1] ?? '', inCurl); }// The previous message's dangling label reads the first non-blank line here.if (i === leadingLine && opts.leadingDangling && line.trim() === '') leadingLine = i + 1;- const dangling = i === leadingLine ? (opts.leadingDangling ?? null) : i > 0 ? danglingFor(lines[i - 1] ?? '', opts) : null;+ const dangling = (i === leadingLine ? (opts.leadingDangling ?? null) : i > 0 ? danglingFor(lines[i - 1] ?? '', opts) : null) ?? (netrc ? 'netrc' : null);const cells = table ? tableCredentialCells(line, lines[i + 1], table) : [];if (line.length < 8 && !dangling && cells.length === 0) continue;// ENG-12: a line wholly inside a private-key claim is already redacted by@@ -984,7 +975,7 @@ function scanInternal(text: string, opts: ScanOpts): RawHit[] {for (const c of cells) if (!taken || !anyTaken(taken, c.start, c.start + c.value.length)) { taken ??= new Uint8Array(line.length); claim('labeled_credential', c.value, c.start, c.start); }// GST-2768: cookie values (transcript lane), union coverage around anything already claimed.if (opts.transcriptRules === true && COOKIE_PRECHECK_RE.test(line)) {- for (const c of cookieHeaderValues(line)) {+ for (const c of cookieHeaderValues(line, inCurl)) {taken ??= new Uint8Array(line.length);for (let a = c.start, k = c.start, end = c.start + c.value.length; k <= end; k++) {if (k < end && !taken[k]) continue;diff --git a/src/core/transcripts/secret-audit.ts b/src/core/transcripts/secret-audit.tsindex 7feff2d5d3..6707ebda8d 100644--- a/src/core/transcripts/secret-audit.ts+++ b/src/core/transcripts/secret-audit.ts@@ -16,8 +16,11 @@import type { BrainEngine } from '../engine.ts';import { TRANSCRIPT_LANE_SCAN_OPTS, scanText } from '../secret-scan.ts';-/** Bump when the transcript-lane detectors change, so doctor asks for a fresh audit. 6: cookie_header, .netrc and the key-label join (GST-2768). */-export const TRANSCRIPT_SECRET_AUDIT_VERSION = 6;+/**+ * Bump when the transcript-lane detectors change, so doctor asks for a fresh audit. 6: cookie_header, .netrc and the+ * key-label join (GST-2768). 7: a multi-line .netrc stanza, a curl -b on a continued line and a JSON-escaped quoted cookie.+ */+export const TRANSCRIPT_SECRET_AUDIT_VERSION = 7;export const TRANSCRIPT_SECRET_AUDIT_KEY = 'transcripts.secret_audit';const BATCH = 25;const LINES_PER_PAGE = 20;diff --git a/test/transcripts-secret-cookie-header.test.ts b/test/transcripts-secret-cookie-header.test.tsindex 9f5b27efbb..381c9f57fa 100644--- a/test/transcripts-secret-cookie-header.test.ts+++ b/test/transcripts-secret-cookie-header.test.ts@@ -116,7 +116,7 @@ describe('audit-secrets uses the transcript-lane rules', () => {await resetPgliteState(engine);});- test('a stored page with a cookie header is affected; the audit version moved to 6', async () => {+ test('a stored page with a cookie header is affected; the audit version moved to 7', async () => {await engine.putPage('conversations/sessions/2026-01-02-codex-cccccccccccc', {type: 'conversation', title: 'session', compiled_truth: `intro\ncurl -H 'Cookie: sid=${SID}' https://app.acme-example.test/`,frontmatter: { transcript_import: { harness: 'codex', session_id: 'cookie' } },@@ -125,6 +125,48 @@ describe('audit-secrets uses the transcript-lane rules', () => {expect(result.by_pattern).toEqual({ cookie_header: 1 });expect(result.pages[0]!.lines).toEqual([2]);expect(JSON.stringify(result)).not.toContain(SID);- expect(TRANSCRIPT_SECRET_AUDIT_VERSION).toBe(6);+ expect(TRANSCRIPT_SECRET_AUDIT_VERSION).toBe(7);+ });++ test('the audit finds the follow-up shapes on a stored page', async () => {+ const pw = ['Synth', 'NetrcMulti4321'].join('');+ await engine.putPage('conversations/sessions/2026-01-03-codex-dddddddddddd', {+ type: 'conversation', title: 'session',+ compiled_truth: [+ 'machine api.acme-example.test', ' login alice-example', ` password ${pw}`,+ "curl 'https://app.acme-example.test/' \\", ` -b 'sid=${SID}'`,+ `{"headers": "Cookie: csrf=\\"${CSRF}\\""}`,+ ].join('\n'),+ frontmatter: { transcript_import: { harness: 'codex', session_id: 'follow-up' } },+ });+ const result = await auditTranscriptSecrets(engine);+ expect(result.by_pattern).toEqual({ labeled_credential: 1, cookie_header: 2 });+ expect(result.pages[0]!.lines).toEqual([3, 5, 6]);+ for (const s of [pw, SID, CSRF]) expect(JSON.stringify(result)).not.toContain(s);});});++describe('GST-2768 follow-up: continued curl lines and JSON-escaped quoted values', () => {+ test('a -b cookie on its own continued line of a multi-line curl command is redacted', () => {+ const input = [`curl 'https://app.acme-example.test/api' \\`, ` -H 'accept: */*' \\`, ` -b 'sid=${SID}; csrf=${CSRF}' \\`, ' --compressed'].join('\n');+ const out = one(input);+ expect(out).not.toContain(SID);+ expect(out).not.toContain(CSRF);+ expect(out).toContain(" -b 'sid=<REDACTED:cookie_header>; csrf=<REDACTED:cookie_header>' \\");+ });+ test('a JSON-escaped quoted cookie value is redacted', () => {+ expect(one(`{"headers": "Cookie: sid=\\"${SID}\\"; csrf=\\"${CSRF}\\""}`)).toBe('{"headers": "Cookie: sid=\\"<REDACTED:cookie_header>\\"; csrf=\\"<REDACTED:cookie_header>\\""}');+ expect(one(`Cookie: sid=\\\\\\"${SID}\\\\\\"`)).toBe('Cookie: sid=\\\\\\"<REDACTED:cookie_header>\\\\\\"');+ });+ test('ordinary text near these shapes stays', () => {+ const notCurl = ['make build \\', ' && git checkout -b feature=abcdefgh12345'].join('\n');+ expect(one(notCurl)).toBe(notCurl);+ const ended = ['curl https://app.acme-example.test/ \\', ' --compressed', 'git checkout -b feature=abcdefgh12345'].join('\n');+ expect(one(ended)).toBe(ended);+ const prefs = '{"headers": "Cookie: theme=\\"dark\\"; lang=\\"en\\""}';+ expect(one(prefs)).toBe(prefs);+ const prose = '{"msg": "tick the cookie: \\"remember me\\" box"}';+ expect(one(prose)).toBe(prose);+ });+});+diff --git a/test/transcripts-secret-corpus.test.ts b/test/transcripts-secret-corpus.test.tsindex 3514e94133..eb724400d0 100644--- a/test/transcripts-secret-corpus.test.ts+++ b/test/transcripts-secret-corpus.test.ts@@ -6,7 +6,7 @@* (test/fixtures/transcripts/synthetic-credentials.ts).*/import { describe, expect, test } from 'bun:test';-import { NAMED_SECRET_RULES } from '../src/core/secret-scan.ts';+import { NAMED_SECRET_RULES, TRANSCRIPT_LANE_SCAN_OPTS, scanText } from '../src/core/secret-scan.ts';import { redactSession } from '../src/core/transcripts/render.ts';import type { ParsedSession } from '../src/core/transcripts/types.ts';import { SYNTHETIC_CREDENTIAL_CORPUS } from './fixtures/transcripts/synthetic-credentials.ts';@@ -31,6 +31,44 @@ describe('synthetic credential corpus', () => {expect(out).toContain('password \\"<REDACTED:labeled_credential>\\"');});+ test('a .netrc entry written over several lines: the password is redacted', () => {+ const pw = ['Synth', 'NetrcMulti4321'].join('');+ for (const text of [+ `machine api.acme-example.test\n login alice-example\n password ${pw}`,+ `machine api.acme-example.test\nlogin alice-example\npassword "${pw}"`,+ `machine api.acme-example.test login alice-example\n password ${pw}`,+ `default\n password ${pw}\n login alice-example`,+ ]) {+ const out = run({ meta: { harness: 'claude-code', sessionId: 'netrc-multi' }, messages: [{ role: 'user', timestamp: TS(0), text }] });+ expect([text, out.includes(pw)]).toEqual([text, false]);+ expect(out).toContain('<REDACTED:labeled_credential>');+ }+ });++ test('a blank or comment line inside a multi-line .netrc stanza does not end it', () => {+ const pw = ['Synth', 'NetrcGap7788'].join('');+ for (const text of [+ `machine api.acme-example.test\nlogin alice-example\n\npassword ${pw}`,+ `machine api.acme-example.test\n# service account\nlogin alice-example\npassword ${pw}`,+ `machine api.acme-example.test\n login alice-example\n # rotated monthly\n\n password "${pw}"`,+ ]) {+ const out = run({ meta: { harness: 'claude-code', sessionId: 'netrc-gap' }, messages: [{ role: 'user', timestamp: TS(0), text }] });+ expect([text, out.includes(pw)]).toEqual([text, false]);+ expect(scanText(text, TRANSCRIPT_LANE_SCAN_OPTS).map((f) => f.pattern)).toEqual(['labeled_credential']);+ }+ });++ test('prose near a .netrc shape is not redacted', () => {+ for (const text of [+ 'machine learning\npassword managers are useful',+ 'machine api.acme-example.test\nthen we talked\npassword hunter2x',+ 'we used the machine api.acme-example.test\npassword Synth4242pw',+ ]) {+ const out = redactSession({ meta: { harness: 'claude-code', sessionId: 'netrc-prose' }, messages: [{ role: 'user', timestamp: TS(0), text }] }, { patterns: [] }).session.messages[0]!.text;+ expect(out).toBe(text);+ }+ });+for (const e of SYNTHETIC_CREDENTIAL_CORPUS) {test(`${e.id} (${e.shape}) in message bodies`, () => {const out = run({