The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
fix(privacy): redact snake_case secret flags and stop query strings bleeding (#1194)
Naman Omare7f5bd614a61
diff --git a/docs/changelog/cli.mdx b/docs/changelog/cli.mdxindex 2c8e9d5c..a7218a5f 100644--- a/docs/changelog/cli.mdx+++ b/docs/changelog/cli.mdx@@ -13,10 +13,11 @@ Stay up to date with the latest changes to the Agent Beacon CLI.Changes on `main` since v1.3.58. They ship in the next release.+- **Redaction covers snake_case secret flags and Stripe webhook secrets, and keeps a URL's other query parameters** · Flags written in snake_case or named for key material (`--client_secret`, `--api_key`, `--private-key`, `--secret-key`) have their values redacted, and so do Stripe webhook signing secrets (`whsec_…`). A secret passed as a URL query parameter (`?api_key=…&user=alice`) is redacted up to the next `&` or `#`, so the parameters after it stay readable; after any other label, `#`, `&` and `?` are still part of the redacted value. A diff line such as `-SECRET_KEY = '…'` keeps its `=`.+- **Run and repository metadata is redacted and capped** · The `run` block (CI provider, workflow, branch, pull-request ref, actor), the `vcs` block, and the top-level `model`, `repository` and `branch` were written as they arrived, so a credential in a branch name or remote URL reached the log, and a branch name larger than the event size limit made the whole event unwritable. They now get the same secret redaction and 2 KB limit as every other attribute string, in both the collector's writer and the CLI's.- **Renamed-binary approval bypass catches rm copies under any name** · `approval-bypass-via-renamed-binary` (v3) no longer requires a temp-dir copy of `rm` to be named like `rm`: after a denied dangerous command, running any executable from `/tmp/`, `/var/tmp/`, or `$(mktemp ...)` at command position with an rm-style recursive+force flag (`-rf`, `-v -rf`, a trailing `-rf`) correlates. Temp paths passed as arguments to other tools (`cp /tmp/build -rf`, `tar -C /tmp/stage -rf`, `find /tmp/build -printf`) do not.-- **A repository cannot turn auto-share on through the hook environment** · The hooks read the auto-share setting from `~/.beacon/endpoint/share.json` in the home of the account they run as, looked up from the account (the passwd entry, or the Windows process token), not from `$HOME` or `USERPROFILE`, so a repository that sets `HOME` for its hooks cannot plant a `share.json` that publishes its sessions. On Linux, an account that is not in `/etc/passwd` (LDAP, SSSD) still falls back to `$HOME`. `beacon share auto` writes the same file, and the restricted-log spool also moved to the account's home. The cloud-sandbox shuttle now uploads on Stop and SessionEnd only with `BEACON_ORIGIN=cloud`, which the cloud agent setups already set; `BEACON_CLOUD_*` without it does nothing.- **Fleet package installs turn the local dashboard off** · An unattended first install of the `.deb` or `.rpm`, the macOS `.pkg` or the Windows MSI (Salt, Ansible, an MDM, a silent `msiexec`) now turns the local dashboard off for every account on the machine, root included, as `BEACON_DASHBOARD=disabled` already did: that install sets up the machine for the people who use it, and they should not be able to open a view of its telemetry unless an administrator says so. A person who installs the package on their own machine (`sudo` at a terminal, including the one-line `install.sh`, Installer.app, or the MSI's full interface) keeps the dashboard, by the same test that adds them to `beacon-log`. Upgrades, and reinstalls over a kept config or log, keep the setting an endpoint already has, so set `BEACON_DASHBOARD=disabled` to turn it off on machines installed from an earlier release. `BEACON_DASHBOARD=enabled` keeps it on an unattended install, and `sudo beacon endpoint repair --system --dashboard enabled` turns it back on. Installs from the `beacon` CLI are unchanged. See [Turning the dashboard off](/cli/dashboard#turning-the-dashboard-off).diff --git a/docs/security/retention-redaction.mdx b/docs/security/retention-redaction.mdxindex 88a32217..da2e66f7 100644--- a/docs/security/retention-redaction.mdx+++ b/docs/security/retention-redaction.mdx@@ -22,12 +22,12 @@ Redaction runs on retained text (prompts, command output, diffs, tool arguments| Form | Examples ||------|----------|-| A value labeled as a secret | `token=…`, `GITHUB_TOKEN=…`, `AWS_SECRET_ACCESS_KEY=…`, `{"api_key": "…"}`, `client_secret: …` |+| A value labeled as a secret | `token=…`, `GITHUB_TOKEN=…`, `AWS_SECRET_ACCESS_KEY=…`, `{"api_key": "…"}`, `client_secret: …`, and a URL query parameter `?api_key=…`, whose value ends at the next `&` or `#` so the other parameters stay || A structured field named for a credential | the value of any field whose name ends in `password`, `secret`, `token`, `api_key`, `access_key`, `private_key`, `authorization`, `credential(s)` or `cookie` |-| Authorization headers and flags | `Authorization: Bearer …`, `Authorization: Basic …`, `--password …`, `--token=…` |+| Authorization headers and flags | `Authorization: Bearer …`, `Authorization: Basic …`, `--password …`, `--token=…`, and snake_case or key flags such as `--client_secret …`, `--api_key …`, `--private-key …`, `--secret-key …` || URL credentials | the password in `postgres://user:…@host`, `https://x-access-token:…@github.com` || Private keys | PEM and OpenSSH `PRIVATE KEY` blocks, including a block cut off before its end line |-| Recognizable tokens (the public type prefix stays, as in `ghp_[REDACTED]`) | Anthropic and OpenAI `sk-…`, AWS `AKIA…`/`ASIA…`, GitHub `ghp_…`/`github_pat_…`, GitLab `glpat-…`, Slack `xox…-` and `xapp-…`, Stripe `sk_live_…`, Google `AIza…`, npm, Hugging Face, HashiCorp Vault `hvs.…`, Groq `gsk_…`, Perplexity `pplx-…`, Replicate `r8_…`, xAI `xai-…`, LangSmith `lsv2_…`, Pinecone `pcsk_…`, DigitalOcean `dop_v1_…`, Databricks `dapi…`, PyPI `pypi-…`, Shopify `shpat_…`, Linear `lin_api_…`, Doppler `dp.pt.…`, SendGrid `SG.…`, Beacon keys and tokens (`bcn_api_…`, `bcn_device_…`, `bcn_enfdev_…`, `bcn_mcp_…`, `bcn_mcprt_…`, and the `beacon login` token `bcn_cli_…` with its `bcn_code_…`, `bcn_client_…` and `bcn_exchange_…` values), and JWTs |+| Recognizable tokens (the public type prefix stays, as in `ghp_[REDACTED]`) | Anthropic and OpenAI `sk-…`, AWS `AKIA…`/`ASIA…`, GitHub `ghp_…`/`github_pat_…`, GitLab `glpat-…`, Slack `xox…-` and `xapp-…`, Stripe `sk_live_…` and webhook signing secrets `whsec_…`, Google `AIza…`, npm, Hugging Face, HashiCorp Vault `hvs.…`, Groq `gsk_…`, Perplexity `pplx-…`, Replicate `r8_…`, xAI `xai-…`, LangSmith `lsv2_…`, Pinecone `pcsk_…`, DigitalOcean `dop_v1_…`, Databricks `dapi…`, PyPI `pypi-…`, Shopify `shpat_…`, Linear `lin_api_…`, Doppler `dp.pt.…`, SendGrid `SG.…`, Beacon keys and tokens (`bcn_api_…`, `bcn_device_…`, `bcn_enfdev_…`, `bcn_mcp_…`, `bcn_mcprt_…`, and the `beacon login` token `bcn_cli_…` with its `bcn_code_…`, `bcn_client_…` and `bcn_exchange_…` values), and JWTs || An AWS secret key beside its key id | the 40-character secret that follows an `AKIA…` or `ASIA…` key id within a few spaces, commas, colons, pipes or quotes, as in a credentials CSV (`AKIA…,wJalr…`) or a table |Redaction is pattern-based. A credential with no label and no recognizable shape, such as a bare random password, is not detected, so treat retained content as sensitive. Metadata-only forwarding is the control for keeping content off Beacon Cloud entirely, and [`beacon share`](/cli/share#metadata-only-endpoints) follows it: a Metadata-only endpoint shares no content either.diff --git a/pkg/asymptoteobserve/privacy.go b/pkg/asymptoteobserve/privacy.goindex c8f0cfea..362495c3 100644--- a/pkg/asymptoteobserve/privacy.go+++ b/pkg/asymptoteobserve/privacy.go@@ -96,17 +96,38 @@ func quotedRules(kind quotedKind, keep func(string) bool) []redactionRule {return rules}-// redactUnquoted is an unquoted value, up to a quote, a comma or whitespace. Brackets, braces and-// backslashes are part of it: a generated password is full of them. It never starts with a quote,-// escaped or not, so it leaves a value the quoted rule already replaced alone.-const redactUnquoted = `(?:[^"'` + "`" + `,\s\\]|\\[^"'` + "`" + `])[^"'` + "`" + `,\s]*`+// redactValueStops are the bytes an unquoted labeled value stops at: a quote, a comma and+// whitespace (regexp's \s). redactUnquoted is built from them, and urlRunsOn reads a label's value+// with them, so the two read the same value.+const redactValueStops = redactQuotes + "," + regexpSpace++// redactUnquoted is an unquoted value, up to one of redactValueStops. Brackets, braces,+// backslashes, `#`, `&` and `?` are part of it: a generated password is full of them. It never+// starts with a quote, escaped or not, so it leaves a value the quoted rule already replaced+// alone.+const redactUnquoted = `(?:[^` + redactValueStops + `\\]|\\[^` + redactQuotes + `])[^` + redactValueStops + `]*`++// redactParamStops are the bytes a URL query parameter's value stops at: redactValueStops, and+// the `&` that opens the next parameter and the `#` that opens the fragment.+const redactParamStops = redactValueStops + "&#"++// redactParamValue is redactUnquoted for the value of a URL query parameter, a label opened by+// `?` or `&`: `?api_key=x&user=alice` keeps `&user=alice`. Only there do `&` and `#` end a value;+// after any other label they are a password's own characters (`DB_PASSWORD=#Xy9&z`).+const redactParamValue = `(?:[^` + redactParamStops + `\\]|\\[^` + redactQuotes + `])[^` + redactParamStops + `]*`// redactBare is a bare value after a quoted label: a JSON number or literal, which ends at the// structure around it.const redactBare = `[^"'` + "`" + `\\\s,{}\[\]][^"'` + "`" + `\\\s,{}\[\]]*`-// redactFlagName is a command-line flag named for a secret: `--password`, `--token`, `-api-key`.-const redactFlagName = `--?[a-z0-9-]*(?:password|passwd|passphrase|token|secret|api-?key|access-?key)`+// redactFlagName is a command-line flag named for a secret: `--password`, `--token`, `-api-key`,+// `--client_secret`, `--private-key`. The prefix class carries `_` as well as `-`, matching+// redactLabel's `[A-Za-z0-9_.-]*?`: snake_case flags (`--client_secret`, `--api_key`,+// `--access_key`) are the default naming style for argparse/click-style CLIs and otherwise passed+// through unredacted. `secret[_-]?key` and `private[_-]?key` join the alternation for the same+// reason redactLabel already has them (`--private-key`, `--secret-key`): openssl, ssh-keygen+// wrappers and KMS/cloud CLIs commonly take a key material flag by that name.+const redactFlagName = `--?[a-z0-9_-]*(?:password|passwd|passphrase|token|secret|api[_-]?key|access[_-]?key|secret[_-]?key|private[_-]?key)`// redactFlag is a redactFlagName and its separator: `--password `, `--token=`, `-api-key `.const redactFlag = `(?:^|\s)` + redactFlagName + `(?:=|\s+)`@@ -280,22 +301,22 @@ var redactListedFlagNeedles = func() *redactNeedles {}()// startsFlag reports whether the text before a flag word ends in what redactFlag puts ahead of it:-// `(?:^|\s)--?[a-z0-9-]*`.+// `(?:^|\s)--?[a-z0-9_-]*`.func startsFlag(before string) bool {start, ok := flagStart(before)return ok && (start == 0 || strings.ContainsRune(regexpSpace, rune(before[start-1])))}// flagStart finds the dash that opens the flag the text before a flag word ends in: the start of-// the run of letters, digits and dashes at its end. A pattern's `--?[a-z0-9-]*` can only begin-// where that maximal run does, because what it requires before the dash (whitespace, a quote) is-// not part of the run. The patterns are case-insensitive, so the run takes capitals too: the-// needle scan reads the text as it is, not lower-cased.+// the run of letters, digits, underscores and dashes at its end. A pattern's `--?[a-z0-9_-]*` can+// only begin where that maximal run does, because what it requires before the dash (whitespace, a+// quote) is not part of the run. The patterns are case-insensitive, so the run takes capitals too:+// the needle scan reads the text as it is, not lower-cased.func flagStart(before string) (int, bool) {start := len(before)for start > 0 {c := before[start-1]- if !(c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || c == '-') {+ if !(c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || c == '-' || c == '_') {break}start--@@ -406,14 +427,22 @@ var redactionRules = concatRules([]redactionRule{expr: `(?i)(\b` + redactLabel + `(?P<q>` + redactOpenQuote + `)\s*[:=]\s*)(?P<value>` + redactBare + `)`,replacement: "${1}${q}" + redactedValue + "${q}", keep: isLiteralOrAuthScheme, needles: redactQuotedLabelNeedles, start: startLabel,},+ // An unquoted value. A label opened by `?` or `&` is a URL query parameter, whose value ends+ // at the `&` of the next one or the `#` of the fragment: `?api_key=x&user=alice` keeps+ // `&user=alice`. The two are alternatives of one rule so that the parameter, which starts a+ // byte earlier, is the match: a rule for the label alone, run after it, would read the+ // redacted parameter as a value running on over `&user=alice`.{- expr: `(?i)(\b` + redactLabel + `\s*[:=]\s*)(?P<value>` + redactUnquoted + `)`,- replacement: "${1}" + redactedValue, keep: isAuthScheme, needles: redactLabelNeedles, start: startLabel,+ expr: `(?i)(?:([?&]` + redactLabel + `=)(?P<value>` + redactParamValue + `)|` ++ `(\b` + redactLabel + `\s*[:=]\s*)(?P<value>` + redactUnquoted + `))`,+ replacement: "${1}${3}" + redactedValue, keep: isAuthScheme, needles: redactLabelNeedles, start: startParam,},// A command-line flag: `--password x`, `--token=x`, `--api-key x`, and quoted, the way agents- // usually write arguments: `--password "x y"`, `--token='x'`.+ // usually write arguments: `--password "x y"`, `--token='x'`. An unquoted value does not start+ // with `=` or `:`, which is an assignment's separator after a word that only looks like a flag:+ // a diff's removed line `-SECRET_KEY = 'x'`.}, quotedRules(quotedFlag, nil), []redactionRule{- {expr: `(?i)(` + redactFlag + `)([^\s"'-][^\s"']*)`, replacement: "${1}" + redactedValue, needles: redactFlagNeedles, start: startFlag},+ {expr: `(?i)(` + redactFlag + `)([^\s"'=:-][^\s"']*)`, replacement: "${1}" + redactedValue, needles: redactFlagNeedles, start: startFlag},// A flag and its value as two quoted items of an argument list, the way an MCP server's args// are written in JSON, TOML and YAML: `"args": ["--api-key", "x"]`.{@@ -452,6 +481,7 @@ var redactionRules = concatRules([]redactionRule{shapeRule(`\b(glpat-)[A-Za-z0-9_-]{20,}`, "${1}"+redactedValue, "glpat-"), // GitLabshapeRule(`\b(xox[abeposr]-|xapp-)[A-Za-z0-9-]{10,}`, "${1}"+redactedValue, "xox", "xapp-"), // Slack, with xoxe- refresh and xapp- app-level tokensshapeRule(`\b([rs]k_(?:live|test)_)[A-Za-z0-9]{16,}`, "${1}"+redactedValue, "k_live_", "k_test_").before(1), // Stripe+ shapeRule(`\b(whsec_)[A-Za-z0-9]{32,}`, "${1}"+redactedValue, "whsec_"), // Stripe webhook signing secret: Stripe's alphanumeric shape only, not other services' base64 whsec_ (Svix)shapeRule(`\b(AIza)[0-9A-Za-z_-]{35}`, "${1}"+redactedValue, "AIza"), // Google API keyshapeRule(`\b(ya29\.)(?:c\.)?[0-9A-Za-z_-]{20,}`, "${1}"+redactedValue, "ya29."), // Google OAuth access tokenshapeRule(`\b(npm_)[A-Za-z0-9]{36}`, "${1}"+redactedValue, "npm_"), // npm@@ -533,11 +563,7 @@ func (rule redactionRule) applyOnce(text *redactGate, slot *redactSlot) string {continue // inside a match that ran on (redactionRule.runsOn)}if keep := use.rule.keep; keep != nil {- index := use.pattern.SubexpIndex("value")- if index < 0 {- index = 2- }- if group := 2 * index; match[group] >= 0 && keep(value[match[group]:match[group+1]]) {+ if group := 2 * valueGroup(use.pattern, match); match[group] >= 0 && keep(value[match[group]:match[group+1]]) {text.hidden = text.hidden || keptLabel(value, match[group+1])continue}@@ -571,13 +597,29 @@ func (rule *redactionRule) replacing(t string, match []int) (string, int) {return rule.replacement, match[1]}+// valueGroup returns the index of the group named value that took part in match, which a rule's+// keep reads: a pattern with alternatives names one in each. A pattern that names none has its+// value in the second group.+func valueGroup(re *regexp.Regexp, match []int) int {+ index := 2+ for i, name := range re.SubexpNames() {+ if name == "value" {+ if index = i; match[2*i] >= 0 {+ return i+ }+ }+ }+ return index+}+// urlRunsOn returns where the password of a URL match runs on to when the URL's user name is a// secret-named label (`https://x-access-token:…@`), which the label rules redact too, or -1. The-// two stop at different bytes, a label's value at a quote, a comma or whitespace and a URL's-// password at an `@`, a slash, a quote or whitespace, and once one has replaced its part, the-// other's runs on over the marker; so each pass of the rules took one more piece of what follows,-// starting with the `@`. The value runs on as far as the two take it in turn, which is where-// every later pass would leave it.+// two stop at different bytes, a label's value at redactValueStops and a URL's password at an `@`,+// a slash, a quote or whitespace, and once one has replaced its part, the other's runs on over the+// marker; so each pass of the rules took one more piece of what follows, starting with the `@`.+// The value runs on as far as the two take it in turn. That is where every later pass would leave+// it because the label's value is read here with the stops redactUnquoted is built from: a label+// in a user name is never a query parameter, whose separator is `=`, not the user name's `:`.//// The shape rules run after this one, and one of them changes where either run stops: an AWS key// id followed by its secret (awsPairAt), a secret recognized only after its key id, which holds@@ -594,7 +636,7 @@ func urlRunsOn(t string, m []int) int {// The label's value: the marker the URL rule wrote and what follows it. A key id it ends// in, before any separators it takes (`:`, `|`), takes its secret with it.for run := end; ; {- for end < len(t) && !strings.ContainsRune("\"'`,", rune(t[end])) && !isRegexpSpace(t[end]) {+ for end < len(t) && strings.IndexByte(redactValueStops, t[end]) < 0 {end++}k := end@@ -681,13 +723,14 @@ func labelEndsAt(s string) bool {// redactLabelEnd matches text that ends in one of redactLabel's markers.var redactLabelEnd = lazyPattern(`(?i)(?:` + strings.TrimPrefix(redactLabel, `[A-Za-z0-9_.-]*?`) + `)$`)-// redactAssignedRule captures the value of a labeled assignment, quoted with ", ' or a backtick-// (${1} to ${3}) or not (${4}), so a bare copy of it elsewhere in the same text is-// removed too.+// redactAssignedRule captures the value of a labeled assignment, so a bare copy of it elsewhere in+// the same text is removed too: a URL query parameter's (${1}), as the label rule reads it, then+// one quoted with ", ' or a backtick, or an unquoted one (the last group).var redactAssignedRule = sync.OnceValue(func() redactionRule {return redactionRule{- expr: `(?i)\b` + redactLabel + redactSeparator + `(?:` + quotedAlternatives() + `|(` + redactUnquoted + `))`,- needles: redactLabelNeedles, start: startLabel,+ expr: `(?i)(?:[?&]` + redactLabel + `=(` + redactParamValue + `)|` ++ `\b` + redactLabel + redactSeparator + `(?:` + quotedAlternatives() + `|(` + redactUnquoted + `)))`,+ needles: redactLabelNeedles, start: startParam,}})diff --git a/pkg/asymptoteobserve/privacy_gate_test.go b/pkg/asymptoteobserve/privacy_gate_test.goindex de80b3d2..badb9e83 100644--- a/pkg/asymptoteobserve/privacy_gate_test.go+++ b/pkg/asymptoteobserve/privacy_gate_test.go@@ -13,13 +13,13 @@ var redactGateFragments = []string{"secret", "client_secret", "secret_key", "private-key", "password", "PassWord", "passwd","passphrase", "authorization", "Authorization", "credential", "credentials", "keyboard", "key","GITHUB_TOKEN", "export ", "--token", "--db-password", "-x-token", "a-token", "9", "_", "-password", "--api-key", "--access-key", "-x", "--", "-",- "=", ":", " ", " ", "\t", "\n", "\r", ",", `"`, "'", "`", `\`, `\"`, `\\\"`, "{", "}", "[", "]",+ "=", ":", " ", " ", "\t", "\n", "\r", ",", `"`, "'", "`", `\`, `\"`, `\\\"`, "{", "}", "[", "]", "?", "&", "#","Bearer ", "bearer", "Basic ", "Digest ", "eyJ", "eyJhbGciOiJIUzI1NiJ9", ".", "abcdefgh12345678","hunter2hunter2", "null", "true", "42", "https://", "postgres://user:", "pw", "@host", "://","sk-", "sk-ant-api03-", "AKIA", "ASIA", "ABCDEFGHIJKLMNOP", "ghp_", "gho_", "ghs_", "github_pat_","glpat-", "xoxb-", "xoxe-", "xapp-", "sk_live_", "rk_test_", "AIza", "ya29.", "npm_", "hf_", "bcn_api_", "bcn_device_","hvs.", "hvb.", "gsk_", "pplx-", "r8_", "xai-", "lsv2_pt_", "lsv2_sk_", "pcsk_", "dop_v1_", "doo_v1_", "dapi",- "pypi-AgEIcHlwaS5vcmc", "pypi-", "shpat_", "shpss_", "lin_api_", "dp.pt.", "SG.",+ "pypi-AgEIcHlwaS5vcmc", "pypi-", "shpat_", "shpss_", "lin_api_", "dp.pt.", "SG.", "whsec_","bcn_cli_", "bcn_mcprt_", "bcn_code_", "bcn_client_", "bcn_exchange_", "bcn_mcp_", "bcn_enfdev_", "bcn_other_","AKIAIOSFODNN7EXAMPLE", "ASIAIOSFODNN7EXAMPLE", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", "|", "/", "+","0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJ", "-----BEGIN RSA PRIVATE KEY-----",@@ -45,6 +45,7 @@ func TestRedactGateNeverChangesAResult(t *testing.T) {"mysql --password hunter2 -u root", "postgres://user:pw@host/db", "input_tokens: 812","ſecret=abcdefgh12345678", "KEY=abcdefgh12345678", "toKen\\\"=\"abc\"", `["--api-key", "abcdefgh12345678"]`,"-----BEGIN RSA PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASC\n-----END RSA PRIVATE KEY-----",+ "https://x/y?token=abcdefgh12345678&user=alice#frag", "a?-token=x&y", "?token: a&b", "x&&api_key=#a&b",}r := rand.New(rand.NewSource(1))for i := 0; i < 3000; i++ {diff --git a/pkg/asymptoteobserve/privacy_test.go b/pkg/asymptoteobserve/privacy_test.goindex 12da6e1c..b1da038a 100644--- a/pkg/asymptoteobserve/privacy_test.go+++ b/pkg/asymptoteobserve/privacy_test.go@@ -131,6 +131,11 @@ func TestRedactStringRedactsCommonCredentialFormats(t *testing.T) {{"bare url password without a user", "redis-cli -u redis://:pa55w0rd@localhost:6379 ping", "pa55w0rd"},{"password flag", "mysql --user root --password hunter2hunter2", "hunter2hunter2"},{"token flag", "tool login --token=opaque-flag-value", "opaque-flag-value"},+ {"snake_case client secret flag", "gcloud auth application-default login --client_secret 1a2b3c4d5e6f7g8h9i0j", "1a2b3c4d5e6f7g8h9i0j"},+ {"snake_case api key flag", "tool --api_key abcdef0123456789abcdef0123456789", "abcdef0123456789abcdef0123456789"},+ {"snake_case access key flag", "gcloud auth --access_key plainOpaqueValueNoShape1234567890", "plainOpaqueValueNoShape1234567890"},+ {"private key flag", "openssl rsautl --private-key pemEncodedPrivateKeyStringValue123", "pemEncodedPrivateKeyStringValue123"},+ {"secret key flag", "vault-tool --secret-key abcdefghijklmnopqrstuvwxyz0123456789", "abcdefghijklmnopqrstuvwxyz0123456789"},{"api key flag as a list item", `"args": ["-y", "@upstash/context7-mcp", "--api-key", "ctx7sk-abcdef0123456789"]`, "ctx7sk-abcdef0123456789"},{"token flag as a toml list item", `args = ['--token', 'opaque-flag-value']`, "opaque-flag-value"},{"curl basic auth", "curl -u admin:hunter2hunter2 https://api.example.com/v1", "hunter2hunter2"},@@ -154,6 +159,8 @@ func TestRedactStringRedactsCommonCredentialFormats(t *testing.T) {// Split so secret scanners do not take the fake fixture for a live Slack token.{"slack refresh token", "rotated: xoxe" + "-1-EXAMPLEnotarealrefreshtoken", "EXAMPLEnotarealrefreshtoken"},{"stripe", "sk_live_AbCdEfGhIjKlMnOpQrSt", "AbCdEfGhIjKlMnOpQrSt"},+ // Split so secret scanners do not take the fake fixture for a live Stripe webhook secret.+ {"stripe webhook secret", "got webhook secret whsec_" + "8f9a7cb1d9e4f2a0b3c6d7e8f9a0b1c2d3e4f5a6", "8f9a7cb1d9e4f2a0b3c6d7e8f9a0b1c2d3e4f5a6"},{"google", "AIzaSyA-abcdefghijklmnopqrstuvwxyz012345", "abcdefghijklmnopqrstuvwxyz012345"},{"google oauth access token", "$ gcloud auth print-access-token\nya29.a0AfB_byC1x2Y3z4-W5v6U7t8S9r0QpOnMlKjIhGfEdCbA\n", "a0AfB_byC1x2Y3z4-W5v6U7t8S9r0QpOnMlKjIhGfEdCbA"},{"google metadata server token", `{"access_token":"x"} ya29.c.Elt0BxAbCdEfGhIjKlMnOpQrStUvWxYz0123456789`, "Elt0BxAbCdEfGhIjKlMnOpQrStUvWxYz0123456789"},@@ -266,6 +273,8 @@ func TestRedactStringKeepsLabelsAndOrdinaryText(t *testing.T) {kept := []string{"input_tokens: 812, output_tokens=95, max_tokens=4096","run go test ./... --max-tokens 10",+ "run go test ./... --max_tokens 10",+ "vault operator init --num_secret_shares 3","the author: Ada","see https://github.com/org/repo/pull/12 and [email protected]:org/repo.git","PWD=/home/ada/project",@@ -291,6 +300,32 @@ func TestRedactStringKeepsLabelsAndOrdinaryText(t *testing.T) {if got := RedactString("git clone https://u:[email protected]/r"); got != "git clone https://u:[REDACTED]@example.com/r" {t.Fatalf("URL credential = %q, want the scheme, user and host kept", got)}+ // A label opened by `?` or `&` is a URL query parameter: its value ends at the next `&` or the+ // fragment's `#`, so the parameters after the secret stay readable. Only there: after any+ // other label `&`, `#` and `?` are the value's own characters (TestRedactStringTakesWholeValues).+ for input, want := range map[string]string{+ "GET https://api.example.com/v1/data?api_key=sk-liveabcdef0123456789&user=alice&format=json": "GET https://api.example.com/v1/data?api_key=[REDACTED]&user=alice&format=json",+ "https://x/y?token=SECRET&user=alice#frag": "https://x/y?token=[REDACTED]&user=alice#frag",+ "https://x/y?user=alice&access_token=abc#x&password=hunter2&page=2": "https://x/y?user=alice&access_token=[REDACTED]#x&password=[REDACTED]&page=2",+ // `token:` is not a parameter, whose separator is `=`.+ "https://x/y?token: abc&def": "https://x/y?token: [REDACTED]",+ } {+ if got := RedactString(input); got != want {+ t.Fatalf("URL query string %q = %q, want %q", input, got, want)+ }+ }+ // A diff's removed line can start with what looks like a flag named for a secret. Its `=` is+ // the assignment's, not the flag's value, so the line keeps it; the value after it is the+ // label rules' to redact.+ for input, kept := range map[string]string{+ "-SECRET_KEY = 'django-insecure-abc'": "-SECRET_KEY = '",+ "-SECRET_KEY = 'x'": "-SECRET_KEY = '",+ `-API_TOKEN = os.environ["X"]`: "-API_TOKEN = ",+ } {+ if got := RedactString(input); !strings.HasPrefix(got, kept) {+ t.Fatalf("diff line %q = %q, want it to start with %q", input, got, kept)+ }+ }// Detection rules match what kind of credential a line held, so the public type prefix and the// Authorization scheme stay.for input, want := range map[string]string{@@ -318,6 +353,22 @@ func TestRedactStringIsIdempotent(t *testing.T) {}}+// A URL whose user name is a secret-named label, with `&` in its password and a query string after+// it. `x-access-token:` is a label, not a query parameter, so `&` is part of its value, which runs+// to the whitespace over the host and the query string, as any label's unquoted value does; the+// URL rule reads that value with the same stops (urlRunsOn), so one pass leaves what a second does.+func TestRedactStringURLLabelPasswordWithQueryString(t *testing.T) {+ const input = "git clone https://x-access-token:ab&[email protected]/o/r?ref=main&user=alice"+ const want = "git clone https://x-access-token:[REDACTED]"+ once := RedactString(input)+ if once != want {+ t.Fatalf("RedactString(%q) = %q, want %q", input, once, want)+ }+ if twice := RedactString(once); twice != once {+ t.Fatalf("RedactString is not idempotent:\nonce %q\ntwice %q", once, twice)+ }+}+// A label hidden inside a value a rule kept is redacted in the same call, as a second call would.func TestRedactStringRedactsALabelAKeptValueHid(t *testing.T) {for input, want := range map[string]string{@@ -391,6 +442,9 @@ var redactIdempotenceInputs = []string{// is told by place, not by count."token=abcdefgh12345678 AKIAIOSFODNN7EXAMPLEabcdefgh12345678[REDACTED]","https://x-access-token:a@b,AKIAIOSFODNN7EXAMPLE,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY@host",+ // `&` in a URL password that a label's value runs on over, and a query string after it.+ "git clone https://x-access-token:ab&[email protected]/o/r?ref=main&user=alice",+ "https://x/y?token=abcdefgh12345678&user=alice#frag&password=x",}// A field named for a credential is one, whatever its value looks like.@@ -460,13 +514,18 @@ func TestCleanStringRedactsBeforeTruncating(t *testing.T) {}}-// A generated password is full of brackets, braces and backslashes, and a quoted one can hold-// spaces and commas. None of it may survive past the first such character.+// A generated password is full of brackets, braces, backslashes, `#`, `&` and `?`, and a quoted one+// can hold spaces and commas. None of it may survive past the first such character.func TestRedactStringTakesWholeValues(t *testing.T) {tests := []struct {name, input, want string}{{"unquoted symbols", "DB_PASSWORD=Xy7]kL9}pQ next", "DB_PASSWORD=[REDACTED] next"},+ {"leading hash", "export DB_PASSWORD=#Xy9kqLmN2pQ", "export DB_PASSWORD=[REDACTED]"},+ {"hash and ampersand", "password: s3cr3t#tail&more", "password: [REDACTED]"},+ {"short hash and ampersand", "password: a#b&c", "password: [REDACTED]"},+ {"query delimiters before a command", "PGPASSWORD=Pa55w0rdAbCd#EfGh&IjKl?MnOp psql", "PGPASSWORD=[REDACTED] psql"},+ {"question mark", "token=x?y", "token=[REDACTED]"},{"unquoted backslash", `password=ab\cdEFgh`, "password=[REDACTED]"},{"leading bracket", "token=[abcDEF123]", "token=[REDACTED]"},{"quoted with spaces", `secret: "correct horse, battery staple"`, `secret: "[REDACTED]"`},diff --git a/pkg/asymptoteobserve/redact_adjacent.go b/pkg/asymptoteobserve/redact_adjacent.goindex a4a9f70a..0e65f97d 100644--- a/pkg/asymptoteobserve/redact_adjacent.go+++ b/pkg/asymptoteobserve/redact_adjacent.go@@ -1192,11 +1192,7 @@ func (c *redactChain) followerAt(p int) ([]int, redactUse) {}rule, re := &redactionRules[f.rule], c.pattern(f.rule)if rule.keep != nil {- index := re.SubexpIndex("value")- if index < 0 {- index = 2- }- if g := 2 * index; m[g] >= 0 && rule.keep(t[m[g]:m[g+1]]) {+ if g := 2 * valueGroup(re, m); m[g] >= 0 && rule.keep(t[m[g]:m[g+1]]) {continue}}diff --git a/pkg/asymptoteobserve/redact_adjacent_test.go b/pkg/asymptoteobserve/redact_adjacent_test.goindex 9a6a3d11..8bbdb810 100644--- a/pkg/asymptoteobserve/redact_adjacent_test.go+++ b/pkg/asymptoteobserve/redact_adjacent_test.go@@ -212,6 +212,7 @@ var adjacentSecrets = []string{"lsv2_pt_" + strings.Repeat("0a", 16) + "_" + strings.Repeat("0a", 5), "pcsk_Ab3Cd_" + strings.Repeat("Ab3", 21),"dapi" + strings.Repeat("0a", 16) + "-2", "pypi-AgEIcHlwaS5vcmc" + strings.Repeat("Ab3_", 13),"dp.pt." + strings.Repeat("Ab3", 14) + "C", "SG." + strings.Repeat("Ab", 11) + "." + strings.Repeat("Ab3", 14) + "C",+ "whsec_" + strings.Repeat("Ab3", 11),}// Idempotence and the gate's equivalence over many random texts where secrets meet, beyond whatdiff --git a/pkg/asymptoteobserve/redact_engine.go b/pkg/asymptoteobserve/redact_engine.goindex 6c835bc9..aca51789 100644--- a/pkg/asymptoteobserve/redact_engine.go+++ b/pkg/asymptoteobserve/redact_engine.go@@ -59,6 +59,11 @@ const (// startScheme: at a letter on a word boundary in the run of [A-Za-z0-9+.-] before the needle,// a URL's `\b[a-z][a-z0-9+.-]*` scheme. The pattern opens with \b.startScheme+ // startParam: at the `?` or `&` right before the run of [A-Za-z0-9_.-] that ends at the+ // needle, where a URL query parameter opens, and where startLabel begins when there is none+ // there or the pattern does not match from it. The pattern is two alternatives: the parameter+ // with its `?` or `&`, and a label that opens with \b.+ startParam// startLine: anywhere on the line of the needle or the blank lines before it. The pattern runs// whole from the start of that line, where its (?m)^ and \b see what they see in the whole// text.@@ -342,7 +347,8 @@ func (set *redactNeedles) passes(engine *redactEngine, t string, h redactHit, wi// Where a match can begin anywhere in a run before the needle (startLabel, startScheme), only the// leftmost place in the run is tried. Whatever a match from a later place in the run takes, a// match from the earlier one can take too, its lazy or starred run of the same characters taking-// the ones between; so when the earlier place does not match, no place in the run does.+// the ones between; so when the earlier place does not match, no place in the run does. A+// startParam rule tries the `?` or `&` before the run first, then that place.//// When the places would hand the pattern more text than the budget allows (redactAnchoredPerByte),// the whole pattern runs once from a safe start before the first of them instead. No match starts@@ -444,7 +450,11 @@ func (rule redactionRule) matches(text *redactGate, slot *redactSlot) iter.Seq2[first = q}last = q- if handed += len(t) - q; handed > budget {+ places := 1+ if rule.opensParam(t, q) {+ places = 2 // the label after it, where the parameter does not match+ }+ if handed += places * (len(t) - q); handed > budget {break}}@@ -455,6 +465,18 @@ func (rule redactionRule) matches(text *redactGate, slot *redactSlot) iter.Seq2[whole(slot.safeStart(t, first))return}+ anchoredAt := func(q int) []int {+ if redactWork != nil {+ redactWork(len(t) - q)+ }+ m := slot.anchored.FindStringSubmatchIndex(t[q:])+ for i := range m {+ if m[i] >= 0 {+ m[i] += q+ }+ }+ return m+ }pos, failed := 0, -1for _, h := range text.hits {if mask&(1<<h.id) == 0 || h.start < pos || !rule.needles.passes(engine, t, h, true) {@@ -464,14 +486,13 @@ func (rule redactionRule) matches(text *redactGate, slot *redactSlot) iter.Seq2[if q < 0 || q == failed {continue}- if redactWork != nil {- redactWork(len(t) - q)- }- m := slot.anchored.FindStringSubmatchIndex(t[q:])- for i := range m {- if m[i] >= 0 {- m[i] += q+ m := anchoredAt(q)+ if m == nil && rule.opensParam(t, q) {+ // No parameter opens at the `?` or `&`: the label after it may still match.+ if q = rule.startFor(t, h, q+1, slot.boundary); q < 0 {+ continue}+ m = anchoredAt(q)}if m == nil && chain != nil {m = chain.rescue(chain.adj.byRule[slot.index], q)@@ -513,7 +534,7 @@ func (rule redactionRule) startFor(t string, h redactHit, pos int, boundary boolreturn -1}return q- case startLabel, startScheme:+ case startLabel, startScheme, startParam:class, last := &labelClass, h.startif rule.start == startScheme {class, last = &schemeClass, h.start-1@@ -525,8 +546,11 @@ func (rule redactionRule) startFor(t string, h redactHit, pos int, boundary boolif redactWork != nil {redactWork(h.start - run)}+ if rule.start == startParam && run > pos && (t[run-1] == '?' || t[run-1] == '&') {+ return run - 1+ }for q := max(run, pos); q <= last; q++ {- if wordBoundary(t, q) && (rule.start == startLabel || isASCIILetter(t[q])) {+ if wordBoundary(t, q) && (rule.start != startScheme || isASCIILetter(t[q])) {return q}}@@ -546,6 +570,12 @@ func (rule redactionRule) startFor(t string, h redactHit, pos int, boundary boolpanic("asymptoteobserve: no start for a redaction rule")}+// opensParam reports whether q, a place startFor returned, is the `?` or `&` a startParam rule's+// query parameter opens with, rather than its label's place: a label holds neither.+func (rule redactionRule) opensParam(t string, q int) bool {+ return rule.start == startParam && (t[q] == '?' || t[q] == '&')+}+// lineStart returns the start of a line at or before every place a startLine match holding one of// the needles can begin: before the needle, an optional `set-`, a quote, whitespace across lines,// and a few bytes (`=`, `--header`) more.