Skip to content

The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu

fix(privacy): redact snake_case secret flags and stop query strings bleeding (#1194)

Naman Omare7f5bd614a61
diff --git a/docs/changelog/cli.mdx b/docs/changelog/cli.mdx
index 2c8e9d5c..a7218a5f 100644
--- a/docs/changelog/cli.mdx
+++ b/docs/changelog/cli.mdx
@@ -13,10 +13,11 @@ Stay up to date with the latest changes to the Agent Beacon CLI.
Changes on `main` since v1.3.58. They ship in the next release.
+- **Redaction covers snake_case secret flags and Stripe webhook secrets, and keeps a URL's other query parameters** · Flags written in snake_case or named for key material (`--client_secret`, `--api_key`, `--private-key`, `--secret-key`) have their values redacted, and so do Stripe webhook signing secrets (`whsec_…`). A secret passed as a URL query parameter (`?api_key=…&user=alice`) is redacted up to the next `&` or `#`, so the parameters after it stay readable; after any other label, `#`, `&` and `?` are still part of the redacted value. A diff line such as `-SECRET_KEY = '…'` keeps its `=`.
+
- **Run and repository metadata is redacted and capped** · The `run` block (CI provider, workflow, branch, pull-request ref, actor), the `vcs` block, and the top-level `model`, `repository` and `branch` were written as they arrived, so a credential in a branch name or remote URL reached the log, and a branch name larger than the event size limit made the whole event unwritable. They now get the same secret redaction and 2 KB limit as every other attribute string, in both the collector's writer and the CLI's.
- **Renamed-binary approval bypass catches rm copies under any name** · `approval-bypass-via-renamed-binary` (v3) no longer requires a temp-dir copy of `rm` to be named like `rm`: after a denied dangerous command, running any executable from `/tmp/`, `/var/tmp/`, or `$(mktemp ...)` at command position with an rm-style recursive+force flag (`-rf`, `-v -rf`, a trailing `-rf`) correlates. Temp paths passed as arguments to other tools (`cp /tmp/build -rf`, `tar -C /tmp/stage -rf`, `find /tmp/build -printf`) do not.
-
- **A repository cannot turn auto-share on through the hook environment** · The hooks read the auto-share setting from `~/.beacon/endpoint/share.json` in the home of the account they run as, looked up from the account (the passwd entry, or the Windows process token), not from `$HOME` or `USERPROFILE`, so a repository that sets `HOME` for its hooks cannot plant a `share.json` that publishes its sessions. On Linux, an account that is not in `/etc/passwd` (LDAP, SSSD) still falls back to `$HOME`. `beacon share auto` writes the same file, and the restricted-log spool also moved to the account's home. The cloud-sandbox shuttle now uploads on Stop and SessionEnd only with `BEACON_ORIGIN=cloud`, which the cloud agent setups already set; `BEACON_CLOUD_*` without it does nothing.
- **Fleet package installs turn the local dashboard off** · An unattended first install of the `.deb` or `.rpm`, the macOS `.pkg` or the Windows MSI (Salt, Ansible, an MDM, a silent `msiexec`) now turns the local dashboard off for every account on the machine, root included, as `BEACON_DASHBOARD=disabled` already did: that install sets up the machine for the people who use it, and they should not be able to open a view of its telemetry unless an administrator says so. A person who installs the package on their own machine (`sudo` at a terminal, including the one-line `install.sh`, Installer.app, or the MSI's full interface) keeps the dashboard, by the same test that adds them to `beacon-log`. Upgrades, and reinstalls over a kept config or log, keep the setting an endpoint already has, so set `BEACON_DASHBOARD=disabled` to turn it off on machines installed from an earlier release. `BEACON_DASHBOARD=enabled` keeps it on an unattended install, and `sudo beacon endpoint repair --system --dashboard enabled` turns it back on. Installs from the `beacon` CLI are unchanged. See [Turning the dashboard off](/cli/dashboard#turning-the-dashboard-off).
diff --git a/docs/security/retention-redaction.mdx b/docs/security/retention-redaction.mdx
index 88a32217..da2e66f7 100644
--- a/docs/security/retention-redaction.mdx
+++ b/docs/security/retention-redaction.mdx
@@ -22,12 +22,12 @@ Redaction runs on retained text (prompts, command output, diffs, tool arguments
| Form | Examples |
|------|----------|
-| A value labeled as a secret | `token=…`, `GITHUB_TOKEN=…`, `AWS_SECRET_ACCESS_KEY=…`, `{"api_key": "…"}`, `client_secret: …` |
+| A value labeled as a secret | `token=…`, `GITHUB_TOKEN=…`, `AWS_SECRET_ACCESS_KEY=…`, `{"api_key": "…"}`, `client_secret: …`, and a URL query parameter `?api_key=…`, whose value ends at the next `&` or `#` so the other parameters stay |
| A structured field named for a credential | the value of any field whose name ends in `password`, `secret`, `token`, `api_key`, `access_key`, `private_key`, `authorization`, `credential(s)` or `cookie` |
-| Authorization headers and flags | `Authorization: Bearer …`, `Authorization: Basic …`, `--password …`, `--token=…` |
+| Authorization headers and flags | `Authorization: Bearer …`, `Authorization: Basic …`, `--password …`, `--token=…`, and snake_case or key flags such as `--client_secret …`, `--api_key …`, `--private-key …`, `--secret-key …` |
| URL credentials | the password in `postgres://user:…@host`, `https://x-access-token:…@github.com` |
| Private keys | PEM and OpenSSH `PRIVATE KEY` blocks, including a block cut off before its end line |
-| Recognizable tokens (the public type prefix stays, as in `ghp_[REDACTED]`) | Anthropic and OpenAI `sk-…`, AWS `AKIA…`/`ASIA…`, GitHub `ghp_…`/`github_pat_…`, GitLab `glpat-…`, Slack `xox…-` and `xapp-…`, Stripe `sk_live_…`, Google `AIza…`, npm, Hugging Face, HashiCorp Vault `hvs.…`, Groq `gsk_…`, Perplexity `pplx-…`, Replicate `r8_…`, xAI `xai-…`, LangSmith `lsv2_…`, Pinecone `pcsk_…`, DigitalOcean `dop_v1_…`, Databricks `dapi…`, PyPI `pypi-…`, Shopify `shpat_…`, Linear `lin_api_…`, Doppler `dp.pt.…`, SendGrid `SG.…`, Beacon keys and tokens (`bcn_api_…`, `bcn_device_…`, `bcn_enfdev_…`, `bcn_mcp_…`, `bcn_mcprt_…`, and the `beacon login` token `bcn_cli_…` with its `bcn_code_…`, `bcn_client_…` and `bcn_exchange_…` values), and JWTs |
+| Recognizable tokens (the public type prefix stays, as in `ghp_[REDACTED]`) | Anthropic and OpenAI `sk-…`, AWS `AKIA…`/`ASIA…`, GitHub `ghp_…`/`github_pat_…`, GitLab `glpat-…`, Slack `xox…-` and `xapp-…`, Stripe `sk_live_…` and webhook signing secrets `whsec_…`, Google `AIza…`, npm, Hugging Face, HashiCorp Vault `hvs.…`, Groq `gsk_…`, Perplexity `pplx-…`, Replicate `r8_…`, xAI `xai-…`, LangSmith `lsv2_…`, Pinecone `pcsk_…`, DigitalOcean `dop_v1_…`, Databricks `dapi…`, PyPI `pypi-…`, Shopify `shpat_…`, Linear `lin_api_…`, Doppler `dp.pt.…`, SendGrid `SG.…`, Beacon keys and tokens (`bcn_api_…`, `bcn_device_…`, `bcn_enfdev_…`, `bcn_mcp_…`, `bcn_mcprt_…`, and the `beacon login` token `bcn_cli_…` with its `bcn_code_…`, `bcn_client_…` and `bcn_exchange_…` values), and JWTs |
| An AWS secret key beside its key id | the 40-character secret that follows an `AKIA…` or `ASIA…` key id within a few spaces, commas, colons, pipes or quotes, as in a credentials CSV (`AKIA…,wJalr…`) or a table |
Redaction is pattern-based. A credential with no label and no recognizable shape, such as a bare random password, is not detected, so treat retained content as sensitive. Metadata-only forwarding is the control for keeping content off Beacon Cloud entirely, and [`beacon share`](/cli/share#metadata-only-endpoints) follows it: a Metadata-only endpoint shares no content either.
diff --git a/pkg/asymptoteobserve/privacy.go b/pkg/asymptoteobserve/privacy.go
index c8f0cfea..362495c3 100644
--- a/pkg/asymptoteobserve/privacy.go
+++ b/pkg/asymptoteobserve/privacy.go
@@ -96,17 +96,38 @@ func quotedRules(kind quotedKind, keep func(string) bool) []redactionRule {
return rules
}
-// redactUnquoted is an unquoted value, up to a quote, a comma or whitespace. Brackets, braces and
-// backslashes are part of it: a generated password is full of them. It never starts with a quote,
-// escaped or not, so it leaves a value the quoted rule already replaced alone.
-const redactUnquoted = `(?:[^"'` + "`" + `,\s\\]|\\[^"'` + "`" + `])[^"'` + "`" + `,\s]*`
+// redactValueStops are the bytes an unquoted labeled value stops at: a quote, a comma and
+// whitespace (regexp's \s). redactUnquoted is built from them, and urlRunsOn reads a label's value
+// with them, so the two read the same value.
+const redactValueStops = redactQuotes + "," + regexpSpace
+
+// redactUnquoted is an unquoted value, up to one of redactValueStops. Brackets, braces,
+// backslashes, `#`, `&` and `?` are part of it: a generated password is full of them. It never
+// starts with a quote, escaped or not, so it leaves a value the quoted rule already replaced
+// alone.
+const redactUnquoted = `(?:[^` + redactValueStops + `\\]|\\[^` + redactQuotes + `])[^` + redactValueStops + `]*`
+
+// redactParamStops are the bytes a URL query parameter's value stops at: redactValueStops, and
+// the `&` that opens the next parameter and the `#` that opens the fragment.
+const redactParamStops = redactValueStops + "&#"
+
+// redactParamValue is redactUnquoted for the value of a URL query parameter, a label opened by
+// `?` or `&`: `?api_key=x&user=alice` keeps `&user=alice`. Only there do `&` and `#` end a value;
+// after any other label they are a password's own characters (`DB_PASSWORD=#Xy9&z`).
+const redactParamValue = `(?:[^` + redactParamStops + `\\]|\\[^` + redactQuotes + `])[^` + redactParamStops + `]*`
// redactBare is a bare value after a quoted label: a JSON number or literal, which ends at the
// structure around it.
const redactBare = `[^"'` + "`" + `\\\s,{}\[\]][^"'` + "`" + `\\\s,{}\[\]]*`
-// redactFlagName is a command-line flag named for a secret: `--password`, `--token`, `-api-key`.
-const redactFlagName = `--?[a-z0-9-]*(?:password|passwd|passphrase|token|secret|api-?key|access-?key)`
+// redactFlagName is a command-line flag named for a secret: `--password`, `--token`, `-api-key`,
+// `--client_secret`, `--private-key`. The prefix class carries `_` as well as `-`, matching
+// redactLabel's `[A-Za-z0-9_.-]*?`: snake_case flags (`--client_secret`, `--api_key`,
+// `--access_key`) are the default naming style for argparse/click-style CLIs and otherwise passed
+// through unredacted. `secret[_-]?key` and `private[_-]?key` join the alternation for the same
+// reason redactLabel already has them (`--private-key`, `--secret-key`): openssl, ssh-keygen
+// wrappers and KMS/cloud CLIs commonly take a key material flag by that name.
+const redactFlagName = `--?[a-z0-9_-]*(?:password|passwd|passphrase|token|secret|api[_-]?key|access[_-]?key|secret[_-]?key|private[_-]?key)`
// redactFlag is a redactFlagName and its separator: `--password `, `--token=`, `-api-key `.
const redactFlag = `(?:^|\s)` + redactFlagName + `(?:=|\s+)`
@@ -280,22 +301,22 @@ var redactListedFlagNeedles = func() *redactNeedles {
}()
// startsFlag reports whether the text before a flag word ends in what redactFlag puts ahead of it:
-// `(?:^|\s)--?[a-z0-9-]*`.
+// `(?:^|\s)--?[a-z0-9_-]*`.
func startsFlag(before string) bool {
start, ok := flagStart(before)
return ok && (start == 0 || strings.ContainsRune(regexpSpace, rune(before[start-1])))
}
// flagStart finds the dash that opens the flag the text before a flag word ends in: the start of
-// the run of letters, digits and dashes at its end. A pattern's `--?[a-z0-9-]*` can only begin
-// where that maximal run does, because what it requires before the dash (whitespace, a quote) is
-// not part of the run. The patterns are case-insensitive, so the run takes capitals too: the
-// needle scan reads the text as it is, not lower-cased.
+// the run of letters, digits, underscores and dashes at its end. A pattern's `--?[a-z0-9_-]*` can
+// only begin where that maximal run does, because what it requires before the dash (whitespace, a
+// quote) is not part of the run. The patterns are case-insensitive, so the run takes capitals too:
+// the needle scan reads the text as it is, not lower-cased.
func flagStart(before string) (int, bool) {
start := len(before)
for start > 0 {
c := before[start-1]
- if !(c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || c == '-') {
+ if !(c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || c == '-' || c == '_') {
break
}
start--
@@ -406,14 +427,22 @@ var redactionRules = concatRules([]redactionRule{
expr: `(?i)(\b` + redactLabel + `(?P<q>` + redactOpenQuote + `)\s*[:=]\s*)(?P<value>` + redactBare + `)`,
replacement: "${1}${q}" + redactedValue + "${q}", keep: isLiteralOrAuthScheme, needles: redactQuotedLabelNeedles, start: startLabel,
},
+ // An unquoted value. A label opened by `?` or `&` is a URL query parameter, whose value ends
+ // at the `&` of the next one or the `#` of the fragment: `?api_key=x&user=alice` keeps
+ // `&user=alice`. The two are alternatives of one rule so that the parameter, which starts a
+ // byte earlier, is the match: a rule for the label alone, run after it, would read the
+ // redacted parameter as a value running on over `&user=alice`.
{
- expr: `(?i)(\b` + redactLabel + `\s*[:=]\s*)(?P<value>` + redactUnquoted + `)`,
- replacement: "${1}" + redactedValue, keep: isAuthScheme, needles: redactLabelNeedles, start: startLabel,
+ expr: `(?i)(?:([?&]` + redactLabel + `=)(?P<value>` + redactParamValue + `)|` +
+ `(\b` + redactLabel + `\s*[:=]\s*)(?P<value>` + redactUnquoted + `))`,
+ replacement: "${1}${3}" + redactedValue, keep: isAuthScheme, needles: redactLabelNeedles, start: startParam,
},
// A command-line flag: `--password x`, `--token=x`, `--api-key x`, and quoted, the way agents
- // usually write arguments: `--password "x y"`, `--token='x'`.
+ // usually write arguments: `--password "x y"`, `--token='x'`. An unquoted value does not start
+ // with `=` or `:`, which is an assignment's separator after a word that only looks like a flag:
+ // a diff's removed line `-SECRET_KEY = 'x'`.
}, quotedRules(quotedFlag, nil), []redactionRule{
- {expr: `(?i)(` + redactFlag + `)([^\s"'-][^\s"']*)`, replacement: "${1}" + redactedValue, needles: redactFlagNeedles, start: startFlag},
+ {expr: `(?i)(` + redactFlag + `)([^\s"'=:-][^\s"']*)`, replacement: "${1}" + redactedValue, needles: redactFlagNeedles, start: startFlag},
// A flag and its value as two quoted items of an argument list, the way an MCP server's args
// are written in JSON, TOML and YAML: `"args": ["--api-key", "x"]`.
{
@@ -452,6 +481,7 @@ var redactionRules = concatRules([]redactionRule{
shapeRule(`\b(glpat-)[A-Za-z0-9_-]{20,}`, "${1}"+redactedValue, "glpat-"), // GitLab
shapeRule(`\b(xox[abeposr]-|xapp-)[A-Za-z0-9-]{10,}`, "${1}"+redactedValue, "xox", "xapp-"), // Slack, with xoxe- refresh and xapp- app-level tokens
shapeRule(`\b([rs]k_(?:live|test)_)[A-Za-z0-9]{16,}`, "${1}"+redactedValue, "k_live_", "k_test_").before(1), // Stripe
+ shapeRule(`\b(whsec_)[A-Za-z0-9]{32,}`, "${1}"+redactedValue, "whsec_"), // Stripe webhook signing secret: Stripe's alphanumeric shape only, not other services' base64 whsec_ (Svix)
shapeRule(`\b(AIza)[0-9A-Za-z_-]{35}`, "${1}"+redactedValue, "AIza"), // Google API key
shapeRule(`\b(ya29\.)(?:c\.)?[0-9A-Za-z_-]{20,}`, "${1}"+redactedValue, "ya29."), // Google OAuth access token
shapeRule(`\b(npm_)[A-Za-z0-9]{36}`, "${1}"+redactedValue, "npm_"), // npm
@@ -533,11 +563,7 @@ func (rule redactionRule) applyOnce(text *redactGate, slot *redactSlot) string {
continue // inside a match that ran on (redactionRule.runsOn)
}
if keep := use.rule.keep; keep != nil {
- index := use.pattern.SubexpIndex("value")
- if index < 0 {
- index = 2
- }
- if group := 2 * index; match[group] >= 0 && keep(value[match[group]:match[group+1]]) {
+ if group := 2 * valueGroup(use.pattern, match); match[group] >= 0 && keep(value[match[group]:match[group+1]]) {
text.hidden = text.hidden || keptLabel(value, match[group+1])
continue
}
@@ -571,13 +597,29 @@ func (rule *redactionRule) replacing(t string, match []int) (string, int) {
return rule.replacement, match[1]
}
+// valueGroup returns the index of the group named value that took part in match, which a rule's
+// keep reads: a pattern with alternatives names one in each. A pattern that names none has its
+// value in the second group.
+func valueGroup(re *regexp.Regexp, match []int) int {
+ index := 2
+ for i, name := range re.SubexpNames() {
+ if name == "value" {
+ if index = i; match[2*i] >= 0 {
+ return i
+ }
+ }
+ }
+ return index
+}
+
// urlRunsOn returns where the password of a URL match runs on to when the URL's user name is a
// secret-named label (`https://x-access-token:…@`), which the label rules redact too, or -1. The
-// two stop at different bytes, a label's value at a quote, a comma or whitespace and a URL's
-// password at an `@`, a slash, a quote or whitespace, and once one has replaced its part, the
-// other's runs on over the marker; so each pass of the rules took one more piece of what follows,
-// starting with the `@`. The value runs on as far as the two take it in turn, which is where
-// every later pass would leave it.
+// two stop at different bytes, a label's value at redactValueStops and a URL's password at an `@`,
+// a slash, a quote or whitespace, and once one has replaced its part, the other's runs on over the
+// marker; so each pass of the rules took one more piece of what follows, starting with the `@`.
+// The value runs on as far as the two take it in turn. That is where every later pass would leave
+// it because the label's value is read here with the stops redactUnquoted is built from: a label
+// in a user name is never a query parameter, whose separator is `=`, not the user name's `:`.
//
// The shape rules run after this one, and one of them changes where either run stops: an AWS key
// id followed by its secret (awsPairAt), a secret recognized only after its key id, which holds
@@ -594,7 +636,7 @@ func urlRunsOn(t string, m []int) int {
// The label's value: the marker the URL rule wrote and what follows it. A key id it ends
// in, before any separators it takes (`:`, `|`), takes its secret with it.
for run := end; ; {
- for end < len(t) && !strings.ContainsRune("\"'`,", rune(t[end])) && !isRegexpSpace(t[end]) {
+ for end < len(t) && strings.IndexByte(redactValueStops, t[end]) < 0 {
end++
}
k := end
@@ -681,13 +723,14 @@ func labelEndsAt(s string) bool {
// redactLabelEnd matches text that ends in one of redactLabel's markers.
var redactLabelEnd = lazyPattern(`(?i)(?:` + strings.TrimPrefix(redactLabel, `[A-Za-z0-9_.-]*?`) + `)$`)
-// redactAssignedRule captures the value of a labeled assignment, quoted with ", ' or a backtick
-// (${1} to ${3}) or not (${4}), so a bare copy of it elsewhere in the same text is
-// removed too.
+// redactAssignedRule captures the value of a labeled assignment, so a bare copy of it elsewhere in
+// the same text is removed too: a URL query parameter's (${1}), as the label rule reads it, then
+// one quoted with ", ' or a backtick, or an unquoted one (the last group).
var redactAssignedRule = sync.OnceValue(func() redactionRule {
return redactionRule{
- expr: `(?i)\b` + redactLabel + redactSeparator + `(?:` + quotedAlternatives() + `|(` + redactUnquoted + `))`,
- needles: redactLabelNeedles, start: startLabel,
+ expr: `(?i)(?:[?&]` + redactLabel + `=(` + redactParamValue + `)|` +
+ `\b` + redactLabel + redactSeparator + `(?:` + quotedAlternatives() + `|(` + redactUnquoted + `)))`,
+ needles: redactLabelNeedles, start: startParam,
}
})
diff --git a/pkg/asymptoteobserve/privacy_gate_test.go b/pkg/asymptoteobserve/privacy_gate_test.go
index de80b3d2..badb9e83 100644
--- a/pkg/asymptoteobserve/privacy_gate_test.go
+++ b/pkg/asymptoteobserve/privacy_gate_test.go
@@ -13,13 +13,13 @@ var redactGateFragments = []string{
"secret", "client_secret", "secret_key", "private-key", "password", "PassWord", "passwd",
"passphrase", "authorization", "Authorization", "credential", "credentials", "keyboard", "key",
"GITHUB_TOKEN", "export ", "--token", "--db-password", "-x-token", "a-token", "9", "_", "-password", "--api-key", "--access-key", "-x", "--", "-",
- "=", ":", " ", " ", "\t", "\n", "\r", ",", `"`, "'", "`", `\`, `\"`, `\\\"`, "{", "}", "[", "]",
+ "=", ":", " ", " ", "\t", "\n", "\r", ",", `"`, "'", "`", `\`, `\"`, `\\\"`, "{", "}", "[", "]", "?", "&", "#",
"Bearer ", "bearer", "Basic ", "Digest ", "eyJ", "eyJhbGciOiJIUzI1NiJ9", ".", "abcdefgh12345678",
"hunter2hunter2", "null", "true", "42", "https://", "postgres://user:", "pw", "@host", "://",
"sk-", "sk-ant-api03-", "AKIA", "ASIA", "ABCDEFGHIJKLMNOP", "ghp_", "gho_", "ghs_", "github_pat_",
"glpat-", "xoxb-", "xoxe-", "xapp-", "sk_live_", "rk_test_", "AIza", "ya29.", "npm_", "hf_", "bcn_api_", "bcn_device_",
"hvs.", "hvb.", "gsk_", "pplx-", "r8_", "xai-", "lsv2_pt_", "lsv2_sk_", "pcsk_", "dop_v1_", "doo_v1_", "dapi",
- "pypi-AgEIcHlwaS5vcmc", "pypi-", "shpat_", "shpss_", "lin_api_", "dp.pt.", "SG.",
+ "pypi-AgEIcHlwaS5vcmc", "pypi-", "shpat_", "shpss_", "lin_api_", "dp.pt.", "SG.", "whsec_",
"bcn_cli_", "bcn_mcprt_", "bcn_code_", "bcn_client_", "bcn_exchange_", "bcn_mcp_", "bcn_enfdev_", "bcn_other_",
"AKIAIOSFODNN7EXAMPLE", "ASIAIOSFODNN7EXAMPLE", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", "|", "/", "+",
"0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJ", "-----BEGIN RSA PRIVATE KEY-----",
@@ -45,6 +45,7 @@ func TestRedactGateNeverChangesAResult(t *testing.T) {
"mysql --password hunter2 -u root", "postgres://user:pw@host/db", "input_tokens: 812",
"ſecret=abcdefgh12345678", "KEY=abcdefgh12345678", "toKen\\\"=\"abc\"", `["--api-key", "abcdefgh12345678"]`,
"-----BEGIN RSA PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASC\n-----END RSA PRIVATE KEY-----",
+ "https://x/y?token=abcdefgh12345678&user=alice#frag", "a?-token=x&y", "?token: a&b", "x&&api_key=#a&b",
}
r := rand.New(rand.NewSource(1))
for i := 0; i < 3000; i++ {
diff --git a/pkg/asymptoteobserve/privacy_test.go b/pkg/asymptoteobserve/privacy_test.go
index 12da6e1c..b1da038a 100644
--- a/pkg/asymptoteobserve/privacy_test.go
+++ b/pkg/asymptoteobserve/privacy_test.go
@@ -131,6 +131,11 @@ func TestRedactStringRedactsCommonCredentialFormats(t *testing.T) {
{"bare url password without a user", "redis-cli -u redis://:pa55w0rd@localhost:6379 ping", "pa55w0rd"},
{"password flag", "mysql --user root --password hunter2hunter2", "hunter2hunter2"},
{"token flag", "tool login --token=opaque-flag-value", "opaque-flag-value"},
+ {"snake_case client secret flag", "gcloud auth application-default login --client_secret 1a2b3c4d5e6f7g8h9i0j", "1a2b3c4d5e6f7g8h9i0j"},
+ {"snake_case api key flag", "tool --api_key abcdef0123456789abcdef0123456789", "abcdef0123456789abcdef0123456789"},
+ {"snake_case access key flag", "gcloud auth --access_key plainOpaqueValueNoShape1234567890", "plainOpaqueValueNoShape1234567890"},
+ {"private key flag", "openssl rsautl --private-key pemEncodedPrivateKeyStringValue123", "pemEncodedPrivateKeyStringValue123"},
+ {"secret key flag", "vault-tool --secret-key abcdefghijklmnopqrstuvwxyz0123456789", "abcdefghijklmnopqrstuvwxyz0123456789"},
{"api key flag as a list item", `"args": ["-y", "@upstash/context7-mcp", "--api-key", "ctx7sk-abcdef0123456789"]`, "ctx7sk-abcdef0123456789"},
{"token flag as a toml list item", `args = ['--token', 'opaque-flag-value']`, "opaque-flag-value"},
{"curl basic auth", "curl -u admin:hunter2hunter2 https://api.example.com/v1", "hunter2hunter2"},
@@ -154,6 +159,8 @@ func TestRedactStringRedactsCommonCredentialFormats(t *testing.T) {
// Split so secret scanners do not take the fake fixture for a live Slack token.
{"slack refresh token", "rotated: xoxe" + "-1-EXAMPLEnotarealrefreshtoken", "EXAMPLEnotarealrefreshtoken"},
{"stripe", "sk_live_AbCdEfGhIjKlMnOpQrSt", "AbCdEfGhIjKlMnOpQrSt"},
+ // Split so secret scanners do not take the fake fixture for a live Stripe webhook secret.
+ {"stripe webhook secret", "got webhook secret whsec_" + "8f9a7cb1d9e4f2a0b3c6d7e8f9a0b1c2d3e4f5a6", "8f9a7cb1d9e4f2a0b3c6d7e8f9a0b1c2d3e4f5a6"},
{"google", "AIzaSyA-abcdefghijklmnopqrstuvwxyz012345", "abcdefghijklmnopqrstuvwxyz012345"},
{"google oauth access token", "$ gcloud auth print-access-token\nya29.a0AfB_byC1x2Y3z4-W5v6U7t8S9r0QpOnMlKjIhGfEdCbA\n", "a0AfB_byC1x2Y3z4-W5v6U7t8S9r0QpOnMlKjIhGfEdCbA"},
{"google metadata server token", `{"access_token":"x"} ya29.c.Elt0BxAbCdEfGhIjKlMnOpQrStUvWxYz0123456789`, "Elt0BxAbCdEfGhIjKlMnOpQrStUvWxYz0123456789"},
@@ -266,6 +273,8 @@ func TestRedactStringKeepsLabelsAndOrdinaryText(t *testing.T) {
kept := []string{
"input_tokens: 812, output_tokens=95, max_tokens=4096",
"run go test ./... --max-tokens 10",
+ "run go test ./... --max_tokens 10",
+ "vault operator init --num_secret_shares 3",
"the author: Ada",
"see https://github.com/org/repo/pull/12 and [email protected]:org/repo.git",
"PWD=/home/ada/project",
@@ -291,6 +300,32 @@ func TestRedactStringKeepsLabelsAndOrdinaryText(t *testing.T) {
if got := RedactString("git clone https://u:[email protected]/r"); got != "git clone https://u:[REDACTED]@example.com/r" {
t.Fatalf("URL credential = %q, want the scheme, user and host kept", got)
}
+ // A label opened by `?` or `&` is a URL query parameter: its value ends at the next `&` or the
+ // fragment's `#`, so the parameters after the secret stay readable. Only there: after any
+ // other label `&`, `#` and `?` are the value's own characters (TestRedactStringTakesWholeValues).
+ for input, want := range map[string]string{
+ "GET https://api.example.com/v1/data?api_key=sk-liveabcdef0123456789&user=alice&format=json": "GET https://api.example.com/v1/data?api_key=[REDACTED]&user=alice&format=json",
+ "https://x/y?token=SECRET&user=alice#frag": "https://x/y?token=[REDACTED]&user=alice#frag",
+ "https://x/y?user=alice&access_token=abc#x&password=hunter2&page=2": "https://x/y?user=alice&access_token=[REDACTED]#x&password=[REDACTED]&page=2",
+ // `token:` is not a parameter, whose separator is `=`.
+ "https://x/y?token: abc&def": "https://x/y?token: [REDACTED]",
+ } {
+ if got := RedactString(input); got != want {
+ t.Fatalf("URL query string %q = %q, want %q", input, got, want)
+ }
+ }
+ // A diff's removed line can start with what looks like a flag named for a secret. Its `=` is
+ // the assignment's, not the flag's value, so the line keeps it; the value after it is the
+ // label rules' to redact.
+ for input, kept := range map[string]string{
+ "-SECRET_KEY = 'django-insecure-abc'": "-SECRET_KEY = '",
+ "-SECRET_KEY = 'x'": "-SECRET_KEY = '",
+ `-API_TOKEN = os.environ["X"]`: "-API_TOKEN = ",
+ } {
+ if got := RedactString(input); !strings.HasPrefix(got, kept) {
+ t.Fatalf("diff line %q = %q, want it to start with %q", input, got, kept)
+ }
+ }
// Detection rules match what kind of credential a line held, so the public type prefix and the
// Authorization scheme stay.
for input, want := range map[string]string{
@@ -318,6 +353,22 @@ func TestRedactStringIsIdempotent(t *testing.T) {
}
}
+// A URL whose user name is a secret-named label, with `&` in its password and a query string after
+// it. `x-access-token:` is a label, not a query parameter, so `&` is part of its value, which runs
+// to the whitespace over the host and the query string, as any label's unquoted value does; the
+// URL rule reads that value with the same stops (urlRunsOn), so one pass leaves what a second does.
+func TestRedactStringURLLabelPasswordWithQueryString(t *testing.T) {
+ const input = "git clone https://x-access-token:ab&[email protected]/o/r?ref=main&user=alice"
+ const want = "git clone https://x-access-token:[REDACTED]"
+ once := RedactString(input)
+ if once != want {
+ t.Fatalf("RedactString(%q) = %q, want %q", input, once, want)
+ }
+ if twice := RedactString(once); twice != once {
+ t.Fatalf("RedactString is not idempotent:\nonce %q\ntwice %q", once, twice)
+ }
+}
+
// A label hidden inside a value a rule kept is redacted in the same call, as a second call would.
func TestRedactStringRedactsALabelAKeptValueHid(t *testing.T) {
for input, want := range map[string]string{
@@ -391,6 +442,9 @@ var redactIdempotenceInputs = []string{
// is told by place, not by count.
"token=abcdefgh12345678 AKIAIOSFODNN7EXAMPLEabcdefgh12345678[REDACTED]",
"https://x-access-token:a@b,AKIAIOSFODNN7EXAMPLE,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY@host",
+ // `&` in a URL password that a label's value runs on over, and a query string after it.
+ "git clone https://x-access-token:ab&[email protected]/o/r?ref=main&user=alice",
+ "https://x/y?token=abcdefgh12345678&user=alice#frag&password=x",
}
// A field named for a credential is one, whatever its value looks like.
@@ -460,13 +514,18 @@ func TestCleanStringRedactsBeforeTruncating(t *testing.T) {
}
}
-// A generated password is full of brackets, braces and backslashes, and a quoted one can hold
-// spaces and commas. None of it may survive past the first such character.
+// A generated password is full of brackets, braces, backslashes, `#`, `&` and `?`, and a quoted one
+// can hold spaces and commas. None of it may survive past the first such character.
func TestRedactStringTakesWholeValues(t *testing.T) {
tests := []struct {
name, input, want string
}{
{"unquoted symbols", "DB_PASSWORD=Xy7]kL9}pQ next", "DB_PASSWORD=[REDACTED] next"},
+ {"leading hash", "export DB_PASSWORD=#Xy9kqLmN2pQ", "export DB_PASSWORD=[REDACTED]"},
+ {"hash and ampersand", "password: s3cr3t#tail&more", "password: [REDACTED]"},
+ {"short hash and ampersand", "password: a#b&c", "password: [REDACTED]"},
+ {"query delimiters before a command", "PGPASSWORD=Pa55w0rdAbCd#EfGh&IjKl?MnOp psql", "PGPASSWORD=[REDACTED] psql"},
+ {"question mark", "token=x?y", "token=[REDACTED]"},
{"unquoted backslash", `password=ab\cdEFgh`, "password=[REDACTED]"},
{"leading bracket", "token=[abcDEF123]", "token=[REDACTED]"},
{"quoted with spaces", `secret: "correct horse, battery staple"`, `secret: "[REDACTED]"`},
diff --git a/pkg/asymptoteobserve/redact_adjacent.go b/pkg/asymptoteobserve/redact_adjacent.go
index a4a9f70a..0e65f97d 100644
--- a/pkg/asymptoteobserve/redact_adjacent.go
+++ b/pkg/asymptoteobserve/redact_adjacent.go
@@ -1192,11 +1192,7 @@ func (c *redactChain) followerAt(p int) ([]int, redactUse) {
}
rule, re := &redactionRules[f.rule], c.pattern(f.rule)
if rule.keep != nil {
- index := re.SubexpIndex("value")
- if index < 0 {
- index = 2
- }
- if g := 2 * index; m[g] >= 0 && rule.keep(t[m[g]:m[g+1]]) {
+ if g := 2 * valueGroup(re, m); m[g] >= 0 && rule.keep(t[m[g]:m[g+1]]) {
continue
}
}
diff --git a/pkg/asymptoteobserve/redact_adjacent_test.go b/pkg/asymptoteobserve/redact_adjacent_test.go
index 9a6a3d11..8bbdb810 100644
--- a/pkg/asymptoteobserve/redact_adjacent_test.go
+++ b/pkg/asymptoteobserve/redact_adjacent_test.go
@@ -212,6 +212,7 @@ var adjacentSecrets = []string{
"lsv2_pt_" + strings.Repeat("0a", 16) + "_" + strings.Repeat("0a", 5), "pcsk_Ab3Cd_" + strings.Repeat("Ab3", 21),
"dapi" + strings.Repeat("0a", 16) + "-2", "pypi-AgEIcHlwaS5vcmc" + strings.Repeat("Ab3_", 13),
"dp.pt." + strings.Repeat("Ab3", 14) + "C", "SG." + strings.Repeat("Ab", 11) + "." + strings.Repeat("Ab3", 14) + "C",
+ "whsec_" + strings.Repeat("Ab3", 11),
}
// Idempotence and the gate's equivalence over many random texts where secrets meet, beyond what
diff --git a/pkg/asymptoteobserve/redact_engine.go b/pkg/asymptoteobserve/redact_engine.go
index 6c835bc9..aca51789 100644
--- a/pkg/asymptoteobserve/redact_engine.go
+++ b/pkg/asymptoteobserve/redact_engine.go
@@ -59,6 +59,11 @@ const (
// startScheme: at a letter on a word boundary in the run of [A-Za-z0-9+.-] before the needle,
// a URL's `\b[a-z][a-z0-9+.-]*` scheme. The pattern opens with \b.
startScheme
+ // startParam: at the `?` or `&` right before the run of [A-Za-z0-9_.-] that ends at the
+ // needle, where a URL query parameter opens, and where startLabel begins when there is none
+ // there or the pattern does not match from it. The pattern is two alternatives: the parameter
+ // with its `?` or `&`, and a label that opens with \b.
+ startParam
// startLine: anywhere on the line of the needle or the blank lines before it. The pattern runs
// whole from the start of that line, where its (?m)^ and \b see what they see in the whole
// text.
@@ -342,7 +347,8 @@ func (set *redactNeedles) passes(engine *redactEngine, t string, h redactHit, wi
// Where a match can begin anywhere in a run before the needle (startLabel, startScheme), only the
// leftmost place in the run is tried. Whatever a match from a later place in the run takes, a
// match from the earlier one can take too, its lazy or starred run of the same characters taking
-// the ones between; so when the earlier place does not match, no place in the run does.
+// the ones between; so when the earlier place does not match, no place in the run does. A
+// startParam rule tries the `?` or `&` before the run first, then that place.
//
// When the places would hand the pattern more text than the budget allows (redactAnchoredPerByte),
// the whole pattern runs once from a safe start before the first of them instead. No match starts
@@ -444,7 +450,11 @@ func (rule redactionRule) matches(text *redactGate, slot *redactSlot) iter.Seq2[
first = q
}
last = q
- if handed += len(t) - q; handed > budget {
+ places := 1
+ if rule.opensParam(t, q) {
+ places = 2 // the label after it, where the parameter does not match
+ }
+ if handed += places * (len(t) - q); handed > budget {
break
}
}
@@ -455,6 +465,18 @@ func (rule redactionRule) matches(text *redactGate, slot *redactSlot) iter.Seq2[
whole(slot.safeStart(t, first))
return
}
+ anchoredAt := func(q int) []int {
+ if redactWork != nil {
+ redactWork(len(t) - q)
+ }
+ m := slot.anchored.FindStringSubmatchIndex(t[q:])
+ for i := range m {
+ if m[i] >= 0 {
+ m[i] += q
+ }
+ }
+ return m
+ }
pos, failed := 0, -1
for _, h := range text.hits {
if mask&(1<<h.id) == 0 || h.start < pos || !rule.needles.passes(engine, t, h, true) {
@@ -464,14 +486,13 @@ func (rule redactionRule) matches(text *redactGate, slot *redactSlot) iter.Seq2[
if q < 0 || q == failed {
continue
}
- if redactWork != nil {
- redactWork(len(t) - q)
- }
- m := slot.anchored.FindStringSubmatchIndex(t[q:])
- for i := range m {
- if m[i] >= 0 {
- m[i] += q
+ m := anchoredAt(q)
+ if m == nil && rule.opensParam(t, q) {
+ // No parameter opens at the `?` or `&`: the label after it may still match.
+ if q = rule.startFor(t, h, q+1, slot.boundary); q < 0 {
+ continue
}
+ m = anchoredAt(q)
}
if m == nil && chain != nil {
m = chain.rescue(chain.adj.byRule[slot.index], q)
@@ -513,7 +534,7 @@ func (rule redactionRule) startFor(t string, h redactHit, pos int, boundary bool
return -1
}
return q
- case startLabel, startScheme:
+ case startLabel, startScheme, startParam:
class, last := &labelClass, h.start
if rule.start == startScheme {
class, last = &schemeClass, h.start-1
@@ -525,8 +546,11 @@ func (rule redactionRule) startFor(t string, h redactHit, pos int, boundary bool
if redactWork != nil {
redactWork(h.start - run)
}
+ if rule.start == startParam && run > pos && (t[run-1] == '?' || t[run-1] == '&') {
+ return run - 1
+ }
for q := max(run, pos); q <= last; q++ {
- if wordBoundary(t, q) && (rule.start == startLabel || isASCIILetter(t[q])) {
+ if wordBoundary(t, q) && (rule.start != startScheme || isASCIILetter(t[q])) {
return q
}
}
@@ -546,6 +570,12 @@ func (rule redactionRule) startFor(t string, h redactHit, pos int, boundary bool
panic("asymptoteobserve: no start for a redaction rule")
}
+// opensParam reports whether q, a place startFor returned, is the `?` or `&` a startParam rule's
+// query parameter opens with, rather than its label's place: a label holds neither.
+func (rule redactionRule) opensParam(t string, q int) bool {
+ return rule.start == startParam && (t[q] == '?' || t[q] == '&')
+}
+
// lineStart returns the start of a line at or before every place a startLine match holding one of
// the needles can begin: before the needle, an optional `set-`, a quote, whitespace across lines,
// and a few bytes (`=`, `--header`) more.