The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
fix(privacy): redact repeated curl passwords, URL passwords after Bearer and labeled AWS secrets; replace long copies in linear time (#1230)
kyle-elliott-asymptote437bb6fcd865
diff --git a/docs/changelog/cli.mdx b/docs/changelog/cli.mdxindex a2ab1c6e..624567de 100644--- a/docs/changelog/cli.mdx+++ b/docs/changelog/cli.mdx@@ -13,6 +13,14 @@ Stay up to date with the latest changes to the Agent Beacon CLI.Changes on `main` since v1.3.58. They ship in the next release.+- **An AWS secret key after a labeled key id is redacted** · Where a label's or a flag's value ended in an AWS access key id and its secret access key followed after a comma, a blank or a quote (`token=AKIA…,<secret>`, `--access-key AKIA… <secret>`, `x-access-token:abc@AKIA…,<secret>`), the value rule replaced the key id and the rule for a key id and its secret then found no key id, so the secret was kept. The secret is now replaced with the value, keeping the separators: `token=[REDACTED],[REDACTED]`. A URL password that runs on over a key id written right after a host name (`…@hostAKIA…,<secret>`) takes the secret too.++- **A URL password after `Bearer` is redacted** · The Bearer rule takes the token after `Bearer` up to a `:`, so for `Bearer postgres://user:pw@host` it replaced the scheme, and the URL rule then found no scheme before `://` and kept the password. A password after a replaced scheme is now redacted too: `Bearer [REDACTED]://user:[REDACTED]@host`.++- **Every curl password and cookie on a command line is redacted** · The curl rules redacted the first `-u`/`--user`/`--proxy-user` password on a line and left the rest (`curl -u a:x -u b:y` kept `y`), and applied the cookie rule at most 16 more times, so a line with more `-b`/`--cookie` strings kept the later ones and redacting the output again changed it. Each rule now redacts every flag on its line in one pass.++- **Redacting a long assigned value takes time in proportion to the text** · After the rules run, redaction removes bare copies of every assigned value (`token=…`) from the rest of the text. For a value longer than a few dozen bytes that search grew with the square of the value's length, so 256 KB of text holding one long value took about 4 seconds. Copies of values longer than 64 bytes are now found in one pass over the text; the result is unchanged.+- **Failures lens** · A new built-in lens lists everything that went wrong in a session, in order: commands that exited non-zero, tool and MCP calls the runtime reported as failed, and error events, each with the output that explains it, or the output's hash and size when the log did not keep it. A call reported by both a hook and OpenTelemetry is listed once. For each failed call it says what the next identical call did -- the same command line, the same tool on the same path, or the same MCP tool with the same arguments: succeeded, failed again, or never came. Long lists page with **Show more**. See [Lenses](/concepts/lenses#built-in-lenses).- **Failed tool calls carry an error type in traces** · Runtimes that report a failed tool call through hooks (Claude Code, Cursor, Qwen Code, OpenHands, Kiro, Kimi Code and others) logged it as `tool.failed` with no `error`, so in the dashboard's trace view, lenses and shared traces the failure had no `error.type`, though the lenses spec promises one for every action the runtime reported as failed. A `tool.failed` without an error type of its own now gets `error.type` `tool_execution_failed`, the type the session-sync mappers already give a failed tool result. Stored dashboard history is re-derived on the next read, and the runtime log is unchanged.diff --git a/pkg/asymptoteobserve/privacy.go b/pkg/asymptoteobserve/privacy.goindex 59bc94f0..6f1e1a46 100644--- a/pkg/asymptoteobserve/privacy.go+++ b/pkg/asymptoteobserve/privacy.go@@ -90,7 +90,7 @@ func quotedRules(kind quotedKind, keep func(string) bool) []redactionRule {}rules = append(rules, redactionRule{expr: `(?i)(` + prefix + quoted[0] + `)(?P<value>` + quoted[1] + `)`, replacement: "${1}" + redactedValue,- keep: keep, needles: needles, start: start,+ keep: keep, needles: needles, start: start, runsOn: awsPairRunsOn,})}return rules@@ -145,10 +145,10 @@ type redactionRule struct {// needles are literals the pattern cannot match without: when the text holds none of them,// the rule is skipped without running the pattern.needles *redactNeedles- // repeat applies the rule until the text stops changing. A rule scoped to a command line- // matches from the command to the value, so a second such value on the line lies inside the- // first match and is reached only by another pass.- repeat bool+ // wholeLine marks a rule scoped to a command line: its pattern opens with the command word and+ // the lazy run of the line (`\bcurl\b[^\n]*?`), so a match takes the first flag after the+ // command, and the rest of the line is searched for the flag again (restOfLine).+ wholeLine bool// start says where, around an occurrence of one of its needles, a match can begin, and offset// how many bytes before the needle a startNeedle match does. See redact_engine.go.start redactStart@@ -157,8 +157,11 @@ type redactionRule struct {// secret after it. Such a rule also redacts a secret written right after its match// (redact_adjacent.go).shape redactShapeKind- // runsOn, when set, returns where the text a match replaces ends, at or past the match's end.- runsOn func(t string, match []int) int+ // runsOn, when set, returns the replacement of a match whose replaced text runs on past the+ // match's end, and where that text ends; an end below 0 means the match is replaced as usual.+ // It is a plain function of the rule, not a closure, so building the rules allocates nothing+ // for it (TestPackageInitStaysCheap).+ runsOn func(rule *redactionRule, t string, match []int) (replacement string, end int)}// pattern is the rule's compiled pattern.@@ -207,10 +210,10 @@ func foldedRule(expr, replacement string, keep func(string) bool, needles ...str}// curlRule is a foldedRule scoped to a curl command line. A flag curl accepts more than once (two-// -b cookie strings) is redacted at every occurrence on the line.+// -b cookie strings, two -u credentials) is redacted at every occurrence on the line.func curlRule(expr, replacement string) redactionRule {r := foldedRule(expr, replacement, nil, "curl")- r.repeat = true+ r.wholeLine = truereturn r}@@ -426,6 +429,7 @@ var redactionRules = concatRules([]redactionRule{{expr: `(?i)(\b` + redactLabel + `(?P<q>` + redactOpenQuote + `)\s*[:=]\s*)(?P<value>` + redactBare + `)`,replacement: "${1}${q}" + redactedValue + "${q}", keep: isLiteralOrAuthScheme, needles: redactQuotedLabelNeedles, start: startLabel,+ runsOn: awsPairRunsOn,},// An unquoted value. A label opened by `?` or `&` is a URL query parameter, whose value ends// at the `&` of the next one or the `#` of the fragment: `?api_key=x&user=alice` keeps@@ -436,18 +440,20 @@ var redactionRules = concatRules([]redactionRule{expr: `(?i)(?:([?&]` + redactLabel + `=)(?P<value>` + redactParamValue + `)|` +`(\b` + redactLabel + `\s*[:=]\s*)(?P<value>` + redactUnquoted + `))`,replacement: "${1}${3}" + redactedValue, keep: isAuthScheme, needles: redactLabelNeedles, start: startParam,+ runsOn: awsPairRunsOn,},// A command-line flag: `--password x`, `--token=x`, `--api-key x`, and quoted, the way agents// usually write arguments: `--password "x y"`, `--token='x'`. An unquoted value does not start// with `=` or `:`, which is an assignment's separator after a word that only looks like a flag:// a diff's removed line `-SECRET_KEY = 'x'`.}, quotedRules(quotedFlag, nil), []redactionRule{- {expr: `(?i)(` + redactFlag + `)([^\s"'=:-][^\s"']*)`, replacement: "${1}" + redactedValue, needles: redactFlagNeedles, start: startFlag},+ {expr: `(?i)(` + redactFlag + `)([^\s"'=:-][^\s"']*)`, replacement: "${1}" + redactedValue, needles: redactFlagNeedles, start: startFlag, runsOn: awsPairRunsOn},// A flag and its value as two quoted items of an argument list, the way an MCP server's args// are written in JSON, TOML and YAML: `"args": ["--api-key", "x"]`.{expr: `(?i)(["']` + redactFlagName + `["']\s*,\s*["'])(?P<value>(?:[^"'\n\\]|\\[^\n])+)`,replacement: "${1}" + redactedValue, keep: isRedactedValue, needles: redactListedFlagNeedles, start: startListedFlag,+ runsOn: awsPairRunsOn,},// curl's basic-auth credentials: `-u user:pass`, `-uuser:pass`, `--user user:pass` and// `--proxy-user user:pass`. The user name and the colon stay. A bare `-u user` (curl prompts for@@ -463,7 +469,7 @@ var redactionRules = concatRules([]redactionRule{foldedRule(`(?im)((?:^\s*|["']\s*|(?:-H|--header)(?:=|\s+)["']?)(?:set-)?cookie["']?\s*:\s*["']?)([^\n"'\\]+)`, "${1}"+redactedValue, isRedactedValue, "cookie").from(startLine),// The password in a URL's userinfo: `postgres://user:pw@host`, `https://x-access-token:[email protected]`,// and with no user name, the way Redis URLs carry a password: `redis://:pw@host`.- foldedRule(`(?i)(\b[a-z][a-z0-9+.-]*://[^/\s:@"'<>]*:)([^@/\s"'<>]+)(@)`, "${1}"+redactedValue+"${3}", nil, "://").from(startScheme).runningOn(urlRunsOn),+ foldedRule(`(?i)(\b[a-z][a-z0-9+.-]*://[^/\s:@"'<>]*:)([^@/\s"'<>]+)(@)`, "${1}"+redactedValue+"${3}", nil, "://").from(startScheme).runningOn(urlRunningOn),// Credentials recognizable by their own shape, wherever they appear. The type prefix is public// and stays (`ghp_[REDACTED]`, `AKIA[REDACTED]`), the way secret scanners report a finding: the// line still says a GitHub token or an AWS key was there, which is what detection rules match on@@ -537,24 +543,13 @@ func (rule redactionRule) segmented() redactionRule {}// runningOn returns the rule with the text its matches replace running on where runsOn says.-func (rule redactionRule) runningOn(runsOn func(t string, match []int) int) redactionRule {+func (rule redactionRule) runningOn(runsOn func(rule *redactionRule, t string, match []int) (string, int)) redactionRule {rule.runsOn = runsOnreturn rule}-// apply replaces every match of the rule in the gate's text, except one whose value keep passes,-// and a repeating rule until the text stops changing.+// apply replaces every match of the rule in the gate's text, except one whose value keep passes.func (rule redactionRule) apply(text *redactGate, slot *redactSlot) string {- out := rule.applyOnce(text, slot)- // Each pass redacts at least one more value, and a redacted value matches no further.- for pass := 0; rule.repeat && out != text.value && pass < 16; pass++ {- text.set(out)- out = rule.applyOnce(text, slot)- }- return out-}--func (rule redactionRule) applyOnce(text *redactGate, slot *redactSlot) string {value := text.valuevar out []bytelast, slashed := 0, false@@ -572,6 +567,9 @@ func (rule redactionRule) applyOnce(text *redactGate, slot *redactSlot) string {replacement, end := use.rule.replacing(value, match)out = use.pattern.ExpandString(out, replacement, value, match)last = end+ if use.rule.wholeLine {+ out, last = use.rule.restOfLine(out, value, match[0], last)+ }// An AWS secret may hold a `/`, where the URL rule, which runs before the shape rules,// stopped a password; once the secret is replaced, the password can run on to its `@`.slashed = slashed || slashedAWSSecret(use.rule.expr, value, match)@@ -585,18 +583,67 @@ func (rule redactionRule) applyOnce(text *redactGate, slot *redactSlot) string {return string(append(out, value[last:]...))}+// restOfLine redacts the flag of a wholeLine rule again and again on the rest of the command line+// a match starting at start took the first flag of, from last, where that match's replacement+// ended: one search over the line, however many flags it holds. A flag must start on the line, or+// at the newline ending it, as the rule's lazy run requires; its value can reach the next lines+// (commandLineEnd). The rule itself cannot find the next flag: its match opens with the command+// word, which lies before the end of the last match.+func (rule *redactionRule) restOfLine(out []byte, t string, start, last int) ([]byte, int) {+ nl := len(t)+ if i := strings.IndexByte(t[start:], '\n'); i >= 0 {+ nl = start + i+ }+ if last > nl {+ return out, last+ }+ end := commandLineEnd(t, last)+ redactWorked(end - last)+ re := rule.flagPattern()+ from := last+ for _, m := range re.FindAllStringSubmatchIndex(t[from:end], -1) {+ for i := range m {+ if m[i] >= 0 {+ m[i] += from+ }+ }+ if m[0] > nl {+ break+ }+ out = append(out, t[last:m[0]]...)+ out = re.ExpandString(out, rule.replacement, t, m)+ last = m[1]+ }+ return out, last+}++// flagPattern is a wholeLine rule's pattern without the command word and the lazy run after it:+// the flag and its value, in the same groups.+func (rule *redactionRule) flagPattern() *regexp.Regexp {+ const run = `\bcurl\b[^\n]*?`+ if !strings.Contains(rule.expr, "("+run) {+ panic("asymptoteobserve: a whole-line rule opens with `curl` and the lazy run of its line: " + rule.expr)+ }+ return compiledPattern(strings.Replace(rule.expr, run, "", 1))+}+// replacing returns the replacement of match in t, and where the text it replaces ends: the-// match's end, unless the rule's match runs on (runsOn), when what the rule keeps after its value-// goes too.+// match's end, unless the rule's match runs on (runsOn).func (rule *redactionRule) replacing(t string, match []int) (string, int) {if rule.runsOn != nil {- if end := rule.runsOn(t, match); end >= 0 {- return "${1}" + redactedValue, end+ if replacement, end := rule.runsOn(rule, t, match); end >= 0 {+ return replacement, end}}return rule.replacement, match[1]}+// urlRunningOn is the URL rule's runsOn: where urlRunsOn says, the text up to there is replaced+// with what the rule keeps before the password, ${1}, and a marker.+func urlRunningOn(_ *redactionRule, t string, m []int) (string, int) {+ return "${1}" + redactedValue, urlRunsOn(t, m)+}+// valueGroup returns the index of the group named value that took part in match, which a rule's// keep reads: a pattern with alternatives names one in each. A pattern that names none has its// value in the second group.@@ -624,8 +671,8 @@ func valueGroup(re *regexp.Regexp, match []int) int {// The shape rules run after this one, and one of them changes where either run stops: an AWS key// id followed by its secret (awsPairAt), a secret recognized only after its key id, which holds// `/` and `+`. A label's value that ends in the key id would leave the secret with no key id-// before it, so it takes the separators and the secret too; the password reads the pair as the-// markers and separators the shape rule leaves of it.+// before it, so it takes the separators and the secret too (awsPairTailEnd); the password+// reads the pair as the markers and separators the shape rule leaves of it.func urlRunsOn(t string, m []int) int {if !labelEndsAt(t[:m[3]]) {return -1@@ -639,18 +686,11 @@ func urlRunsOn(t string, m []int) int {for end < len(t) && strings.IndexByte(redactValueStops, t[end]) < 0 {end++}- k := end- for k > run && end-k < 8 && (t[k-1] == ':' || t[k-1] == '|') {- k--- }- if k-run < awsKeyIDLen {- break- }- _, secretEnd := awsPairAt(t, k-awsKeyIDLen)- if secretEnd < 0 {+ tailEnd := awsPairTailEnd(t, run, end)+ if tailEnd < 0 {break}- end, run = secretEnd, secretEnd+ end, run = tailEnd, tailEnd}// The URL's password: that value and what follows it up to an `@`.at := end@@ -674,12 +714,101 @@ func urlRunsOn(t string, m []int) int {}}+// awsPairRunsOn is the runsOn of a rule for a labeled value or a flag's value: where the value+// ends in an AWS access key id that its secret follows within a few separators, the replaced text+// runs on to the secret's end. The value stops at a comma, a blank or a quote between them+// (`token=AKIA…,<secret>`), and once it is replaced the rule for a key id and its secret finds no+// key id before the secret. The match is replaced with the rule's replacement, the separators after+// the value stay and the secret gets a marker of its own, as that rule writes a pair, so the value+// reads the same to the next pass: `token=[REDACTED],[REDACTED]`. The separators are blanks,+// commas and quotes, which an expansion template reads as they are.+func awsPairRunsOn(rule *redactionRule, t string, match []int) (string, int) {+ end := awsPairTailEnd(t, match[0], match[1])+ if end < 0 {+ return "", -1+ }+ return rule.replacement + t[match[1]:end-awsPairKeyLen] + redactedValue, end+}++// awsPairTailEnd returns where the secret ends that follows an AWS access key id the run+// t[run:end] ends in, before any separators `:` and `|` it takes, or -1. The key id may be glued to+// what comes before it in the run (`@hostAKIA…`), where the rules for key ids would not take it for+// one: it is redacted as part of the run, and so is the secret.+//+// Forty bytes that a secret-named label with its separator starts in are no secret+// (`<34 bytes>secret: x`, `<40 bytes>--db-password=x`): running on over them would hide the+// label's own match, which starts inside what the run replaces, from its rule. A label is spelled+// in bytes the secret holds too, and may go on past it (labelRunEnd), so the label is looked for+// where that run ends.+func awsPairTailEnd(t string, run, end int) int {+ k := end+ for k > run && end-k < 8 && (t[k-1] == ':' || t[k-1] == '|') {+ k--+ }+ if k-run < awsKeyIDLen {+ return -1+ }+ id := t[k-awsKeyIDLen : k]+ if !strings.HasPrefix(id, "AKIA") && !strings.HasPrefix(id, "ASIA") {+ return -1+ }+ for i := 4; i < awsKeyIDLen; i++ {+ if c := id[i]; (c < 'A' || c > 'Z') && (c < '0' || c > '9') {+ return -1+ }+ }+ stop := min(len(t), k+awsPairTailSpan)+ redactWorked(stop - k)+ m := redactAWSPairTail().FindStringSubmatchIndex(t[k:stop])+ if m == nil {+ return -1+ }+ tail := k + m[3]+ label := labelRunEnd(t, tail)+ redactWorked(label - tail)+ if keptLabel(t, label) {+ return -1+ }+ return tail+}++// labelRunEnd returns the end of the run from i of the bytes a label is spelled with, as the+// case-insensitive label rules read them: letters, digits, `_`, `.` and `-`, and ſ and K, which+// fold into s and k.+func labelRunEnd(t string, i int) int {+ for i < len(t) {+ switch {+ case labelClass[t[i]]:+ i+++ case strings.HasPrefix(t[i:], "\u017f"):+ i += len("\u017f")+ case strings.HasPrefix(t[i:], "\u212a"):+ i += len("\u212a")+ default:+ return i+ }+ }+ return i+}++// redactAWSPairTail is redactAWSPair after its key id: the separators, the secret (${1}) and+// what ends it. The key id ends in a word byte and the separators hold none, so redactAWSPair's \b+// holds where it starts.+var redactAWSPairTail = lazyPattern(`^[\s,:|"']{1,8}([A-Za-z0-9/+]{40})(?:[^A-Za-z0-9/+=]|$)`)++// awsPairTailSpan is the most redactAWSPairTail reads: eight separators, the secret and the byte+// after it. Its `$` is the end of the text only where the slice is.+const awsPairTailSpan = 8 + 40 + 1+// redactAWSPair is an AWS access key id followed by its secret access key within a few separators:// ${1} the key id's type prefix, ${2} the separators, ${3} what ends the secret.const redactAWSPair = `(\b(?:AKIA|ASIA))[A-Z0-9]{16}(\b[\s,:|"']{1,8})[A-Za-z0-9/+]{40}([^A-Za-z0-9/+=]|$)`-// awsKeyIDLen is the length of an AWS access key id.-const awsKeyIDLen = 20+// awsKeyIDLen is the length of an AWS access key id, and awsPairKeyLen of its secret access key.+const (+ awsKeyIDLen = 20+ awsPairKeyLen = 40+)var redactAWSPairAt = lazyPattern(`^` + redactAWSPair)@@ -774,11 +903,12 @@ func RedactString(value string) string {// (`token` names an auth scheme) and never sees `token":0`; the rule for unquoted labels then// replaces `seCret`, the label that hid it, and only the next pass redacted `0`. Copies of// assigned values, replaced after every rule has run, can be where a match starts or ends-// (copyHidesMatch); and the URL rule stops a password at a `/` in an AWS secret, which the shape-// rules replace after it. Such text is redacted again until it stops changing, as every later call-// would redact it. A pass can only uncover what the one before hid; texts the tests and fuzzing-// have tried took at most three passes that change them, and redactMaxPasses keeps the cost-// linear whatever the text.+// (copyHidesMatch); the URL rule stops a password at a `/` in an AWS secret, which the shape+// rules replace after it; and a password after a replaced scheme is replaced only once every rule+// has run (redactUserinfoAfterMarkers). Such text is redacted again until it stops changing, as+// every later call would redact it. A pass can only uncover what the one before hid; texts the+// tests and fuzzing have tried took at most three passes that change them, and redactMaxPasses+// keeps the cost linear whatever the text.func redactString(value string, gated bool) string {for pass := 1; ; pass++ {out, hidden := redactPass(value, gated)@@ -809,7 +939,9 @@ func redactPass(value string, gated bool) (string, bool) {var kept boolif text.holds(redactLabelNeedles) {assigned, kept = captureAssigned(&text, engine, assigned)- text.copies = byFirstBytes(assigned)+ if len(assigned) > 0 {+ text.copies = ©Index{values: assigned}+ }}original := text.valuefor i, rule := range redactionRules {@@ -831,9 +963,63 @@ func redactPass(value string, gated bool) (string, bool) {replaced, glued := replaceAssigned(text.value, assigned)hidden := text.hidden || replaced != text.value && copyHidesMatch(assigned, glued)out, slashed := engine.redactAfterMarkers(replaced)- return out, hidden || slashed && strings.Contains(out, "://")+ hidden = hidden || slashed && strings.Contains(out, "://")+ if after := redactUserinfoAfterMarkers(out); after != out {+ return after, true+ }+ return out, hidden+}++// redactUserinfoAfterMarkers replaces the password of a URL whose scheme a rule replaced: the+// Bearer rule's token stops at a `:`, so of `Bearer postgres://user:pw@host` it takes the scheme,+// and the URL rule, which needs a scheme before `://`, finds none in what is left. A copy of an+// assigned value or a token can be replaced right before `://` too. It runs once every rule has+// run and copies are replaced, on what a pass would return, so it only ever adds a replacement:+// `Bearer [REDACTED]://user:[REDACTED]@host`. The user name and password are read as the URL rule+// reads them.+func redactUserinfoAfterMarkers(t string) string {+ const scheme = redactedValue + "://"+ var out []byte+ last := 0+ for from := 0; ; {+ i := strings.Index(t[from:], scheme)+ if i < 0 {+ break+ }+ user := from + i + len(scheme)+ from = user+ colon := user+ for colon < len(t) && strings.IndexByte(redactUserinfoNameStops, t[colon]) < 0 {+ colon+++ }+ if colon == len(t) || t[colon] != ':' {+ redactWorked(colon - user)+ continue+ }+ at := colon + 1+ for at < len(t) && strings.IndexByte(redactUserinfoValueStops, t[at]) < 0 {+ at+++ }+ redactWorked(at - user)+ if at == colon+1 || at == len(t) || t[at] != '@' {+ continue+ }+ out = append(append(append(out, t[last:colon+1]...), redactedValue...), '@')+ last, from = at+1, at+1+ }+ if out == nil {+ return t+ }+ return string(append(out, t[last:]...))}+// redactUserinfoNameStops and redactUserinfoValueStops end a URL's user name and password as the URL rule reads+// them: `[^/\s:@"'<>]*:` and `[^@/\s"'<>]+@`.+const (+ redactUserinfoNameStops = `/:@"'<>` + regexpSpace+ redactUserinfoValueStops = `@/"'<>` + regexpSpace+)+// copyHidesMatch reports whether replacing copies of assigned values, which happens after every// rule has run, may have made a match where the rules found none: where a copy that starts with a// byte no word starts with was, which some rules' values cannot start with and their marker can;@@ -935,11 +1121,8 @@ func replaceAssigned(value string, assigned map[string]struct{}) (string, bool)if strings.IndexByte(value, 0) >= 0 {place = redactedValue}- pairs := make([]string, 0, 2*len(values))- for _, copied := range values {- pairs = append(pairs, copied, place)- }- replaced := strings.NewReplacer(pairs...).Replace(value)+ replaced, work := replaceLiterals(value, values, place)+ redactWorked(work)if place == redactedValue {return replaced, markersAfterWords(replaced) > markersAfterWords(value)}@@ -958,26 +1141,6 @@ func replaceAssigned(value string, assigned map[string]struct{}) (string, bool)// redactCopyMin is the least length of an assigned value whose copies are removed.const redactCopyMin = 8-// redactCopies are the assigned values by their first redactCopyMin bytes, so a place in the text-// is looked up once to tell whether a copy may start there, and the length of the longest-// (redactChain.overCopy).-type redactCopies struct {- byFirst map[string][]string- longest int-}--func byFirstBytes(assigned map[string]struct{}) *redactCopies {- if len(assigned) == 0 {- return nil- }- copies := &redactCopies{byFirst: map[string][]string{}}- for s := range assigned {- copies.byFirst[s[:redactCopyMin]] = append(copies.byFirst[s[:redactCopyMin]], s)- copies.longest = max(copies.longest, len(s))- }- return copies-}-// IsSecretKeyName reports whether a structured field's name has a word naming a credential, for// callers that judge a field by its name alone. IsSecretField judges a value held under it.func IsSecretKeyName(key string) bool {diff --git a/pkg/asymptoteobserve/privacy_test.go b/pkg/asymptoteobserve/privacy_test.goindex b1da038a..f41ddfe3 100644--- a/pkg/asymptoteobserve/privacy_test.go+++ b/pkg/asymptoteobserve/privacy_test.go@@ -128,6 +128,8 @@ func TestRedactStringRedactsCommonCredentialFormats(t *testing.T) {{"database url", "DATABASE_URL=postgres://app:[email protected]:5432/app", "pa55w0rd"},{"bare database url", "connect to postgres://app:[email protected]:5432/app", "pa55w0rd"},{"url password without a user", "REDIS_URL=redis://:[email protected]:6379/0", "pa55w0rd"},+ {"database url after a bearer token", "Authorization header: Bearer postgres://app:[email protected]:5432/app", "pa55w0rd"},+ {"url password without a user after a bearer token", "bearer redis://:[email protected]:6379/0", "pa55w0rd"},{"bare url password without a user", "redis-cli -u redis://:pa55w0rd@localhost:6379 ping", "pa55w0rd"},{"password flag", "mysql --user root --password hunter2hunter2", "hunter2hunter2"},{"token flag", "tool login --token=opaque-flag-value", "opaque-flag-value"},@@ -148,6 +150,9 @@ func TestRedactStringRedactsCommonCredentialFormats(t *testing.T) {{"curl long cookie flag", "curl --cookie sessionid=abcdef123456789 https://x", "abcdef123456789"},{"curl cookie flag in a cluster", `curl -sSb "sessionid=abcdef123456789" https://x`, "abcdef123456789"},{"curl second cookie flag on a line", "curl -b 'a=1' --cookie sessionid=abcdef123456789 https://x", "abcdef123456789"},+ {"curl basic auth given twice", "curl -u a:hunter2hunter2 -u b:opaque-second-pw https://x", "opaque-second-pw"},+ {"curl user and proxy user", "curl --user a:hunter2hunter2 --proxy-user p:opaque-proxy-pw https://x", "opaque-proxy-pw"},+ {"curl with many cookie flags", "curl " + strings.Repeat("-b a=1 ", 20) + "-b sessionid=abcdef123456789 https://x", "abcdef123456789"},{"cookie header", "curl -H 'Cookie: sessionid=abcdef123456789; csrftoken=zzz' https://x", "abcdef123456789"},{"set-cookie header", "Set-Cookie: sid=abcdef123456789; HttpOnly", "abcdef123456789"},{"json cookie header", `{"cookie": "sid=abcdef123456789"}`, "abcdef123456789"},@@ -174,6 +179,11 @@ func TestRedactStringRedactsCommonCredentialFormats(t *testing.T) {{"aws credentials csv", "Access key ID,Secret access key\nAKIAIOSFODNN7EXAMPLE,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\n", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"},{"aws key pair", "use AKIAIOSFODNN7EXAMPLE wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY for the bucket", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"},{"aws key pair ending in a slash", `["ASIAIOSFODNN7EXAMPLE", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKE/"]`, "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKE/"},+ {"aws key pair as a labeled value", "aws_credentials=AKIAIOSFODNN7EXAMPLE,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"},+ {"aws key pair as a quoted labeled value", `token="AKIAIOSFODNN7EXAMPLE","wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"`, "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"},+ {"aws key pair as a flag's value", "tool --access-key AKIAIOSFODNN7EXAMPLE wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"},+ {"aws key pair after a labeled user name", "x-access-token:abc@AKIAIOSFODNN7EXAMPLE,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"},+ {"aws key pair glued to a host", "https://x-access-token:a,b@hostAKIAIOSFODNN7EXAMPLE,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"},{"rsa key", "-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEAsecretbody\n-----END RSA PRIVATE KEY-----", "MIIEowIBAAKCAQEAsecretbody"},{"openssh key cut short", "cat ~/.ssh/id_ed25519\n-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNzaC1rZXktdjEAAAAA", "b3BlbnNzaC1rZXktdjEAAAAA"},}@@ -369,6 +379,53 @@ func TestRedactStringURLLabelPasswordWithQueryString(t *testing.T) {}}+// awsPairLabelRun is 34 bytes of an AWS secret access key's alphabet: with a 6-byte label word+// after it, the label is exactly as long as a secret access key, 40 bytes.+const awsPairLabelRun = "Ab3Ab3Ab3Ab3Ab3Ab3Ab3Ab3Ab3Ab3Ab3x"++// A labeled or flag value that ends in an AWS access key id takes the secret access key written+// after it (awsPairRunsOn). This guards that a 40-byte run after the key id that is itself a+// secret-named label with its separator is not taken for the secret: the value rule would run on+// over the label, the label's own match would then lie inside that run-on and be skipped, and a+// value under 8 bytes, which copy removal leaves alone, would be kept. The label's value is+// redacted, as on main. The quoted and the `:` cases are the ones that kept the value; with `=`,+// which no secret ends at, and after a flag, whose value the label rules redact first, it was+// redacted already, and stays so. A label may also start in the 40 bytes and go on past them, in+// bytes a secret does not hold (`-`, `_`, ſ), and a real secret access key can be where it starts.+func TestRedactStringRedactsALabelAfterAnAWSKeyPair(t *testing.T) {+ for input, want := range map[string]string{+ `{"token": "AKIAIOSFODNN7EXAMPLE", "` + awsPairLabelRun + `Secret": "pin123"}`: `{"token": "[REDACTED]", "` + awsPairLabelRun + `Secret": "[REDACTED]"}`,+ "token=AKIAIOSFODNN7EXAMPLE," + awsPairLabelRun + "secret: pin123": "token=[REDACTED]," + awsPairLabelRun + "secret: [REDACTED]",+ "token=AKIAIOSFODNN7EXAMPLE," + awsPairLabelRun + "secret=pin123": "token=[REDACTED]," + awsPairLabelRun + "secret=[REDACTED]",+ "tool --access-key AKIAIOSFODNN7EXAMPLE " + awsPairLabelRun + "token: pin123": "tool --access-key [REDACTED] " + awsPairLabelRun + "token: [REDACTED]",+ "token=AKIAIOSFODNN7EXAMPLE," + awsPairLabelRun + "Secret--db-password=pin12": "token=[REDACTED]," + awsPairLabelRun + "Secret--db-password=[REDACTED]",+ "token=AKIAIOSFODNN7EXAMPLE,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY--db-password: pin12": "token=[REDACTED],wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY--db-password: [REDACTED]",+ "token=AKIAIOSFODNN7EXAMPLE," + awsPairLabelRun + "secret\u017fecret_apikey : pin12": "token=[REDACTED]," + awsPairLabelRun + "secret\u017fecret_apikey : [REDACTED]",+ } {+ got := RedactString(input)+ if strings.Contains(got, "pin12") || got != want {+ t.Errorf("RedactString(%q) = %q, want %q", input, got, want)+ }+ }+}++// A URL's password after a scheme a rule replaced is redacted: the Bearer rule takes a scheme+// written after it, and a copy or a token's tail can end right before `://`. A copy that holds+// the password is still removed whole.+func TestRedactStringRedactsAPasswordAfterAReplacedScheme(t *testing.T) {+ for input, want := range map[string]string{+ "Bearer postgres://user:pw@host/db": "Bearer [REDACTED]://user:[REDACTED]@host/db",+ "bearer redis://:[email protected]:6379/0": "bearer [REDACTED]://:[REDACTED]@cache.internal:6379/0",+ "token=00000000 00000000://:0@": "token=[REDACTED] [REDACTED]://:[REDACTED]@",+ "ghp_0123456789abcdefghijklmnopqrstuvwxyz://u:hunter2@h": "ghp_[REDACTED]z://u:[REDACTED]@h",+ "Bearer api_key-x-token://c:dnpm_@curl ABC": "Bearer [REDACTED]:[REDACTED] ABC",+ } {+ if got := RedactString(input); got != want {+ t.Errorf("RedactString(%q) = %q, want %q", input, got, want)+ }+ }+}+// A label hidden inside a value a rule kept is redacted in the same call, as a second call would.func TestRedactStringRedactsALabelAKeptValueHid(t *testing.T) {for input, want := range map[string]string{@@ -391,6 +448,23 @@ var redactIdempotenceInputs = []string{"postgres://app:pa55w0rd@db/app","curl -u admin:hunter2hunter2 -H 'Cookie: sid=abcdef123456789' https://x","curl -b 'sessionid=abcdef123456789' --cookie sid=abcdef123456789 https://x",+ "curl -u a:hunter2hunter2 -u b:opaque-second-pw --proxy-user p:opaque-proxy-pw https://x",+ "curl " + strings.Repeat("-b a=1 ", 20) + "-b sessionid=abcdef123456789 https://x",+ "Bearer postgres://app:pa55w0rd@db/app",+ // A scheme replaced right before `://` by a copy of an assigned value, and by a token's tail.+ "token=00000000 00000000://:0@",+ "ghp_0123456789abcdefghijklmnopqrstuvwxyz://u:hunter2@h",+ "token=AKIAIOSFODNN7EXAMPLE,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY}AKIAIOSFODNN7EXAMPLE",+ "https://x-access-token:a,b@hostAKIAIOSFODNN7EXAMPLE,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",+ "token=abcdefgh12345678AKIAIOSFODNN7EXAMPLE,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY-----BEGIN PRIVATE KEY-----",+ // A 40-byte run after a labeled key id that is itself a label with its separator.+ `{"token": "AKIAIOSFODNN7EXAMPLE", "` + awsPairLabelRun + `Secret": "pin123"}`,+ "token=AKIAIOSFODNN7EXAMPLE," + awsPairLabelRun + "secret: pin123",+ "token=AKIAIOSFODNN7EXAMPLE," + awsPairLabelRun + "secret=pin123",+ "tool --access-key AKIAIOSFODNN7EXAMPLE " + awsPairLabelRun + "token: pin123",+ "token=AKIAIOSFODNN7EXAMPLE," + awsPairLabelRun + "Secret--db-password=pin12",+ "token=AKIAIOSFODNN7EXAMPLE,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY--db-password: pin12",+ "token=AKIAIOSFODNN7EXAMPLE," + awsPairLabelRun + "secret\u017fecret_apikey : pin12",// A URL whose user name is a secret-named label: the label rules and the URL rule stop at// different bytes, and each used to take one more piece per pass."https://x-access-token:[email protected]_api_`@host",diff --git a/pkg/asymptoteobserve/redact_adjacent.go b/pkg/asymptoteobserve/redact_adjacent.goindex 0e65f97d..6a68fd6a 100644--- a/pkg/asymptoteobserve/redact_adjacent.go+++ b/pkg/asymptoteobserve/redact_adjacent.go@@ -101,8 +101,8 @@ const (followRun redactFollowerKind = iota// followScheme: a URL, which holds no `/` and no whitespace after its `://`.followScheme- // followLine: a curl rule, which reads its whole command line and is applied to it until it- // stops changing (redactLinesAfterMarkers).+ // followLine: a curl rule, which reads its whole command line and redacts every flag on it+ // (redactLinesAfterMarkers).followLine)@@ -138,7 +138,7 @@ func (e *redactEngine) adjacency() *redactAdjacency {switch {case rule.start == startScheme:f.kind = followScheme- case rule.repeat:+ case rule.wholeLine:f.kind = followLinef.lead = leadPattern(rule)default:@@ -348,14 +348,13 @@ type redactChain struct {// last place followerAt was asked about and its answer.runFrom, runTo intasked []redactAsked- // The assigned values whose copies are removed after every rule has run (overCopy), what- // running on over them may still read (copyBudget), and whether running on over one, or- // over an AWS secret that holds a `/`, may hide a match from this pass, as replacing a copy- // may (copyHidesMatch); matches reports it to the gate.- copies *redactCopies- copyBudget int- hid bool- slashed bool+ // The pass's assigned values whose copies are removed after every rule has run (overCopy),+ // and whether running on over a copy, or over an AWS secret that holds a `/`, may hide a+ // match from this pass, as replacing a copy may (copyHidesMatch); matches reports it to the+ // gate.+ copies *copyIndex+ hid bool+ slashed bool}// redactAsked is followerAt's answer for one follower at a place.@@ -366,17 +365,9 @@ type redactAsked struct {func newRedactChain(engine *redactEngine, t string, hits []redactHit) *redactChain {return &redactChain{engine: engine, adj: engine.adjacency(), t: t, hits: hits, segFrom: -1, restAt: -1,- schemeFrom: -1, schemeTo: -1, schemeAt: -1, lineFrom: -1, lineTo: -1, lineFirst: -1, runFrom: -1, runTo: -1,- copyBudget: redactCopyWorkPerByte*len(t) + redactCopyWorkBase}+ schemeFrom: -1, schemeTo: -1, schemeAt: -1, lineFrom: -1, lineTo: -1, lineFirst: -1, runFrom: -1, runTo: -1}}-// overCopy may read redactCopyWorkPerByte bytes and copies per byte of the text, and-// redactCopyWorkBase more; past that it leaves items as they are, as main does.-const (- redactCopyWorkPerByte = 8- redactCopyWorkBase = 1 << 16-)-func redactWorked(n int) {if redactWork != nil {redactWork(n)@@ -621,7 +612,7 @@ func (c *redactChain) run(si int, m []int, use redactUse, yield func([]int, redaitem[1] = sm[2*g]}// The item is yielded with the first secret's rule: an AWS secret it takes is not seen- // as one where the replacements are made (redactionRule.applyOnce).+ // as one where the replacements are made (redactionRule.apply).c.slashed = c.slashed || slashedAWSSecret(redactionRules[c.adj.shapes[sj].rule].expr, c.t, sm)si, m = sj, smcontinue@@ -671,34 +662,78 @@ func (c *redactChain) run(si int, m []int, use redactUse, yield func([]int, reda// ends past it, the item having taken the copy's first bytes; or item. The copy is removed after// every rule has run, which finds it only whole: of `hf_…opaque-flag-value` with `opaque-flag-value`// assigned, the token's pattern takes `opaque`, and `-flag-value` was left. Such a copy starts-// less than the longest copy's length before the item's end. What it reads, places, copies looked-// at and bytes compared, comes out of the chain's copyBudget, linear in the text; once that is-// spent, items are left as they are.+// less than the longest copy's length before the item's end, and once the item runs on, a copy+// that starts before its new end and ends past it takes it further.+//+// What it reads of the text is what copyIndex.longestAt finds near the items, for every chain of+// the pass that reads the same text, and each item reads its own bytes, which no other item of the+// chain holds: what overCopy reads stays linear in the text.func (c *redactChain) overCopy(item []int) []int {- if c.copies == nil || c.copyBudget <= 0 {+ if c.copies == nil {return item}- t, work := c.t, 0- for x := max(item[0]+1, item[1]-c.copies.longest+1); x < item[1] && x+redactCopyMin <= len(t) && work < c.copyBudget; x++ {- work++- for _, v := range c.copies.byFirst[t[x:x+redactCopyMin]] {- work++- if x+len(v) <= item[1] {- continue- }- work += len(v)- if strings.HasPrefix(t[x:], v) {- item = slices.Clone(item)- item[1] = x + len(v)- c.hid = true- }+ idx := c.copies+ from, end := max(item[0]+1, item[1]-idx.built().maxLen+1), item[1]+ for x := from; x < end; x++ {+ if e := x + int(idx.longestAt(c.t, x, end)); e > end {+ end, c.hid = e, true}}- c.copyBudget -= work- redactWorked(work)+ redactWorked(max(0, end-from))+ if end == item[1] {+ return item+ }+ item = slices.Clone(item)+ item[1] = endreturn item}+// copyIndex is what the chains of one redaction pass find of the assigned values in the text+// (overCopy): their finder, built the first time a chain asks, and, for the text the gate holds,+// the length of the longest copy starting at each byte of the stretch [lo, lo+len(at)) that the+// chains asked about last. A rule that changes the text moves its bytes, so redactGate.set drops+// what was found of it.+type copyIndex struct {+ values map[string]struct{}+ finder *literalFinder+ lo int+ at []int32+}++// built returns the finder over the values, building it the first time.+func (idx *copyIndex) built() *literalFinder {+ if idx.finder == nil {+ values := make([]string, 0, len(idx.values))+ for v := range idx.values {+ values = append(values, v)+ }+ idx.finder = newLiteralFinder(values)+ redactWorked(len(idx.finder.label) + idx.finder.work)+ }+ return idx.finder+}++// longestAt returns the length of the longest copy starting at x in t, the text of the gate the+// index belongs to, after built. Where it has not found that yet, it finds the bytes from x, or+// from the end of the stretch it found last when that ends less than the longest value's length+// before x, up to upto and at least that length past where it starts (literalFinder.longestIn). The+// chains ask about places in order, so it finds each byte of the text at most once per chain, and+// reads at most twice what it finds: only the stretches near items, however long the text.+func (idx *copyIndex) longestAt(t string, x, upto int) int32 {+ if hi := idx.lo + len(idx.at); idx.lo <= x && x < hi {+ return idx.at[x-idx.lo]+ }+ f := idx.finder+ hi := idx.lo + len(idx.at)+ if len(idx.at) == 0 || x < idx.lo || x > hi+f.maxLen {+ idx.lo, idx.at, hi = x, idx.at[:0], x+ }+ end := min(len(t), max(upto, x+1, hi+f.maxLen))+ idx.at = slices.Grow(idx.at, end-hi)[:end-idx.lo]+ redactWorked(f.longestIn(t, idx.at[hi-idx.lo:], hi, end))+ return idx.at[x-idx.lo]+}+// slashedAWSSecret reports whether m, a match of the pattern expr, is one of the rule for an AWS// key id and its secret whose secret holds a `/`, where the URL rule, which runs before the shape// rules, stopped a password: once the secret is replaced, the password can run on to its `@`.@@ -864,8 +899,8 @@ func markerBeforeWord(t string, from int) int {// before a word byte, the followers and then the shape rules, in rule order, are tried there; the// first that matches is applied, and a shape rule's chain after it. Each place is tried once, left// to right, so a chain that alternates between them is redacted in one pass. The curl rules,-// which read a whole command line and are applied to it until it stops changing, are then-// applied to the line from the first such place where a curl command starts.+// which read a whole command line and redact every flag on it, are then applied to the line from+// the first such place where a curl command starts.//// It also reports whether it replaced an AWS secret that holds a `/` (slashedAWSSecret).func (e *redactEngine) redactAfterMarkers(t string) (string, bool) {diff --git a/pkg/asymptoteobserve/redact_adjacent_test.go b/pkg/asymptoteobserve/redact_adjacent_test.goindex 8bbdb810..bb36edab 100644--- a/pkg/asymptoteobserve/redact_adjacent_test.go+++ b/pkg/asymptoteobserve/redact_adjacent_test.go@@ -246,3 +246,75 @@ func TestRedactIsIdempotentWhereSecretsMeet(t *testing.T) {t.Fatalf("only %d texts redacted a secret right after another", met)}}++// overCopyByFirstBytes is overCopy as it first searched, the reference+// TestOverCopyFindsWhatItsFirstSearchFound holds it to: at each place of the window, the values+// that share its first redactCopyMin bytes, compared whole, the window growing with every copy+// that runs past its end.+func overCopyByFirstBytes(t string, values map[string]struct{}, item []int) (int, bool) {+ byFirst, longest := map[string][]string{}, 0+ for v := range values {+ byFirst[v[:redactCopyMin]] = append(byFirst[v[:redactCopyMin]], v)+ longest = max(longest, len(v))+ }+ end, hid := item[1], false+ for x := max(item[0]+1, end-longest+1); x < end && x+redactCopyMin <= len(t); x++ {+ for _, v := range byFirst[t[x:x+redactCopyMin]] {+ if x+len(v) > end && strings.HasPrefix(t[x:], v) {+ end, hid = x+len(v), true+ }+ }+ }+ return end, hid+}++// overCopy takes an item to where the first search took it, item after item along one chain:+// values that overlap, contain each other and start alike, in texts made of their pieces, so+// copies cross items' ends and run on into each other.+func TestOverCopyFindsWhatItsFirstSearchFound(t *testing.T) {+ r := rand.New(rand.NewPCG(5, 11))+ piece := func(n int) string {+ b := make([]byte, n)+ for i := range b {+ b[i] = "ab"[r.IntN(2)]+ }+ return string(b)+ }+ ranOn := 0+ for range 20000 {+ values := map[string]struct{}{}+ var list []string+ for range 1 + r.IntN(4) {+ v := piece(redactCopyMin + r.IntN(12))+ values[v] = struct{}{}+ list = append(list, v)+ }+ var text strings.Builder+ for range 1 + r.IntN(10) {+ if r.IntN(3) == 0 {+ text.WriteString(piece(1 + r.IntN(6)))+ } else {+ text.WriteString(list[r.IntN(len(list))])+ }+ }+ s := text.String()+ c := &redactChain{t: s, copies: ©Index{values: values}}+ for pos := 0; pos < len(s); {+ start := pos + r.IntN(len(s)-pos)+ item := []int{start, start + 1 + r.IntN(len(s)-start)}+ wantEnd, wantHid := overCopyByFirstBytes(s, values, item)+ c.hid = false+ got := c.overCopy(item)+ if got[1] != wantEnd || c.hid != wantHid || got[0] != item[0] {+ t.Fatalf("overCopy(%v) in %q with %q: got end %d hid %v, want %d %v", item, s, list, got[1], c.hid, wantEnd, wantHid)+ }+ if wantEnd > item[1] {+ ranOn+++ }+ pos = got[1]+ }+ }+ if ranOn < 1000 {+ t.Fatalf("only %d items ran on over a copy: the texts no longer exercise overCopy", ranOn)+ }+}diff --git a/pkg/asymptoteobserve/redact_cost_test.go b/pkg/asymptoteobserve/redact_cost_test.goindex 5d5f6357..6da902fd 100644--- a/pkg/asymptoteobserve/redact_cost_test.go+++ b/pkg/asymptoteobserve/redact_cost_test.go@@ -16,6 +16,18 @@ func BenchmarkRedactStringRepeatedAssignments(b *testing.B) {}}+// BenchmarkRedactStringLongAssignedValue covers copies of a value longer than replaceLiteralsShort: a+// token as long as a JWT, assigned once and quoted again later in 8 KiB of output.+func BenchmarkRedactStringLongAssignedValue(b *testing.B) {+ value := strings.Repeat("abcdefghijklmnopqrstuvwxyz0123456789-_ABCDEFGHIJKLMNOPQRSTUVWXYZ", 10)+ line := "ok \tgithub.com/example/project/internal/endpoint\t0.123s\n"+ text := "access_token=" + value + "\n" + strings.Repeat(line, 60) + "echo " + value + "\n" + strings.Repeat(line, 60)+ b.SetBytes(int64(len(text)))+ for b.Loop() {+ RedactString(text)+ }+}+func TestRedactStringRemovesBareCopiesLongestFirst(t *testing.T) {out := RedactString("token=hunter2hunter2 password=hunter2hunter2xyz later hunter2hunter2xyz and hunter2hunter2")if strings.Contains(out, "hunter2") || strings.Contains(out, "xyz") {diff --git a/pkg/asymptoteobserve/redact_engine.go b/pkg/asymptoteobserve/redact_engine.goindex aca51789..3e581c38 100644--- a/pkg/asymptoteobserve/redact_engine.go+++ b/pkg/asymptoteobserve/redact_engine.go@@ -244,13 +244,17 @@ type redactGate struct {// neither looks into; a copy a token ran on over, as replacing a copy may (copyHidesMatch); or a// replaced AWS secret's `/`, where a URL's password stopped.hidden bool- // copies are the assigned values whose copies are removed after every rule has run.- copies *redactCopies+ // copies are the assigned values whose copies are removed after every rule has run, and what+ // the chains of this pass found of them in the text (copyIndex).+ copies *copyIndex}func (g *redactGate) set(value string) {if value != g.value {g.value, g.scanned, g.known, g.present = value, false, 0, 0+ if g.copies != nil {+ g.copies.lo, g.copies.at = 0, g.copies.at[:0]+ }}}diff --git a/pkg/asymptoteobserve/redact_engine_test.go b/pkg/asymptoteobserve/redact_engine_test.goindex afeef1db..ac9d6344 100644--- a/pkg/asymptoteobserve/redact_engine_test.go+++ b/pkg/asymptoteobserve/redact_engine_test.go@@ -264,6 +264,78 @@ func TestRedactionWorkIsLinear(t *testing.T) {}return b.String() + repeatTo("ghp_"+strings.Repeat("a", 40)+" ", n/2)}+ // Long assigned values, whose copies are searched for in the whole text: one as long as half+ // the text, many of fixed length, and many ending the same way, which share a run of states+ // with many edges out of it.+ texts["one value as long as half the text"] = func(n int) string {+ a := strings.Repeat("a", n/2)+ return "token=" + a + " hf_" + a+ }+ texts["many long values"] = func(n int) string {+ var b strings.Builder+ for i := 0; b.Len() < n; i++ {+ fmt.Fprintf(&b, "token=%070d\n", i)+ }+ return b.String()+ }+ texts["many long values ending the same way"] = func(n int) string {+ var b strings.Builder+ tail := strings.Repeat("a", 70)+ for i := 0; b.Len() < n/2; i++ {+ fmt.Fprintf(&b, "token=%d%s\n", i, tail)+ }+ return b.String() + repeatTo("a", n/2)+ }+ // Every flag on a curl line is redacted in one pass: many on one line, after values that never+ // close, and many lines.+ texts["curl passwords on one line"] = func(n int) string { return "curl" + repeatTo(" -u a:b", n) }+ texts["curl cookies on one line"] = func(n int) string { return "curl" + repeatTo(" -b a=1", n) }+ texts["curl cookies that never close"] = func(n int) string { return "curl -b a=1" + repeatTo(" -b 'x", n) }+ texts["curl lines with flags"] = func(n int) string { return repeatTo("curl -b a=1 -u a:b -b 'x\n", n) }+ // URLs whose scheme a Bearer token took, and replaced schemes with no password after them.+ texts["URLs after Bearer tokens"] = func(n int) string { return repeatTo("Bearer a://u:p@h ", n) }+ texts["replaced schemes without passwords"] = func(n int) string { return "[REDACTED]://u:" + repeatTo("[REDACTED]://u:", n) }+ // Labeled values ending in AWS key ids, with and without their secrets after them.+ texts["labeled key ids and secrets"] = func(n int) string {+ return repeatTo("token=AKIAIOSFODNN7EXAMPLE,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY ", n)+ }+ texts["labeled key ids without secrets"] = func(n int) string { return repeatTo("token=xAKIAIOSFODNN7EXAMPLE::|,", n) }+ // Key pairs after labeled key ids, each followed by a run a label could be spelled in.+ texts["labeled key pairs before label runs"] = func(n int) string {+ return repeatTo("token=AKIAIOSFODNN7EXAMPLE,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY-"+strings.Repeat("_", 200)+": ", n/2) ++ "token=AKIAIOSFODNN7EXAMPLE,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" + repeatTo("-", n/2)+ }+ // Copies that end alike after one byte each, X+a^9 and X+a^10 for every printable X, so two+ // states of the copy finder have as many children; a token of every shape, each a chain that+ // asks for the copies; and a long run of `a`, over which the finder would go between those two+ // states at every byte, with the tokens before it and after it. Their work per byte, that of+ // the whole redaction, is held to a ceiling too, not only its growth: about 90 now, where it+ // was over 3,000 when every chain read the whole text through child lists.+ wideCopies := func(tokensFirst bool) func(n int) string {+ return func(n int) string {+ var b, tokens strings.Builder+ for c := byte('!'); c <= '~'; c++ {+ if c == 'a' || strings.IndexByte("\"'`,\\", c) >= 0 {+ continue+ }+ fmt.Fprintf(&b, "token=%c%s\ntoken=%c%s\n", c, strings.Repeat("a", 9), c, strings.Repeat("a", 10))+ }+ for _, token := range adjacentSecrets {+ tokens.WriteString(token + " ")+ }+ run := repeatTo("a", n-b.Len()-tokens.Len())+ if tokensFirst {+ return b.String() + tokens.String() + run+ }+ return b.String() + run + tokens.String()+ }+ }+ texts["copies that end alike, tokens of every shape, a long run"] = wideCopies(true)+ texts["copies that end alike, a long run, tokens of every shape"] = wideCopies(false)+ perByte := map[string]int{+ "copies that end alike, tokens of every shape, a long run": 128,+ "copies that end alike, a long run, tokens of every shape": 128,+ }for name, text := range texts {work := func(n int) int {total := 0@@ -279,6 +351,10 @@ func TestRedactionWorkIsLinear(t *testing.T) {t.Errorf("%s: work %d on %d bytes, %d on half as many: not linear", name, w, n, previous)break}+ if ceiling := perByte[name]; ceiling > 0 && w > ceiling*n {+ t.Errorf("%s: work %d on %d bytes, %d a byte, more than %d", name, w, n, w/n, ceiling)+ break+ }previous = w}}diff --git a/pkg/asymptoteobserve/replacer.go b/pkg/asymptoteobserve/replacer.gonew file mode 100644index 00000000..a12c4716--- /dev/null+++ b/pkg/asymptoteobserve/replacer.go@@ -0,0 +1,276 @@+package asymptoteobserve++import "strings"++// This file replaces literal strings in a text in time linear in the text whatever the literals+// hold. It is what strings.NewReplacer(v1, place, v2, place, …).Replace(t) returns, which for long+// literals is not linear: for one literal Go builds a Boyer-Moore table in time quadratic in its+// length, and for several it reads up to the longest literal's length at each byte of the text.++// replaceLiteralsShort is the longest literal strings.Replacer is used for. Up to it, its search+// reads at most that many bytes at each byte of the text, and its one-literal table costs at most+// its square: linear in the text. Past it, one literal is found by replaceLiteral and several by+// literalFinder, each reading every byte of the text a bounded number of times.+const replaceLiteralsShort = 64++// replaceLiterals replaces the occurrences of values, a non-empty set sorted longest first, with+// place, as strings.NewReplacer(values[0], place, values[1], place, …).Replace(t) does: left to+// right, the longest where several start at one byte, none overlapping. It also returns the work+// done, an upper bound on the bytes and steps read, for callers that account for it.+func replaceLiterals(t string, values []string, place string) (string, int) {+ longest, total := len(values[0]), 0+ for _, v := range values {+ total += len(v)+ }+ switch {+ case longest > replaceLiteralsShort && len(values) == 1:+ return replaceLiteral(t, values[0], place)+ case longest > replaceLiteralsShort:+ return newLiteralFinder(values).replace(t, place)+ }+ pairs := make([]string, 0, 2*len(values))+ for _, v := range values {+ pairs = append(pairs, v, place)+ }+ return strings.NewReplacer(pairs...).Replace(t), (len(t) + total) * longest+}++// replaceLiteral replaces the occurrences of v in t with place, left to right and none+// overlapping, and returns the work done. It is Knuth-Morris-Pratt's search: where no part of v+// is matched it skips to the next byte v starts with, and otherwise each byte read either extends+// the part matched or shortens it.+func replaceLiteral(t, v, place string) (string, int) {+ // border[j] is the length of the longest proper prefix of v[:j+1] that is also its suffix.+ border := make([]int32, len(v))+ for j, k := 1, 0; j < len(v); j++ {+ for k > 0 && v[j] != v[k] {+ k = int(border[k-1])+ }+ if v[j] == v[k] {+ k+++ }+ border[j] = int32(k)+ }+ var b strings.Builder+ last, k, steps := 0, 0, 0+ for i := 0; i < len(t); i++ {+ if k == 0 {+ j := strings.IndexByte(t[i:], v[0])+ if j < 0 {+ break+ }+ i += j+ }+ for k > 0 && t[i] != v[k] {+ k = int(border[k-1])+ steps+++ }+ if t[i] == v[k] {+ k+++ }+ if k == len(v) {+ if last == 0 {+ b.Grow(len(t))+ }+ b.WriteString(t[last : i+1-len(v)])+ b.WriteString(place)+ last, k = i+1, 0+ }+ }+ work := 2*len(v) + len(t) + steps+ if last == 0 {+ return t, work+ }+ b.WriteString(t[last:])+ return b.String(), work+}++// literalFinder is an Aho-Corasick automaton over literals written backwards. Reading a text from+// its end, the state it is in after reading the byte at i tells the longest literal that starts+// at i, so one pass finds, for every byte, the occurrence strings.Replacer would take there.+//+// A state's children are a list, the root's a table: literals are few and most states have one+// child, while a table per state would cost a row per byte of every literal. A state with more+// than literalListMax children also has them in a map, so following an edge reads at most+// literalListMax list entries or one map entry, however many children a state has. A step+// follows at most one edge forward and, on the way, failure links that each shorten the state, so+// a pass costs at most twice the text's length in edge lookups.+//+// A table per wide state would make each lookup one load too, but it costs 1 KiB however few of+// its 256 entries are used, and literals can be as long as the text: a set of literals made to+// branch widely at every depth would cost that much per few bytes of them. The map costs a few+// words per child.+type literalFinder struct {+ root [256]int32 // the root's child by byte; 0 for none+ label []byte // per state, the byte of the edge into it+ child []int32 // per state, its first child; 0 for none+ sibling []int32 // per state, its parent's next child; 0 for none+ fanout []int32 // per state, its number of children+ wide map[uint64]int32+ fail []int32 // per state, the longest proper suffix of its string that is a state+ longest []int32 // per state, the length of the longest literal its string ends with+ maxLen int // the longest literal's length+ work int // list entries and map entries read, and links followed+}++// literalListMax is the most children a state's list is searched for an edge; a state with more+// has them in literalFinder.wide.+const literalListMax = 8++// wideEdge is the key of state s's edge by c in literalFinder.wide.+func wideEdge(s int32, c byte) uint64 {+ return uint64(s)<<8 | uint64(c)+}++func newLiteralFinder(values []string) *literalFinder {+ f := &literalFinder{label: []byte{0}, child: []int32{0}, sibling: []int32{0}, fanout: []int32{0}, longest: []int32{0}}+ for _, v := range values {+ s := int32(0)+ for i := len(v) - 1; i >= 0; i-- {+ c := v[i]+ next := f.next(s, c)+ if next <= 0 {+ next = f.add(s, c)+ }+ s = next+ }+ f.longest[s] = int32(len(v))+ f.maxLen = max(f.maxLen, len(v))+ }+ // Failure links, breadth first, so a state's link and its link's longest are known before+ // its children's.+ f.fail = make([]int32, len(f.label))+ queue := make([]int32, 0, len(f.label))+ for _, s := range f.root {+ if s != 0 {+ queue = append(queue, s)+ }+ }+ for len(queue) > 0 {+ s := queue[0]+ queue = queue[1:]+ for x := f.child[s]; x != 0; x = f.sibling[x] {+ f.fail[x] = f.step(f.fail[s], f.label[x])+ if f.longest[x] == 0 {+ f.longest[x] = f.longest[f.fail[x]]+ }+ queue = append(queue, x)+ }+ }+ return f+}++// add makes a child of s by c and returns it. The state that gets its literalListMax+1st child+// has all of them put in the map.+func (f *literalFinder) add(s int32, c byte) int32 {+ x := int32(len(f.label))+ f.label = append(f.label, c)+ f.child = append(f.child, 0)+ f.fanout = append(f.fanout, 0)+ f.longest = append(f.longest, 0)+ if s == 0 {+ f.sibling = append(f.sibling, 0)+ f.root[c] = x+ return x+ }+ f.sibling = append(f.sibling, f.child[s])+ f.child[s] = x+ f.fanout[s]+++ switch {+ case f.fanout[s] == literalListMax+1:+ if f.wide == nil {+ f.wide = map[uint64]int32{}+ }+ for y := f.child[s]; y != 0; y = f.sibling[y] {+ f.wide[wideEdge(s, f.label[y])] = y+ }+ case f.fanout[s] > literalListMax+1:+ f.wide[wideEdge(s, c)] = x+ }+ return x+}++// next returns s's child by c: 0 from the root, and -1 from another state, when it has none.+func (f *literalFinder) next(s int32, c byte) int32 {+ if s == 0 {+ return f.root[c]+ }+ if f.fanout[s] > literalListMax {+ f.work+++ if x, ok := f.wide[wideEdge(s, c)]; ok {+ return x+ }+ return -1+ }+ for x := f.child[s]; x != 0; x = f.sibling[x] {+ f.work+++ if f.label[x] == c {+ return x+ }+ }+ return -1+}++// step returns the state after reading c in state s.+func (f *literalFinder) step(s int32, c byte) int32 {+ for {+ if next := f.next(s, c); next >= 0 {+ return next+ }+ f.work+++ s = f.fail[s]+ }+}++// longestAt returns, for every byte of t, the length of the longest literal that starts there (0+// for none), and the work done: one pass over t from its end.+func (f *literalFinder) longestAt(t string) ([]int32, int) {+ at := make([]int32, len(t))+ return at, f.longestIn(t, at, 0, len(t))+}++// longestIn sets at[i-lo], for every i in [lo, hi), to the length of the longest literal that+// starts at i in t, and returns the work done. The state the pass from the end is in at a byte+// depends only on the bytes up to the longest literal's length after it, so the pass starts that+// far past hi, or at the end: it reads at most that many bytes more than it sets.+func (f *literalFinder) longestIn(t string, at []int32, lo, hi int) int {+ before := f.work+ from := min(len(t), hi+f.maxLen)+ s := int32(0)+ for i := from - 1; i >= lo; i-- {+ s = f.step(s, t[i])+ if i < hi {+ at[i-lo] = f.longest[s]+ }+ }+ return from - lo + f.work - before+}++// replace replaces the literals in t with place, as replaceLiterals does, and returns the work+// done.+func (f *literalFinder) replace(t string, place string) (string, int) {+ at, _ := f.longestAt(t)+ // The states built, the text, and the edges and links followed building and reading.+ work := len(f.label) + len(t) + f.work+ var b strings.Builder+ last := 0+ for i := 0; i < len(t); {+ n := int(at[i])+ if n == 0 {+ i+++ continue+ }+ if last == 0 {+ b.Grow(len(t))+ }+ b.WriteString(t[last:i])+ b.WriteString(place)+ i += n+ last = i+ }+ if last == 0 {+ return t, work+ }+ b.WriteString(t[last:])+ return b.String(), work+}diff --git a/pkg/asymptoteobserve/replacer_test.go b/pkg/asymptoteobserve/replacer_test.gonew file mode 100644index 00000000..b13c82b7--- /dev/null+++ b/pkg/asymptoteobserve/replacer_test.go@@ -0,0 +1,74 @@+package asymptoteobserve++import (+ "maps"+ "math/rand/v2"+ "slices"+ "strings"+ "testing"+)++// The literal searches replace what strings.Replacer does, over literals that overlap, contain+// each other, start or end alike, and texts made of their pieces: short literals, the same+// literals stretched past replaceLiteralsShort so replaceLiterals takes its own searches too, and+// many literals over a wider alphabet, whose states have more children than a list is searched for+// (literalListMax).+func TestLiteralSearchesAgreeWithReplacer(t *testing.T) {+ r := rand.New(rand.NewPCG(3, 9))+ wide := 0+ for _, c := range []struct {+ alphabet string+ scale, values int+ }{{"ab", 1, 5}, {"ab", 10, 5}, {"abcdefghijklmnopqrst", 1, 60}, {"abcdefghijklmnopqrst", 10, 60}} {+ piece := func(n int) string {+ b := make([]byte, 0, n*c.scale)+ for range n {+ b = append(b, strings.Repeat(string(c.alphabet[r.IntN(len(c.alphabet))]), c.scale)...)+ }+ return string(b)+ }+ for range 10000 {+ set := map[string]struct{}{}+ for range 1 + r.IntN(c.values) {+ set[piece(1+r.IntN(4))+"ab"[:r.IntN(3)]] = struct{}{}+ }+ values := slices.SortedFunc(maps.Keys(set), func(a, b string) int {+ if len(a) != len(b) {+ return len(b) - len(a)+ }+ return strings.Compare(a, b)+ })+ var text strings.Builder+ for range r.IntN(12) {+ if r.IntN(3) == 0 {+ text.WriteString(piece(1 + r.IntN(4)))+ } else {+ text.WriteString(values[r.IntN(len(values))])+ }+ }+ pairs := make([]string, 0, 2*len(values))+ for _, v := range values {+ pairs = append(pairs, v, "#")+ }+ want := strings.NewReplacer(pairs...).Replace(text.String())+ f := newLiteralFinder(values)+ if f.wide != nil {+ wide+++ }+ if got, _ := f.replace(text.String(), "#"); got != want {+ t.Fatalf("literals %q in %q: got %q, want %q", values, text.String(), got, want)+ }+ if got, _ := replaceLiterals(text.String(), values, "#"); got != want {+ t.Fatalf("replaceLiterals(%q, %q): got %q, want %q", text.String(), values, got, want)+ }+ if len(values) == 1 {+ if got, _ := replaceLiteral(text.String(), values[0], "#"); got != want {+ t.Fatalf("literal %q in %q: got %q, want %q", values[0], text.String(), got, want)+ }+ }+ }+ }+ if wide < 1000 {+ t.Fatalf("only %d literal sets had a state with more than %d children", wide, literalListMax)+ }+}