Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier3): mask judge and finding evidence whose quotes are escaped again (#577)
Narendran Raghavan3a70bb2d0027
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7f911a4b..d4a60f36 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -454,6 +454,62 @@ All notable changes to SkillEvaluator are documented in this file.A RAGAS `replied` entry for a reply with no text and no tool call, such as arefusal or a content-filter stop, now saves the whole reply body instead of anempty reply.+- Judge evidence (`judge_attempts.json`) and security finding evidence now mask+ credentials whose quotes are escaped twice or three times, as when a RAGAS+ request repeats a tool call's JSON arguments as JSON, and single-quoted values+ whose quotes are escaped once or twice, as in Python source, JSON or not.+ - Newly masked: an argv element after a credential flag, `-u`, `-b`, or a+ `login` or `sshpass` `-p`+ (`[\\\"tool\\\", \\\"--password\\\", \\\"...\\\"]`, or+ `[\'tool\', \'--password\', \'...\']` in a quoted Python list), which was+ saved in clear. An attached `-p...` after a `login` or MySQL client element,+ which was saved in clear when escaped twice or in a JSON-escaped quoted+ Python list (a quoted Python list that is not JSON-escaped already masked+ it). A double- or single-quoted value after a credential flag, `-u`, `-p`,+ `.netrc` `password`, `aws configure set`, `git config`, `npm config set` or+ `curl -b` (`--password \\\"two words\\\"`, `--password \'two words\'`,+ `curl -u \'user:...\'`) and a JSON member such as+ `\\\"password\\\": \\\"...\\\"`, which were saved in clear or with only+ their first word masked.+ - How: the text is also read with one, then two levels of string escapes+ removed (`\\`, `\"`, `\'`, `\/`, `\n`, `\r`, `\t`, `\uXXXX`, inside a quoted+ string or not). What the same patterns mask in a reading is masked in the+ saved text too, where the text kept around it places it for certain, so a+ value escaped again is masked as it is when escaped once, also where the+ patterns mask more than a credential (`"total_tokens": self.total_tokens,`+ in a Write call a RAGAS request repeats). Markers are only added: the text+ as written is masked exactly as before. A value in a reading ends at a quote+ that ends a string of the text it was read from, and `\\"` ends a string, as+ in JSON, so JSON inside a Python `repr`, where `\\"` is an escaped quote, is+ masked as before.+ - Still kept: values in text escaped more than three times; single-quoted+ values whose quotes are escaped three times; values in text that holds a NUL+ character, which is not read again (an escaped `\u0000` is read as written);+ the part of a value that once-escaped text keeps, such as the words after+ the first in `"password: \"two words\""` at the start of a JSON string; and+ the words after an escaped quote in a quoted Python argv value+ (`[\'--token\', \'a\\\'b two\']`).+ - Known over-masks: `{"command": "mysql -u root -p", "timeout": 30}` is saved+ as `-p<redacted>timeout": 30}`, as before, and now also when escaped twice+ or three times. The judge's line pass can still mask the JSON after a value+ on a piece that starts inside a string+ (`{"command": "run(['C:\\\\tmp\\\\']); password: ...", "note": "keep me"}`+ loses `"note"`). And as escapes are decoded outside strings too, in text+ that was never escaped, an escape that leaves a quote unmatched in a reading+ can mask the rest of its line after a later credential assignment+ (`export GITHUB_TOKEN=$(...)`, `password: ...`): a `\n`, `\r` or `\u000a`+ inside a quoted string (`printf '%s\n'`, `tr -d '\n'`, `echo "a\nb"`,+ `cd "C:\repo\x"`), or an odd number of `\"` in a double-quoted string+ (`echo "it\"s"`). Finding evidence can then lose the `curl` or `rm` the+ finding is about. Judge evidence already lost the rest of+ such a line after a `\n`, `\r` or `\t` escape, and now also loses it after+ an odd `\"` or a `\u000a`.+ - Cost: text with a backslash is masked up to three times, in time and memory+ linear in its length. Judge evidence takes two to four times as long to mask+ (about 0.2 s instead of 0.06 s for a 70 KB RAGAS prompt of Bash calls), and+ up to about eight times as long on adversarial text. Peak memory can grow by+ about 250 MB per MB of input for one long quoted value escaped twice, as the+ patterns then read that value decoded.- Tier 3 tasks with Compose sidecars no longer fail with `Docker Composeinterpolation variable ... requires protected execution state` because theshell exports a short flag with a sensitive-looking name. A host variable suchdiff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdxindex 2d08a7fa..f02402d3 100644--- a/docs/tier3-live-evaluation.mdx+++ b/docs/tier3-live-evaluation.mdx@@ -1472,10 +1472,13 @@ the case is judged and scored normally.two characters `\n`, and inside JSON-escaped quotes (`-u \"user:password\"`),as in tool-call arguments. That includes a value whose flag or header namewas masked together with an unquoted credential value that ends the line- before (`--password ...\n--token ...`). Long prompts and replies are- shortened. The prompts contain the task question, expected answer, and agent- evidence. Pass `--harbor-keep-jobs` to keep the Harbor trial directories that- hold this file.+ before (`--password ...\n--token ...`). Values whose quotes are escaped twice+ or three times, as when a RAGAS prompt repeats a tool call's arguments, and+ single-quoted values whose quotes are escaped once or twice, as in Python+ source (`[\'--password\', \'...\']`), are masked too. Long prompts and+ replies are shortened. The prompts contain the task question, expected+ answer, and agent evidence. Pass `--harbor-keep-jobs` to keep the Harbor+ trial directories that hold this file.- **Reasoning models.** GPT-5 and GPT-6 judges get no `temperature` and atleast 32,768 output tokens, because reasoning tokens count against thelimit; other judges get 16,384. GPT-6 judges and Claude 5.5 or later judges@@ -2071,12 +2074,25 @@ query parameters named only `key`, `sig`, or `signature`, a `redis-cli -a` orpassword after an unknown flag, an unquoted `PASS: value`, an `npm config set`option with a separate value (`--location global`), or all but the first wordof a value with spaces on an unquoted here-document line or inside a `sh -c`-or `ssh` command string. Some ordinary words are masked, such as the-path in `grep -r token: /etc`, the words after a `.netrc`-like-`login NAME password` phrase in prose, or, after an unmatched quote, the rest-of the line after a credential name. Quotes are read from the start of each-line, and unlike in a shell, a line break ends a quoted string here, so a-quoted value ends at its closing quote or at the end of its line.+or `ssh` command string. Escaped text is read with at most two levels of+escapes removed, so a value can also appear in text escaped more than three+times, in a single-quoted value whose quotes are escaped three times, in text+that holds a NUL character (which is not read again), in JSON inside a Python+`repr` (where `\\"` is an escaped quote), after an escaped quote inside a quoted+Python argv value, and where text escaped once keeps part of a value+(`"password: \"two words\""` at the start of a JSON string). Some ordinary+words are masked, such as the path in `grep -r token: /etc`, the words after a+`.netrc`-like `login NAME password` phrase in prose, or, after an unmatched+quote, the rest of the line after a credential name. As escapes are also+decoded in text that was never escaped, an escape that leaves a quote+unmatched there can mask the rest of its line after a later credential+assignment such as `export GITHUB_TOKEN=$(...)`: a `\n`, `\r` or `\u000a`+inside a single- or double-quoted string (`printf '%s\n'`, `echo "a\nb"`,+`cd "C:\repo\x"`), or an odd number of `\"` in a double-quoted string+(`echo "it\"s"`). That can hide the rest of a command that a destructive or+network finding is about. Quotes are read from the start of each line, and+unlike in a shell, a line break ends a quoted string here, so a quoted value+ends at its closing quote or at the end of its line.For security findings, only the first 2,000 characters are scanned. Judgeevidence retains the separate judge evidence budgets described above.A word that the security-finding scan limit cuts indiff --git a/src/skillevaluator/tier3/eval_core/checks.py b/src/skillevaluator/tier3/eval_core/checks.pyindex fe0695f8..95e778b9 100644--- a/src/skillevaluator/tier3/eval_core/checks.py+++ b/src/skillevaluator/tier3/eval_core/checks.py@@ -20,6 +20,7 @@ import jsonimport posixpathimport reimport shlex+from array import arrayfrom collections.abc import Callable, Mappingfrom fnmatch import fnmatchcasefrom typing import Any@@ -3263,7 +3264,9 @@ def _security_finding(# assigned to a credential name (_authToken=..., api_key: ..., DB_PASS=...);# and common token prefixes. Only the first _FINDING_EVIDENCE_SCAN_CHARS# characters are scanned, and each pass reads that text in linear time. A-# quoted value ends at its closing quote or at the end of its line.+# quoted value ends at its closing quote or at the end of its line. The same+# forms are masked in text whose quotes are escaped up to three times, as tool+# calls repeated in a RAGAS prompt are (see _mask_unescaped_views)._FINDING_EVIDENCE_SCAN_CHARS = 2000_FINDING_EVIDENCE_TRUNCATED = "<truncated>"_FINDING_EVIDENCE_PRIVATE_KEY_MARKER = "<redacted private key>"@@ -3637,6 +3640,41 @@ _FINDING_EVIDENCE_TOKEN_PREFIX_RE = re.compile(r"(?<![A-Za-z0-9_-])(gh[pousr]_|github_pat_|glpat-|npm_|xox[abeoprs]-|hf_)([A-Za-z0-9_-]{8,})")_FINDING_EVIDENCE_IDENTIFIER_BODY_RE = re.compile(r"[a-z]+(?:_[a-z]+)+")+# Tool-call arguments are JSON, and a RAGAS request writes them into its prompt+# as JSON again, so a command can reach evidence with each quote escaped twice+# ([\\\"--password\\\", \\\"VALUE\\\"], --password \\\"VALUE MORE\\\") or three+# times. The patterns above read a quote with at most one backslash before it,+# so they kept such a value, or masked only its first word. They did the same+# with a single-quoted value whose quotes are escaped once, as in Python source+# (\'--password\', \'VALUE\'; --password \'VALUE MORE\'), JSON or not.+# So the text is also read with one level of string escapes removed, and then+# with two: \\, \", \', \/, \n, \r, \t and \uXXXX are decoded wherever they+# appear, inside a quoted string or not, and any other backslash is read as+# written, as is \u0000, since the Harbor verifier's masker drops NUL+# characters. A double quote after an even run of backslashes (none, or an+# escaped backslash: "C:\\tmp\\") ends a string of the text it is read from,+# so that run is kept as written. Decoded, the quote would read as escaped in+# the first reading, and a value that the second reading masks there would not+# end at that string's end: a JSON string that ends in a shell line+# continuation, "tool --token \\", would lose its closing quote. Each reading+# is masked with the same patterns, and what they mask there is masked in the+# text as written too, where that still reads in clear (see+# _mask_unescaped_views). Nothing the text as written masks is kept, so a+# reading only adds markers.+_BACKSLASH_RUN_RE = re.compile(r"(\\+)(u[0-9A-Fa-f]{4}|[\s\S])?")+# A double quote after an even run of backslashes, which ends (or opens) a+# string of the text it is in.+_STRING_END_RE = re.compile(r'(?<!\\)(?:\\\\)*"')+_UNESCAPED_CHARS = {"n": "\n", "r": "\r", "t": "\t", '"': '"', "'": "'", "/": "/"}+_UNESCAPE_LEVELS = 2+# A masker only replaces values with markers, so the text it keeps between two+# markers is found in the text it read, in order, and what lies between was+# masked. The JWT pattern writes "jwt-" before its marker, the Harbor verifier+# writes [REDACTED] for a credential its process holds, and markers next to+# each other are read as one.+_MASKED_MARKERS_RE = re.compile(+ r"((?:(?:jwt-)?<redacted>|" + re.escape(_FINDING_EVIDENCE_PRIVATE_KEY_MARKER) + r"|\[REDACTED\])+)"+)def _redact_finding_evidence(text: Any, limit: int = 500) -> str:@@ -3660,6 +3698,210 @@ def _redact_finding_evidence(text: Any, limit: int = 500) -> str:def _mask_finding_evidence_credentials(text: str, *, preserve_header_format: bool = False) -> str:+ """Mask supported credential forms, also in text escaped again, without imposing a caller's evidence budget."""++ def mask(view: str) -> str:+ return _mask_finding_evidence_credentials_as_written(view, preserve_header_format=preserve_header_format)++ return _mask_unescaped_views(text, mask(text), mask)+++def _mask_unescaped_views(text: str, masked: str, mask: Callable[[str], str]) -> str:+ """Return *masked*, which *mask* wrote for *text*, also masking what *mask* masks in *text* unescaped.++ *text* is read with one level of escapes removed, then with two. A span is+ masked only where the text that *masked* keeps around it places it for+ certain, and only where *masked* still keeps it: a marker is never taken+ back, and a span next to a marker joins it.+ """+ # The Harbor verifier's masker drops NUL characters, so the text it keeps+ # would no longer be found in the text it read. A reading keeps \u0000 as+ # written, so it holds a NUL only where the text does.+ if "\\" not in text or "\x00" in text:+ return masked+ spans: list[tuple[int, int]] = []+ view = text+ view_starts: list[array[int]] = []+ for _ in range(_UNESCAPE_LEVELS):+ source = view+ view, starts = _unescaped(source)+ if starts is None:+ break+ string_ends: list[int] | None = None+ for start, end in _masked_spans(view, mask(view)):+ start, end = starts[start], starts[end]+ # The text the reading decodes lies inside strings of the text it+ # was read from, so a value of the reading ends at a quote that+ # ends such a string, and one that starts right after that quote+ # was read from the text around the strings.+ if string_ends is None:+ string_ends = [match.end() - 1 for match in _STRING_END_RE.finditer(source)]+ index = bisect.bisect_left(string_ends, start - 1)+ if index < len(string_ends) and string_ends[index] == start - 1:+ continue+ if index < len(string_ends) and string_ends[index] < end:+ end = string_ends[index]+ if start == end:+ continue+ # Back through each earlier reading to the text as written.+ for level_starts in reversed(view_starts):+ start, end = level_starts[start], level_starts[end]+ spans.append((start, end))+ view_starts.append(starts)+ if "\\" not in view:+ break+ return _mask_spans(text, masked, spans) if spans else masked+++def _unescaped(text: str) -> tuple[str, array[int] | None]:+ """Return *text* with one level of string escapes decoded, and where each of its characters starts in *text*.++ The starts end with len(text), so the span [start, end) of the result is+ [starts[start], starts[end]) in *text*. They are None when no escape is+ decoded.+ """+ parts: list[str] = []+ starts = array("q")+ pos = 0+ for match in _BACKSLASH_RUN_RE.finditer(text):+ run, after = match.group(1), match.group(2) or ""+ escapes = len(run) % 2 == 1 and (after in _UNESCAPED_CHARS or (len(after) == 5 and after != "u0000"))+ if (len(run) == 1 and not escapes) or (after == '"' and len(run) % 2 == 0):+ continue+ parts.append(text[pos : match.start()])+ starts.extend(range(pos, match.start()))+ # Each escaped backslash is one backslash, and an odd one escapes the+ # character after it.+ parts.append("\\" * (len(run) // 2))+ starts.extend(range(match.start(), match.start() + len(run) - 1, 2))+ pos = match.start() + len(run) // 2 * 2+ if escapes:+ parts.append(chr(int(after[1:], 16)) if len(after) == 5 else _UNESCAPED_CHARS[after])+ starts.append(pos)+ pos = match.end()+ if not parts:+ return text, None+ parts.append(text[pos:])+ starts.extend(range(pos, len(text) + 1))+ return "".join(parts), starts+++def _masker_reading(text: str) -> tuple[str, array[int] | None]:+ """Return *text* as the masker reads it, without shell line continuations, and where each of its characters is.++ The positions are None when *text* holds no line continuation.+ """+ if _LINE_CONTINUATION_RE.search(text) is None:+ return text, None+ starts = array("q")+ pos = 0+ for match in _LINE_CONTINUATION_RE.finditer(text):+ starts.extend(range(pos, match.start()))+ pos = match.end()+ starts.extend(range(pos, len(text)))+ return _LINE_CONTINUATION_RE.sub("", text), starts+++def _masked_spans(text: str, masked: str) -> list[tuple[int, int]]:+ """Return the spans of *text* that *masked*, a masker's output for it, masks for certain."""+ read, read_starts = _masker_reading(text)+ kept = _MASKED_MARKERS_RE.split(masked)[::2]+ starts = _place_kept_text(read, kept)+ spans: list[tuple[int, int]] = []+ for index in range(len(kept) - 1 if starts else 0):+ start, end = starts[index], starts[index + 1]+ if start < 0 or end < 0 or start + len(kept[index]) >= end:+ continue+ start += len(kept[index])+ spans.append((start, end) if read_starts is None else (read_starts[start], read_starts[end - 1] + 1))+ return spans+++def _place_kept_text(read: str, kept: list[str]) -> list[int] | None:+ """Return where each piece of *kept*, the text a masker kept between its markers, starts in *read*, the text it read.++ The pieces are placed in order, once as early and once as late as they+ fit. A piece placed at the same index both times is there in every+ placement; any other is -1. A first piece that starts *read* and a last+ piece that ends it are placed there, as a masker at most strips whitespace+ from the ends of its result. None: a piece is missing.+ """+ count = len(kept)+ early: list[int] = []+ pos = 0+ for index, piece in enumerate(kept):+ found = read.find(piece, pos)+ if index == count - 1 and read.endswith(piece):+ found = len(read) - len(piece) if len(read) - len(piece) >= pos else -1+ if found < 0:+ return None+ early.append(found)+ pos = found + len(piece)+ # The latest places are found from the end, in the reversed text, so that+ # each search reads it once.+ backwards = read[::-1]+ late = [0] * count+ pos = 0+ for index in range(count - 1, -1, -1):+ found = backwards.find(kept[index][::-1], pos)+ late[index] = len(read) - found - len(kept[index])+ pos = found + len(kept[index])+ if read.startswith(kept[0]):+ late[0] = 0+ return [start if start == late[index] else -1 for index, start in enumerate(early)]+++def _mask_spans(text: str, masked: str, spans: list[tuple[int, int]]) -> str:+ """Return *masked*, a masker's output for *text*, with the parts of *spans* (of *text*) that it keeps masked."""+ read, read_starts = _masker_reading(text)+ if read_starts is not None:+ spans = [(bisect.bisect_left(read_starts, start), bisect.bisect_left(read_starts, end)) for start, end in spans]+ merged: list[list[int]] = []+ for start, end in sorted(span for span in spans if span[0] < span[1]):+ if merged and start <= merged[-1][1]:+ merged[-1][1] = max(merged[-1][1], end)+ else:+ merged.append([start, end])+ pieces = _MASKED_MARKERS_RE.split(masked)+ starts = _place_kept_text(read, pieces[::2]) if merged else None+ if starts is None:+ return masked+ parts: list[str] = []+ # Whether the result so far ends with a marker, and whether that marker+ # is one this function added. A marker that follows another one joins it.+ marked = added = False+ first_span = 0+ for index, piece in enumerate(pieces):+ if index % 2:+ if added:+ parts.pop()+ if not marked or added:+ parts.append(piece)+ marked, added = True, False+ continue+ start = starts[index // 2]+ pos = 0+ if start >= 0:+ while first_span < len(merged) and merged[first_span][1] <= start:+ first_span += 1+ probe = first_span+ while probe < len(merged) and merged[probe][0] < start + len(piece):+ cut_start, cut_end = max(merged[probe][0] - start, 0), min(merged[probe][1] - start, len(piece))+ probe += 1+ if cut_start > pos:+ parts.append(piece[pos:cut_start])+ marked = added = False+ if not marked:+ parts.append("<redacted>")+ marked = added = True+ pos = cut_end+ if pos < len(piece):+ parts.append(piece[pos:])+ marked = added = False+ return "".join(parts)+++def _mask_finding_evidence_credentials_as_written(text: str, *, preserve_header_format: bool = False) -> str:"""Mask supported credential forms without imposing a caller's evidence budget."""# Remove shell continuations before matching values: they can join an# attached password to its flag. Ordinary line breaks stay intact.diff --git a/src/skillevaluator/tier3/harbor/templates/eval.py b/src/skillevaluator/tier3/harbor/templates/eval.pyindex 714ff751..f414d80f 100644--- a/src/skillevaluator/tier3/harbor/templates/eval.py+++ b/src/skillevaluator/tier3/harbor/templates/eval.py@@ -2888,7 +2888,7 @@ def _mask_judge_evidence_text(text):"""Mask credentials in *text* read as one piece."""# Share finding credential semantics without imposing the finding scanner's# 2,000-character input budget on the longer judge prompt and reply budgets.- redacted = _shared_checks._mask_finding_evidence_credentials(text, preserve_header_format=True)+ redacted = _shared_checks._mask_finding_evidence_credentials_as_written(text, preserve_header_format=True)redacted = _redact_configured_credentials(_redact_evidence_text(redacted))for pattern in _SECRET_PATTERNS:redacted = pattern.sub("<redacted>", redacted)@@ -2897,7 +2897,7 @@ def _mask_judge_evidence_text(text):def _mask_judge_evidence_line(text):"""Mask credentials in the text between two escapes, which is not trimmed."""- redacted = _shared_checks._mask_finding_evidence_credentials(text, preserve_header_format=True)+ redacted = _shared_checks._mask_finding_evidence_credentials_as_written(text, preserve_header_format=True)redacted = redact_secrets_in_log_line(redacted)for pattern in _SECRET_PATTERNS:redacted = pattern.sub("<redacted>", redacted)@@ -3016,6 +3016,19 @@ def _mask_judge_evidence_value_sites(text, sites):def _redact_judge_evidence(text):"""Mask credentials in a judge prompt, reply, or problem before it is saved."""text = str(text)+ # A RAGAS prompt or reply holds tool-call arguments escaped twice, so the+ # passes below also read the text with one and two levels of escapes+ # removed (see _mask_unescaped_views in checks.py). That reading only adds+ # markers to their result, so each pass reads the text as it did before:+ # masking a value earlier could move a later one out of a pattern's window+ # or hide the name that a later pass reads.+ return _shared_checks._mask_unescaped_views(+ text, _redact_judge_evidence_as_written(text), _redact_judge_evidence_as_written+ )+++def _redact_judge_evidence_as_written(text):+ """Mask credentials in a judge prompt, reply, or problem as written."""# The whole text is masked first, so a form that runs over an escape (a# private key block, a continued or quoted value, a value that holds "\n"# or "\u00e4") is masked whole before the text is split at the escapes.diff --git a/tests/tier3/test_judge_contracts.py b/tests/tier3/test_judge_contracts.pyindex cbb4436a..b82b8b0b 100644--- a/tests/tier3/test_judge_contracts.py+++ b/tests/tier3/test_judge_contracts.py@@ -8,12 +8,21 @@ from __future__ import annotationsimport jsonimport randomimport string+import timefrom pathlib import Pathfrom types import ModuleTypefrom typing import Anyimport pytestfrom tests.conftest import load_harbor_eval_template+from tests.tier3.test_security_finding_evidence import (+ _CONFIGURED_CREDENTIAL,+ _JWT,+ _KEY_BLOCK,+ _READINGS_THAT_START_THE_TEXT,+ _generated_texts,+ _only_adds_markers,+)from skillevaluator.tier3.eval_core import atif_helpers, llm_judgefrom skillevaluator.tier3.harbor.metrics import RESERVED_METRIC_NAMES@@ -1412,6 +1421,27 @@ def test_judge_evidence_keeps_copies_of_line_values_elsewhere(text: str, kept: l# The same with the quotes escaped twice, after a plain MySQL client element.'run(["mysql", "-e", "x"]); '+ json.dumps(json.dumps('run(["mkdir", "-p", "data"]); run(["mysql", "-u", "root", "-p", "appdb"])')),+ # Ordinary argv flags with the quotes escaped twice, also one element+ # per line, and in a quoted Python string, plain and JSON-escaped.+ json.dumps(json.dumps('["date", "-u", "+%H:%M"]; ["pip", "install", "-U", "git+https://github.com/org/repo"]')),+ json.dumps(json.dumps('["rsync", "-u", "host:/data", "."]; ["ssh", "-p", "2222", "host"]')),+ json.dumps(json.dumps('["find", ".", "-path", "x"]; ["tool", "--token", "--verbose"]')),+ json.dumps(json.dumps('["tool", "--password-file", "pw.txt"]; ["curl", "-b", "jar.txt"]')),+ json.dumps(json.dumps('subprocess.run([\n "date",\n "-u",\n "+%H:%M",\n])')),+ "code = '[\\'date\\', \\'-u\\', \\'+%H:%M\\'] + [\\'mkdir\\', \\'-p\\', \\'out\\']'",+ json.dumps("code = '[\\'date\\', \\'-u\\', \\'+%H:%M\\'] + [\\'ssh\\', \\'-p\\', \\'2222\\']'"),+ # Quoted flag values escaped twice that hold no credential: a URL, a+ # remote path and a date format after -u, -u values with no colon, and+ # a flag that names no credential.+ json.dumps(+ json.dumps(+ 'pip install -U "git+https://github.com/org/repo" && rsync -u "host:/data" . && date -u "+%H:%M"'+ )+ ),+ json.dumps(json.dumps('curl -u "bot" https://x && git push -u "origin" main && judge --token-limit "5 10"')),+ # An escaped backslash before a closing quote, in JSON escaped once and twice.+ json.dumps({"path": "C:\\tmp\\", "cmd": "tool --password-file C:\\tmp\\pw.txt", "note": "keep me"}),+ json.dumps(json.dumps({"path": "C:\\tmp\\", "cmd": 'echo "C:\\new\\" && ls', "note": "keep me"})),],)def test_judge_evidence_keeps_ordinary_text(text: str) -> None:@@ -1459,6 +1489,300 @@ def test_judge_evidence_masks_twice_escaped_attached_password(context: str) -> Nassert eval_template._redact_judge_evidence(text) == text.replace(_QUOTED_SECRET, "<redacted>")+def _escaped_twice(text: str) -> str:+ return json.dumps(json.dumps({"command": text}))+++def _in_quoted_python_string(text: str) -> str:+ """Write *text*, a list with double-quoted elements, as a single-quoted list inside a single-quoted string."""+ source = text.replace('"', "'")+ return "code = '" + source.replace("\\", "\\\\").replace("'", "\\'") + "'"+++# A RAGAS request that repeats a tool call writes the call's JSON arguments as+# JSON again, so an argv list there has three backslashes before each quote+# ([\\\"tool\\\", \\\"--password\\\", \\\"VALUE\\\"]). A list inside a quoted+# Python string has one before each quote ([\'tool\', \'--password\',+# \'VALUE\']), two once that string is JSON. Each list is masked as in plain+# text, with its other elements and escapes intact, also when the value holds+# an escaped quote or ends in a backslash.+ ("argv", "masked"),+ [+ pytest.param(+ f'["tool", "--password", "{_QUOTED_SECRET}"]', '["tool", "--password", <redacted>]', id="password"+ ),+ pytest.param(f'["curl", "-u", "admin:{_QUOTED_SECRET}", url]', '["curl", "-u", <redacted>, url]', id="user"),+ pytest.param(+ f'["docker", "login", "-u", "bot", "-p", "{_QUOTED_SECRET}"]',+ '["docker", "login", "-u", "bot", "-p", <redacted>]',+ id="login",+ ),+ pytest.param(+ f'["sshpass", "-p", "{_QUOTED_SECRET}", "ssh", "host"]',+ '["sshpass", "-p", <redacted>, "ssh", "host"]',+ id="sshpass",+ ),+ pytest.param(f'["curl", "-b", "sid={_QUOTED_SECRET}", url]', '["curl", "-b", <redacted>, url]', id="cookie"),+ pytest.param(+ f'["mysql", "-uroot", "-p{_QUOTED_SECRET}"]', '["mysql", "-uroot", "-p<redacted>"]', id="mysql-attached"+ ),+ pytest.param(+ f'["docker", "login", "-p{_QUOTED_SECRET}"]', '["docker", "login", "-p<redacted>"]', id="login-attached"+ ),+ pytest.param(+ f'["tool", "--token", "{_QUOTED_SECRET}\\\\", "next"]',+ '["tool", "--token", <redacted>, "next"]',+ id="value-ends-in-backslash",+ ),+ ],+)+def test_judge_evidence_masks_argv_lists_escaped_again(argv: str, masked: str) -> None:+ for form in (_escaped_twice, _in_quoted_python_string, lambda text: json.dumps(_in_quoted_python_string(text))):+ text = form(f"subprocess.run({argv})")++ assert eval_template._redact_judge_evidence(text) == form(f"subprocess.run({masked})")+++# A double-quoted element that holds an escaped quote is masked whole in an+# argv list escaped twice, where that quote follows seven backslashes.+def test_judge_evidence_masks_an_argv_value_with_an_escaped_quote_escaped_twice() -> None:+ text = _escaped_twice(f'subprocess.run(["tool", "--token", "a\\"b {_QUOTED_SECRET}", "next"])')++ assert eval_template._redact_judge_evidence(text) == _escaped_twice(+ 'subprocess.run(["tool", "--token", <redacted>, "next"])'+ )+++# An argv list escaped twice may hold one element per line, where a line+# break is the three characters "\\n".+def test_judge_evidence_masks_argv_lines_escaped_twice() -> None:+ argv = f'[\n "tool",\n "--token",\n "{_QUOTED_SECRET}",\n]'+ masked = '[\n "tool",\n "--token",\n <redacted>,\n]'++ assert eval_template._redact_judge_evidence(_escaped_twice(f"subprocess.run({argv})")) == _escaped_twice(+ f"subprocess.run({masked})"+ )+++# A RAGAS request that repeats a Write call writes the file's JSON content as+# JSON twice. The value of a member named for a credential is masked as in JSON+# escaped once: with its quotes, or between them after a scheme word for a+# header. The other members are kept.+def test_judge_evidence_masks_credentials_in_json_content_escaped_twice() -> None:+ def prompt(content: str) -> str:+ arguments = json.dumps({"file_path": "config.json", "content": content})+ return "input: " + json.dumps(+ {"user_input": [{"content": f"Agent called: Write({arguments})", "type": "ai"}]}, indent=4+ )++ content = json.dumps(+ {"token": _QUOTED_SECRET, "token_type": "bearer", "headers": {"Authorization": f"Bearer {_PASSPHRASE_WORD}"}}+ )+ masked = content.replace(f'"{_QUOTED_SECRET}"', "<redacted>").replace(_PASSPHRASE_WORD, "<redacted>")++ assert eval_template._redact_judge_evidence(prompt(content)) == prompt(masked)+++# A RAGAS request that repeats a tool call writes the call's JSON arguments as+# JSON again, so a double-quoted shell value there opens and closes with a run+# of three backslashes and a quote (\\\"VALUE\\\"), or seven when the command+# is itself a quoted shell string or the request is quoted once more. The whole+# value is masked, also past a space, and the text around it is kept.+ ("text", "value", "times"),+ [+ pytest.param(f'tool --password "{_QUOTED_SECRET} two" -x', f'"{_QUOTED_SECRET} two"', 2, id="password-flag"),+ pytest.param(f'tool --password "{_QUOTED_SECRET} two" -x', f'"{_QUOTED_SECRET} two"', 3, id="password-flag-3"),+ pytest.param(f'deploy --token "a {_QUOTED_SECRET}" -x', f'"a {_QUOTED_SECRET}"', 2, id="token-flag"),+ pytest.param(f'curl -u "admin:{_QUOTED_SECRET}" https://x', f'"admin:{_QUOTED_SECRET}"', 2, id="user-flag"),+ pytest.param(f'curl -u "admin:{_QUOTED_SECRET}" https://x', f'"admin:{_QUOTED_SECRET}"', 3, id="user-flag-3"),+ pytest.param(+ f'curl --user="admin:{_QUOTED_SECRET}" https://x', f'"admin:{_QUOTED_SECRET}"', 2, id="user-flag-equals"+ ),+ pytest.param(+ f'curl --proxy-user "admin:{_QUOTED_SECRET} two" https://x',+ f'"admin:{_QUOTED_SECRET} two"',+ 2,+ id="proxy-user-flag",+ ),+ pytest.param(f'sshpass -p "{_QUOTED_SECRET} two" ssh host', f'"{_QUOTED_SECRET} two"', 2, id="sshpass"),+ pytest.param(+ f'docker login -u bot -p "{_QUOTED_SECRET} two" registry.example.com',+ f'"{_QUOTED_SECRET} two"',+ 2,+ id="login-password",+ ),+ pytest.param(f'mysql -uroot -p"{_QUOTED_SECRET} two" app', f'"{_QUOTED_SECRET} two"', 2, id="mysql-password"),+ pytest.param(+ f'machine db login bot password "{_QUOTED_SECRET} two"', f'"{_QUOTED_SECRET} two"', 2, id="netrc-password"+ ),+ pytest.param(+ f'bash -c "tool --password \\"{_QUOTED_SECRET} two\\" -x"',+ f'\\"{_QUOTED_SECRET} two\\"',+ 2,+ id="shell-string-2",+ ),+ ],+)+def test_judge_evidence_masks_quoted_values_escaped_again(text: str, value: str, times: int) -> None:+ escaped = json.dumps({"command": text})+ expected = json.dumps({"command": text.replace(value, "<redacted>")})+ for _ in range(times - 1):+ escaped, expected = json.dumps(escaped), json.dumps(expected)++ assert eval_template._redact_judge_evidence(escaped) == expected+ assert eval_template._redact_judge_evidence(f"ok\n{escaped}\ndone") == f"ok\n{expected}\ndone"+++# An argv value escaped twice that the passes mask only in part, at a token+# that ends it, is masked whole: the marker of its reading and the token's+# marker are one marker, and none of the token is kept.+ "value",+ [+ pytest.param("{token}", id="token"),+ pytest.param("word {token}", id="word-and-token"),+ pytest.param("user:{token}", id="user-and-token"),+ ],+)+def test_judge_evidence_masks_a_partly_masked_value_escaped_twice_as_one_value(value: str) -> None:+ token = "gh" + "p_" + "Ab3" * 12+ text = _escaped_twice(f'subprocess.run(["tool", "--token", "{value.format(token=token)}"], check=True)')++ assert eval_template._redact_judge_evidence_as_written(text) != text+ assert eval_template._redact_judge_evidence(text) == _escaped_twice(+ 'subprocess.run(["tool", "--token", <redacted>], check=True)'+ )+++# A value of a reading ends at the quote that ends the string it was read+# from. A double-quoted value that does not close before the end of its JSON+# string is masked to that end, and the quote and the members after it stay.[email protected]("times", [1, 2])+def test_judge_evidence_masks_an_unclosed_quoted_value_to_the_end_of_its_string(times: int) -> None:+ escaped = json.dumps({"command": f'tool --password "{_QUOTED_SECRET} two && echo done', "note": "keep me"})+ masked = json.dumps({"command": "tool --password <redacted>", "note": "keep me"})+ for _ in range(times - 1):+ escaped, masked = json.dumps(escaped), json.dumps(masked)++ assert eval_template._redact_judge_evidence(escaped) == masked+++# A double quote after an escaped backslash ends a JSON string, also in the+# readings: a line continuation that ends a string after a credential flag is+# masked, and the string's closing quote and the members after it stay.+def test_judge_evidence_keeps_the_quote_after_a_continuation_that_ends_a_string() -> None:+ text = json.dumps({"lines": ["tool --token \\", " --verbose"], "note": "keep me"})++ assert eval_template._redact_judge_evidence(text) == json.dumps(+ {"lines": ["tool --token <redacted>", " --verbose"], "note": "keep me"}+ )+++# Tool output with a NUL byte reaches a RAGAS prompt as \u0000, which the+# readings keep as written, so a tool call elsewhere in the prompt is still+# read with its escapes removed.+def test_judge_evidence_reads_a_prompt_with_an_escaped_nul_again() -> None:+ def prompt(command: str) -> str:+ call = json.dumps({"command": command})+ return "input: " + json.dumps(+ {+ "user_input": [+ {"content": f"Agent called: Bash({call})", "type": "ai"},+ {"content": "tool output: bin\x00ary", "type": "tool"},+ ]+ }+ )++ text = prompt(f'tool --password "{_QUOTED_SECRET} two" && echo done')++ assert "\\u0000" in text+ assert eval_template._redact_judge_evidence(text) == prompt("tool --password <redacted> && echo done")+++# The readings place what they mask by the text kept between markers, so a+# JWT's jwt-<redacted>, a private key's marker and the [REDACTED] written for a+# configured credential are read as markers too: a value escaped again in the+# same prompt is still masked.+ ("output", "masked_output"),+ [+ pytest.param(f"session {_JWT} saved", "session jwt-<redacted> saved", id="jwt"),+ pytest.param(f"key:\n{_KEY_BLOCK}\ndone", "key:\n<redacted private key>\ndone", id="private-key"),+ pytest.param(+ f"auth failed for {_CONFIGURED_CREDENTIAL}", "auth failed for [REDACTED]", id="configured-credential"+ ),+ ],+)+def test_judge_evidence_masks_a_value_escaped_again_next_to_a_marker_of_another_kind(+ output: str, masked_output: str, monkeypatch: pytest.MonkeyPatch+) -> None:+ monkeypatch.setenv("ANTHROPIC_API_KEY", _CONFIGURED_CREDENTIAL)++ def prompt(value: str, output: str) -> str:+ call = json.dumps({"command": f'subprocess.run(["tool", "--password", {value}])'})+ return json.dumps({"call": call, "output": output})++ text = prompt(f'"{_QUOTED_SECRET}"', output)++ assert _QUOTED_SECRET in eval_template._redact_judge_evidence_as_written(text)+ assert eval_template._redact_judge_evidence(text) == prompt("<redacted>", masked_output)+++# The text that a reading keeps before its first marker is placed at the+# start of the reading, also when it is found again inside the masked value+# or is empty (see test_security_finding_evidence.py).[email protected](("text", "expected"), _READINGS_THAT_START_THE_TEXT)+def test_judge_evidence_places_the_text_a_reading_keeps_before_its_first_marker(text: str, expected: str) -> None:+ assert eval_template._redact_judge_evidence_as_written(text) != expected+ assert eval_template._redact_judge_evidence(text) == expected+++# Reading the text with its escapes removed only adds markers to what the+# judge evidence passes write for the text as written, also next to the+# [REDACTED] written for a configured credential.+def test_judge_evidence_masking_escaped_again_only_adds_markers(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setenv("ANTHROPIC_API_KEY", _CONFIGURED_CREDENTIAL)+ changed = 0+ for text in _generated_texts(400):+ as_written = eval_template._redact_judge_evidence_as_written(text)+ masked = eval_template._redact_judge_evidence(text)++ assert _only_adds_markers(as_written, masked), (text, as_written, masked)+ changed += masked != as_written+ assert changed > 50+++# The judge passes read whole prompts and replies, with no scan window, so+# their readings also take time in proportion to the text's length on long+# adversarial input.+ "unit",+ [+ pytest.param('\\\\\\"', id="escaped-quotes"),+ pytest.param("\\n\\t\\u00e4\\/\\'", id="escape-runs"),+ pytest.param('password=\\\\\\"', id="credential-names-before-escaped-quotes"),+ pytest.param('--password \\\\\\"a b\\\\\\" ', id="values-escaped-twice"),+ pytest.param('<redacted>\\\\\\"', id="markers-and-escaped-quotes"),+ ],+)+def test_judge_evidence_masking_escaped_again_stays_linear_on_adversarial_input(unit: str) -> None:+ def elapsed(size: int) -> float:+ text = unit * (size // len(unit))+ times = []+ for _ in range(2):+ started = time.perf_counter()+ eval_template._redact_judge_evidence(text)+ times.append(time.perf_counter() - started)+ return min(times)++ small, large = elapsed(20_000), elapsed(200_000)+ # Ten times the text takes about ten times as long. The margin allows for+ # a slow CI runner.+ assert large < 16 * small + 0.05++def test_salvaged_behavior_score_is_saved_as_a_salvage_attempt(tmp_path: Path) -> None:verifier = _stage_verifier(tmp_path)truncated = '{"results":[{"step":1,"passed":true,"reason":"ran the script"},{"step":2,"passed":tr'diff --git a/tests/tier3/test_ragas_judge_evidence.py b/tests/tier3/test_ragas_judge_evidence.pyindex d97c51d8..53a4eabd 100644--- a/tests/tier3/test_ragas_judge_evidence.py+++ b/tests/tier3/test_ragas_judge_evidence.py@@ -503,6 +503,108 @@ def test_ragas_evidence_masks_consecutive_credential_lines_in_tool_calls(assert secret not in saved+def test_ragas_evidence_masks_argv_lists_in_tool_calls(verifier: ModuleType, monkeypatch: pytest.MonkeyPatch) -> None:+ """A tool call's argv list reaches the RAGAS prompt and reply as JSON in JSON, so its quotes are escaped twice.++ Each credential element is masked there as in plain text, and the other+ elements stay.+ """+ trajectory = {+ "steps": [+ {+ "source": "agent",+ "tool_calls": [+ {+ "function_name": "Bash",+ "arguments": {+ "command": "python -c 'import subprocess; "+ 'subprocess.run(["tool", "--password", "aaaaaaaaaaaa"])\''+ },+ },+ {+ "function_name": "Write",+ "arguments": {+ "file_path": "db.py",+ "content": 'subprocess.run([\n "mysql",\n "-uroot",\n "-pbbbbbbbbbbbb",\n])\n'+ 'subprocess.run(["curl", "-u", "admin:cccccccccccc", url])\n',+ },+ },+ ],+ },+ ]+ }+ secrets = ("aaaaaaaaaaaa", "bbbbbbbbbbbb", "cccccccccccc")+ agent_text = verifier.build_conversation_summary(trajectory, "Set up the database.")+ assert all(secret in agent_text for secret in secrets)+ prompt = "Judge whether the agent reached the goal.\ninput: " + json.dumps(+ {"user_input": [{"content": agent_text, "type": "ai"}]}, indent=4+ )+ _install_ragas(monkeypatch, messages=[{"role": "user", "content": prompt}])+ received = _stub_provider(monkeypatch, [_chat_reply(json.dumps({"end_state": agent_text, "verdict": 1}))])++ with _judge_evidence(verifier):+ result = verifier.judge_goal_accuracy("question", "expected", "agent answer")++ assert (result["score"], result["method"]) == (1.0, "ragas")+ assert len(received) == 1+ exchange, outcome = _attempts(verifier)+ assert [exchange["status"], outcome["status"]] == ["replied", "accepted"]+ for field in ("prompt", "reply"):+ assert '[\\\\\\"tool\\\\\\", \\\\\\"--password\\\\\\", <redacted>]' in exchange[field]+ assert '\\\\\\"-uroot\\\\\\",\\\\n \\\\\\"-p<redacted>\\\\\\"' in exchange[field]+ assert '[\\\\\\"curl\\\\\\", \\\\\\"-u\\\\\\", <redacted>, url]' in exchange[field]+ saved = (verifier.VERIFIER_DIR / "judge_attempts.json").read_text(encoding="utf-8")+ for secret in secrets:+ assert secret not in saved+++def test_ragas_evidence_masks_quoted_tool_call_values_escaped_twice(+ verifier: ModuleType, monkeypatch: pytest.MonkeyPatch+) -> None:+ """A double-quoted value in a tool call's arguments reaches the RAGAS prompt and reply with its quotes escaped twice.++ The whole value is masked, past the space in it, and the command around it+ is kept.+ """+ trajectory = {+ "steps": [+ {+ "source": "agent",+ "tool_calls": [+ {"function_name": "Bash", "arguments": {"command": 'deploy --password "aaaa cccccccccccc" -n'}},+ # Built from parts, so the source holds no curl user:password literal.+ {+ "function_name": "Bash",+ "arguments": {"command": "curl -u " + '"admin:' + "b" * 12 + '" https://x.example'},+ },+ ],+ },+ ]+ }+ secrets = ("cccccccccccc", "bbbbbbbbbbbb")+ agent_text = verifier.build_conversation_summary(trajectory, "Deploy the app.")+ assert all(secret in agent_text for secret in secrets)+ prompt = "Judge whether the agent reached the goal.\ninput: " + json.dumps(+ {"user_input": [{"content": agent_text, "type": "ai"}]}, indent=4+ )+ _install_ragas(monkeypatch, messages=[{"role": "user", "content": prompt}])+ received = _stub_provider(monkeypatch, [_chat_reply(json.dumps({"end_state": agent_text, "verdict": 1}))])++ with _judge_evidence(verifier):+ result = verifier.judge_goal_accuracy("question", "expected", "agent answer")++ assert (result["score"], result["method"]) == (1.0, "ragas")+ assert len(received) == 1+ exchange, outcome = _attempts(verifier)+ assert [exchange["status"], outcome["status"]] == ["replied", "accepted"]+ for field in ("prompt", "reply"):+ assert "deploy --password <redacted> -n" in exchange[field]+ assert "curl -u <redacted> https://x.example" in exchange[field]+ saved = (verifier.VERIFIER_DIR / "judge_attempts.json").read_text(encoding="utf-8")+ for secret in secrets:+ assert secret not in saved++def test_ragas_evidence_masks_private_key_blocks_in_json_prompts_and_replies(verifier: ModuleType, monkeypatch: pytest.MonkeyPatch) -> None:diff --git a/tests/tier3/test_security_finding_evidence.py b/tests/tier3/test_security_finding_evidence.pyindex 4cec8dd2..2045857c 100644--- a/tests/tier3/test_security_finding_evidence.py+++ b/tests/tier3/test_security_finding_evidence.py@@ -14,7 +14,10 @@ from __future__ import annotationsimport importlib.utilimport json+import random+import reimport time+from collections.abc import Callablefrom pathlib import Pathfrom typing import Any@@ -1151,6 +1154,28 @@ def test_finding_evidence_redaction_masks_credentials(text: str, secrets: tuple[# The same with the quotes escaped twice, after a plain MySQL client element.'run(["mysql", "-e", "x"]); '+ json.dumps(json.dumps('run(["mkdir", "-p", "data"]); run(["mysql", "-u", "root", "-p", "appdb"])')),+ # Argument lists without a credential, with the quotes escaped twice,+ # also one element per line, and in a quoted Python string, plain and+ # JSON-escaped.+ json.dumps(json.dumps('["date", "-u", "+%H:%M:%S"]; ["pip", "install", "-U", "git+https://h.example/r"]')),+ json.dumps(json.dumps('["sshpass", "-f", "pw.txt", "ssh", "-p", "2222", "host.example"]')),+ json.dumps(json.dumps('["curl", "-b", "cookies.txt"]; ["find", "/var/lib/mysql", "-path", "*.ibd", "-print"]')),+ json.dumps(json.dumps('subprocess.run([\n "rsync",\n "-u",\n "host:/data",\n ".",\n])')),+ "code = '[\\'date\\', \\'-u\\', \\'+%H:%M:%S\\'] + [\\'mkdir\\', \\'-p\\', \\'out\\']'",+ json.dumps("code = '[\\'tool\\', \\'--token\\', \\'--verbose\\'] + [\\'ssh\\', \\'-p\\', \\'2222\\']'"),+ # A URL, a remote path and a date format after -u, and -u values with+ # no colon, in double quotes escaped twice and three times.+ json.dumps(+ json.dumps('pip install -U "git+https://github.com/o/r" && rsync -u "host:/data" . && date -u "+%H"')+ ),+ json.dumps(+ json.dumps(json.dumps('rsync -avu "host:/loot" /tmp && date -u "+%H:%M" && curl -u "bot" https://x'))+ ),+ json.dumps(json.dumps('git push -u "origin" main && echo "done" && judge --token-limit "5 10"')),+ # A Windows path and an escaped backslash before a closing quote, in+ # JSON escaped once and twice.+ json.dumps({"path": "C:\\tmp\\", "cmd": "tool --password-file C:\\tmp\\pw.txt", "note": "keep"}),+ json.dumps(json.dumps({"path": "C:\\tmp\\", "cmd": 'echo "C:\\new\\" && ls', "note": "keep"})),],)def test_finding_evidence_redaction_keeps_ordinary_text(text: str) -> None:@@ -1198,6 +1223,476 @@ def test_finding_evidence_masks_twice_escaped_attached_password(context: str) ->assert eval_core_checks._redact_finding_evidence(text) == text.replace(_BASIC_PASSWORD, "<redacted>")+def _escaped_twice(text: str) -> str:+ return json.dumps(json.dumps({"command": text}))+++def _in_quoted_python_string(text: str) -> str:+ """Write *text*, a list with double-quoted elements, as a single-quoted list inside a single-quoted string."""+ source = text.replace('"', "'")+ return "code = '" + source.replace("\\", "\\\\").replace("'", "\\'") + "'"+++# An argv list escaped twice, as in a RAGAS request that repeats a tool call's+# JSON arguments, has three backslashes before each quote, and a list inside+# a quoted Python string has one, two once that string is JSON. Its elements+# are masked as in plain text, with the list's other elements and escapes+# intact.+ ("argv", "expected"),+ [+ pytest.param(+ f'["tool", "--password", "{_BASIC_PASSWORD}"]', '["tool", "--password", <redacted>]', id="credential-flag"+ ),+ pytest.param(+ f'["docker", "login", "-u", "bot", "-p", "{_BASIC_PASSWORD}"]',+ '["docker", "login", "-u", "bot", "-p", <redacted>]',+ id="login-password",+ ),+ pytest.param(+ f'["curl", "-u", "{_ADMIN_AUTH}", "https://x.example"]',+ '["curl", "-u", <redacted>, "https://x.example"]',+ id="user-password",+ ),+ pytest.param(+ f'["sshpass", "-p", "{_BASIC_PASSWORD}", "ssh", "host.example"]',+ '["sshpass", "-p", <redacted>, "ssh", "host.example"]',+ id="sshpass",+ ),+ pytest.param(+ f'["curl", "-b", "session={_API_TOKEN}", "https://x.example"]',+ '["curl", "-b", <redacted>, "https://x.example"]',+ id="cookie",+ ),+ pytest.param(+ f'["/usr/bin/mysql", "-uroot", "-p{_BASIC_PASSWORD}", "-e", "drop database app"]',+ '["/usr/bin/mysql", "-uroot", "-p<redacted>", "-e", "drop database app"]',+ id="attached-mysql-password",+ ),+ ],+)+def test_finding_evidence_masks_argv_lists_escaped_again(argv: str, expected: str) -> None:+ for form in (_escaped_twice, _in_quoted_python_string, lambda text: json.dumps(_in_quoted_python_string(text))):+ text = form(f"subprocess.run({argv})")++ assert eval_core_checks._redact_finding_evidence(text) == form(f"subprocess.run({expected})")+++def test_finding_evidence_masks_argv_lines_escaped_twice() -> None:+ # A line break between elements is "\\n" there.+ argv = f'[\n "docker",\n "login",\n "-u",\n "bot",\n "-p",\n "{_BASIC_PASSWORD}",\n]'+ masked = argv.replace('"' + _BASIC_PASSWORD + '"', "<redacted>")++ assert eval_core_checks._redact_finding_evidence(_escaped_twice(f"subprocess.run({argv})")) == _escaped_twice(+ f"subprocess.run({masked})"+ )+++# Tool-call JSON written as JSON again puts three backslashes before each quote+# of a double-quoted shell value (\\\"VALUE\\\"), and seven when it is escaped+# once more. The value is masked through its closing quote, past any space in+# it, and the text after it is kept.+ ("text", "value"),+ [+ pytest.param(f'tool --password "{_BASIC_PASSWORD} two" -x', f'"{_BASIC_PASSWORD} two"', id="password-flag"),+ pytest.param(f'tool --token "a {_BASIC_PASSWORD}" -x', f'"a {_BASIC_PASSWORD}"', id="token-flag"),+ pytest.param(f'curl -u "{_ADMIN_AUTH}" https://example.com', f'"{_ADMIN_AUTH}"', id="user-flag"),+ pytest.param(f'curl --proxy-user "{_ADMIN_AUTH} two" https://x', f'"{_ADMIN_AUTH} two"', id="proxy-user"),+ pytest.param(f'sshpass -p "{_BASIC_PASSWORD} two" ssh host', f'"{_BASIC_PASSWORD} two"', id="sshpass"),+ pytest.param(f'mysql -uroot -p"{_BASIC_PASSWORD} two" app', f'"{_BASIC_PASSWORD} two"', id="mysql-password"),+ pytest.param(+ f'machine db login bot password "{_BASIC_PASSWORD} two"', f'"{_BASIC_PASSWORD} two"', id="netrc-password"+ ),+ pytest.param(+ f'git config github.token "{_BASIC_PASSWORD} two" && ls', f'"{_BASIC_PASSWORD} two"', id="git-config-set"+ ),+ pytest.param(+ f'npm config set _authToken "{_BASIC_PASSWORD} two" && ls', f'"{_BASIC_PASSWORD} two"', id="npm-config-set"+ ),+ pytest.param(+ f'aws configure set aws_secret_access_key "{_BASIC_PASSWORD} two" && ls',+ f'"{_BASIC_PASSWORD} two"',+ id="aws-configure-set",+ ),+ pytest.param(+ f'curl -b "sid={_BASIC_PASSWORD} two" https://x.example', f'"sid={_BASIC_PASSWORD} two"', id="cookie-flag"+ ),+ ],+)[email protected]("times", [2, 3])+def test_finding_evidence_masks_quoted_values_escaped_again(text: str, value: str, times: int) -> None:+ escaped, expected = text, text.replace(value, "<redacted>")+ for _ in range(times):+ escaped, expected = json.dumps(escaped), json.dumps(expected)++ assert eval_core_checks._redact_finding_evidence(escaped) == expected+++# A value escaped again is masked whole also when a later pattern masks a+# word in it, here a GitHub token.+def test_finding_evidence_masks_a_value_escaped_again_with_a_token_in_it() -> None:+ token = "gh" + "p_" + "Ab3" * 12+ text = f'bash -lc "bash -c \\"tool --password \\\\\\"word {token} two words\\\\\\" --verbose\\""'++ assert (+ eval_core_checks._redact_finding_evidence(text)+ == 'bash -lc "bash -c \\"tool --password <redacted> --verbose\\""'+ )+++def _ragas_tool_call(tool: str, arguments: dict[str, str]) -> str:+ return "input: " + json.dumps(+ {"user_input": [{"content": f"Agent called: {tool}({json.dumps(arguments)})", "type": "ai"}]}, indent=4+ )+++# A RAGAS request that repeats a Write call writes the file's JSON content as+# JSON twice, so a member there has three backslashes before each quote of its+# name and its value, as does a curl -d JSON body in a repeated Bash call. The+# value of a member whose name names a credential is masked with its quotes,+# as in JSON escaped once, and the other members are kept.+ "content",+ [+ pytest.param({"db": {"host": "h", "password": _BASIC_PASSWORD}}, id="json-password"),+ pytest.param(+ {"token": _BASIC_PASSWORD, "token_type": "bearer", "max_tokens": 5, "author": "bob"}, id="json-token"+ ),+ pytest.param({"api_key": _BASIC_PASSWORD, "key_file": "/k.pem"}, id="json-api-key"),+ pytest.param({"headers": {"Authorization": f"Bearer {_BASIC_PASSWORD}"}}, id="json-authorization"),+ ],+)+def test_finding_evidence_masks_credentials_in_json_content_escaped_twice(content: dict[str, Any]) -> None:+ written = json.dumps(content, indent=2)+ masked = written.replace(f'"{_BASIC_PASSWORD}"', "<redacted>").replace(f'"Bearer {_BASIC_PASSWORD}"', "<redacted>")+ text = _ragas_tool_call("Write", {"file_path": "config.json", "content": written})++ assert eval_core_checks._redact_finding_evidence(text, 10**9) == _ragas_tool_call(+ "Write", {"file_path": "config.json", "content": masked}+ )+++def test_finding_evidence_masks_a_json_body_escaped_twice() -> None:+ def command(body: str) -> str:+ return f"curl -X POST -d '{body}' https://h.example/login"++ body = json.dumps({"username": "bot", "password": _BASIC_PASSWORD})+ text = _ragas_tool_call("Bash", {"command": command(body)})++ assert eval_core_checks._redact_finding_evidence(text, 10**9) == _ragas_tool_call(+ "Bash", {"command": command(body.replace(f'"{_BASIC_PASSWORD}"', "<redacted>"))}+ )+++# A -p element whose "login" element lies more than 256 characters back names+# no password, also when the list is JSON-escaped: a port mapping is kept.+def test_finding_evidence_keeps_a_port_mapping_far_from_a_login_element() -> None:+ script = (+ f'subprocess.run(["docker", "login", "-u", "bot", "-p", "{_BASIC_PASSWORD}{"A" * 80}", "registry.example.com"])\n'+ 'subprocess.run(["docker", "pull", "registry.example.com/team/app:1.4.2"], check=True)\n'+ 'subprocess.run(["docker", "run", "-d", "-p", "8080:80", "registry.example.com/team/app:1.4.2"])\n'+ )+ for text in (script, json.dumps({"code": script}), json.dumps(json.dumps({"code": script}))):+ redacted = eval_core_checks._redact_finding_evidence(text, 10**9)++ assert _BASIC_PASSWORD not in redacted+ assert '"-p", "8080:80"' in redacted.replace("\\", "")+++# An argv value escaped twice that the patterns mask only in part, at a token+# that ends it, is masked whole: the marker of its reading and the token's+# marker are one marker, and none of the token is kept.+ "value",+ [+ pytest.param(_GHP, id="token"),+ pytest.param(f"word {_GHP}", id="word-and-token"),+ pytest.param(f"user:{_GHP}", id="user-and-token"),+ ],+)+def test_finding_evidence_masks_a_partly_masked_value_escaped_twice_as_one_value(value: str) -> None:+ text = _escaped_twice(f'subprocess.run(["tool", "--token", "{value}"], check=True)')++ assert eval_core_checks._mask_finding_evidence_credentials_as_written(text) != text+ assert eval_core_checks._redact_finding_evidence(text, 10**9) == _escaped_twice(+ 'subprocess.run(["tool", "--token", <redacted>], check=True)'+ )+++# A value of a reading ends at the quote that ends the string it was read+# from. A double-quoted value that does not close before the end of its JSON+# string is masked to that end, and the quote and the members after it stay.[email protected]("times", [1, 2])+def test_finding_evidence_masks_an_unclosed_quoted_value_to_the_end_of_its_string(times: int) -> None:+ text = {"command": f'tool --password "{_BASIC_PASSWORD} two && echo done', "note": "keep me"}+ expected = {"command": "tool --password <redacted>", "note": "keep me"}+ escaped, masked = json.dumps(text), json.dumps(expected)+ for _ in range(times - 1):+ escaped, masked = json.dumps(escaped), json.dumps(masked)++ assert eval_core_checks._redact_finding_evidence(escaped, 10**9) == masked+++# A double quote after an escaped backslash ends a JSON string, also in the+# readings, where that backslash is kept as written. A credential flag before+# a line continuation that ends a string has that backslash masked, and the+# string's closing quote and the members after it stay.+def test_finding_evidence_keeps_the_quote_after_a_continuation_that_ends_a_string() -> None:+ text = json.dumps({"lines": ["tool --token \\", " --verbose"], "note": "keep me"})++ assert eval_core_checks._redact_finding_evidence(text, 10**9) == json.dumps(+ {"lines": ["tool --token <redacted>", " --verbose"], "note": "keep me"}+ )+++# A reading can hold a shell line continuation that the text as written+# escapes, and the patterns read the reading without it. What they mask there+# is still placed in the text: here an argv list in a python -c command whose+# Bash call continues on the next line.+def test_finding_evidence_masks_an_argv_value_escaped_again_before_a_line_continuation() -> None:+ def call(value: str) -> str:+ script = f'import subprocess; subprocess.run(["tool", "--password", {value}])'+ return json.dumps({"command": f"python -c {json.dumps(script)} \\\n && echo done"})++ assert eval_core_checks._redact_finding_evidence(call(f'"{_BASIC_PASSWORD}"'), 10**9) == call("<redacted>")+++# A NUL character written as \u0000 elsewhere in the text, as in tool output+# that a RAGAS request repeats, is read as written, so the text is still read+# with its escapes removed.+def test_finding_evidence_reads_text_with_an_escaped_nul_again() -> None:+ def prompt(command: str) -> str:+ return json.dumps({"call": json.dumps({"command": command}), "output": "bin\x00ary"})++ text = prompt(f'tool --password "{_BASIC_PASSWORD} two" && echo done')++ assert "\\u0000" in text+ assert eval_core_checks._redact_finding_evidence(text, 10**9) == prompt("tool --password <redacted> && echo done")+++# A synthetic JWT and private key block, built from parts so that neither+# appears in this file as one literal.+_JWT = "ey" + "JhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9." + "ey" + "JzdWIiOiIxMjM0NTY3ODkwIn0." + "c3ludGhldGljLXNpZ25hdHVyZQ"+_KEY_BLOCK = f"-----BEGIN RSA {_PRIVATE_KEY}-----\nMIIEbodyLineOne\n-----END RSA {_PRIVATE_KEY}-----"+# A value the judge tests configure as the judge's credential, which the Harbor+# verifier masks as [REDACTED] wherever it appears. Finding evidence keeps it.+_CONFIGURED_CREDENTIAL = "dummy-configured-" + "judge-credential"+++# The readings place what they mask by the text kept between markers, so a+# JWT's jwt-<redacted> and a private key's marker are read as markers too: a+# value escaped again in the same text is still masked.+ ("output", "masked_output"),+ [+ pytest.param(f"session {_JWT} saved", "session jwt-<redacted> saved", id="jwt"),+ pytest.param(f"key:\n{_KEY_BLOCK}\ndone", "key:\n<redacted private key>\ndone", id="private-key"),+ ],+)+def test_finding_evidence_masks_a_value_escaped_again_next_to_a_marker_of_another_kind(+ output: str, masked_output: str+) -> None:+ def prompt(value: str, output: str) -> str:+ call = json.dumps({"command": f'subprocess.run(["tool", "--password", {value}])'})+ return json.dumps({"call": call, "output": output})++ text = prompt(f'"{_BASIC_PASSWORD}"', output)++ assert _BASIC_PASSWORD in eval_core_checks._mask_finding_evidence_credentials_as_written(text)+ assert eval_core_checks._redact_finding_evidence(text, 10**9) == prompt("<redacted>", masked_output)+++def _jwt_continued(times: int) -> str:+ """A JWT split by a shell line continuation, inside a JSON string escaped *times* times, without its quotes."""+ text = f"{_JWT[:40]}\\\n{_JWT[40:]} done"+ for _ in range(times):+ text = json.dumps(text)[1:-1]+ return text+++# The text that a reading keeps before its first marker starts the reading,+# and is placed there also when it is found again inside the value masked+# after it (a value that repeats its flag) or is empty, as the reading starts+# with a marker (evidence that starts with a JWT split by a line continuation,+# escaped once or twice). The text as written masks these in part or not at+# all, and a reading masks them whole.+_READINGS_THAT_START_THE_TEXT = (+ pytest.param(+ f'--password \\\\\\"--password {_BASIC_PASSWORD} two\\\\\\" -x',+ "--password <redacted> -x",+ id="value-repeats-its-flag",+ ),+ pytest.param(_jwt_continued(1), "<redacted> done", id="jwt-continued-escaped-once"),+ pytest.param(_jwt_continued(2), "<redacted> done", id="jwt-continued-escaped-twice"),+)++[email protected](("text", "expected"), _READINGS_THAT_START_THE_TEXT)+def test_finding_evidence_places_the_text_a_reading_keeps_before_its_first_marker(text: str, expected: str) -> None:+ assert eval_core_checks._mask_finding_evidence_credentials_as_written(text) != expected+ for preserve_header_format in (False, True):+ assert (+ eval_core_checks._mask_finding_evidence_credentials(text, preserve_header_format=preserve_header_format)+ == expected+ )+++# A reading's markers are mapped back by the text kept between them. Where+# that text fits the reading in more than one way, the span of the reading+# that a marker stands for is not known, and nothing is masked for it; a+# marker whose kept text fits one way is masked.+def test_masking_escaped_again_skips_a_marker_its_kept_text_does_not_place() -> None:+ text = "pw: one\\ntwo\\nthree\\nfour"++ def mask(masked_reading: str) -> Callable[[str], str]:+ return lambda view: masked_reading if view == "pw: one\ntwo\nthree\nfour" else view++ # "one\ntwo" and "three", or "one" and "two\nthree".+ ambiguous = mask("pw: <redacted>\n<redacted>\nfour")+ assert eval_core_checks._mask_unescaped_views(text, text, ambiguous) == text+ placed = mask("pw: <redacted>\nfour")+ assert eval_core_checks._mask_unescaped_views(text, text, placed) == "pw: <redacted>\\nfour"+++_MARKERS = re.compile(r"(?:(?:jwt-)?<redacted>|<redacted private key>|\[REDACTED\])+")+++def _only_adds_markers(base: str, new: str) -> bool:+ """Whether *new* is *base* with more of its text replaced by markers, and none taken back."""+ if new == base:+ return True+ marks = [match.span() for match in _MARKERS.finditer(base)]+ kept = _MARKERS.split(new)+ if len(kept) == 1 or not base.startswith(kept[0]) or not base.endswith(kept[-1]):+ return False+ pos = 0+ for index, piece in enumerate(kept):+ found = 0 if index == 0 else len(base) - len(piece) if index == len(kept) - 1 else base.find(piece, pos)+ # A piece of kept text never overlaps a marker of *base*.+ while found >= pos and any(start < found + len(piece) and found < end for start, end in marks):+ found = base.find(piece, found + 1) if 0 < index < len(kept) - 1 else -1+ if found < pos:+ return False+ pos = found + len(piece)+ return True+++_ESCAPE_FORMS = (+ json.dumps,+ repr,+ lambda text: json.dumps({"command": text}),+ lambda text: f"Agent called: Bash({json.dumps({'command': text})})",+)+_TEXT_PIECES = (+ "tool --password VALUE more",+ 'tool --token "VALUE two" -x',+ # Built from parts, so the source holds no curl user:password literal.+ "curl -u " + "admin" + ":VALUE https://x.example",+ 'subprocess.run(["mysql", "-uroot", "-pVALUE"])',+ '["docker", "login", "-u", "bot", "-p", "VALUE"]',+ "machine h login bot password VALUE",+ "git config github.token VALUE && ls",+ "curl -b sid=VALUE https://x.example",+ "Authorization: Bearer VALUE",+ '{"password": "VALUE", "note": "keep"}',+ "export DB_PASSWORD=VALUE; ls",+ "mysql -u root -p",+ "C:\\tmp\\new\\",+ "it's 'open",+ 'echo "done" && ls -la',+ "<redacted> kept",+ "gh" + "p_" + "Ab3" * 12,+ # Tokens the patterns mask wherever they appear, at the JSON members after+ # a command: a reading must not write them back in clear.+ '{"command": "tool --password VALUE more", "key": "AKIA' + _AWS_KEY_BODY + '", "token": "' + _GLPAT + '"}',+ '["' + _NPM_TOKEN + '", "' + "xo" + "xb-" + "1234567890-AbCdEfGhIj" + '", "' + _HF_TOKEN + '"]',+ # Text masked with a marker of another kind, which the readings must read+ # as a marker: a JWT, a private key block and a configured credential.+ f"session {_JWT} saved",+ f"key:\n{_KEY_BLOCK}\ndone",+ f"auth failed for {_CONFIGURED_CREDENTIAL}",+)+_NOISE = ("\\", "\\\\", '\\"', '"', "'", "\\n", '\\\\\\"', "\\u00e4", "\\\n")+++def _generated_texts(count: int) -> list[str]:+ """Commands with and without credentials, escaped up to three times, some with stray escapes and quotes."""+ rng = random.Random(547)+ texts = []+ for _ in range(count):+ text = " ".join(rng.choice(_TEXT_PIECES) for _ in range(rng.randint(1, 4)))+ text = text.replace("VALUE", f"Kq7{rng.randint(0, 99)}Secret")+ for _ in range(rng.randint(0, 3)):+ text = rng.choice(_ESCAPE_FORMS)(text)+ for _ in range(rng.randint(0, 2)):+ pos = rng.randrange(len(text) + 1)+ text = text[:pos] + rng.choice(_NOISE) * rng.randint(1, 3) + text[pos:]+ texts.append(text)+ return texts+++# Reading the text with its escapes removed only adds markers: the result is+# the result of the patterns read on the text as written, with more of its+# text masked and none of its markers taken back.+def test_finding_evidence_masking_escaped_again_only_adds_markers() -> None:+ changed = 0+ for text in _generated_texts(800):+ for preserve_header_format in (False, True):+ as_written = eval_core_checks._mask_finding_evidence_credentials_as_written(+ text, preserve_header_format=preserve_header_format+ )+ masked = eval_core_checks._mask_finding_evidence_credentials(+ text, preserve_header_format=preserve_header_format+ )++ assert _only_adds_markers(as_written, masked), (text, as_written, masked)+ changed += masked != as_written+ assert changed > 100+++def test_only_adds_markers_rejects_a_marker_taken_back() -> None:+ assert _only_adds_markers("a <redacted> b", "a <redacted>")+ assert _only_adds_markers("a sk-<redacted> b", "a <redacted> b")+ assert not _only_adds_markers("a <redacted> b", "a VALUE b")+ assert not _only_adds_markers("a <redacted> b", "a <redacted> b c")+ assert not _only_adds_markers("a b", "a b c")+++# Reading a long text with deep backslash runs, many escaped quotes, long+# escape runs, values escaped twice in JSON strings that end in an escaped+# backslash, argv elements escaped twice or many markers takes time in+# proportion to its length.+ "unit",+ [+ pytest.param("\\", id="one-backslash-run"),+ pytest.param('\\\\\\"', id="escaped-quotes"),+ pytest.param('{"c": "--password \\\\\\"a b\\\\\\" C:\\\\"}, ', id="strings-ending-in-escaped-backslashes"),+ pytest.param("\\n\\t\\u00e4\\/\\'", id="escape-runs"),+ pytest.param('--password \\\\\\"a b\\\\\\" ', id="values-escaped-twice"),+ pytest.param("<redacted>x\\n", id="markers"),+ pytest.param('<redacted>\\\\\\"', id="markers-and-escaped-quotes"),+ pytest.param('\\\\\\"--token\\\\\\", \\\\\\"a\\\\\\", ', id="argv-elements-escaped-twice"),+ ],+)+def test_masking_escaped_again_stays_linear_on_adversarial_input(unit: str) -> None:+ def elapsed(size: int) -> float:+ text = unit * (size // len(unit))+ times = []+ for _ in range(3):+ started = time.perf_counter()+ eval_core_checks._mask_finding_evidence_credentials(text)+ times.append(time.perf_counter() - started)+ return min(times)++ small, large = elapsed(25_000), elapsed(250_000)+ # Ten times the text takes about ten times as long. A step quadratic in+ # the number of markers took twenty times as long on the escaped quotes+ # between markers. The margin allows for a slow CI runner.+ assert large < 16 * small + 0.05++@pytest.mark.parametrize(("text", "expected"),[