Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier3): score unrecovered failures and align recovery verdicts (#217)
Narendran Raghavancf23b0d1e29f
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4b9a9ba8..42e3773b 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -40,14 +40,27 @@ All notable changes to SkillEvaluator are documented in this file.### Fixed- Tier 3 error recovery now scores unrecovered command failures as zero and- includes them in fault counts and failure verdicts/report reasons.- Host and bundled verifier detection agree and prefer explicit exit status,- avoiding penalties for successful output such as `0 errors`, Python exception- handlers, and numeric timeout configuration. Printed source text containing- exit codes or error strings is not treated as execution status; quiet exit-1- search and file-test probes remain clean, including Codex unified-exec- observations. Beyond the original five-call retry window, only a successful- repeat of the same command and arguments counts as recovery.+ includes them in fault counts and failure verdicts/report reasons. Host and+ bundled verifier detection agree and use a result's execution status when it+ has one: a Codex exit code, a whole `exit code N` line, or Harbor marking a+ Claude Code result as successful. Only results without a status, such as+ OpenCode's, are judged by their text: tracebacks, `error:` lines, pytest,+ jest, go test, tsc and npm failures and OpenCode's timeout note count, while+ `0 errors`, exception handlers, numeric timeout settings and type annotations+ do not. Exit-1 probes with no other output (`grep`, `rg`, `test`, `diff`,+ `cmp`, `command -v`, `which`, `git diff --quiet`, including `grep -c` zero+ counts, `/dev/null` redirects, a leading `cd` or `pwd` and pipelines) remain+ clean. Recovery matches whole command words after a leading `cd`; beyond the+ original five-call retry window, only a successful repeat of the same command+ and arguments counts.+- Catalog validation now includes nested and lowercase skill manifests, skips+ generated directories, and stops at skill boundaries to avoid validating+ bundled templates as separate skills. External linked directories containing+ skills and unreadable directories appear as failures without stopping other+ skills; links back into the catalog are ignored. Linked manifests retain+ direct-target behavior, plugin discovery errors produce findings, and reports+ and external Tier 3 results preserve catalog-relative paths for skills with+ the same basename.- Changing finding severities through a policy no longer clears an independentquality-score threshold failure. A score below `--min-score` still fails,and its reason appears in the terminal, HTML, benchmark, and JSON reports.@@ -208,13 +221,17 @@ All notable changes to SkillEvaluator are documented in this file.### Fixed-- Tier 3 error recovery now scores unrecovered command failures as zero and- includes them in fault counts and failure verdicts/report reasons.- Host and bundled verifier detection agree and prefer explicit exit status,- avoiding penalties for successful output such as `0 errors`, Python exception- handlers, and numeric timeout configuration.--+- Tier 3 reports preserve judge-model exclusions when calculating dimension and+ overall lift. Native multi-step results also retain the judge provenance needed+ for those comparisons, so fallback judges cannot turn a regression into an+ apparent improvement. Judge-free cases retain valid lift, bounded report loads+ use the collector's recorded comparisons, and FINAL-strategy trials use only+ the final step's judges. The CLI displays unavailable comparisons as `n/a`.+- The LLM judge treats a schema option reported as `unavailable` as a schema+ rejection and retries without it, so OpenAI-compatible endpoints such as+ DeepSeek, which answer `response_format: json_schema` with HTTP 400 and+ "This response_format type is unavailable now", can score Tier 3 instead+ of failing every judge call (fixes #207).- Tier 3 reports an inconclusive result when every observed positive with-skilltrial has zero activation, and omits skill-lift comparisons for that agent.Activation counts are reported separately; negative controls and missingdiff --git a/src/skillevaluator/tier3/eval_core/atif_helpers.py b/src/skillevaluator/tier3/eval_core/atif_helpers.pyindex 9b531780..0ca00422 100644--- a/src/skillevaluator/tier3/eval_core/atif_helpers.py+++ b/src/skillevaluator/tier3/eval_core/atif_helpers.py@@ -20,6 +20,7 @@ from skillevaluator.evidence import evidence_ref_identityfrom skillevaluator.tier3.eval_core.codex_tool_call_normalizer import (atif_content_has_text,atif_content_text,+ normalized_tool_call_error_flag,normalized_tool_call_is_error,)from skillevaluator.tier3.eval_core.codex_tool_call_normalizer import (@@ -1062,8 +1063,10 @@ def extract_tool_calls_as_dicts(traj: dict[str, Any]) -> list[dict[str, Any]]:"action_input": tc.get("arguments") or {},"observation": _tool_call_observation(step, tc),}- if normalized_tool_call_is_error(step, tc):- call["is_error"] = True+ # Keep Harbor's explicit success flag too: Claude Code results that+ # exited 0 carry no exit status line.+ if (is_error := normalized_tool_call_error_flag(step, tc)) is not None:+ call["is_error"] = is_errorif status := tc.get("_atif_normalization_status"):call["normalization_status"] = statusif status := tc.get("_atif_observation_status"):diff --git a/src/skillevaluator/tier3/eval_core/checks.py b/src/skillevaluator/tier3/eval_core/checks.pyindex 0c499b6d..086d8197 100644--- a/src/skillevaluator/tier3/eval_core/checks.py+++ b/src/skillevaluator/tier3/eval_core/checks.py@@ -5245,14 +5245,28 @@ def check_error_recovery(r"\]?[ \t]*[=:]?[ \t]*(-?\d+)\]?[ \t]*\r?$",re.IGNORECASE | re.MULTILINE,)- # Match diagnostic forms, not prose such as "0 errors" or "error handling".+ # A type annotation such as `error: str | None = None` or `timeout: float = 30.0`+ # is displayed source, not a diagnostic.+ not_annotation = (+ r"(?![ \t]*[A-Za-z_][\w.]*(?:\[[^\]\n]*\])?"+ r"(?:[ \t]*\|[ \t]*[A-Za-z_][\w.]*(?:\[[^\]\n]*\])?)*[ \t]*[=|;,)])"+ )+ # Results without any exit status (Harbor's OpenCode trajectories keep only+ # the output) are judged by diagnostic forms, not prose such as "0 errors"+ # or "error handling", or a `--timeout:` option description.failure_text_re = re.compile(- r"^\s*(?:[\w.+/-]+:\s*)*(?:error|exception)\s*:"- r"|\btimeout\s*:(?!\s*\d)|^\s*(?:\w*(?:error|exception)|timeout|connectionrefused)\s*$"- r"|^\s*\w*(?:Error|Exception)\s*:|traceback\s*\(most recent call last\)"+ rf"^\s*(?:[\w.+/-]+:\s*)*(?:error|exception)\s*:{not_annotation}"+ rf"|(?<![-\w])timeout\s*:(?!\s*\d){not_annotation}"+ r"|^\s*(?:\w*(?:error|exception)|timeout|connectionrefused)\s*$"+ rf"|^\s*\w*(?:Error|Exception)\s*:{not_annotation}|traceback\s*\(most recent call last\)"r"|\bstatus\s*=\s*(?:failed|error)\b"r"|\b(?:command not found|no such file(?: or directory)?|permission denied)\b"- r"|\b(?:connection refused|connectionrefused|timed out)\b|:\s*not found\b",+ r"|\b(?:connection refused|connectionrefused|timed out)\b|:\s*not found\b"+ # Test and build failures (pytest, unittest, jest and go test summaries,+ # tsc, npm) and OpenCode's note for a command stopped at its timeout.+ r"|(?-i:^[ \t]*(?:---[ \t]+)?(?:FAIL(?:ED)?|ERROR)\b(?![ \t]*=))"+ r"|\b[1-9]\d* failed\b|\berror TS\d+\b|^npm (?:ERR!|error\b)"+ r"|\bbash tool terminated command after exceeding timeout\b",re.IGNORECASE | re.MULTILINE,)skill_fault_keywords = [@@ -5266,22 +5280,67 @@ def check_error_recovery("modulenotfounderror",]+ quiet_probe_programs = {"grep", "egrep", "fgrep", "rg", "test", "[", "[[", "diff", "cmp", "which"}+ devnull_redirect_re = re.compile(+ r"(?<!\S)(?:[012]?>{1,2}|&>{1,2})[ \t]*/dev/null(?=$|[\s;&|])|(?<!\S)[12]?>&[12](?=$|[\s;&|])"+ )++ def _quiet_probe_program(words: list[str]) -> str | None:+ while words and re.fullmatch(r"[A-Za-z_]\w*=.*", words[0], flags=re.DOTALL):+ words = words[1:] # VAR=value prefix+ if not words:+ return None+ program = words[0].rsplit("/", 1)[-1]+ if program in quiet_probe_programs:+ return program+ if program == "command" and words[1:2] in (["-v"], ["-V"]):+ return program+ if program == "git" and words[1:2] == ["diff"] and "--quiet" in words[2:]:+ return program+ return None+def _is_quiet_negative_probe(tc: dict[str, Any], observation: str) -> bool:- # Exit 1 is a normal negative answer for these simple commands. Be- # conservative: keep diagnostics and compound shell commands as failures.+ # Exit 1 is a normal negative answer for these probes. Be conservative:+ # keep diagnostics and other compound shell commands as failures.command = _command_text(tc)if any(marker in command for marker in ("\n", "\r", "`", "$(")):return False- lexer = shlex.shlex(command, posix=True, punctuation_chars=True)+ lexer = shlex.shlex(devnull_redirect_re.sub(" ", command), posix=True, punctuation_chars=True)lexer.whitespace_split = Truelexer.commenters = ""try:words = list(lexer)except ValueError:return False- if not words or words[0].rsplit("/", 1)[-1] not in {"grep", "rg", "test", "[", "diff", "cmp"}:+ segments: list[list[str]] = [[]]+ separators: list[str] = []+ for word in words:+ if word and all(char in ";&|<>()" for char in word):+ if word not in {"&&", ";", "|"} or not segments[-1]:+ return False+ separators.append(word)+ segments.append([])+ else:+ segments[-1].append(word)+ if not segments[-1]:+ return False+ # Skip leading `cd <dir>` and `pwd` commands joined by `&&` or `;` (a+ # failed cd prints a diagnostic) and a trailing `&& echo ...`, which+ # runs only when the probe succeeds.+ start, end = 0, len(segments) - 1+ while (+ start < end+ and separators[start] != "|"+ and (segments[start] == ["pwd"] or (segments[start][0] == "cd" and len(segments[start]) <= 2))+ ):+ start += 1+ while end > start and separators[end - 1] == "&&" and segments[end][0] in {"echo", "printf"}:+ end -= 1+ # A pipeline exits with its last command's status, so only that one must be a probe.+ if any(separator != "|" for separator in separators[start:end]):return False- if any(word and all(char in ";&|<>()" for char in word) for word in words):+ program = _quiet_probe_program(segments[end])+ if program is None:return False# Harbor's Codex unified-exec envelope is metadata, but everything# after Output: is command output, even if it resembles another header.@@ -5295,49 +5354,72 @@ def check_error_recovery(flags=re.DOTALL,)if unified_output is not None:- return not unified_output.group(1).strip()- # Other converters emit status/exit_code lines without an envelope.- output = exit_status_re.sub("", observation)- output = re.sub(- r"^[ \t]*status=(?:completed|failed|error)[ \t]*\r?$",- "",- output,- flags=re.IGNORECASE | re.MULTILINE,+ output = unified_output.group(1)+ else:+ # Other converters emit status/exit_code lines without an envelope.+ output = exit_status_re.sub("", observation)+ output = re.sub(+ r"^[ \t]*status=(?:completed|failed|error)[ \t]*\r?$",+ "",+ output,+ flags=re.IGNORECASE | re.MULTILINE,+ )+ # Harbor appends this exact marker to Claude Code is_error results,+ # including ordinary negative probes. Preserve all other output lines.+ output = re.sub(r"^\[error\] tool reported failure\r?$", "", output, flags=re.MULTILINE)+ lines = [line.strip() for line in output.splitlines() if line.strip()]+ # Each skipped `pwd` printed one absolute directory line first.+ directories = sum(segment == ["pwd"] for segment in segments[:start])+ if all(line.startswith("/") for line in lines[:directories]):+ lines = lines[directories:]+ # `grep -c` prints a zero count (per file) when nothing matches.+ counts = program in {"grep", "egrep", "fgrep", "rg"} and any(+ word in {"--count", "--count-matches"} or re.fullmatch(r"-[A-Za-z]*c[A-Za-z]*", word)+ for word in segments[end])- # Harbor appends this exact marker to Claude Code is_error results,- # including ordinary negative probes. Preserve all other output lines.- output = re.sub(r"^\[error\] tool reported failure\r?$", "", output, flags=re.MULTILINE)- return not output.strip()+ return all(counts and re.fullmatch(r"(?:.*:)?0", line) for line in lines)++ def _exit_status(tc: dict[str, Any]) -> int | None:+ # Harbor marks Claude Code results that exited 0 with tool_result_is_error+ # false and adds an exit status line only to failures.+ if tc.get("is_error") is False:+ return 0+ status = exit_status_re.search(str(tc.get("observation", "")))+ return int(status.group(1)) if status else Nonedef _is_failure(tc: dict[str, Any]) -> bool:obs = str(tc.get("observation", ""))- if status := exit_status_re.search(obs):- exit_code = int(status.group(1))- if exit_code == 1 and _is_quiet_negative_probe(tc, obs):- return False- return exit_code != 0+ exit_code = _exit_status(tc)+ if exit_code is not None:+ return exit_code != 0 and not (exit_code == 1 and _is_quiet_negative_probe(tc, obs))+ # Only results without an exit status are judged by their text.return failure_text_re.search(obs) is not Nonedef _cmd_text(tc: dict[str, Any]) -> str:return _command_text(tc)+ def _command_words(cmd: str) -> list[str]:+ # Skip a leading `cd <dir> &&` or `cd <dir>;` and split into whole words.+ while prefix := re.match(r"\s*cd\s+(?:\"[^\"]*\"|'[^']*'|[^\s;&|]+)\s*(?:&&|;)", cmd):+ cmd = cmd[prefix.end() :]+ return re.findall(r"[^\s;&|()<>'\"`]+", cmd)+def _commands_similar(cmd1: str, cmd2: str) -> bool:- if not cmd1 or not cmd2:+ words1, words2 = _command_words(cmd1), _command_words(cmd2)+ if not words1 or not words2:return False- base1 = cmd1.split(maxsplit=1)[0] if cmd1.split() else ""- base2 = cmd2.split(maxsplit=1)[0] if cmd2.split() else ""- return base1 == base2 or base1 in cmd2 or base2 in cmd1+ # Compare whole words, so `ls` does not match `skills/...`; a program+ # matches across paths and versions (python3 retries python).+ program1, program2 = (re.sub(r"[\d.]+$", "", words[0].rsplit("/", 1)[-1]) for words in (words1, words2))+ return (bool(program1) and program1 == program2) or words1[0] in words2 or words2[0] in words1def _is_skill_fault(error_obs: str) -> bool:obs_lower = error_obs.lower()return any(kw in obs_lower for kw in skill_fault_keywords)failed = [_is_failure(call) for _, call in exec_calls]- succeeded: list[bool] = []- for i, (_, call) in enumerate(exec_calls):- status = exit_status_re.search(str(call.get("observation", "")))- # A benign negative query is not a successful retry of an earlier error.- succeeded.append(not failed[i] and (status is None or int(status.group(1)) == 0))+ # A benign negative query is not a successful retry of an earlier error.+ succeeded = [not failed[i] and _exit_status(call) in (None, 0) for i, (_, call) in enumerate(exec_calls)]corrections: list[dict[str, Any]] = []failures: list[dict[str, Any]] = []diff --git a/src/skillevaluator/tier3/eval_core/codex_tool_call_normalizer.py b/src/skillevaluator/tier3/eval_core/codex_tool_call_normalizer.pyindex a8c1af9c..d406f431 100644--- a/src/skillevaluator/tier3/eval_core/codex_tool_call_normalizer.py+++ b/src/skillevaluator/tier3/eval_core/codex_tool_call_normalizer.py@@ -101,20 +101,34 @@ def normalized_tool_call_observation(step: dict[str, Any], tool_call: dict[str,)-def normalized_tool_call_is_error(step: dict[str, Any], tool_call: dict[str, Any]) -> bool:- """Preserve explicit failure metadata only for an attributable tool result."""+def normalized_tool_call_error_flag(step: dict[str, Any], tool_call: dict[str, Any]) -> bool | None:+ """Return an attributable tool result's explicit ``tool_result_is_error`` flag.++ ``True`` if any attributable result reports an error, ``False`` if they+ explicitly report success, and ``None`` when none carries the flag.+ """if tool_call.get("_atif_observation_status") not in (None, MAPPED_OUTER_EXEC_OBSERVATION):- return False+ return Nonetool_call_id = tool_call.get("tool_call_id")+ flag = Nonefor result in (step.get("observation") or {}).get("results") or []:source_call_id = result.get("source_call_id")# Unscoped observations cannot prove that this particular call failed.if not tool_call_id or source_call_id != tool_call_id:continueextra = result.get("extra") or {}- if isinstance(extra, dict) and extra.get("tool_result_is_error") is True:+ if not isinstance(extra, dict):+ continue+ if extra.get("tool_result_is_error") is True:return True- return False+ if extra.get("tool_result_is_error") is False:+ flag = False+ return flag+++def normalized_tool_call_is_error(step: dict[str, Any], tool_call: dict[str, Any]) -> bool:+ """Preserve explicit failure metadata only for an attributable tool result."""+ return normalized_tool_call_error_flag(step, tool_call) is Truedef normalized_tool_call_wrapper_observation(step: dict[str, Any], tool_call: dict[str, Any]) -> str:diff --git a/src/skillevaluator/tier3/harbor/templates/eval.py b/src/skillevaluator/tier3/harbor/templates/eval.pyindex 2fdcaa56..9b59041e 100644--- a/src/skillevaluator/tier3/harbor/templates/eval.py+++ b/src/skillevaluator/tier3/harbor/templates/eval.py@@ -77,6 +77,7 @@ try:atif_content_has_text,atif_content_text,iter_normalized_tool_calls,+ normalized_tool_call_error_flag,normalized_tool_call_is_error,normalized_tool_call_observation,normalized_tool_call_wrapper_observation,@@ -89,6 +90,7 @@ except ImportError: # pragma: no cover -- source-tree import onlyatif_content_has_text,atif_content_text,iter_normalized_tool_calls,+ normalized_tool_call_error_flag,normalized_tool_call_is_error,normalized_tool_call_observation,normalized_tool_call_wrapper_observation,@@ -471,8 +473,10 @@ def extract_tool_calls_as_dicts(traj):"action_input": tc.get("arguments") or {},"observation": _tool_call_observation(step, tc),}- if normalized_tool_call_is_error(step, tc):- call["is_error"] = True+ # Keep Harbor's explicit success flag too: Claude Code results that+ # exited 0 carry no exit status line.+ if (is_error := normalized_tool_call_error_flag(step, tc)) is not None:+ call["is_error"] = is_errorif status := tc.get("_atif_normalization_status"):call["normalization_status"] = statusif status := tc.get("_atif_observation_status"):@@ -8046,14 +8050,28 @@ def check_error_recovery(tool_calls, expected_script=None):r"\]?[ \t]*[=:]?[ \t]*(-?\d+)\]?[ \t]*\r?$",re.IGNORECASE | re.MULTILINE,)- # Match diagnostic forms, not prose such as "0 errors" or "error handling".+ # A type annotation such as `error: str | None = None` or `timeout: float = 30.0`+ # is displayed source, not a diagnostic.+ not_annotation = (+ r"(?![ \t]*[A-Za-z_][\w.]*(?:\[[^\]\n]*\])?"+ r"(?:[ \t]*\|[ \t]*[A-Za-z_][\w.]*(?:\[[^\]\n]*\])?)*[ \t]*[=|;,)])"+ )+ # Results without any exit status (Harbor's OpenCode trajectories keep only+ # the output) are judged by diagnostic forms, not prose such as "0 errors"+ # or "error handling", or a `--timeout:` option description.failure_text_re = re.compile(- r"^\s*(?:[\w.+/-]+:\s*)*(?:error|exception)\s*:"- r"|\btimeout\s*:(?!\s*\d)|^\s*(?:\w*(?:error|exception)|timeout|connectionrefused)\s*$"- r"|^\s*\w*(?:Error|Exception)\s*:|traceback\s*\(most recent call last\)"+ rf"^\s*(?:[\w.+/-]+:\s*)*(?:error|exception)\s*:{not_annotation}"+ rf"|(?<![-\w])timeout\s*:(?!\s*\d){not_annotation}"+ r"|^\s*(?:\w*(?:error|exception)|timeout|connectionrefused)\s*$"+ rf"|^\s*\w*(?:Error|Exception)\s*:{not_annotation}|traceback\s*\(most recent call last\)"r"|\bstatus\s*=\s*(?:failed|error)\b"r"|\b(?:command not found|no such file(?: or directory)?|permission denied)\b"- r"|\b(?:connection refused|connectionrefused|timed out)\b|:\s*not found\b",+ r"|\b(?:connection refused|connectionrefused|timed out)\b|:\s*not found\b"+ # Test and build failures (pytest, unittest, jest and go test summaries,+ # tsc, npm) and OpenCode's note for a command stopped at its timeout.+ r"|(?-i:^[ \t]*(?:---[ \t]+)?(?:FAIL(?:ED)?|ERROR)\b(?![ \t]*=))"+ r"|\b[1-9]\d* failed\b|\berror TS\d+\b|^npm (?:ERR!|error\b)"+ r"|\bbash tool terminated command after exceeding timeout\b",re.IGNORECASE | re.MULTILINE,)skill_fault_kw = [@@ -8067,22 +8085,67 @@ def check_error_recovery(tool_calls, expected_script=None):"modulenotfounderror",]+ quiet_probe_programs = {"grep", "egrep", "fgrep", "rg", "test", "[", "[[", "diff", "cmp", "which"}+ devnull_redirect_re = re.compile(+ r"(?<!\S)(?:[012]?>{1,2}|&>{1,2})[ \t]*/dev/null(?=$|[\s;&|])|(?<!\S)[12]?>&[12](?=$|[\s;&|])"+ )++ def _quiet_probe_program(words):+ while words and re.fullmatch(r"[A-Za-z_]\w*=.*", words[0], flags=re.DOTALL):+ words = words[1:] # VAR=value prefix+ if not words:+ return None+ program = words[0].rsplit("/", 1)[-1]+ if program in quiet_probe_programs:+ return program+ if program == "command" and words[1:2] in (["-v"], ["-V"]):+ return program+ if program == "git" and words[1:2] == ["diff"] and "--quiet" in words[2:]:+ return program+ return None+def _is_quiet_negative_probe(tc, observation):- # Exit 1 is a normal negative answer for these simple commands. Be- # conservative: keep diagnostics and compound shell commands as failures.+ # Exit 1 is a normal negative answer for these probes. Be conservative:+ # keep diagnostics and other compound shell commands as failures.command = _command_text(tc)if any(marker in command for marker in ("\n", "\r", "`", "$(")):return False- lexer = shlex.shlex(command, posix=True, punctuation_chars=True)+ lexer = shlex.shlex(devnull_redirect_re.sub(" ", command), posix=True, punctuation_chars=True)lexer.whitespace_split = Truelexer.commenters = ""try:words = list(lexer)except ValueError:return False- if not words or words[0].rsplit("/", 1)[-1] not in {"grep", "rg", "test", "[", "diff", "cmp"}:+ segments = [[]]+ separators = []+ for word in words:+ if word and all(char in ";&|<>()" for char in word):+ if word not in {"&&", ";", "|"} or not segments[-1]:+ return False+ separators.append(word)+ segments.append([])+ else:+ segments[-1].append(word)+ if not segments[-1]:return False- if any(word and all(char in ";&|<>()" for char in word) for word in words):+ # Skip leading `cd <dir>` and `pwd` commands joined by `&&` or `;` (a+ # failed cd prints a diagnostic) and a trailing `&& echo ...`, which+ # runs only when the probe succeeds.+ start, end = 0, len(segments) - 1+ while (+ start < end+ and separators[start] != "|"+ and (segments[start] == ["pwd"] or (segments[start][0] == "cd" and len(segments[start]) <= 2))+ ):+ start += 1+ while end > start and separators[end - 1] == "&&" and segments[end][0] in {"echo", "printf"}:+ end -= 1+ # A pipeline exits with its last command's status, so only that one must be a probe.+ if any(separator != "|" for separator in separators[start:end]):+ return False+ program = _quiet_probe_program(segments[end])+ if program is None:return False# Harbor's Codex unified-exec envelope is metadata, but everything# after Output: is command output, even if it resembles another header.@@ -8096,45 +8159,68 @@ def check_error_recovery(tool_calls, expected_script=None):flags=re.DOTALL,)if unified_output is not None:- return not unified_output.group(1).strip()- # Other converters emit status/exit_code lines without an envelope.- output = exit_status_re.sub("", observation)- output = re.sub(- r"^[ \t]*status=(?:completed|failed|error)[ \t]*\r?$",- "",- output,- flags=re.IGNORECASE | re.MULTILINE,+ output = unified_output.group(1)+ else:+ # Other converters emit status/exit_code lines without an envelope.+ output = exit_status_re.sub("", observation)+ output = re.sub(+ r"^[ \t]*status=(?:completed|failed|error)[ \t]*\r?$",+ "",+ output,+ flags=re.IGNORECASE | re.MULTILINE,+ )+ # Harbor appends this exact marker to Claude Code is_error results,+ # including ordinary negative probes. Preserve all other output lines.+ output = re.sub(r"^\[error\] tool reported failure\r?$", "", output, flags=re.MULTILINE)+ lines = [line.strip() for line in output.splitlines() if line.strip()]+ # Each skipped `pwd` printed one absolute directory line first.+ directories = sum(segment == ["pwd"] for segment in segments[:start])+ if all(line.startswith("/") for line in lines[:directories]):+ lines = lines[directories:]+ # `grep -c` prints a zero count (per file) when nothing matches.+ counts = program in {"grep", "egrep", "fgrep", "rg"} and any(+ word in {"--count", "--count-matches"} or re.fullmatch(r"-[A-Za-z]*c[A-Za-z]*", word)+ for word in segments[end])- # Harbor appends this exact marker to Claude Code is_error results,- # including ordinary negative probes. Preserve all other output lines.- output = re.sub(r"^\[error\] tool reported failure\r?$", "", output, flags=re.MULTILINE)- return not output.strip()+ return all(counts and re.fullmatch(r"(?:.*:)?0", line) for line in lines)++ def _exit_status(tc):+ # Harbor marks Claude Code results that exited 0 with tool_result_is_error+ # false and adds an exit status line only to failures.+ if tc.get("is_error") is False:+ return 0+ status = exit_status_re.search(str(tc.get("observation", "")))+ return int(status.group(1)) if status else Nonedef _is_failure(tc):obs = str(tc.get("observation", ""))- if status := exit_status_re.search(obs):- exit_code = int(status.group(1))- if exit_code == 1 and _is_quiet_negative_probe(tc, obs):- return False- return exit_code != 0+ exit_code = _exit_status(tc)+ if exit_code is not None:+ return exit_code != 0 and not (exit_code == 1 and _is_quiet_negative_probe(tc, obs))+ # Only results without an exit status are judged by their text.return failure_text_re.search(obs) is not Nonedef _cmd_text(tc):return _command_text(tc)+ def _command_words(cmd):+ # Skip a leading `cd <dir> &&` or `cd <dir>;` and split into whole words.+ while prefix := re.match(r"\s*cd\s+(?:\"[^\"]*\"|'[^']*'|[^\s;&|]+)\s*(?:&&|;)", cmd):+ cmd = cmd[prefix.end() :]+ return re.findall(r"[^\s;&|()<>'\"`]+", cmd)+def _cmds_similar(c1, c2):- if not c1 or not c2:+ w1, w2 = _command_words(c1), _command_words(c2)+ if not w1 or not w2:return False- b1 = c1.split()[0] if c1.split() else ""- b2 = c2.split()[0] if c2.split() else ""- return b1 == b2 or b1 in c2 or b2 in c1+ # Compare whole words, so `ls` does not match `skills/...`; a program+ # matches across paths and versions (python3 retries python).+ p1, p2 = (re.sub(r"[\d.]+$", "", words[0].rsplit("/", 1)[-1]) for words in (w1, w2))+ return (bool(p1) and p1 == p2) or w1[0] in w2 or w2[0] in w1failed = [_is_failure(call) for _, call in exec_calls]- succeeded = []- for i, (_, call) in enumerate(exec_calls):- status = exit_status_re.search(str(call.get("observation", "")))- # A benign negative query is not a successful retry of an earlier error.- succeeded.append(not failed[i] and (status is None or int(status.group(1)) == 0))+ # A benign negative query is not a successful retry of an earlier error.+ succeeded = [not failed[i] and _exit_status(call) in (None, 0) for i, (_, call) in enumerate(exec_calls)]corrections = []failures = []diff --git a/tests/tier3/test_error_recovery.py b/tests/tier3/test_error_recovery.pyindex 3dee3fca..5f642abe 100644--- a/tests/tier3/test_error_recovery.py+++ b/tests/tier3/test_error_recovery.py@@ -11,6 +11,7 @@ from __future__ import annotationsimport pytest+from skillevaluator.tier3.eval_core.atif_helpers import extract_tool_calls_as_dictsfrom skillevaluator.tier3.eval_core.checks import check_error_recoveryfrom skillevaluator.tier3.harbor.report import _collect_fail_reasonsfrom skillevaluator.tier3.harbor.templates import eval as eval_template@@ -516,3 +517,398 @@ def test_codex_output_does_not_strip_claude_marker(checker):result = checker([_call("rg TODO src", _codex_output(1, "[error] tool reported failure"))])assert result["passed"] is Falseassert result["score"] == 0.0+++ params=[+ (extract_tool_calls_as_dicts, check_error_recovery),+ (eval_template.extract_tool_calls_as_dicts, eval_template.check_error_recovery),+ ],+ ids=["host", "harbor-verifier"],+)+def trajectory_checker(request):+ """Extract tool calls from an ATIF trajectory and score them with one implementation."""+ extract, check = request.param++ def run(trajectory: dict) -> dict:+ return check(extract(trajectory))++ run.extract = extract+ return run+++def _claude_code_trajectory(*commands: tuple[str, str, int]) -> dict:+ """Build Harbor Claude Code ATIF steps for (command, output, exit code) tuples."""+ from harbor.agents.installed.claude_code import ClaudeCode++ steps = []+ for index, (command, output, exit_code) in enumerate(commands):+ if exit_code:+ content = f"Exit code {exit_code}\n{output}".rstrip()+ block = {"type": "tool_result", "content": content, "is_error": True}+ observation, metadata = ClaudeCode._format_tool_result(block, f"Error: {content}")+ else:+ block = {"type": "tool_result", "content": output, "is_error": False}+ tool_use_result = {"stdout": output, "stderr": "", "interrupted": False, "isImage": False}+ observation, metadata = ClaudeCode._format_tool_result(block, tool_use_result)+ call_id = f"toolu_{index}"+ steps.append(+ {+ "step_id": index + 1,+ "source": "agent",+ "message": "",+ "tool_calls": [{"tool_call_id": call_id, "function_name": "Bash", "arguments": {"command": command}}],+ "observation": {+ "results": [+ {+ "source_call_id": call_id,+ "content": observation,+ # Harbor stores the result's is_error flag here.+ "extra": {"tool_result_metadata": metadata, "tool_result_is_error": block["is_error"]},+ }+ ]+ },+ }+ )+ return {"schema_version": "ATIF-v1.7", "agent": {"name": "claude-code"}, "steps": steps}+++ "command,output",+ [+ (+ 'python3 -m py_compile repository.py && echo OK; python3 -c "import sqlalchemy" 2>&1 | tail -1',+ "OK\nModuleNotFoundError: No module named 'sqlalchemy'",+ ),+ ("python scripts/fetch.py --url https://example.com/missing", '{"error": "HTTP 404: Not Found"}'),+ ("cat scripts/call_api.py", 'result["error"] = f"Request timed out after {timeout} seconds"'),+ ("cat scripts/parse.py", " Raises:\n ValueError: If the spec is malformed."),+ ("ls missing; ls skills", "ls: missing: No such file or directory\nreport-builder"),+ ("cat build.log", "step 3\nexit code 1\nretrying"),+ ],+)+def test_claude_code_success_flag_skips_text_check(trajectory_checker, command, output):+ trajectory = _claude_code_trajectory((command, output, 0))+ assert trajectory_checker.extract(trajectory)[0]["is_error"] is False+ result = trajectory_checker(trajectory)+ assert result["score"] == 1.0+ assert result["passed"] is True+ assert result["failures"] == []+++def test_claude_code_success_flag_counts_as_recovery(trajectory_checker):+ trajectory = _claude_code_trajectory(+ ("python scripts/build.py", "Traceback (most recent call last):\nKeyError: 'x'", 1),+ ("python scripts/build.py --in data.csv", "lint step failed, skipped\nexit code 1\nwrote report.html", 0),+ )+ result = trajectory_checker(trajectory)+ assert result["score"] == 0.9+ assert result["passed"] is True+ assert len(result["corrections"]) == 1+ assert result["unrecovered_failures"] == []+++ "command,output,exit_code",+ [+ ("pytest -q", "F.\nFAILED tests/test_a.py::test_a - assert 1 == 2\n1 failed, 1 passed", 1),+ ("python run.py", "Traceback (most recent call last):\nKeyError: 'x'", 1),+ ("npx tsc --noEmit", "src/a.ts(3,7): error TS2322: Type 'string' is not assignable to type 'number'.", 2),+ ("python run.py", "", 1),+ ],+)+def test_claude_code_error_flag_still_fails(trajectory_checker, command, output, exit_code):+ trajectory = _claude_code_trajectory((command, output, exit_code))+ assert trajectory_checker.extract(trajectory)[0]["is_error"] is True+ result = trajectory_checker(trajectory)+ assert result["score"] == 0.0+ assert result["passed"] is False+ assert len(result["unrecovered_failures"]) == 1+++ "content,score",+ [+ ("Command timed out after 2m 0.0s", 0.0),+ ("Permission to use Bash with command rm -rf build has been denied.", 1.0),+ ],+)+def test_claude_code_error_flag_without_exit_status_uses_text(trajectory_checker, content, score):+ trajectory = _claude_code_trajectory(("rm -rf build", "", 0))+ result_entry = trajectory["steps"][0]["observation"]["results"][0]+ result_entry["content"] = f"{content}\n\n[error] tool reported failure"+ result_entry["extra"] = {"tool_result_is_error": True}+ assert trajectory_checker.extract(trajectory)[0]["is_error"] is True+ assert trajectory_checker(trajectory)["score"] == score+++def test_unflagged_result_keeps_text_check(trajectory_checker):+ trajectory = _claude_code_trajectory(("python run.py", "", 0))+ result_entry = trajectory["steps"][0]["observation"]["results"][0]+ result_entry["content"] = "Traceback (most recent call last):\nKeyError: 'x'"+ del result_entry["extra"]+ assert "is_error" not in trajectory_checker.extract(trajectory)[0]+ result = trajectory_checker(trajectory)+ assert result["score"] == 0.0+ assert result["passed"] is False+++def _opencode_trajectory(tmp_path, *commands: tuple[str, str, int | None]) -> dict:+ """Convert OpenCode bash events with Harbor's converter, which keeps only the output."""+ from harbor.agents.installed.opencode import OpenCode++ events = []+ for index, (command, output, exit_code) in enumerate(commands):+ state = {+ "status": "completed",+ "input": {"command": command, "description": "run"},+ "output": output,+ "metadata": {"exit": exit_code, "output": output, "description": "run"},+ }+ events.extend(+ [+ {"type": "step_start", "timestamp": 1000 + index, "sessionID": "s1", "part": {}},+ {+ "type": "tool_use",+ "timestamp": 1000 + index,+ "sessionID": "s1",+ "part": {"type": "tool", "tool": "bash", "callID": f"call_{index}", "state": state},+ },+ {+ "type": "step_finish",+ "timestamp": 1000 + index,+ "sessionID": "s1",+ "part": {"tokens": {}, "cost": 0},+ },+ ]+ )+ agent = OpenCode(logs_dir=tmp_path, model_name="openai/gpt-5")+ return agent._convert_events_to_trajectory(events).to_json_dict()+++OPENCODE_PYTEST_FAIL = (+ "F.\n=================================== FAILURES ===================================\n"+ "___ test_total ___\n\n def test_total():\n> assert total([1, 2]) == 4\nE assert 3 == 4\n\n"+ "tests/test_calc.py:4: AssertionError\n"+ "=========================== short test summary info ============================\n"+ "FAILED tests/test_calc.py::test_total - assert 3 == 4\n1 failed, 1 passed in 0.03s\n"+)+OPENCODE_FAILURES = {+ "pytest": ("pytest -q", OPENCODE_PYTEST_FAIL, 1),+ "pytest-collection": (+ "pytest -q",+ "==== short test summary info ====\nERROR tests/test_calc.py\n!!! Interrupted: 1 error during collection !!!\n",+ 2,+ ),+ "unittest": ("python -m unittest", "F\n----\nRan 1 test in 0.001s\n\nFAILED (failures=1)\n", 1),+ "tsc": (+ "npx tsc --noEmit",+ "src/a.ts(3,7): error TS2322: Type 'string' is not assignable to type 'number'.\n\nFound 1 error in src/a.ts:3\n",+ 2,+ ),+ "jest": (+ "npm test",+ "> [email protected] test\n> jest\n\n FAIL src/sum.test.js\n x adds 1 + 2 to equal 3 (3 ms)\n\nTests: 1 failed, 1 total\n",+ 1,+ ),+ "npm-err": ("npm install", "npm ERR! code E404\nnpm ERR! 404 Not Found - GET https://registry.npmjs.org/nope\n", 1),+ "npm-error": ("npm run build", "npm error Lifecycle script `build` failed with error:\nnpm error code 1\n", 1),+ "go-test": (+ "go test ./...",+ "--- FAIL: TestTotal (0.00s)\n calc_test.go:8: got 3, want 4\nFAIL\nFAIL\tdemo\t0.01s\n",+ 1,+ ),+ "timeout": (+ "python long_job.py",+ "partial output\n\n<bash_metadata>\nbash tool terminated command after exceeding timeout 120000 ms\n</bash_metadata>",+ None,+ ),+}++[email protected]("name", sorted(OPENCODE_FAILURES))+def test_opencode_unrecovered_test_and_build_failures(trajectory_checker, tmp_path, name):+ trajectory = _opencode_trajectory(tmp_path, OPENCODE_FAILURES[name])+ assert "is_error" not in trajectory_checker.extract(trajectory)[0]+ result = trajectory_checker(trajectory)+ assert result["score"] == 0.0+ assert result["passed"] is False+ assert result["first_attempt_clean"] is False+ assert len(result["unrecovered_failures"]) == 1++[email protected]("name", ["pytest", "tsc"])+def test_opencode_recovered_test_and_build_failures(trajectory_checker, tmp_path, name):+ failure = OPENCODE_FAILURES[name]+ trajectory = _opencode_trajectory(tmp_path, failure, (failure[0], "..\n2 passed in 0.02s\n", 0))+ result = trajectory_checker(trajectory)+ assert result["score"] == 0.9+ assert result["passed"] is True+ assert result["first_attempt_clean"] is False+ assert len(result["corrections"]) == 1+++ "output",+ [+ "..\n2 passed in 0.02s\n",+ "Tests: 0 failed, 4 passed, 4 total\n",+ "test result: ok. 3 passed; 0 failed; 0 ignored\n",+ "FAIL_FAST = True\nFAILED = 'failed'\nERROR = 2\n",+ "2 passed, 1 xfailed in 0.04s\n",+ ],+)+def test_opencode_passing_test_output_is_not_failure(trajectory_checker, tmp_path, output):+ result = trajectory_checker(_opencode_trajectory(tmp_path, ("pytest -q", output, 0)))+ assert result["score"] == 1.0+ assert result["passed"] is True+ assert result["failures"] == []+++ "observation",+ [+ 'def call_api(url: str, method: str = "GET", timeout: int = 30) -> dict:',+ " Args:\n --timeout: Request timeout in seconds.",+ "class Settings(BaseModel):\n retries: int = 3\n timeout: float = 30.0",+ "@dataclass\nclass Result:\n ok: bool\n error: str | None = None",+ "export interface Result {\n ok: boolean;\n error: string | null;\n}",+ "def run(\n cmd: list[str],\n timeout: int,\n last_error: Optional[str] = None,\n):",+ ],+)+def test_displayed_annotations_without_status_are_not_failures(checker, observation):+ result = checker([_call("cat scripts/call_api.py", observation)])+ assert result["score"] == 1.0+ assert result["passed"] is True+ assert result["failures"] == []+++ "observation",+ [+ "timeout: invalid time interval 'abc'",+ "error: externally-managed-environment",+ "error: unrecognized arguments: --bogus",+ "RuntimeError: boom",+ "ValueError: invalid literal for int() with base 10: 'x'",+ "requests.exceptions.Timeout: HTTPSConnectionPool(host='api.example.com', port=443)",+ "Error: Cannot find module 'jest'",+ ],+)+def test_diagnostics_without_status_still_fail(checker, observation):+ result = checker([_call("python run.py", observation)])+ assert result["score"] == 0.0+ assert result["passed"] is False+++def _exit_one_observations(output: str, exit_code: int = 1) -> list[str]:+ """The same failed command as a plain status line, a Harbor Claude Code result and a Codex result."""+ from harbor.agents.installed.claude_code import ClaudeCode++ content = f"Exit code {exit_code}\n{output}".rstrip()+ claude_code, _ = ClaudeCode._format_tool_result(+ {"type": "tool_result", "content": content, "is_error": True}, f"Error: {content}"+ )+ return [content, claude_code, _codex_output(exit_code, output)]+++ "command,output",+ [+ ("grep -c TODO a.py", "0"),+ ("grep -rc TODO src", "src/a.py:0\nsrc/b.py:0"),+ ("rg --count TODO src", ""),+ ("grep -rn TODO src 2>/dev/null", ""),+ ("grep -q foo a.txt >/dev/null 2>&1", ""),+ ("rg TODO src &>/dev/null", ""),+ ("cd /app && grep -rn TODO src", ""),+ ('cd "my app"; rg TODO', ""),+ ("pwd && rg --files -g AGENTS.md .", "/workspace"),+ ("cd /app && pwd; grep -rn TODO src", "/app"),+ ("LC_ALL=C grep -q foo a.txt", ""),+ ("ls | grep foo", ""),+ ("ls src | grep -c test_", "0"),+ ("test -f out.json && echo exists", ""),+ ("[[ -f out.json ]]", ""),+ ("command -v jq", ""),+ ("which jq", ""),+ ("git diff --quiet", ""),+ ("git diff --quiet -- src/app.py", ""),+ ],+)+def test_quiet_probe_variants_are_not_failures(checker, command, output):+ for observation in _exit_one_observations(output):+ result = checker([_call(command, observation)])+ assert result["score"] == 1.0, observation+ assert result["passed"] is True+ assert result["failures"] == []+++ "command,output,exit_code",+ [+ ("grep TODO a.py", "0", 1),+ ("grep -c TODO a.py", "0\ngrep: b.py: Permission denied", 1),+ ("grep -c TODO a.py", "3", 1),+ ("grep -rn TODO src 2>/dev/null", "", 2),+ ("cd /app && rg TODO src", "rg: src: No such file or directory", 1),+ ("cd /app && python run.py", "", 1),+ ("cd /app && grep -rn TODO src; python run.py", "", 1),+ ("pwd && rg --files -g AGENTS.md .", "/workspace\nrg: ./private: Permission denied", 1),+ ("pwd; python run.py", "/workspace", 1),+ ("ls | python run.py", "", 1),+ ("test -f out.json && python run.py", "", 1),+ ("pytest -q 2>/dev/null", "", 1),+ ("command jq", "", 1),+ ("which jq", "which: no jq in (/usr/bin:/bin)", 1),+ ("git diff", "", 1),+ ("git diff --exit-code", "diff --git a/app.py b/app.py", 1),+ ("git status --short", "fatal: not a git repository (or any of the parent directories): .git", 128),+ ],+)+def test_quiet_probe_variants_keep_real_failures(checker, command, output, exit_code):+ for observation in _exit_one_observations(output, exit_code):+ result = checker([_call(command, observation)])+ assert result["score"] == 0.0, observation+ assert result["passed"] is False+ assert len(result["unrecovered_failures"]) == 1+++ "command,retry",+ [+ ("python skills/report-builder/scripts/build.py --in data.csv", "ls /tmp"),+ ("cd /app && pytest -q", "cd /app && git status --short"),+ ("cd /app; python run.py", "cd /app; ls"),+ ("pytest tests/test_api.py", "cat pytest.ini"),+ ],+)+def test_unrelated_command_in_window_does_not_recover(checker, command, retry):+ calls = [+ _call(command, "Traceback (most recent call last):\nKeyError: 'x'\nexit code 1"),+ _call(retry, _codex_output(0, "skills")),+ ]+ result = checker(calls)+ assert result["score"] == 0.0+ assert result["passed"] is False+ assert result["corrections"] == []+ assert [failure["failed_cmd"] for failure in result["unrecovered_failures"]] == [command]+++ "command,retry",+ [+ ("cd /app && pytest -q", "pytest -q tests/test_api.py"),+ ("pytest -q", "cd /app && pytest -q"),+ ("python run.py", "python3 run.py --verbose"),+ ("/usr/bin/python3 run.py", "python run.py"),+ ("pytest -q", "python -m pytest -q"),+ ("bash scripts/build.sh", "scripts/build.sh --clean"),+ ],+)+def test_related_command_in_window_recovers(checker, command, retry):+ result = checker([_call(command, "exit code 1"), _call(retry, _codex_output(0, "done"))])+ assert result["score"] == 0.9+ assert result["passed"] is True+ assert len(result["corrections"]) == 1+ assert result["unrecovered_failures"] == []