Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier3): score unrecovered failures and align recovery verdicts (#217)
Maverick697bc441fe07
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c25a5024..4b9a9ba8 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -39,14 +39,15 @@ All notable changes to SkillEvaluator are documented in this file.### Fixed-- Catalog validation now includes nested and lowercase skill manifests, skips- generated directories, and stops at skill boundaries to avoid validating- bundled templates as separate skills. External linked directories containing- skills and unreadable directories appear as failures without stopping other- skills; links back into the catalog are ignored. Linked manifests retain- direct-target behavior, plugin discovery errors produce findings, and reports- and external Tier 3 results preserve catalog-relative paths for skills with- the same basename.+- Tier 3 error recovery now scores unrecovered command failures as zero and+ includes them in fault counts and failure verdicts/report reasons.+ Host and bundled verifier detection agree and prefer explicit exit status,+ avoiding penalties for successful output such as `0 errors`, Python exception+ handlers, and numeric timeout configuration. Printed source text containing+ exit codes or error strings is not treated as execution status; quiet exit-1+ search and file-test probes remain clean, including Codex unified-exec+ observations. Beyond the original five-call retry window, only a successful+ repeat of the same command and arguments counts as recovery.- Changing finding severities through a policy no longer clears an independentquality-score threshold failure. A score below `--min-score` still fails,and its reason appears in the terminal, HTML, benchmark, and JSON reports.@@ -207,17 +208,13 @@ All notable changes to SkillEvaluator are documented in this file.### Fixed-- Tier 3 reports preserve judge-model exclusions when calculating dimension and- overall lift. Native multi-step results also retain the judge provenance needed- for those comparisons, so fallback judges cannot turn a regression into an- apparent improvement. Judge-free cases retain valid lift, bounded report loads- use the collector's recorded comparisons, and FINAL-strategy trials use only- the final step's judges. The CLI displays unavailable comparisons as `n/a`.-- The LLM judge treats a schema option reported as `unavailable` as a schema- rejection and retries without it, so OpenAI-compatible endpoints such as- DeepSeek, which answer `response_format: json_schema` with HTTP 400 and- "This response_format type is unavailable now", can score Tier 3 instead- of failing every judge call (fixes #207).+- Tier 3 error recovery now scores unrecovered command failures as zero and+ includes them in fault counts and failure verdicts/report reasons.+ Host and bundled verifier detection agree and prefer explicit exit status,+ avoiding penalties for successful output such as `0 errors`, Python exception+ handlers, and numeric timeout configuration.++- Tier 3 reports an inconclusive result when every observed positive with-skilltrial has zero activation, and omits skill-lift comparisons for that agent.Activation counts are reported separately; negative controls and missingdiff --git a/src/skillevaluator/tier3/eval_core/checks.py b/src/skillevaluator/tier3/eval_core/checks.pyindex 8460acee..0c499b6d 100644--- a/src/skillevaluator/tier3/eval_core/checks.py+++ b/src/skillevaluator/tier3/eval_core/checks.py@@ -5238,23 +5238,23 @@ def check_error_recovery("agent_faults": 0,}- error_keywords = [- "error",- "traceback",- "exception",- "status=failed",- "status=error",- "not found",- "command not found",- "permission denied",- "no such file",- "filenotfounderror",- "modulenotfounderror",- "connectionrefused",- "timeout",- ]- # Match exit_code=N / "exit code N" for any nonzero N (not just 1/2).- nonzero_exit_re = re.compile(r"(?:exit_code|exit\s+code)\s*[=:]?\s*(?!0\b)(\d+)", re.IGNORECASE)+ # Only complete status lines are execution evidence; source code and prose+ # may mention exit_code=1 without describing this command's outcome.+ exit_status_re = re.compile(+ r"^[ \t]*\[?(?:exit_code|exit[ \t]+code|process[ \t]+exited[ \t]+with[ \t]+code)"+ r"\]?[ \t]*[=:]?[ \t]*(-?\d+)\]?[ \t]*\r?$",+ re.IGNORECASE | re.MULTILINE,+ )+ # Match diagnostic forms, not prose such as "0 errors" or "error handling".+ failure_text_re = re.compile(+ r"^\s*(?:[\w.+/-]+:\s*)*(?:error|exception)\s*:"+ r"|\btimeout\s*:(?!\s*\d)|^\s*(?:\w*(?:error|exception)|timeout|connectionrefused)\s*$"+ r"|^\s*\w*(?:Error|Exception)\s*:|traceback\s*\(most recent call last\)"+ r"|\bstatus\s*=\s*(?:failed|error)\b"+ r"|\b(?:command not found|no such file(?: or directory)?|permission denied)\b"+ r"|\b(?:connection refused|connectionrefused|timed out)\b|:\s*not found\b",+ re.IGNORECASE | re.MULTILINE,+ )skill_fault_keywords = ["no such file","filenotfounderror",@@ -5266,11 +5266,57 @@ def check_error_recovery("modulenotfounderror",]+ def _is_quiet_negative_probe(tc: dict[str, Any], observation: str) -> bool:+ # Exit 1 is a normal negative answer for these simple commands. Be+ # conservative: keep diagnostics and compound shell commands as failures.+ command = _command_text(tc)+ if any(marker in command for marker in ("\n", "\r", "`", "$(")):+ return False+ lexer = shlex.shlex(command, posix=True, punctuation_chars=True)+ lexer.whitespace_split = True+ lexer.commenters = ""+ try:+ words = list(lexer)+ except ValueError:+ return False+ if not words or words[0].rsplit("/", 1)[-1] not in {"grep", "rg", "test", "[", "diff", "cmp"}:+ return False+ if any(word and all(char in ";&|<>()" for char in word) for word in words):+ return False+ # Harbor's Codex unified-exec envelope is metadata, but everything+ # after Output: is command output, even if it resembles another header.+ unified_output = re.fullmatch(+ r"(?:Chunk ID: [0-9a-fA-F]{6}\r?\n)?"+ r"Wall time: \d+(?:\.\d+)? seconds\r?\n"+ r"Process exited with code -?\d+\r?\n"+ r"(?:Original token count: \d+\r?\n)?"+ r"Output:(?:\r?\n|\Z)(.*)",+ observation,+ flags=re.DOTALL,+ )+ if unified_output is not None:+ return not unified_output.group(1).strip()+ # Other converters emit status/exit_code lines without an envelope.+ output = exit_status_re.sub("", observation)+ output = re.sub(+ r"^[ \t]*status=(?:completed|failed|error)[ \t]*\r?$",+ "",+ output,+ flags=re.IGNORECASE | re.MULTILINE,+ )+ # Harbor appends this exact marker to Claude Code is_error results,+ # including ordinary negative probes. Preserve all other output lines.+ output = re.sub(r"^\[error\] tool reported failure\r?$", "", output, flags=re.MULTILINE)+ return not output.strip()+def _is_failure(tc: dict[str, Any]) -> bool:- obs = str(tc.get("observation", "")).lower()- if nonzero_exit_re.search(obs):- return True- return any(kw in obs for kw in error_keywords)+ obs = str(tc.get("observation", ""))+ if status := exit_status_re.search(obs):+ exit_code = int(status.group(1))+ if exit_code == 1 and _is_quiet_negative_probe(tc, obs):+ return False+ return exit_code != 0+ return failure_text_re.search(obs) is not Nonedef _cmd_text(tc: dict[str, Any]) -> str:return _command_text(tc)@@ -5286,6 +5332,13 @@ def check_error_recovery(obs_lower = error_obs.lower()return any(kw in obs_lower for kw in skill_fault_keywords)+ failed = [_is_failure(call) for _, call in exec_calls]+ succeeded: list[bool] = []+ for i, (_, call) in enumerate(exec_calls):+ status = exit_status_re.search(str(call.get("observation", "")))+ # A benign negative query is not a successful retry of an earlier error.+ succeeded.append(not failed[i] and (status is None or int(status.group(1)) == 0))+corrections: list[dict[str, Any]] = []failures: list[dict[str, Any]] = []seen_fail_indices: set[int] = set()@@ -5293,7 +5346,7 @@ def check_error_recovery(for i, (orig_idx, call) in enumerate(exec_calls):if orig_idx in seen_fail_indices:continue- if not _is_failure(call):+ if not failed[i]:continuecmd = _cmd_text(call)@@ -5310,10 +5363,17 @@ def check_error_recovery(}failures.append(failure)- for j in range(i + 1, min(i + 6, len(exec_calls))):+ for j in range(i + 1, len(exec_calls)):retry_idx, retry_call = exec_calls[j]retry_cmd = _cmd_text(retry_call)- if _commands_similar(cmd, retry_cmd) and not _is_failure(retry_call):+ # Keep the original local correction heuristic, but require the+ # same command and arguments for retries beyond the five-call window.+ matches = (+ _commands_similar(cmd, retry_cmd)+ if j - i <= 5+ else bool(cmd.split()) and cmd.split() == retry_cmd.split()+ )+ if matches and succeeded[j]:corrections.append({"failed_cmd": cmd[:200],@@ -5328,8 +5388,8 @@ def check_error_recovery(breakfirst_attempt_clean = len(failures) == 0- skill_faults = sum(1 for c in corrections if c["fault"] == "skill")- agent_faults = sum(1 for c in corrections if c["fault"] == "agent")+ skill_faults = sum(1 for f in failures if f["fault"] == "skill")+ agent_faults = sum(1 for f in failures if f["fault"] == "agent")unrecovered_failures = [f for f in failures if not f["recovered"]]if first_attempt_clean:@@ -5347,7 +5407,7 @@ def check_error_recovery(reason = f"{agent_faults} agent error(s), no skill defects"return {- "passed": first_attempt_clean or skill_faults == 0,+ "passed": first_attempt_clean or (skill_faults == 0 and not unrecovered_failures),"score": round(score, 4),"reason": reason,"first_attempt_clean": first_attempt_clean,diff --git a/src/skillevaluator/tier3/harbor/templates/eval.py b/src/skillevaluator/tier3/harbor/templates/eval.pyindex 00d67cc2..2fdcaa56 100644--- a/src/skillevaluator/tier3/harbor/templates/eval.py+++ b/src/skillevaluator/tier3/harbor/templates/eval.py@@ -8039,21 +8039,23 @@ def check_error_recovery(tool_calls, expected_script=None):"agent_faults": 0,}- error_kw = [- "error",- "traceback",- "exception",- "status=failed",- "status=error",- "not found",- "command not found",- "permission denied",- "no such file",- "filenotfounderror",- "modulenotfounderror",- ]- # Match exit_code=N / "exit code N" for any nonzero N (not just 1/2).- nonzero_exit_re = re.compile(r"(?:exit_code|exit\s+code)\s*[=:]?\s*(?!0\b)(\d+)", re.IGNORECASE)+ # Only complete status lines are execution evidence; source code and prose+ # may mention exit_code=1 without describing this command's outcome.+ exit_status_re = re.compile(+ r"^[ \t]*\[?(?:exit_code|exit[ \t]+code|process[ \t]+exited[ \t]+with[ \t]+code)"+ r"\]?[ \t]*[=:]?[ \t]*(-?\d+)\]?[ \t]*\r?$",+ re.IGNORECASE | re.MULTILINE,+ )+ # Match diagnostic forms, not prose such as "0 errors" or "error handling".+ failure_text_re = re.compile(+ r"^\s*(?:[\w.+/-]+:\s*)*(?:error|exception)\s*:"+ r"|\btimeout\s*:(?!\s*\d)|^\s*(?:\w*(?:error|exception)|timeout|connectionrefused)\s*$"+ r"|^\s*\w*(?:Error|Exception)\s*:|traceback\s*\(most recent call last\)"+ r"|\bstatus\s*=\s*(?:failed|error)\b"+ r"|\b(?:command not found|no such file(?: or directory)?|permission denied)\b"+ r"|\b(?:connection refused|connectionrefused|timed out)\b|:\s*not found\b",+ re.IGNORECASE | re.MULTILINE,+ )skill_fault_kw = ["no such file","filenotfounderror",@@ -8065,11 +8067,57 @@ def check_error_recovery(tool_calls, expected_script=None):"modulenotfounderror",]+ def _is_quiet_negative_probe(tc, observation):+ # Exit 1 is a normal negative answer for these simple commands. Be+ # conservative: keep diagnostics and compound shell commands as failures.+ command = _command_text(tc)+ if any(marker in command for marker in ("\n", "\r", "`", "$(")):+ return False+ lexer = shlex.shlex(command, posix=True, punctuation_chars=True)+ lexer.whitespace_split = True+ lexer.commenters = ""+ try:+ words = list(lexer)+ except ValueError:+ return False+ if not words or words[0].rsplit("/", 1)[-1] not in {"grep", "rg", "test", "[", "diff", "cmp"}:+ return False+ if any(word and all(char in ";&|<>()" for char in word) for word in words):+ return False+ # Harbor's Codex unified-exec envelope is metadata, but everything+ # after Output: is command output, even if it resembles another header.+ unified_output = re.fullmatch(+ r"(?:Chunk ID: [0-9a-fA-F]{6}\r?\n)?"+ r"Wall time: \d+(?:\.\d+)? seconds\r?\n"+ r"Process exited with code -?\d+\r?\n"+ r"(?:Original token count: \d+\r?\n)?"+ r"Output:(?:\r?\n|\Z)(.*)",+ observation,+ flags=re.DOTALL,+ )+ if unified_output is not None:+ return not unified_output.group(1).strip()+ # Other converters emit status/exit_code lines without an envelope.+ output = exit_status_re.sub("", observation)+ output = re.sub(+ r"^[ \t]*status=(?:completed|failed|error)[ \t]*\r?$",+ "",+ output,+ flags=re.IGNORECASE | re.MULTILINE,+ )+ # Harbor appends this exact marker to Claude Code is_error results,+ # including ordinary negative probes. Preserve all other output lines.+ output = re.sub(r"^\[error\] tool reported failure\r?$", "", output, flags=re.MULTILINE)+ return not output.strip()+def _is_failure(tc):- obs = str(tc.get("observation", "")).lower()- if nonzero_exit_re.search(obs):- return True- return any(kw in obs for kw in error_kw)+ obs = str(tc.get("observation", ""))+ if status := exit_status_re.search(obs):+ exit_code = int(status.group(1))+ if exit_code == 1 and _is_quiet_negative_probe(tc, obs):+ return False+ return exit_code != 0+ return failure_text_re.search(obs) is not Nonedef _cmd_text(tc):return _command_text(tc)@@ -8081,12 +8129,19 @@ def check_error_recovery(tool_calls, expected_script=None):b2 = c2.split()[0] if c2.split() else ""return b1 == b2 or b1 in c2 or b2 in c1+ failed = [_is_failure(call) for _, call in exec_calls]+ succeeded = []+ for i, (_, call) in enumerate(exec_calls):+ status = exit_status_re.search(str(call.get("observation", "")))+ # A benign negative query is not a successful retry of an earlier error.+ succeeded.append(not failed[i] and (status is None or int(status.group(1)) == 0))+corrections = []failures = []seen = set()for i, (orig_idx, call) in enumerate(exec_calls):- if orig_idx in seen or not _is_failure(call):+ if orig_idx in seen or not failed[i]:continuecmd = _cmd_text(call)obs = str(call.get("observation", ""))@@ -8101,10 +8156,17 @@ def check_error_recovery(tool_calls, expected_script=None):"recovered": False,}failures.append(failure)- for j in range(i + 1, min(i + 6, len(exec_calls))):+ for j in range(i + 1, len(exec_calls)):retry_idx, retry_call = exec_calls[j]retry_cmd = _cmd_text(retry_call)- if _cmds_similar(cmd, retry_cmd) and not _is_failure(retry_call):+ # Keep the original local correction heuristic, but require the+ # same command and arguments for retries beyond the five-call window.+ matches = (+ _cmds_similar(cmd, retry_cmd)+ if j - i <= 5+ else bool(cmd.split()) and cmd.split() == retry_cmd.split()+ )+ if matches and succeeded[j]:corrections.append({"failed_cmd": cmd[:200],@@ -8119,8 +8181,8 @@ def check_error_recovery(tool_calls, expected_script=None):breakfirst_attempt_clean = len(failures) == 0- skill_faults = sum(1 for c in corrections if c["fault"] == "skill")- agent_faults = sum(1 for c in corrections if c["fault"] == "agent")+ skill_faults = sum(1 for f in failures if f["fault"] == "skill")+ agent_faults = sum(1 for f in failures if f["fault"] == "agent")unrecovered_failures = [f for f in failures if not f["recovered"]]if first_attempt_clean:@@ -8137,7 +8199,7 @@ def check_error_recovery(tool_calls, expected_script=None):reason = f"{agent_faults} agent error(s), no skill defects"return {- "passed": first_attempt_clean or skill_faults == 0,+ "passed": first_attempt_clean or (skill_faults == 0 and not unrecovered_failures),"score": round(score, 4),"reason": reason,"first_attempt_clean": first_attempt_clean,diff --git a/tests/tier3/test_error_recovery.py b/tests/tier3/test_error_recovery.pyindex f24dbe89..3dee3fca 100644--- a/tests/tier3/test_error_recovery.py+++ b/tests/tier3/test_error_recovery.py@@ -9,18 +9,41 @@ command must NOT be scored as first-attempt clean.from __future__ import annotations+import pytest+from skillevaluator.tier3.eval_core.checks import check_error_recovery+from skillevaluator.tier3.harbor.report import _collect_fail_reasons+from skillevaluator.tier3.harbor.templates import eval as eval_template++[email protected](params=[check_error_recovery, eval_template.check_error_recovery], ids=["host", "harbor-verifier"])+def checker(request):+ def check(*args, **kwargs):+ result = request.param(*args, **kwargs)+ peer = eval_template.check_error_recovery if request.param is check_error_recovery else check_error_recovery+ assert result == peer(*args, **kwargs)+ return result++ return checkdef _call(cmd: str, observation: str) -> dict:return {"action": "bash", "action_input": {"command": cmd}, "observation": observation}-def test_failure_with_no_retry_is_not_first_attempt_clean():- """A failure that is never retried must not score 1.0 / passed-clean."""- result = check_error_recovery(- [_call("pytest tests/test_api.py -x", "ERROR: test_create_user failed\nexit code 1")]+def _codex_output(exit_code: int, output: str = "") -> str:+ return (+ "Chunk ID: a1b2c3\n"+ "Wall time: 0.125 seconds\n"+ f"Process exited with code {exit_code}\n"+ f"Original token count: {len(output.split())}\n"+ f"Output:\n{output}")+++def test_failure_with_no_retry_is_not_first_attempt_clean(checker):+ """A failure that is never retried must not score 1.0 / passed-clean."""+ result = checker([_call("pytest tests/test_api.py -x", "ERROR: test_create_user failed\nexit code 1")])assert result["first_attempt_clean"] is Falseassert result["score"] == 0.0assert len(result["unrecovered_failures"]) == 1@@ -30,9 +53,9 @@ def test_failure_with_no_retry_is_not_first_attempt_clean():assert "pytest tests/test_api.py -x" in result["reason"]-def test_failed_retries_are_not_first_attempt_clean():+def test_failed_retries_are_not_first_attempt_clean(checker):"""Repeated failures of a similar command are all recorded as unrecovered."""- result = check_error_recovery(+ result = checker([_call("pytest tests/test_api.py -x", "ERROR: test_create_user failed\nexit code 1"),_call("pytest tests/test_api.py -x -v", "ERROR: test_create_user failed\nexit code 1"),@@ -45,9 +68,9 @@ def test_failed_retries_are_not_first_attempt_clean():assert result["corrections"] == []-def test_recovered_failure_behavior_unchanged():+def test_recovered_failure_behavior_unchanged(checker):"""A failure followed by a successful similar command keeps the old scoring."""- result = check_error_recovery(+ result = checker([_call("pytest tests/test_api.py", "pytest: error: unrecognized arguments: --bogus\nexit code 2"),_call("pytest tests/test_api.py -q", "12 passed"),@@ -63,9 +86,9 @@ def test_recovered_failure_behavior_unchanged():assert result["passed"] is True-def test_unrecovered_failure_after_earlier_success():+def test_unrecovered_failure_after_earlier_success(checker):"""An unrecovered failure later in the log still breaks first_attempt_clean."""- result = check_error_recovery(+ result = checker([_call("ls -la", "total 4\ndrwxr-xr-x 2 root root 4096 ."),_call("make build", "gcc: error: missing.c: No such file or directory"),@@ -77,3 +100,419 @@ def test_unrecovered_failure_after_earlier_success():assert len(result["unrecovered_failures"]) == 1assert result["unrecovered_failures"][0]["fault"] == "skill"assert "make build" in result["reason"]+++def test_unrecovered_failure_after_earlier_recovery(checker):+ result = checker(+ [+ _call("pytest tests/test_api.py", "pytest: error: unrecognized arguments: --bogus\nexit code 2"),+ _call("pytest tests/test_api.py -q", "12 passed"),+ _call("make build", "gcc: error: missing.c: No such file or directory"),+ ]+ )+ assert len(result["corrections"]) == 1+ assert result["first_attempt_clean"] is False+ assert [f["failed_cmd"] for f in result["unrecovered_failures"]] == ["make build"]+ assert result["score"] == 0.0+ assert result["passed"] is False+ assert result["skill_faults"] == 1+ assert result["agent_faults"] == 1+++ "observation",+ [+ _codex_output(0, "0 errors"),+ "exit_code=0\nERROR: example diagnostic",+ "exit code: 0\nTraceback (most recent call last):",+ "0 errors",+ "No errors found",+ "error handling example",+ "timeout configured to 30 seconds",+ "try:\n open(p)\nexcept FileNotFoundError:\n pass",+ "server:\n timeout: 30",+ ],+)+def test_successful_output_is_not_failure(checker, observation):+ result = checker([_call("make build", observation)])+ assert result["score"] == 1.0+ assert result["passed"] is True+ assert result["failures"] == []+++ "observation",+ [+ _codex_output(7, "All done"),+ "exit_code=127",+ "exit code: 42",+ "exit code -9",+ "Timeout: command exceeded 30 seconds",+ "TimeoutError",+ "FileNotFoundError",+ "bash: foo: not found",+ "ConnectionRefused",+ "timeout",+ "ValueError: invalid value",+ "Traceback (most recent call last):",+ "status=failed",+ "command not found",+ ],+)+def test_failure_detection_agrees(checker, observation):+ result = checker([_call("run.py", observation)])+ assert result["score"] == 0.0+ assert result["first_attempt_clean"] is False+ assert len(result["unrecovered_failures"]) == 1+ assert result["skill_faults"] + result["agent_faults"] == 1+++def test_unrecovered_skill_fault_report_reason(checker):+ result = checker([_call("make build", "gcc: error: missing.c: No such file or directory")])+ assert result["passed"] is False+ assert result["skill_faults"] == 1+ assert result["agent_faults"] == 0+ reasons = _collect_fail_reasons("skill_execution", [{"detail": {"error_recovery": result}}])+ assert result["reason"] in reasons+ assert "make build" in result["reason"]+++def test_recovered_skill_fault_counted_once(checker):+ result = checker(+ [+ _call("make build", "No such file or directory"),+ _call("make build", "done"),+ ]+ )+ assert result["skill_faults"] == 1+ assert result["score"] == 0.75+ assert result["passed"] is False+ assert result["unrecovered_failures"] == []+++ "observation",+ [+ "ERROR: test failed\nexit code 1",+ "Traceback (most recent call last):\nValueError: bad input",+ ],+)+def test_unrecovered_agent_fault_report_reason(checker, observation):+ result = checker([_call("pytest", observation)])+ assert result["score"] == 0.0+ assert result["passed"] is False+ assert result["agent_faults"] == 1+ assert result["skill_faults"] == 0+ reasons = _collect_fail_reasons("skill_execution", [{"detail": {"error_recovery": result}}])+ assert result["reason"] in reasons+++ "observation",+ [+ "def failure(stage, message, exit_code=1):",+ " translate unexpected failures to exit code 2",+ "try:\n open(p)\nexcept Exception:\n pass",+ 'print("Error: expected <spec_url>")',+ 'f"{self.prog}: error: {message}"',+ ],+)+def test_displayed_source_is_not_execution_failure(checker, observation):+ result = checker([_call("cat loader.py", observation)])+ assert result["score"] == 1.0+ assert result["passed"] is True+ assert result["failures"] == []++[email protected]("inspections", [4, 5, 6, 8])+def test_recovery_after_intervening_inspections(checker, inspections):+ calls = [_call("pytest -q", _codex_output(1, "ERROR: test failed"))]+ calls.extend(_call(f"sed -n '1,20p' file{i}.py", _codex_output(0, "source")) for i in range(inspections))+ calls.append({"action": "apply_patch", "action_input": {}, "observation": "Done"})+ calls.append(_call("pytest -q", _codex_output(0, "12 passed")))+ result = checker(calls)+ assert result["passed"] is True+ assert result["score"] == 0.9+ assert len(result["corrections"]) == 1+ assert result["corrections"][0]["steps_to_fix"] == inspections + 2+ assert result["unrecovered_failures"] == []+++ "command",+ [+ "rg TODO src",+ "grep needle file.txt",+ "test -f out.json",+ "[ -e out.json ]",+ "diff -q a b",+ "cmp -s a b",+ "/usr/bin/grep needle file.txt",+ ],+)[email protected]("observation", ["Exit code 1", "status=completed\nexit_code=1", _codex_output(1)])+def test_quiet_negative_probe_is_not_failure(checker, command, observation):+ result = checker([_call(command, observation)])+ assert result["passed"] is True+ assert result["score"] == 1.0+ assert result["failures"] == []+++ "command,observation",+ [+ ("pytest -q", "Exit code 1"),+ ("python run.py", "Exit code 1"),+ ("rg TODO src", "Exit code 2"),+ ("rg TODO src", "Exit code 1\nrg: src: Permission denied"),+ ("test -f out.json", "Exit code 1\ntest: invalid argument"),+ ("rg TODO src; python run.py", "Exit code 1"),+ ("rg TODO src && python run.py", "Exit code 1"),+ ("rg TODO src\npython run.py", "Exit code 1"),+ ("rg TODO src | python run.py", "Exit code 1"),+ ("git status", "Exit code 128\nfatal: not a git repository"),+ ],+)+def test_quiet_probe_exception_does_not_hide_real_failures(checker, command, observation):+ result = checker([_call(command, observation)])+ assert result["passed"] is False+ assert result["score"] == 0.0+ assert len(result["unrecovered_failures"]) == 1+++ "status", ["Exit code 2", "exit_code=2", "[exit_code=2]", "[exit_code] 2", "Process exited with code 2"]+)+def test_whole_status_lines_remain_failures(checker, status):+ result = checker([_call("python run.py", status)])+ assert result["passed"] is False+ assert result["score"] == 0.0+++def test_negative_probe_is_not_successful_recovery(checker):+ result = checker([_call("rg TODO src", "Exit code 2\nrg: Permission denied"), _call("rg TODO src", "Exit code 1")])+ assert result["passed"] is False+ assert result["corrections"] == []+ assert len(result["failures"]) == 1+ assert len(result["unrecovered_failures"]) == 1+++ "command,retry,inspections,observation",+ [+ ("pytest -q", "test -f report.json", 6, "Exit code 1"),+ ("pytest tests/test_tools.py", "ls -la", 6, "Exit code 1"),+ ("pytest -q", "cat pytest.ini", 8, "Exit code 1"),+ (+ "python scripts/convert.py in.pdf out.json",+ 'python3 -c "print(1)"',+ 6,+ "Traceback (most recent call last):\nValueError: bad input",+ ),+ ("python scripts/convert.py in.pdf out.json", 'python -c "print(1)"', 6, "Exit code 1"),+ ("pytest tests/test_api.py", "pytest tests/test_other.py", 6, "Exit code 1"),+ ],+)+def test_unrelated_distant_success_does_not_recover(checker, command, retry, inspections, observation):+ calls = [_call(command, observation)]+ calls.extend(_call(f"head -n 5 source{i}.py", _codex_output(0, "source")) for i in range(inspections))+ calls.append(_call(retry, _codex_output(0)))+ result = checker(calls)+ assert result["score"] == 0.0+ assert result["passed"] is False+ assert result["corrections"] == []+ assert [failure["failed_cmd"] for failure in result["unrecovered_failures"]] == [command]++[email protected]("command", ["pytest -q", "python scripts/convert.py in.pdf out.json"])+def test_same_command_recovers_after_distant_edit(checker, command):+ calls = [_call(command, _codex_output(1, "ValueError: bad input"))]+ calls.extend(_call(f"head -n 5 source{i}.py", _codex_output(0, "source")) for i in range(8))+ calls.append({"action": "apply_patch", "action_input": {}, "observation": "Done"})+ calls.append(_call(command, _codex_output(0, "done")))+ result = checker(calls)+ assert result["score"] == 0.9+ assert result["passed"] is True+ assert result["corrections"][0]["steps_to_fix"] == 10+ assert result["unrecovered_failures"] == []++[email protected]("command", ["rg TODO src", "test -f out.json"])[email protected]("line_ending", ["\n", "\r\n"])+def test_codex_quiet_probe_header_is_not_diagnostic(checker, command, line_ending):+ result = checker([_call(command, _codex_output(1).replace("\n", line_ending))])+ assert result["score"] == 1.0+ assert result["passed"] is True+ assert result["failures"] == []+++ "output",+ [+ "Permission denied",+ "status=failed",+ "exit_code=2",+ "Output:",+ "Chunk ID: a1b2c3",+ "Wall time: 0.125 seconds",+ "Original token count: 0",+ ],+)+def test_codex_probe_output_is_never_stripped_as_metadata(checker, output):+ result = checker([_call("rg TODO src", _codex_output(1, output))])+ assert result["score"] == 0.0+ assert result["passed"] is False+++ "command,exit_code,output",+ [+ ("rg TODO src; python run.py", 1, ""),+ ("rg TODO src | python run.py", 1, ""),+ ('rg "$(python run.py)" src', 1, ""),+ ("rg TODO src", 2, ""),+ ("python run.py", 1, ""),+ ("test -f out.json", 1, "test: invalid argument"),+ ],+)+def test_codex_probe_header_does_not_hide_real_failures(checker, command, exit_code, output):+ result = checker([_call(command, _codex_output(exit_code, output))])+ assert result["score"] == 0.0+ assert result["passed"] is False+++def test_codex_quiet_probe_is_not_successful_retry(checker):+ result = checker(+ [_call("rg TODO src", _codex_output(2, "Permission denied")), _call("rg TODO src", _codex_output(1))]+ )+ assert result["score"] == 0.0+ assert result["passed"] is False+ assert len(result["failures"]) == 1+ assert result["corrections"] == []+ assert len(result["unrecovered_failures"]) == 1++[email protected]("command,retry", [("pytest -q", "pytest -q"), ("pytest\t-q", "pytest -q")])+def test_distant_retry_ignores_only_whitespace(checker, command, retry):+ calls = [_call(command, "Exit code 1")]+ calls.extend(_call(f"head -n 5 source{i}.py", _codex_output(0, "source")) for i in range(6))+ calls.append(_call(retry, _codex_output(0, "passed")))+ result = checker(calls)+ assert result["score"] == 0.9+ assert result["passed"] is True+ assert result["corrections"][0]["steps_to_fix"] == 7+ assert result["unrecovered_failures"] == []+++ "command,retry",+ [+ ("pytest tests/test_app.py -x", "pytest tests/test_app.py"),+ ("cd /app && pytest -q", "pytest -q"),+ ("pytest -q 2>&1 | tail -20", "pytest -q 2>&1 | tail -50"),+ ("", ""),+ ("", " \t "),+ ],+)+def test_distant_retry_keeps_tokens_strict_and_nonempty(checker, command, retry):+ calls = [_call(command, "Exit code 1")]+ calls.extend(_call(f"head -n 5 source{i}.py", _codex_output(0, "source")) for i in range(6))+ calls.append(_call(retry, _codex_output(0)))+ result = checker(calls)+ assert result["score"] == 0.0+ assert result["passed"] is False+ assert result["corrections"] == []+ assert len(result["unrecovered_failures"]) == 1++[email protected]("has_chunk_id,has_token_count", [(True, True), (True, False), (False, True), (False, False)])[email protected]("line_ending", ["\n", "\r\n"])[email protected]("output", ["", "Permission denied", "status=failed", "Original token count: 0"])+def test_codex_optional_headers_preserve_output(checker, has_chunk_id, has_token_count, line_ending, output):+ observation = _codex_output(1, output)+ if not has_chunk_id:+ observation = observation.replace("Chunk ID: a1b2c3\n", "", 1)+ if not has_token_count:+ observation = observation.replace(f"Original token count: {len(output.split())}\n", "", 1)+ observation = observation.replace("\n", line_ending)+ result = checker([_call("rg TODO src", observation)])+ assert result["passed"] is (not output)+ assert result["score"] == (0.0 if output else 1.0)+ assert len(result["failures"]) == (1 if output else 0)++[email protected](scope="module")+def claude_error_observation():+ from harbor.agents.installed.claude_code import ClaudeCode++ observation, metadata = ClaudeCode._format_tool_result(+ {"type": "tool_result", "content": "Exit code 1", "is_error": True},+ "Error: Exit code 1",+ )+ assert observation == "Exit code 1\n\n[error] tool reported failure"+ assert metadata["is_error"] is True+ return observation+++ "command", ["rg TODO src", "grep needle file.txt", "test -f out.json", "[ -e out.json ]", "diff a b", "cmp -s a b"]+)[email protected]("line_ending", ["\n", "\r\n"])+def test_claude_error_marker_allows_quiet_negative_probe(checker, claude_error_observation, command, line_ending):+ observation = claude_error_observation.replace("\n", line_ending)+ result = checker([_call(command, observation)])+ assert result["passed"] is True+ assert result["score"] == 1.0+ assert result["failures"] == []+++ "command,exit_code,extra",+ [+ ("rg TODO src", 1, "rg: src: Permission denied"),+ ("rg TODO src", 1, "[error] tool reported failure\nrg: src: Permission denied"),+ ("rg TODO src", 1, "[stderr] Permission denied"),+ ("rg TODO src", 1, '[metadata] {"interrupted": true}'),+ ("rg TODO src", 1, "[error] tool reported failure: interrupted"),+ ("rg TODO src", 1, "[ERROR] tool reported failure"),+ ("rg TODO src", 1, " [error] tool reported failure"),+ ("rg TODO src", 2, ""),+ ("python run.py", 1, ""),+ ("rg TODO src; python run.py", 1, ""),+ ("rg TODO src && python run.py", 1, ""),+ ("rg TODO src | python run.py", 1, ""),+ ('rg "$(python run.py)" src', 1, ""),+ ("rg `python run.py` src", 1, ""),+ ],+)[email protected]("line_ending", ["\n", "\r\n"])+def test_claude_error_marker_does_not_hide_failure(+ checker, claude_error_observation, command, exit_code, extra, line_ending+):+ observation = claude_error_observation.replace("Exit code 1", f"Exit code {exit_code}", 1)+ if extra:+ observation += "\n" + extra+ result = checker([_call(command, observation.replace("\n", line_ending))])+ assert result["passed"] is False+ assert result["score"] == 0.0+ assert len(result["unrecovered_failures"]) == 1+++def test_claude_quiet_negative_probe_is_not_successful_retry(checker, claude_error_observation):+ result = checker(+ [+ _call("rg TODO src", claude_error_observation.replace("Exit code 1", "Exit code 2", 1)),+ _call("rg TODO src", claude_error_observation),+ ]+ )+ assert result["passed"] is False+ assert result["score"] == 0.0+ assert result["corrections"] == []+ assert len(result["failures"]) == 1+ assert len(result["unrecovered_failures"]) == 1+++def test_codex_output_does_not_strip_claude_marker(checker):+ result = checker([_call("rg TODO src", _codex_output(1, "[error] tool reported failure"))])+ assert result["passed"] is False+ assert result["score"] == 0.0