Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
docs(tier3): say that a refused attempt fails the arm's attempt coverage (#347)
Narendran Raghavan2248e69f67ae
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex b145db94..5e92c480 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -682,12 +682,18 @@ All notable changes to SkillEvaluator are documented in this file.trial was scored as the selected model although another model finished it.Generated and native tasks now set `CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK=1`in the task environment for every env mode, and neither `harbor.runtime_env`- nor a native task's `[environment.env]` can turn the fallback back on. The- refused attempt is not scored, like any other failed trial, and counts- against the arm's scored-attempt coverage. Other agents ignore the variable.- The variable was checked with Claude Code 2.1.292, the release Harbor- installs; local mode keeps the fallback off only if the host's Claude Code- honors it.+ nor a native task's `[environment.env]` can turn the fallback back on. Other+ agents ignore the variable. The variable was checked with Claude Code+ 2.1.292, the release Harbor installs; local mode keeps the fallback off only+ if the host's Claude Code honors it. A refused attempt is not scored, like+ any other failed trial, so the arm no longer has complete attempt coverage:+ after one refusal in 20 cases with 3 attempts each, it reports+ `Scored attempt coverage is 59/60` and its execution fails.+ `tier3 evaluate` then exits 1, and so does `validate --block-on-agent-eval`.+ On a route that serves the fallback model, such as the Anthropic API, the+ retry used to finish such an attempt and the arm succeeded, so a run that+ completed before can now fail. Rerun it, or evaluate the refused case with+ another model.- Tier 1 security scans no longer fail a skill for an exact read-only`/etc/passwd` bind mount such as `docker run -v /etc/passwd:/etc/passwd:ro`.diff --git a/docs/agents-and-sandboxes.mdx b/docs/agents-and-sandboxes.mdxindex b5246403..72a356e6 100644--- a/docs/agents-and-sandboxes.mdx+++ b/docs/agents-and-sandboxes.mdx@@ -68,11 +68,18 @@ and gateways. With the pinned Claude Code 2.1.292, a refused turn is notretried on another model, and the refusal ends the attempt with Harbor's`AgentSafetyRefusalError`. The variable was checked with that release only.[Local mode](#local-mode) runs the Claude Code already on your machine, so it-keeps the fallback off only if that release honors the variable. Like any-other failed trial, a refused attempt is not scored and counts against the-arm's scored-attempt coverage. Neither `harbor.runtime_env` nor a native-task's `[environment.env]` can turn the fallback back on, and other agents-ignore the variable.+keeps the fallback off only if that release honors the variable. Neither+`harbor.runtime_env` nor a native task's `[environment.env]` can turn the+fallback back on, and other agents ignore the variable.++Like any other failed trial, a refused attempt is not scored, so the arm no+longer has complete attempt coverage: after one refusal in 20 cases with 3+attempts each, it reports `Scored attempt coverage is 59/60` and its+execution fails. `tier3 evaluate` then exits 1, and so does+`validate --block-on-agent-eval`. On a route that serves the fallback model,+such as the Anthropic API, the retry used to finish such an attempt and the+arm succeeded, so a run that completed before can now fail. See+[Troubleshooting](tier3-live-evaluation.mdx#troubleshooting) for what to do.The `claude` alias is canonicalized to `claude-code` everywhere an agentname can appear — `-a/--agents`, `--agent-model`, and `harbor.agents` indiff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdxindex 451d8334..d4551581 100644--- a/docs/tier3-live-evaluation.mdx+++ b/docs/tier3-live-evaluation.mdx@@ -1043,10 +1043,13 @@ machine-readable JSON contract, and how to judge whether a given lift is good.case. SkillEvaluator turns Claude Code's refusal model fallback off(`CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK=1` in every staged task), so theattempt ends on the selected model instead of retrying on another one.- Like any other failed trial, the attempt is not scored and counts against- the arm's scored-attempt coverage. Review the case prompt and the skill- content that the agent read, or evaluate the case with another model. See- [Agents & Sandboxes](agents-and-sandboxes.mdx#supported-agents).+ Like any other failed trial, the attempt is not scored, so the arm no+ longer has complete attempt coverage: after one refusal in 20 cases with 3+ attempts each, it reports `Scored attempt coverage is 59/60` and its+ execution fails. `tier3 evaluate` then exits 1, and so does+ `validate --block-on-agent-eval`. Review the case prompt and the skill+ content that the agent read, then rerun, or evaluate the case with another+ model. See [Agents & Sandboxes](agents-and-sandboxes.mdx#supported-agents).</Accordion><Accordion title="Results look wrong or a case is hard to diagnose">Re-run with `--harbor-keep-jobs` — Harbor artifacts are transient anddiff --git a/tests/test_harbor_claude_refusal_fallback.py b/tests/test_harbor_claude_refusal_fallback.pyindex 22cfdd93..4a54e542 100644--- a/tests/test_harbor_claude_refusal_fallback.py+++ b/tests/test_harbor_claude_refusal_fallback.py@@ -18,6 +18,7 @@ import asyncioimport importlibimport jsonimport os+import refrom collections.abc import Callablefrom pathlib import Pathfrom typing import NamedTuple@@ -46,7 +47,9 @@ from tests.claude_refusal_fixtures import (result_event,tool_round,)+from tests.test_harbor_collector_runtime_failures import _write_actual_harbor_022_result+from skillevaluator.evaluation import EvaluationServicefrom skillevaluator.provider_config import resolve_llm_providerfrom skillevaluator.tier3.harbor import CONTAINER_AGENT_IMPORT_PATHS, LOCAL_AGENT_IMPORT_PATHSfrom skillevaluator.tier3.harbor.adapter import (@@ -54,6 +57,7 @@ from skillevaluator.tier3.harbor.adapter import (generate_harbor_tasks,stage_native_harbor_tasks,)+from skillevaluator.tier3.harbor.collector import collect_harbor_resultsfrom skillevaluator.tier3.harbor.local_environment import SkillEvaluatorLocalEnvironmentfrom skillevaluator.tier3.harbor.runner import (_CLAUDE_CODE_VERSION,@@ -63,6 +67,8 @@ from skillevaluator.tier3.harbor.runner import ()_SWITCH = "CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK"+# The collector's arm-level error when scored attempts fall short of expected ones.+_COVERAGE_ERROR = "Scored attempt coverage is"_FAKE_CREDENTIAL = "test-key"_VERTEX_ENV = {"CLAUDE_CODE_USE_VERTEX": "1",@@ -217,6 +223,66 @@ def test_docs_scope_the_switch_to_the_pinned_claude_code_release() -> None:assert "[Local mode](#local-mode) runs the Claude Code already on your machine" in paragraph+def _refusal_doc_texts() -> dict[str, str]:+ """Return each place that documents the switch, whitespace-normalized."""+ root = Path(__file__).resolve().parents[1]+ changelog = (root / "CHANGELOG.md").read_text(encoding="utf-8")+ unreleased = changelog.split("\n## Unreleased\n", 1)[1].split("\n## ", 1)[0]+ [entry] = [block for block in unreleased.split("\n\n") if _SWITCH in block]+ blocks = (root / "docs" / "agents-and-sandboxes.mdx").read_text(encoding="utf-8").split("\n\n")+ [index] = [index for index, block in enumerate(blocks) if f"{_SWITCH}=1" in block]+ troubleshooting = (root / "docs" / "tier3-live-evaluation.mdx").read_text(encoding="utf-8")+ [accordion] = re.findall(+ r'<Accordion title="A claude-code trial fails with AgentSafetyRefusalError">(.*?)</Accordion>',+ troubleshooting,+ flags=re.DOTALL,+ )+ texts = {+ "CHANGELOG.md": entry,+ "agents-and-sandboxes.mdx": "\n\n".join(blocks[index : index + 2]),+ "tier3-live-evaluation.mdx": accordion,+ }+ return {name: " ".join(text.split()) for name, text in texts.items()}+++def test_a_refused_attempt_fails_attempt_coverage_and_the_docs_say_so(tmp_path: Path) -> None:+ # Where the fallback used to finish a refused attempt on another model, the+ # arm succeeded. Now the attempt is an unscored trial failure, so the arm's+ # attempt coverage is incomplete, its execution fails and the command exits 1.+ jobs_dir = tmp_path / "jobs"+ _write_actual_harbor_022_result(+ jobs_dir / "demo-claude-code-with", reward=0.0, exception_type="AgentSafetyRefusalError"+ )++ results = collect_harbor_results(+ skill_name="demo",+ agents=["claude-code"],+ output_dir=tmp_path / "results",+ jobs_dir=jobs_dir,+ skip_baseline=True,+ expected_cases=1,+ expected_case_ids=["case-001"],+ expected_trials=1,+ )++ arm = results["agents"]["claude-code"]+ assert arm["execution_status"] == "failed"+ assert (arm["expected_attempts"], arm["scored_attempts"]) == (1, 0)+ assert f"{_COVERAGE_ERROR} 0/1" in arm["execution_errors"]+ assert results["execution_status"] == "failed"+ # `tier3 evaluate` exits 1 whenever this reports a failure.+ assert EvaluationService.failure_reason(results) is not None++ # Every place that documents the switch names that result in the collector's+ # own words and uses the gate's term, "complete attempt coverage".+ for name, text in _refusal_doc_texts().items():+ assert re.search(rf"`{_COVERAGE_ERROR} \d+/\d+`", text), name+ assert "complete attempt coverage" in text, name+ assert "scored-attempt coverage" not in text, name+ assert "execution fails" in text, name+ assert "`tier3 evaluate` then exits 1, and so does `validate --block-on-agent-eval`" in text, name++def test_refusal_fallback_switch_is_not_redacted_as_a_secret() -> None:# Harbor replaces each sensitive value across the trial directory; a value of# "1" must never be treated as one.