Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
docs(tier3): say where a refused claude-code attempt's coverage error appears (#360)
Narendran Raghavan7776eb60722d
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 69fb45a6..63bc8d49 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -343,14 +343,14 @@ All notable changes to SkillEvaluator are documented in this file.agents ignore the variable. The variable was checked with Claude Code2.1.292, the release Harbor installs; local mode keeps the fallback off onlyif the host's Claude Code honors it. A refused attempt is not scored, like- any other failed trial, so the arm no longer has complete attempt coverage:- after one refusal in 20 cases with 3 attempts each, it reports- `Scored attempt coverage is 59/60` and its execution fails.+ any other failed trial, so the arm no longer has complete attempt coverage+ and its execution fails: after one refusal in 20 cases with 3 attempts+ each, the run's `result.json` records `Scored attempt coverage is 59/60`.`tier3 evaluate` then exits 1, and so does `validate --block-on-agent-eval`.On a route that serves the fallback model, such as the Anthropic API, the- retry used to finish such an attempt and the arm succeeded, so a run that- completed before can now fail. Rerun it, or evaluate the refused case with- another model.+ retry used to finish such an attempt, and the attempt was scored, so a run+ that completed before can now fail. Rerun it, or run the arm with another+ model through `--agent-model claude-code=MODEL`.- Tier 3 staging no longer fails with `chmod: cannot use dir_fd andfollow_symlinks together` when it replaces or merges into an existingdiff --git a/docs/agents-and-sandboxes.mdx b/docs/agents-and-sandboxes.mdxindex 72a356e6..054a3b60 100644--- a/docs/agents-and-sandboxes.mdx+++ b/docs/agents-and-sandboxes.mdx@@ -73,12 +73,12 @@ keeps the fallback off only if that release honors the variable. Neitherfallback back on, and other agents ignore the variable.Like any other failed trial, a refused attempt is not scored, so the arm no-longer has complete attempt coverage: after one refusal in 20 cases with 3-attempts each, it reports `Scored attempt coverage is 59/60` and its-execution fails. `tier3 evaluate` then exits 1, and so does-`validate --block-on-agent-eval`. On a route that serves the fallback model,-such as the Anthropic API, the retry used to finish such an attempt and the-arm succeeded, so a run that completed before can now fail. See+longer has complete attempt coverage and its execution fails: after one+refusal in 20 cases with 3 attempts each, the run's `result.json` records+`Scored attempt coverage is 59/60`. `tier3 evaluate` then exits 1, and so+does `validate --block-on-agent-eval`. On a route that serves the fallback+model, such as the Anthropic API, the retry used to finish such an attempt,+and the attempt was scored, so a run that completed before can now fail. See[Troubleshooting](tier3-live-evaluation.mdx#troubleshooting) for what to do.The `claude` alias is canonicalized to `claude-code` everywhere an agentdiff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdxindex 293ea51e..5685c5cd 100644--- a/docs/tier3-live-evaluation.mdx+++ b/docs/tier3-live-evaluation.mdx@@ -1114,12 +1114,17 @@ machine-readable JSON contract, and how to judge whether a given lift is good.(`CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK=1` in every staged task), so theattempt ends on the selected model instead of retrying on another one.Like any other failed trial, the attempt is not scored, so the arm no- longer has complete attempt coverage: after one refusal in 20 cases with 3- attempts each, it reports `Scored attempt coverage is 59/60` and its- execution fails. `tier3 evaluate` then exits 1, and so does- `validate --block-on-agent-eval`. Review the case prompt and the skill- content that the agent read, then rerun, or evaluate the case with another- model. See [Agents & Sandboxes](agents-and-sandboxes.mdx#supported-agents).+ longer has complete attempt coverage and its execution fails: after one+ refusal in 20 cases with 3 attempts each, the run's `result.json` records+ `Scored attempt coverage is 59/60`. `tier3 evaluate` then exits 1, and so+ does `validate --block-on-agent-eval`. `tier3 evaluate --verbose` lists the+ refused trial and its `AgentSafetyRefusalError` in the terminal. The+ execution errors in `result.json` call it an `Unscoreable reward`, not an+ agent runtime failure, because a refusal is the model's response to the+ case rather than an infrastructure failure. Review the case prompt and the+ skill content that the agent read, then rerun, or run the arm with another+ model through `--agent-model claude-code=MODEL`. See+ [Agents & Sandboxes](agents-and-sandboxes.mdx#supported-agents).</Accordion><Accordion title="Results look wrong or a case is hard to diagnose">Re-run with `--harbor-keep-jobs` — Harbor artifacts are transient anddiff --git a/tests/test_harbor_claude_refusal_fallback.py b/tests/test_harbor_claude_refusal_fallback.pyindex fb0fa108..58973bd4 100644--- a/tests/test_harbor_claude_refusal_fallback.py+++ b/tests/test_harbor_claude_refusal_fallback.py@@ -19,11 +19,16 @@ import importlibimport jsonimport osimport re+import shutilfrom collections.abc import Callable+from datetime import UTC, datetime, timedelta+from io import StringIOfrom pathlib import Path-from typing import NamedTuple+from typing import Any, NamedTuple+from uuid import UUIDimport pytest+from click.testing import CliRunnerfrom harbor.agents.installed.base import (AgentSafetyRefusalError,ApiOverloadedError,@@ -33,9 +38,12 @@ from harbor.agents.installed.base import (from harbor.agents.installed.claude_code import ClaudeCodefrom harbor.environments.base import BaseEnvironment, ExecResultfrom harbor.models.agent.context import AgentContext+from harbor.models.job.result import JobResult, JobStatsfrom harbor.models.task.config import EnvironmentConfig, TaskConfigfrom harbor.models.trial.paths import TrialPaths+from harbor.models.trial.result import TrialResultfrom harbor.utils.env import is_sensitive_env_key+from rich.console import Consolefrom tests.claude_refusal_fixtures import (NO_FALLBACK_TEXT,SELECTED_MODEL,@@ -47,8 +55,8 @@ from tests.claude_refusal_fixtures import (result_event,tool_round,)-from tests.test_harbor_collector_runtime_failures import _write_actual_harbor_022_result+from skillevaluator.cli import clifrom skillevaluator.evaluation import EvaluationServicefrom skillevaluator.provider_config import resolve_llm_providerfrom skillevaluator.tier3.harbor import CONTAINER_AGENT_IMPORT_PATHS, LOCAL_AGENT_IMPORT_PATHS@@ -65,10 +73,15 @@ from skillevaluator.tier3.harbor.runner import (_resolve_agent_runtime_plan,_resolve_runtime_env,)+from skillevaluator.tier3.result_display import render_evaluation_result_SWITCH = "CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK"# The collector's arm-level error when scored attempts fall short of expected ones._COVERAGE_ERROR = "Scored attempt coverage is"+# The switch's CHANGELOG entry, found by its opening words so that a later entry+# that names the variable does not change which entry the docs test reads.+_CHANGELOG_ENTRY_LEAD = "- A Tier 3 `claude-code` trial that the model provider's safety classifier stops"+_SIMPLE_SKILL = Path(__file__).resolve().parent / "fixtures" / "skills" / "simple"_FAKE_CREDENTIAL = "test-key"_VERTEX_ENV = {"CLAUDE_CODE_USE_VERTEX": "1",@@ -227,11 +240,11 @@ def _refusal_doc_texts() -> dict[str, str]:"""Return each place that documents the switch, whitespace-normalized."""root = Path(__file__).resolve().parents[1]changelog = (root / "CHANGELOG.md").read_text(encoding="utf-8")- # The switch shipped in 0.8.0: one release section and one entry in it name it.[release] = [section for section in changelog.split("\n## ")[1:] if _SWITCH in section]assert release.startswith("0.8.0 - "), release.splitlines()[0]- entries = re.findall(r"^- .*(?:\n .*)*", release, flags=re.MULTILINE)- [entry] = [item for item in entries if _SWITCH in item]+ entries = [block for block in release.split("\n\n") if " ".join(block.split()).startswith(_CHANGELOG_ENTRY_LEAD)]+ assert entries, "the 0.8.0 CHANGELOG entry for the refusal fallback switch is missing"+ entry = entries[0]blocks = (root / "docs" / "agents-and-sandboxes.mdx").read_text(encoding="utf-8").split("\n\n")[index] = [index for index, block in enumerate(blocks) if f"{_SWITCH}=1" in block]troubleshooting = (root / "docs" / "tier3-live-evaluation.mdx").read_text(encoding="utf-8")@@ -248,42 +261,183 @@ def _refusal_doc_texts() -> dict[str, str]:return {name: " ".join(text.split()) for name, text in texts.items()}-def test_a_refused_attempt_fails_attempt_coverage_and_the_docs_say_so(tmp_path: Path) -> None:- # Where the fallback used to finish a refused attempt on another model, the- # arm succeeded. Now the attempt is an unscored trial failure, so the arm's- # attempt coverage is incomplete, its execution fails and the command exits 1.- jobs_dir = tmp_path / "jobs"- _write_actual_harbor_022_result(- jobs_dir / "demo-claude-code-with", reward=0.0, exception_type="AgentSafetyRefusalError"+def _write_harbor_job(job_dir: Path, case_ids: list[str], *, attempts: int, refused: str | None = None) -> None:+ """Persist a Harbor 0.24 job whose trials all score 1.0, except the ``refused`` attempt."""+ started = datetime(2026, 8, 25, tzinfo=UTC)+ trials: list[TrialResult] = []+ for case_id in case_ids:+ for attempt in range(1, attempts + 1):+ name = f"{case_id}__attempt{attempt}"+ trial_dir = job_dir / name+ trial_dir.mkdir(parents=True)+ at = started + timedelta(seconds=len(trials))+ task = {"path": str(job_dir / "task" / case_id)}+ refusal = {+ "exception_type": "AgentSafetyRefusalError",+ "exception_message": "Command failed (exit 1)",+ "exception_traceback": "",+ "occurred_at": at,+ }+ trial = TrialResult.model_validate(+ {+ "id": UUID(int=len(trials) + 2),+ "task_name": case_id,+ "trial_name": name,+ "trial_uri": trial_dir.as_uri(),+ "task_id": task,+ "task_checksum": "refusal-fixture",+ "config": {"task": task, "trial_name": name},+ "agent_info": {"name": "claude-code", "version": "test", "model_info": {"name": SELECTED_MODEL}},+ "agent_result": {"n_input_tokens": 7, "n_cache_tokens": 2, "n_output_tokens": 3},+ "started_at": at,+ "finished_at": at,+ "verifier_result": None if name == refused else {"rewards": {"overall": 1.0}},+ "exception_info": refusal if name == refused else None,+ }+ )+ (trial_dir / "result.json").write_text(trial.model_dump_json(indent=2), encoding="utf-8")+ trials.append(trial)+ job = JobResult(+ id=UUID(int=1),+ started_at=started,+ updated_at=started,+ finished_at=started,+ n_total_trials=len(trials),+ stats=JobStats.from_trial_results(trials, n_total_trials=len(trials)),+ trial_results=trials,)+ (job_dir / "result.json").write_text(job.model_dump_json(indent=2), encoding="utf-8")- results = collect_harbor_results(++def _collect_refusal(+ tmp_path: Path, *, cases: int, attempts: int, refused: str, skip_baseline: bool = True+) -> dict[str, Any]:+ """Collect a claude-code run in which the safety classifier stopped only ``refused``."""+ case_ids = [f"case-{index:03d}" for index in range(1, cases + 1)]+ jobs_dir = tmp_path / "jobs"+ _write_harbor_job(jobs_dir / "demo-claude-code-with", case_ids, attempts=attempts, refused=refused)+ if not skip_baseline:+ _write_harbor_job(jobs_dir / "demo-claude-code-without", case_ids, attempts=attempts)+ return collect_harbor_results(skill_name="demo",agents=["claude-code"],output_dir=tmp_path / "results",jobs_dir=jobs_dir,- skip_baseline=True,- expected_cases=1,- expected_case_ids=["case-001"],- expected_trials=1,+ skip_baseline=skip_baseline,+ n_attempts=attempts,+ expected_cases=cases,+ expected_case_ids=case_ids,+ expected_trials=cases * attempts,+ case_id_by_task_selector={case_id: case_id for case_id in case_ids},)++def test_a_refused_attempt_fails_attempt_coverage_as_the_docs_say(tmp_path: Path) -> None:+ # The documented example: one refusal in 20 cases with 3 attempts each.+ # Where the fallback used to finish a refused attempt on another model, the+ # attempt was scored. Now it is an unscored trial failure, so the arm's+ # attempt coverage is incomplete, its execution fails and the command exits 1.+ refused = "case-007__attempt2"+ results = _collect_refusal(tmp_path, cases=20, attempts=3, refused=refused)+arm = results["agents"]["claude-code"]- assert arm["execution_status"] == "failed"- assert (arm["expected_attempts"], arm["scored_attempts"]) == (1, 0)- assert f"{_COVERAGE_ERROR} 0/1" in arm["execution_errors"]- assert results["execution_status"] == "failed"+ assert arm["execution_status"] == results["execution_status"] == "failed"+ assert (arm["expected_attempts"], arm["scored_attempts"]) == (60, 59)+ coverage = f"{_COVERAGE_ERROR} 59/60"+ # The run's result.json persists these execution errors.+ assert coverage in results["execution_errors"]+ # A refusal is a trial failure, reported as an unscoreable reward, and never+ # an agent runtime (infrastructure) failure.+ assert arm["agent_runtime_failures"]["with_skill"] == []+ assert [failure["trial"] for failure in arm["trial_failures"]["with_skill"]] == [refused]+ assert (+ f"Unscoreable reward in {refused}: AgentSafetyRefusalError: Command failed (exit 1)"+ in results["execution_errors"]+ )# `tier3 evaluate` exits 1 whenever this reports a failure.assert EvaluationService.failure_reason(results) is not None- # Every place that documents the switch names that result in the collector's- # own words and uses the gate's term, "complete attempt coverage".- for name, text in _refusal_doc_texts().items():- assert re.search(rf"`{_COVERAGE_ERROR} \d+/\d+`", text), name- assert "complete attempt coverage" in text, name- assert "scored-attempt coverage" not in text, name- assert "execution fails" in text, name- assert "`tier3 evaluate` then exits 1, and so does `validate --block-on-agent-eval`" in text, name+ texts = _refusal_doc_texts()+ for name, text in texts.items():+ # Each place quotes the persisted message for exactly this example.+ assert re.findall(rf"`({_COVERAGE_ERROR} \d+/\d+)`", text) == [coverage], name+ for token in (+ "`result.json`",+ "complete attempt coverage",+ "`tier3 evaluate`",+ "`validate --block-on-agent-eval`",+ "exits 1",+ ):+ assert token in text, (name, token)+ # The model is chosen per agent, not per case.+ for name in ("CHANGELOG.md", "tier3-live-evaluation.mdx"):+ assert "`--agent-model claude-code=MODEL`" in texts[name], name+ assert "`Unscoreable reward`" in texts["tier3-live-evaluation.mdx"]++[email protected]("skip_baseline", [True, False], ids=["with-skill-only", "with-baseline"])+def test_verbose_evaluate_output_lists_the_refused_trial(tmp_path: Path, skip_baseline: bool) -> None:+ # Troubleshooting points to `tier3 evaluate --verbose`. Without it, a run with+ # a completed baseline arm prints only a failure count, and the terminal never+ # quotes the coverage message when no arm is usable.+ refused = "case-001__attempt1"+ results = _collect_refusal(tmp_path, cases=1, attempts=1, refused=refused, skip_baseline=skip_baseline)+ stream = StringIO()++ render_evaluation_result(results, console=Console(file=stream, force_terminal=False, width=200), verbose=True)++ assert f"{refused}: AgentSafetyRefusalError: Command failed (exit 1)" in stream.getvalue()+ assert "`tier3 evaluate --verbose`" in _refusal_doc_texts()["tier3-live-evaluation.mdx"]+++ ("gate", "expected_exit_code"),+ [("--block-on-agent-eval", 1), ("--no-block-on-agent-eval", 0)],+)+def test_validate_block_on_agent_eval_fails_a_run_with_a_refused_attempt(+ monkeypatch: pytest.MonkeyPatch, tmp_path: Path, gate: str, expected_exit_code: int+) -> None:+ # The docs say `validate --block-on-agent-eval` exits 1 too. The engine+ # returns the collector's own result for a refused attempt, so the gate reads+ # the failed execution and the coverage error that a refusal produces.+ _clear_route_env(monkeypatch)+ results = _collect_refusal(tmp_path, cases=1, attempts=1, refused="case-001__attempt1")+ monkeypatch.setattr(EvaluationService, "evaluate", lambda _self, _options, **_kwargs: results)+ skill = tmp_path / "simple"+ shutil.copytree(_SIMPLE_SKILL, skill)+ (skill / "evals").mkdir()+ (skill / "evals" / "evals.json").write_text(+ json.dumps([{"id": "case-001", "prompt": "Use the skill."}]), encoding="utf-8"+ )+ output_dir = tmp_path / "reports"++ result = CliRunner().invoke(+ cli,+ [+ "validate",+ "--no-autopilot",+ str(skill),+ "--no-llm",+ "--no-tier2",+ "--tier3",+ gate,+ "--agents",+ "claude-code",+ "--checks",+ "schema",+ "--report",+ "json",+ "--output-dir",+ str(output_dir),+ ],+ )++ assert result.exit_code == expected_exit_code, result.output+ report_text = next(output_dir.glob("skillevaluator-output-*.json")).read_text(encoding="utf-8")+ report = json.loads(report_text)+ assert report["overall_passed"] is (expected_exit_code == 0)+ assert report["gating"]["tiers"]["3"]["blocking"] is (expected_exit_code == 1)+ assert f"{_COVERAGE_ERROR} 0/1" in report_textdef test_refusal_fallback_switch_is_not_redacted_as_a_secret() -> None: