Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier1): keep the hook gate failure FAIL beside an incomplete scan under a policy override (#563)
Narendran Raghavanaeb3dbb60509
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex dfec0395..5ae74edb 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -943,10 +943,14 @@ All notable changes to SkillEvaluator are documented in this file.folder or plugin, still records the SkillSpector scan as incomplete(`incomplete_scans` in the JSON report, and publication eligibility`Incomplete`), with an error that names only that gap. The check itself is- reported as `FAIL`, not `INCOMPLETE`, because the Critical is a confirmed- failure. With SkillSpector v2.13.4, a hook with a `"timeout"` key also gets+ reported as `FAIL`, not `INCOMPLETE`, because the Critical and its gate+ failure are a confirmed failure, also when a `--policy` override lowers the+ Critical. With SkillSpector v2.13.4, a hook with a `"timeout"` key also getsa `static_parse_limit` row on the manifest, so that scan is recorded as- incomplete in the same way.+ incomplete in the same way. In the terminal Tier 1 summary, "what to fix"+ for a check that failed beside an incomplete scan also lists its other gate+ failures, after its first one and any unresolved file reference, so it+ shows the hook gate failure, not only the error for the gap.- `validate` no longer repeats its header box in short terminal windows orwhen a log warning prints mid-run. Only the tier that is still running isdiff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdxindex ba8d92bd..7530d19c 100644--- a/docs/tier1-validation.mdx+++ b/docs/tier1-validation.mdx@@ -334,7 +334,7 @@ For `validate`, the default set is `schema`, `version`, `security`, `pii`, `liceFrom v2.13.4, SkillSpector no longer reports such a scan as `partial`. When it inspected every bundled file and the only gaps are references to files the skill does not bundle, it reports the status `complete_with_caveats` and lists each reference as a caveat. SkillEvaluator accepts these caveats under the same conditions and shows the missing-reference warning above, and the findings of an accepted report are checked as in any other complete scan. A caveat on a path that is not a bundled file, or one that fails a condition above, such as a line that names a file left out of the scan, leaves the scan `INCOMPLETE`; as for a `partial` report, the report's findings are still checked and reported. Unlike a v2.13.3 caveat, these caveats are never accepted from the report alone, so they leave a scan without a known target `INCOMPLETE`, and an accepted report carries no `skillspector_reference_caveats` entry. SkillSpector gives this status only when every gap is a nonfatal `reference_missing` row, so a report with no caveat, or with a caveat that is not a nonfatal `reference_missing` row in phase `reference_resolution`, contradicts itself: SkillEvaluator does not trust it, and the scan is `INCOMPLETE` without its findings. The same holds for `complete_with_caveats` from a release before v2.13.4.- SkillSpector v2.12.0 and later cannot model what an ordinary command or HTTP hook, such as one that runs `echo hi` or a bundled script, does with its hook input. The exception is a hook that SkillSpector recognizes as sending hook content to a remote host, which it reports as **BH2**; a manifest whose command and HTTP hooks are all of that kind counts as modeled. For other hooks it discovers in manifests such as `hooks/hooks.json` and `.claude/settings.json`, SkillSpector reports a **BH1** finding whose evidence says the payload is `unmodeled`, records a nonfatal `partial` `opaque_content` row from `bundled_execution_surface` on the manifest, and reports the analysis as `partial` although the scan finished. This rule applies only to manifests SkillSpector actually reports with that evidence and ledger row; it does not establish coverage for inline plugin-manifest hooks or other hook locations the scanner does not report. The scan's findings cannot show that such hooks are safe: a hook script that sends its input over a socket, `nc` or `/dev/tcp` gets no blocking SkillSpector finding. SkillEvaluator therefore reports the manifest as the Critical `skillspector_hook_payload_unverified`, with a gate failure that neither a `--policy` severity override nor `--llm-verify` clears, instead of `INCOMPLETE` (see [Troubleshooting](#troubleshooting)). This applies when the row is a whole-file row that names only `bundled_execution_surface` and no error class, on a manifest whose BH1 says `unmodeled` (also when a baseline suppresses that BH1), and the report accounts for it exactly: one such row per manifest, `bundled_execution_surface` degraded by exactly those rows with nothing failed or skipped, its `Analyzer bundled_execution_surface status: degraded.` limitation, and one partially inspected file for each manifest that no other row names. The rest of the report is then judged on its own: it is complete when it shows no other gap, and missing references and referenced images can still be accepted with their warnings as described above. The v2.13.3 missing-reference caveat that is accepted from the report alone does not apply, because that check reads the whole report, hook row included. Any other gap still records the SkillSpector scan as incomplete (`incomplete_scans` in the JSON report, and publication eligibility **Incomplete**), with an error that names the other rows, or the other gaps when no other row remains. The check itself is then reported as `FAIL`, not `INCOMPLETE`, because the Critical is a confirmed failure, which takes precedence over missing evidence (see [Reading the final status](/skills/skillevaluator/reports#reading-the-final-status)). A row that does not fit, such as one for a manifest in which SkillSpector parsed no hook (no `unmodeled` BH1), or a report that does not account for it, leaves the whole scan `INCOMPLETE` as before.+ SkillSpector v2.12.0 and later cannot model what an ordinary command or HTTP hook, such as one that runs `echo hi` or a bundled script, does with its hook input. The exception is a hook that SkillSpector recognizes as sending hook content to a remote host, which it reports as **BH2**; a manifest whose command and HTTP hooks are all of that kind counts as modeled. For other hooks it discovers in manifests such as `hooks/hooks.json` and `.claude/settings.json`, SkillSpector reports a **BH1** finding whose evidence says the payload is `unmodeled`, records a nonfatal `partial` `opaque_content` row from `bundled_execution_surface` on the manifest, and reports the analysis as `partial` although the scan finished. This rule applies only to manifests SkillSpector actually reports with that evidence and ledger row; it does not establish coverage for inline plugin-manifest hooks or other hook locations the scanner does not report. The scan's findings cannot show that such hooks are safe: a hook script that sends its input over a socket, `nc` or `/dev/tcp` gets no blocking SkillSpector finding. SkillEvaluator therefore reports the manifest as the Critical `skillspector_hook_payload_unverified`, with a gate failure that neither a `--policy` severity override nor `--llm-verify` clears, instead of `INCOMPLETE` (see [Troubleshooting](#troubleshooting)). This applies when the row is a whole-file row that names only `bundled_execution_surface` and no error class, on a manifest whose BH1 says `unmodeled` (also when a baseline suppresses that BH1), and the report accounts for it exactly: one such row per manifest, `bundled_execution_surface` degraded by exactly those rows with nothing failed or skipped, its `Analyzer bundled_execution_surface status: degraded.` limitation, and one partially inspected file for each manifest that no other row names. The rest of the report is then judged on its own: it is complete when it shows no other gap, and missing references and referenced images can still be accepted with their warnings as described above. The v2.13.3 missing-reference caveat that is accepted from the report alone does not apply, because that check reads the whole report, hook row included. Any other gap still records the SkillSpector scan as incomplete (`incomplete_scans` in the JSON report, and publication eligibility **Incomplete**), with an error that names the other rows, or the other gaps when no other row remains. The check itself is then reported as `FAIL`, not `INCOMPLETE`, because the Critical and its gate failure are a confirmed failure, which takes precedence over missing evidence, also when a `--policy` override lowers the Critical (see [Reading the final status](/skills/skillevaluator/reports#reading-the-final-status)). A row that does not fit, such as one for a manifest in which SkillSpector parsed no hook (no `unmodeled` BH1), or a report that does not account for it, leaves the whole scan `INCOMPLETE` as before.SkillSpector records the same row when it could not parse part of the manifest or read only its first 2,048 hook, permission and settings records. How much of that SkillEvaluator can see depends on the release:@@ -868,7 +868,8 @@ Tier 1 checks always gate the exit code inside `validate`. Tier 2 gates when itthe manifest and every file its hooks run. SkillEvaluator has no way torecord that review: removing the hooks is the only way to pass the check.A `--policy` severity override can lower the finding's severity, but the- gate failure keeps the check failed. `--llm-verify` does not send the+ gate failure keeps the check failed, also when another gap records the+ scan as incomplete. `--llm-verify` does not send thefinding to the verifier, so it keeps its severity and gets no verdict. Thefinding's metadata records the manifest's SHA-256 (`manifest_sha256`) andSkillSpector's row for it.@@ -888,7 +889,8 @@ Tier 1 checks always gate the exit code inside `validate`. Tier 2 gates when itscan of such a manifest is also recorded as incomplete (`incomplete_scans`in the JSON report, and publication eligibility **Incomplete**), with anerror that names only that row. The check itself is still reported as- `FAIL` because of the Critical. v2.13.3 does not record the row.+ `FAIL` because of the Critical and its gate failure. v2.13.3 does not+ record the row.</Accordion><Accordion title="SkillSpector times out">Each SkillSpector run (the static stage, and the LLM stage with `--llm`)diff --git a/src/skillevaluator/models/result.py b/src/skillevaluator/models/result.pyindex 6012770a..78a56eb6 100644--- a/src/skillevaluator/models/result.py+++ b/src/skillevaluator/models/result.py@@ -750,6 +750,10 @@ class ValidationResult:gate_failures = [*self.gate_failures, *(f"[{prefix}] {message}" for message in other.gate_failures)]if gate_failures:self.metadata["gate_failures"] = gate_failures+ # A member's confirmed failure outranks an incomplete scan in the+ # merged result's reported status, as it does in the member's own.+ if other.metadata.get("confirmed_failure") is True:+ self.metadata["confirmed_failure"] = Trueif scanners := {**self.scanners, **other.scanners}:self.metadata["scanners"] = scanners# Keep each skill's unparsed files, prefixed like errors and findings sodiff --git a/src/skillevaluator/reporting/console_ui.py b/src/skillevaluator/reporting/console_ui.pyindex 39eed9f1..72fe17e9 100644--- a/src/skillevaluator/reporting/console_ui.py+++ b/src/skillevaluator/reporting/console_ui.py@@ -1272,6 +1272,7 @@ def summarize_tier1(results: list[ValidationResult], lineup: list[str] | None =)items = [item for r in broken for item in _tier1_fixes(r)]items += [item for r in failed for item in _unresolved_reference_fixes(r)]+ items += [item for r in broken for item in _later_gate_fixes(r)]if setup := [r for r in incomplete if not _only_llm_stages_missing(r) and not _unresolved_reference_fixes(r)]:items.append(_setup_fix(setup))if llm := [r for r in incomplete if _only_llm_stages_missing(r)]:@@ -1299,6 +1300,17 @@ def _unresolved_reference_fixes(result: ValidationResult) -> list[FixItem]:return []+def _later_gate_fixes(result: ValidationResult) -> list[FixItem]:+ """One item for each gate failure after the first of a check that failed beside an incomplete scan.++ The check's own item names only its first gate failure, which is often the error for the gap. Listing+ the others keeps a failure that outranks the gap, such as the hook gate, from being hidden behind it.+ """+ if not result.is_incomplete:+ return []+ return [_tier1_fix(result, gate) for gate in list(dict.fromkeys(result.gate_failures))[1:]]++def _tier1_fixes(result: ValidationResult) -> list[FixItem]:"""One item per distinct critical/high finding, so a check that failed on several shows each one."""serious = [@@ -1316,8 +1328,8 @@ def _tier1_fixes(result: ValidationResult) -> list[FixItem]:return items-def _tier1_fix(result: ValidationResult) -> FixItem:- problem = _first_error(result) or f"{result.validator_name} failed"+def _tier1_fix(result: ValidationResult, problem: str = "") -> FixItem:+ problem = problem or _first_error(result) or f"{result.validator_name} failed"action = next((f.suggestion for f in result.findings if f.message == problem and f.suggestion), None)if action is None and not result.gate_failures:action = first_fix([result])diff --git a/src/skillevaluator/validators/security.py b/src/skillevaluator/validators/security.pyindex b8e505ba..6c1a624b 100644--- a/src/skillevaluator/validators/security.py+++ b/src/skillevaluator/validators/security.py@@ -5217,6 +5217,11 @@ class SecurityValidator(ValidatorBase):result.fail_gate(f"{manifest}: SkillSpector could not model these hooks; this check cannot pass while they are present")+ # The gate is a confirmed failure, so it outranks another gap that+ # records the scan as incomplete, also when a --policy override+ # lowers the Critical. Other gate failures, such as the residual+ # incomplete-analysis error, do not set this.+ result.metadata["confirmed_failure"] = Truemanifests.append(manifest)return manifestsdiff --git a/tests/reporting/test_shared_run_status.py b/tests/reporting/test_shared_run_status.pyindex a4ffb4ea..7b625c84 100644--- a/tests/reporting/test_shared_run_status.py+++ b/tests/reporting/test_shared_run_status.py@@ -103,6 +103,26 @@ def test_confirmed_legacy_failure_survives_separate_missing_evidence():assert status.of(result) == Status.FAILED+def test_confirmed_failure_survives_a_folder_merge():+ """A folder keeps a member's confirmed failure; a member's gate failure alone stays missing evidence."""+ confirmed = ValidationResult(validator_name="Security Scan")+ confirmed.fail_gate("hooks/hooks.json: these hooks cannot pass this check")+ confirmed.metadata["confirmed_failure"] = True+ confirmed.mark_scan_incomplete("scanner")+ incomplete = ValidationResult(validator_name="Security Scan")+ incomplete.fail_gate("scanner reports incomplete analysis")+ incomplete.mark_scan_incomplete("scanner")++ for members, expected in (((confirmed, incomplete), Status.FAILED), ((incomplete,), Status.INCOMPLETE)):+ folder = ValidationResult(validator_name="Security Scan")+ for index, member in enumerate(members):+ folder.merge_with_prefix(member, f"skill-{index}")+ status = resolve_run_status([folder])+ assert status.of(folder) == expected+ assert status.headline == expected+ assert status.missing == ("scanner",)++def test_missing_scanner_process_error_alone_remains_incomplete():result = ValidationResult(validator_name="Scanner")result.add_error("The scanner produced no trustworthy report")diff --git a/tests/validators/test_skillspector_hook_surface.py b/tests/validators/test_skillspector_hook_surface.pyindex 6afcfe3f..cc10bfc0 100644--- a/tests/validators/test_skillspector_hook_surface.py+++ b/tests/validators/test_skillspector_hook_surface.py@@ -49,6 +49,8 @@ from unittest.mock import patchimport pytest+from skillevaluator.reporting.console_ui import summarize_tier1+from skillevaluator.reporting.json_reporter import JSONReporterfrom skillevaluator.reporting.run_status import Status, resolve_run_statusfrom skillevaluator.utils.tool_runner import ToolResultfrom skillevaluator.validators.base import Severity, ValidationResult@@ -370,6 +372,50 @@ def _assert_failed_with_incomplete_scan(result: ValidationResult, stage: str = "assert run.headline == Status.FAILEDassert run.missing == (stage,)assert run.publication == Status.INCOMPLETE+ _assert_what_to_fix_lists_the_hook_gate(result)+++def _assert_what_to_fix_lists_the_hook_gate(result: ValidationResult) -> None:+ """The "what to fix" items of the Tier 1 terminal summary show the hook gate failure.++ The check's own item names its first gate failure, which beside a gap that+ records the scan as incomplete is often the error for the gap. An+ unresolved reference keeps its item next, and each later gate failure+ follows, so the hook gate failure is not hidden behind the gap.+ """+ _passed, rows = summarize_tier1([result], lineup=["security"])+ fixes = ["".join(text for text, _style in row.segments) for row in rows if row.glyph.isdigit()]+ gates = list(dict.fromkeys(result.gate_failures))+ references = [fix for fix in fixes if fix.startswith("SkillSpector ran, but couldn't resolve the file reference")]+ assert bool(references) == any("[reference_resolution; reference_missing]" in gate for gate in gates)+ assert fixes == [gates[0], *references, *gates[1:]][: len(fixes)]+ assert [fix for fix in fixes if "could not model these hooks" in fix]+++def _assert_terminal_summary_lists_a_failure(result: ValidationResult) -> None:+ """The Tier 1 terminal summary counts the check as failed and points at a gate failure.++ A check that is not counted as a confirmed failure is listed as one that+ "didn't run", with advice to install the scanner or skip the check. The+ scanners row and the first "what to fix" item follow the rules for any+ incomplete scan, so this does not pin the scanners row, or which of+ several gate failures comes first.+ """+ result.validator_name = "Security Scan" # run_validation names the result this way+ passed, rows = summarize_tier1([result], lineup=["security"])+ shown = {row.label: "".join(text for text, _style in row.segments) for row in rows}+ everything = "\n".join(shown.values())++ assert not passed+ assert shown["checks"] == "✗ security"+ assert shown["summary"].startswith("0/1 checks passed")+ assert " · 1 failed" in shown["summary"]+ assert "didn't run" not in shown["summary"]+ [fix] = [row for row in rows if row.glyph == "1"]+ assert fix.label == "security"+ assert "".join(text for text, _style in fix.segments) in result.gate_failures+ assert "--skip" not in everything+ assert "not installed" not in everythingclass TestCapturedReports:@@ -456,20 +502,42 @@ class TestUnmodeledHooks:pytest.param({"SECURITY.*": Severity.LOW}, id="security-low"),],)+ @pytest.mark.parametrize(+ ("name", "build"),+ [+ pytest.param(_PLUGIN_REPORT, _plugin, id="complete"),+ # The v2.13.4 parse-limit row records the scan as incomplete.+ pytest.param(_TIMEOUT_REPORT, _timeout_plugin, id="timeout-parse-limit"),+ ],+ )@patch("skillevaluator.validators.security.Tools")def test_policy_overrides_cannot_pass_the_hooks(- self, mock_tools, tmp_path: Path, overrides: dict[str, Severity]+ self,+ mock_tools,+ tmp_path: Path,+ overrides: dict[str, Severity],+ name: str,+ build: Callable[[Path], Path],) -> None:- root = _plugin(tmp_path)- result = _validate(mock_tools, root, _report(_PLUGIN_REPORT))+ """The gate failure keeps the check failed, also when another gap records the scan as incomplete."""+ root = build(tmp_path)+ result = _validate(mock_tools, root, _report(name))apply_policy([result], ValidationPolicy(severity_overrides=overrides))[finding] = _unverified(result)assert finding.severity in {Severity.INFO, Severity.LOW}assert not result.passed- assert result.status == "failed"- assert result.errors == [_gate("hooks/hooks.json")]+ assert result.errors.count(_gate("hooks/hooks.json")) == 1+ if name == _TIMEOUT_REPORT:+ _assert_failed_with_incomplete_scan(result)+ else:+ assert result.status == "failed"+ assert result.errors == [_gate("hooks/hooks.json")]+ assert resolve_run_status([result]).of(result) == Status.FAILED+ exported = json.loads(JSONReporter(include_timestamp=False).render_all([result]))+ assert (exported["overall_status"], exported["results"][0]["status"]) == ("failed", "failed")+ _assert_terminal_summary_lists_a_failure(result)@patch("skillevaluator.validators.security.Tools")def test_llm_verification_cannot_downgrade_the_hook_critical(self, mock_tools, tmp_path: Path) -> None:@@ -735,6 +803,39 @@ class TestFolderScans:f"[{name}] .claude/settings.json" for name in ("a", "b")]+ @pytest.mark.parametrize("method", ["validate_security_only", "validate"])+ @patch("skillevaluator.validators.security.Tools")+ def test_a_policy_override_keeps_a_folder_scan_failed_beside_a_missing_reference(+ self, mock_tools, tmp_path: Path, method: str+ ) -> None:+ """Each skill's gate failure is a confirmed failure that the folder result keeps.++ A ``--policy`` override lowers each Critical, and each skill's+ reference row records the scan as incomplete, but the check is still+ reported as failed, not as a scan that did not run.+ """+ folder = tmp_path / "folder"+ for name in ("a", "b"):+ _write(folder / name, {**_SETTINGS_SKILL, "SKILL.md": _SETTINGS_MISSING_REFERENCE_SKILL_MD})+ report = _report(_settings_missing_reference_report("2.13.4"))+ mock_tools.skillspector.is_available = True+ mock_tools.skillspector.run.return_value = ToolResult(+ success=True, stdout=json.dumps(report), stderr="", exit_code=0+ )+ result = getattr(SecurityValidator(use_llm=False), method)(folder)++ overrides = {"SECURITY.skillspector_hook_payload_unverified": Severity.INFO}+ apply_policy([result], ValidationPolicy(severity_overrides=overrides))++ assert {finding.severity for finding in _unverified(result)} == {Severity.INFO}+ assert [gate for gate in result.gate_failures if "could not model" in gate] == [+ f"[{name}] {_gate('.claude/settings.json')}" for name in ("a", "b")+ ]+ _assert_failed_with_incomplete_scan(result)+ exported = json.loads(JSONReporter(include_timestamp=False).render_all([result]))+ assert (exported["overall_status"], exported["results"][0]["status"]) == ("failed", "failed")+ _assert_terminal_summary_lists_a_failure(result)+class TestPartlyReadManifests:@pytest.mark.parametrize(@@ -890,6 +991,8 @@ class TestOtherGapsStayIncomplete:"skillspector JSON field 'analysis_completeness' reports incomplete analysis (status 'partial'): "f"{label}; security scan did not complete")+ # The incomplete-analysis gate alone is missing evidence, not a confirmed failure.+ assert resolve_run_status([result]).of(result) == Status.INCOMPLETEdef test_fatal_row_is_not_trusted(self) -> None:report = _report(_PLUGIN_REPORT)