Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(reporting): include required case checks in the HTML verdict rule (#651)
Christopher Kevin16ef5cc70897
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 0f662921..09b5517f 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -326,6 +326,7 @@ All notable changes to SkillEvaluator are documented in this file.- Benchmark methodology describes the Overall row's shared comparison basisrather than the headline with-skill score, in Tier 3 and publication cards.- BENCHMARK cards explain the required with-skill goal and security case checks+- HTML reports explain the required with-skill goal and security case checksthat can fail quality even when all dimensions pass.- The Tier 2 scale guide names the fixed 8 MiB selected-manifest input budgetand 1 MiB per-manifest limit, including in description-only embedding mode.diff --git a/src/skillevaluator/reporting/templates/report.html.j2 b/src/skillevaluator/reporting/templates/report.html.j2index a53d36b5..6ed5ddef 100644--- a/src/skillevaluator/reporting/templates/report.html.j2+++ b/src/skillevaluator/reporting/templates/report.html.j2@@ -2690,7 +2690,7 @@<summary>What each number means</summary><div class="t3-details-body"><ul class="t3-reasoning-list">- <li><strong>Verdict</strong>: the live-evaluation result from the five dimensions. It passes when one agent scores at least {{ "%.2f" | format(tier3_dimension_pass_threshold) }} on every dimension with the skill.</li>+ <li><strong>Verdict</strong>: the live-evaluation result from the five dimensions and required case checks. It passes when one agent scores at least {{ "%.2f" | format(tier3_dimension_pass_threshold) }} on every dimension with the skill and has no with-skill case failing a required goal-accuracy or security check. A scored unmet goal or security issue in any attempt fails its case and that agent's quality verdict, even if all dimensions pass. Baseline-arm required-check failures do not change this quality verdict.</li><li><strong>Overall Score</strong>: the equal-weight mean of the five dimension scores with the skill, from 0.00 to 1.00.</li><li><strong>Skill Lift</strong>: compares Overall on the metrics and cases both arms scored, so it can differ from subtracting the displayed Overall scores. It shows whether the skill helped and does not change the verdict; N/A means no comparable baseline, not zero.</li><li><strong>pass@k</strong> (Trials): the share of eval cases with at least one passing attempt out of k. It measures how repeatable success is.</li>diff --git a/tests/reporting/test_dimension_explanations.py b/tests/reporting/test_dimension_explanations.pyindex 4b117dd9..26ff2834 100644--- a/tests/reporting/test_dimension_explanations.py+++ b/tests/reporting/test_dimension_explanations.py@@ -124,6 +124,19 @@ def test_benchmark_verdict_methodology_includes_with_skill_required_checks() ->assert "Baseline-arm required-check failures do not change this quality verdict." in markdown+def test_html_verdict_methodology_includes_with_skill_required_checks() -> None:+ result = _validation_result_from_payload(_payload({"agent": _agent()}))+ assert result is not None+ html = HTMLReporter(include_timestamp=False).render_all([result])++ assert "has no with-skill case failing a required goal-accuracy or security check" in html+ assert (+ "A scored unmet goal or security issue in any attempt fails its case and that agent's quality "+ "verdict, even if all dimensions pass."+ ) in html+ assert "Baseline-arm required-check failures do not change this quality verdict." in html++@pytest.mark.parametrize("scope", ["publication", "tier3"])@pytest.mark.parametrize("baseline", [True, False])def test_overall_row_methodology_describes_comparison_basis(scope: str, baseline: bool) -> None: