Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(reporting): name the skill in single-skill JSON reports and record the evaluator version (#424)
Narendran Raghavan3d93f2e0fa92
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 264d0191..6ad875ab 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -62,6 +62,15 @@ All notable changes to SkillEvaluator are documented in this file.versioned labels, with offline suite checks, repeated-grading statistics,provenance, optional rate gates and a bundled public reference suite.+- JSON reports record the SkillEvaluator `version` that wrote them, the+ `target_path` (the absolute path or Git remote URL from `validate`, the path+ as given from `tier1` and `tier2`, and `null` from single-check commands such+ as `security-scan`), and `top_issues`, the grouped issue rows across every+ tier that the HTML report data carries, without their page anchors. Each+ `gating.tiers` entry adds `validator_policies`, which says for each check+ whether it gates the exit code; the tier's `blocking` is `true` when any one+ of them does. The keys are additive, and existing keys keep their shape.+- Import production trace exports into redacted review candidates with`tier3 import-traces`, then promote only explicitly human-reviewed expectationsand replay fixtures with `tier3 accept-traces`.@@ -549,6 +558,16 @@ All notable changes to SkillEvaluator are documented in this file.use schema 1.1; valid schema 1.0 snapshots retain their original datasets andintegrity checks, with corrected summaries projected without rewriting files.+- The JSON report of `skillevaluator validate`, `tier1` or `tier2` on one skill+ folder lists that skill in `skills` and under its author in `contributors`.+ A clean run with a narrowed check list, such as `--checks schema`, has no+ finding path or quality score to name the skill by, so the report listed one+ skill per check ID (`manifest_exists`, `frontmatter_valid` and so on). It+ now uses the folder's name. The `BENCHMARK.md` of a run without a recorded+ Tier 3 version, such as `--tiers 1`, now records the evaluator version+ instead of `Evaluator version: not recorded (legacy or non-live result)`; a+ version recorded by a Tier 3 run still wins.+- Tier 3 improvement recommendations respect each case's expected skill routing:passing negative controls no longer inherit activation-loss advice from positivecases, while failed negatives retain task, environment, and run-error diagnostics.@@ -903,6 +922,7 @@ All notable changes to SkillEvaluator are documented in this file.with a `destructive_command` finding, already gets FAIL from that check (seeChanged).+## 0.8.0 - 2026-11-05### Addeddiff --git a/ci/run_tiers.py b/ci/run_tiers.pyindex 8c9c7897..cf53b463 100644--- a/ci/run_tiers.py+++ b/ci/run_tiers.py@@ -337,7 +337,16 @@ def _retain(raw: Path, destination: Path, tier: int, tokens: set[str], skill: Pafor result, row in zip(results, report["results"], strict=True):result.metadata["gating"] = {**result.metadata.get("gating", {}), "tier": 2, "blocking": False}row["gating"] = result.metadata["gating"]- report["gating"] = {"tiers": {"2": {"blocking": False, "validators": [r.validator_name for r in results]}}}+ report["gating"] = {+ "tiers": {+ "2": {+ "blocking": False,+ "validators": [r.validator_name for r in results],+ # Keep the reporter's per-check policy list, now advisory too.+ "validator_policies": [{"validator": r.validator_name, "blocking": False} for r in results],+ }+ }+ }run_status = resolve_run_status(results)if tier == 2:report.update(overall_passed=run_status.headline.value == "passed", overall_status=run_status.headline.value)diff --git a/docs/ci-integration.mdx b/docs/ci-integration.mdxindex f367c633..c4d1a68d 100644--- a/docs/ci-integration.mdx+++ b/docs/ci-integration.mdx@@ -242,6 +242,8 @@ When the exit code is not enough — dashboards, custom thresholds, badge genera| Field | Type | Contents || --- | --- | --- |+| `version` | string | SkillEvaluator version that wrote the report |+| `target_path` | string or null | The validated target. `validate` records its absolute path, or its browsable URL when it is in a Git checkout with an `origin` remote. `tier1` and `tier2` record the path as given on the command line. The single-check commands such as `security-scan` give `null` || `overall_passed` | boolean | `true` when every check passed || `overall_status` | string | `passed`, `failed`, or `incomplete` (a required scanner produced no evidence) || `publication_status` | string | `pass`, `neutral`, `fail`, or `incomplete`; use this field for publication eligibility |@@ -257,7 +259,9 @@ When the exit code is not enough — dashboards, custom thresholds, badge genera| `finding_groups` | array | One entry per unique finding: `finding_id`, highest `severity`, `message`, `validators`, and every location in `occurrences[]` with its `occurrence_id`, including each location a finding lists in `metadata.occurrences` when its check folds repeats of one value into it (the PII scan does) || `message_counts` | object | `independent_errors` and `independent_warnings`: error and warning strings with no finding behind them, such as `gate_failures` and scanner or execution errors (strings that mirror a finding, and the `Workflow scope:` note for a stage that configuration skips, are not counted); `informational`: informational messages, not counting the `[OK]` line recorded for each passed check || `results[].findings[].finding_id`, `results[].findings[].occurrence_id` | string | Stable finding and location identities (see below) |-| `skills` | array | Per-skill `{ name, passed, issue_count }`; `issue_count` counts raw finding records |+| `skills` | array | Per-skill `{ name, passed, issue_count }`; `issue_count` counts raw finding records. `validate`, `tier1` and `tier2` on one skill folder list that skill once, under its folder name, also when a narrowed run such as `--checks schema` reports no finding |+| `top_issues` | array | Top issues across every tier in the report, most severe first: `message` (a security finding is grouped under the text before its first `: `), highest `severity`, `total_count` of raw finding records (the unit of `skills[].issue_count`, so a PII finding that folds repeats of one value counts once although `finding_groups` lists each of its locations), `skills_affected`, `category` and `suggestion`. The HTML report data carries the same rows; the HTML summary panel lists the Tier 1 rows only |+| `gating.tiers` | object | `validate`, `tier1` and `tier2` runs: one entry per tier that ran (`"1"`, `"2"`, `"3"`). `blocking` is `true` when any of the tier's checks gates the exit code, `validators` lists the checks, and `validator_policies` gives each check's own `{ validator, blocking }` || `quality_summary` | array | Quality details with `overall_score` (0–100) and `grade` (A–F); a folder target reports the collection average plus a `skill_count` |Every finding carries two stable IDs. `finding_id` (`SEF-` plus 20 hexdiff --git a/docs/reports.mdx b/docs/reports.mdxindex 8c57052b..d0a0b6b9 100644--- a/docs/reports.mdx+++ b/docs/reports.mdx@@ -406,10 +406,11 @@ same evaluator container publish cards whose only recorded revision is the sharecontainer tag, and a reader cannot tell which version of a skill was evaluated orwhether the card has gone stale.-The identity is supplied by the orchestration input and carried through the-evaluation-result contract unchanged. It is never inferred from repository state-while rendering, because the tree that renders a card is the evaluator checkout-rather than the evaluated skill's source. The supported seams are:+The evaluated source identity is supplied by the orchestration input and carried+through the evaluation-result contract unchanged. It is never inferred from+repository state while rendering, because the tree that renders a card is the+evaluator checkout rather than the evaluated skill's source. The supported seams+are:- `--evaluated-source-repository`, `--evaluated-source-revision` and`--evaluator-container-revision`, accepted by both `validate` and@@ -443,6 +444,12 @@ revision, because a tag can be repointed after the card is published. Fieldsthe orchestration did not supply render as`not recorded (not supplied by the orchestration input)`.+Unlike the evaluated source identity, `Evaluator version` names the evaluator+build. It is the version recorded in the Tier 3 result, because that build ran+the live evaluation. When the run has no Tier 3 result that records one, as in+`validate --tiers 1`, `validate` writes its own version there instead of+`not recorded`.+Keep the file with the skill and refresh it whenever the skill, eval dataset,agent/model, evaluator version, scoring policy, attempt policy, or executionenvironment materially changes. Follow thediff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.pyindex 4d521129..91c698db 100644--- a/src/skillevaluator/cli.py+++ b/src/skillevaluator/cli.py@@ -2786,6 +2786,9 @@ def validate(policy=policy,target_path=target_display,expected_skill_name=publication_name,+ # A Git remote URL in ``target_display`` cannot be probed for SKILL.md,+ # and a clean run has no finding path to infer the skill from.+ explicit_skill_name=publication_name if resolved_type == CONTENT_TYPE_SKILL and target_path.is_dir() else None,content_label=content_label,announce_paths=not quiet,sarif_scan_root=target_path,@@ -2807,7 +2810,9 @@ def validate(# stays as the last line of defence: the renderer re-validates the# carriers it is handed, and a producer can record one after the fact.try:- BenchmarkReporter(skill_name=publication_name).save(results, output_dir / BENCHMARK_FILENAME)+ BenchmarkReporter(skill_name=publication_name, evaluator_version=__version__).save(+ results, output_dir / BENCHMARK_FILENAME+ )except EvaluatedSourceConflict as exc:# Publication fails closed on a contradictory identity, so report which# values disagreed rather than letting the card write a guess.diff --git a/src/skillevaluator/reporting/benchmark.py b/src/skillevaluator/reporting/benchmark.pyindex ea475dd4..81a7d958 100644--- a/src/skillevaluator/reporting/benchmark.py+++ b/src/skillevaluator/reporting/benchmark.py@@ -147,12 +147,17 @@ class BenchmarkReporter(ReporterBase):max_findings_shown: int = 5,skill_name: str | None = None,report_scope: str = "publication",+ evaluator_version: str | None = None,) -> None:self.include_timestamp = include_timestampself.max_findings_shown = max_findings_shownvalidate_report_scope(report_scope, [])self.report_scope = report_scopeself.skill_name = skill_name+ # The version of the evaluator writing this card. A version recorded in+ # the Tier 3 payload still wins, because that is the build that ran the+ # live evaluation; this fills in a run without one, such as Tier 1 only.+ self.evaluator_version = evaluator_version@propertydef name(self) -> str:@@ -360,7 +365,7 @@ class BenchmarkReporter(ReporterBase):summary = _mapping((ae or {}).get("summary"))version = _first_publication_safe_label(- ((ae or {}).get("evaluator_version"), summary.get("evaluator_version")),+ ((ae or {}).get("evaluator_version"), summary.get("evaluator_version"), self.evaluator_version),)lines.append(f"- Evaluator version: `{version}`"diff --git a/src/skillevaluator/reporting/json_reporter.py b/src/skillevaluator/reporting/json_reporter.pyindex 63bb338f..544079f5 100644--- a/src/skillevaluator/reporting/json_reporter.py+++ b/src/skillevaluator/reporting/json_reporter.py@@ -18,6 +18,7 @@ import jsonfrom datetime import UTC, datetimefrom typing import TYPE_CHECKING, Any+from skillevaluator import __version__from skillevaluator.publication_evidence import (TIER1_REQUIRED_CHECKS_KEY,TIER1_UNSCANNED_PATHS_KEY,@@ -96,16 +97,26 @@ class JSONReporter(ReporterBase):indent: int | None = 2,include_timestamp: bool = True,expected_skill_name: str | None = None,+ target_path: str | None = None,+ explicit_skill_name: str | None = None,) -> None:"""Initialize JSON reporter.Args:indent: JSON indentation level (None for compact)include_timestamp: Whether to include generation timestamp+ expected_skill_name: Publication target name the evidence must bind to+ target_path: Display path or Git remote URL of the validated target+ explicit_skill_name: Name of the one skill a single-skill run+ validated. It names the ``skills`` entry, which is otherwise+ inferred from finding paths and quality metadata and so falls+ back to one entry per check ID for a clean run without them."""self.indent = indentself.include_timestamp = include_timestampself.expected_skill_name = expected_skill_name+ self.target_path = target_path+ self.explicit_skill_name = explicit_skill_name@propertydef name(self) -> str:@@ -137,10 +148,25 @@ class JSONReporter(ReporterBase):result_finding_summaries = [summarize_findings([result]) for result in results]finding_summary = merge_finding_summaries(result_finding_summaries)- # Reuse HTML reporter's skill reorganization and contributor extraction- html_reporter = HTMLReporter()+ # Reuse HTML reporter's skill reorganization and contributor extraction.+ # The explicit name keeps a single-skill run to one ``skills`` entry+ # when no finding path or quality score names the skill, and the+ # target path is the same fallback the HTML report uses.+ html_reporter = HTMLReporter(+ target_path=self.target_path,+ explicit_skill_name=self.explicit_skill_name,+ expected_skill_name=self.expected_skill_name,+ )skills_by_name = html_reporter._reorganize_by_skill(results)contributors = HTMLReporter._extract_contributors(skills_by_name, results)+ # Top issues across every tier, the rows the HTML report data carries+ # (its summary panel lists the Tier 1 rows only), without ``issue_key``:+ # that is an HTML anchor hashed from the finding text, and CI redaction+ # masks the text in this report but cannot refresh a hash it never sees.+ top_issues = [+ {key: value for key, value in issue.items() if key != "issue_key"}+ for issue in html_reporter._compute_top_issues(skills_by_name)+ ]# Build per-skill summaryskills_summary = []@@ -156,6 +182,8 @@ class JSONReporter(ReporterBase):)data: dict[str, Any] = {+ "version": __version__,+ "target_path": self.target_path,"overall_passed": all_passed,"overall_status": overall_status,"incomplete_scans": incomplete_scans,@@ -177,6 +205,7 @@ class JSONReporter(ReporterBase):"message_counts": summarize_messages(results),"skills": skills_summary,"contributors": contributors,+ "top_issues": top_issues,"results": [self._result_to_dict(result, finding_summary=result_summary)for result, result_summary in zip(results, result_finding_summaries, strict=True)@@ -254,9 +283,18 @@ class JSONReporter(ReporterBase):if not isinstance(gating, dict) or gating.get("tier") is None:continuetier = str(gating["tier"])- tier_entry = gating_by_tier.setdefault(tier, {"blocking": False, "validators": []})+ tier_entry = gating_by_tier.setdefault(+ tier,+ {"blocking": False, "validators": [], "validator_policies": []},+ )tier_entry["blocking"] = bool(tier_entry["blocking"] or gating.get("blocking"))tier_entry["validators"].append(result.validator_name)+ # ``blocking`` above is true when any validator of the tier blocks;+ # this records each validator's own policy, so an advisory Tier 2+ # check is not read as blocking because another one is.+ tier_entry["validator_policies"].append(+ {"validator": result.validator_name, "blocking": bool(gating.get("blocking"))}+ )if gating_by_tier:data["gating"] = {"tiers": gating_by_tier}diff --git a/src/skillevaluator/tier1/commands.py b/src/skillevaluator/tier1/commands.pyindex ff9a9e6a..9b04bcab 100644--- a/src/skillevaluator/tier1/commands.py+++ b/src/skillevaluator/tier1/commands.py@@ -615,6 +615,7 @@ def emit_reports(policy: ValidationPolicy | None = None,target_path: str | None = None,expected_skill_name: str | None = None,+ explicit_skill_name: str | None = None,content_label: str = "Skill",announce_paths: bool = True,sarif_scan_root: Path | None = None,@@ -633,6 +634,10 @@ def emit_reports(forwarded to the HTML reporter so the report shows a Target link and thecorrect content noun. HTML navigation tabs are derived from the tierspresent in *results*, matching SkillEvaluator's combined report.++ The JSON report records *target_path* too. *explicit_skill_name* names+ the one skill a single-skill run validated, so the JSON ``skills`` list+ names it even when no finding path or quality score does."""if policy is not None:apply_policy(results, policy)@@ -666,6 +671,12 @@ def emit_reports(workspace_root=sarif_repository_root,scan_root=sarif_scan_root,)+ elif fmt == "json":+ reporter = reporter_cls(+ expected_skill_name=expected_skill_name,+ target_path=target_path,+ explicit_skill_name=explicit_skill_name,+ )else:reporter = reporter_cls(expected_skill_name=expected_skill_name)output_path = output_dir / f"{basename}{reporter.get_file_extension()}"diff --git a/tests/ci/test_native_reports.py b/tests/ci/test_native_reports.pyindex ff9ea348..e5a8066a 100644--- a/tests/ci/test_native_reports.py+++ b/tests/ci/test_native_reports.py@@ -435,7 +435,15 @@ def test_runtime_retains_public_json_and_regenerates_each_native_tier(tmp_path,if tier == 2:# CI supplies an advisory policy even for native JSON without gating.payload["results"][0]["gating"] = {"tier": 2, "blocking": False}- payload["gating"] = {"tiers": {"2": {"blocking": False, "validators": ["Context Optimization"]}}}+ payload["gating"] = {+ "tiers": {+ "2": {+ "blocking": False,+ "validators": ["Context Optimization"],+ "validator_policies": [{"validator": "Context Optimization", "blocking": False}],+ }+ }+ }assert json.loads((output / f"skillevaluator-tier{tier}.json").read_text()) == payloaddocument = (output / f"skillevaluator-tier{tier}.html").read_text()assert "ARTIFACT HTML" not in documentdiff --git a/tests/ci/test_run_tiers.py b/tests/ci/test_run_tiers.pyindex e83c9384..92d80db6 100644--- a/tests/ci/test_run_tiers.py+++ b/tests/ci/test_run_tiers.py@@ -740,6 +740,9 @@ def test_retained_tier2_reports_apply_ci_advisory_policy(consumer, tmp_path, faioriginal = json.loads(document)assert original["overall_passed"] is Falseassert original["results"][0]["gating"] == {"tier": 2, "blocking": True}+ assert original["gating"]["tiers"]["2"]["validator_policies"] == [+ {"validator": "Context Deduplication", "blocking": True}+ ]raw = tmp_path / "raw"raw.mkdir()(raw / "skillevaluator-tier2.json").write_text(document)@@ -751,6 +754,9 @@ def test_retained_tier2_reports_apply_ci_advisory_policy(consumer, tmp_path, faiassert outcome == {"status": status, "metrics": {"findings": int(failed)}, "complete": not incomplete}retained = json.loads((destination / "skillevaluator-tier2.json").read_text())assert retained["gating"]["tiers"]["2"]["blocking"] is False+ assert retained["gating"]["tiers"]["2"]["validator_policies"] == [+ {"validator": "Context Deduplication", "blocking": False}+ ]assert retained["overall_passed"] is Trueassert retained["overall_status"] == "passed"assert retained["results"][0]["gating"] == {"tier": 2, "blocking": False}diff --git a/tests/cli/test_tier_workflows.py b/tests/cli/test_tier_workflows.pyindex 8d65747a..27517e9b 100644--- a/tests/cli/test_tier_workflows.py+++ b/tests/cli/test_tier_workflows.py@@ -240,6 +240,45 @@ def test_tier2_without_catalog_does_not_manufacture_a_similarity_pass(skill, tmpassert "--catalog" in result.output+ ("tier", "validator"),+ [("tier1", "Schema Validation"), ("tier2", "Context Deduplication")],+)+def test_direct_tier_json_names_the_clean_skill_and_records_the_target_as_given(+ tier, validator, skill, tmp_path, monkeypatch+):+ """A clean result names no file and has no quality score, so the JSON+ names the skill from the target path, which records the path as typed."""+ from skillevaluator.tier1 import commands as tier1_commands+ from skillevaluator.tier2 import commands as tier2_commands++ options = []+ if tier == "tier1":+ monkeypatch.setattr(tier1_commands, "run_validation", lambda *_a, **_kw: _result(validator))+ options = ["--no-llm"]+ else:+ monkeypatch.setattr(tier2_commands, "run_context_optimization_check", lambda *_a, **_kw: _result(validator))+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "nv_build")+ monkeypatch.setenv("NVIDIA_API_KEY", "test-credential")+ monkeypatch.chdir(skill.parent)++ result = CliRunner().invoke(cli, [tier, skill.name, *options, "-r", "json", "-o", str(tmp_path / "reports")])++ assert result.exit_code == 0, result.output+ payload = _report(tmp_path / "reports")+ assert payload["skills"] == [{"name": "sample", "passed": True, "issue_count": 0}]+ assert [item["name"] for contributor in payload["contributors"] for item in contributor["items"]] == ["sample"]+ assert payload["target_path"] == "sample"+ number = tier.removeprefix("tier")+ assert payload["gating"]["tiers"] == {+ number: {+ "blocking": True,+ "validators": [validator],+ "validator_policies": [{"validator": validator, "blocking": True}],+ }+ }++def test_tier_name_collision_requires_explicit_path(tmp_path, monkeypatch):monkeypatch.chdir(tmp_path)path = tmp_path / "quality-check"diff --git a/tests/golden/test_benchmark_card.py b/tests/golden/test_benchmark_card.pyindex 62b1e8b6..0a8e2b30 100644--- a/tests/golden/test_benchmark_card.py+++ b/tests/golden/test_benchmark_card.py@@ -441,6 +441,37 @@ def test_benchmark_requires_tier3_by_default() -> None:assert "- Environment: not recorded (legacy or non-live result)" in rendered+def test_benchmark_records_the_writer_version_when_no_tier3_version_is_recorded() -> None:+ rendered = BenchmarkReporter(+ include_timestamp=False,+ skill_name="demo-skill",+ evaluator_version="1.2.3",+ ).render_all(_deterministic_results())++ assert "- Skill: `demo-skill`" in rendered+ assert "- Evaluator version: `1.2.3`" in rendered+ assert "- Evaluation date: not recorded (legacy or non-live result)" in rendered+++def test_benchmark_keeps_the_recorded_tier3_evaluator_version() -> None:+ rendered = BenchmarkReporter(include_timestamp=False, evaluator_version="9.9.9").render_all(+ [*_deterministic_results(), _live_tier3_result()]+ )++ assert "- Evaluator version: `0.8.2`" in rendered+ assert "9.9.9" not in rendered++[email protected]("retired_identity", ["SkillsEval", "SkillsE\u0301val"])+def test_benchmark_writer_version_is_normalized_like_a_recorded_one(retired_identity: str) -> None:+ rendered = BenchmarkReporter(include_timestamp=False, evaluator_version=retired_identity).render_all(+ _deterministic_results()+ )++ assert retired_identity not in rendered+ assert "- Evaluator version: `SkillEvaluator`" in rendered++def test_tier1_only_cli_result_cannot_publish_without_tier3() -> None:rendered = BenchmarkReporter(include_timestamp=False).render_all([_deterministic_results()[0]])diff --git a/tests/reporting/test_reporters.py b/tests/reporting/test_reporters.pyindex 31ce744b..8f874cbf 100644--- a/tests/reporting/test_reporters.py+++ b/tests/reporting/test_reporters.py@@ -506,6 +506,116 @@ class TestJSONReporter:assert data["total_validators"] == 2assert len(data["results"]) == 2+ @staticmethod+ def _clean_schema_result() -> ValidationResult:+ """A single-skill schema result with no finding, as a clean skill gives."""+ result = ValidationResult(+ validator_name="Schema & Repository Governance",+ validator_description="Validate SKILL.md frontmatter and repository structure",+ )+ result.add_success("manifest_exists", "Found skill manifest: SKILL.md")+ result.add_success("frontmatter_valid", "Valid frontmatter for skill 'simple'")+ result.add_success("naming_convention", "Folder name 'simple' follows kebab-case convention")+ return result++ def test_render_all_names_the_explicit_skill_of_a_clean_single_skill_run(self) -> None:+ data = json.loads(+ JSONReporter(+ include_timestamp=False,+ target_path="https://github.com/example/skills/tree/main/skills/simple",+ explicit_skill_name="simple",+ ).render_all([self._clean_schema_result()])+ )++ assert data["skills"] == [{"name": "simple", "passed": True, "issue_count": 0}]++ def test_render_all_names_the_skill_at_a_local_target_path(self, tmp_path: Path) -> None:+ skill = tmp_path / "simple"+ skill.mkdir()+ (skill / "SKILL.md").write_text("---\nname: simple\n---\n", encoding="utf-8")++ data = json.loads(+ JSONReporter(include_timestamp=False, target_path=str(skill)).render_all([self._clean_schema_result()])+ )++ assert [entry["name"] for entry in data["skills"]] == ["simple"]+ assert data["target_path"] == str(skill)++ def test_render_all_records_the_evaluator_version_and_target(self, success_result: ValidationResult) -> None:+ from skillevaluator import __version__++ target = "https://github.com/example/skills/tree/main/skills/test-skill"+ data = json.loads(JSONReporter(include_timestamp=False, target_path=target).render_all([success_result]))+ programmatic = json.loads(JSONReporter(include_timestamp=False).render_all([success_result]))++ assert data["version"] == __version__+ assert data["target_path"] == target+ assert programmatic["version"] == __version__+ assert programmatic["target_path"] is None++ def test_render_all_lists_the_html_top_issues_without_the_html_anchor(+ self, mixed_results: list[ValidationResult]+ ) -> None:+ output = JSONReporter(include_timestamp=False).render_all(mixed_results)+ data = json.loads(output)+ html_rows = _html_report_data(HTMLReporter(include_timestamp=False).render_all(mixed_results))["top_issues"]++ assert [(row["message"], row["severity"], row["total_count"]) for row in data["top_issues"]] == [+ ("AWS Access Key ID detected", "critical", 1),+ ("Generic API key pattern detected", "high", 1),+ ]+ assert all(+ set(row) == {"message", "severity", "total_count", "skills_affected", "category", "suggestion"}+ for row in data["top_issues"]+ )+ assert data["top_issues"] == [+ {key: value for key, value in row.items() if key != "issue_key"} for row in html_rows+ ]+ # The anchor hashes finding text, which CI redaction cannot refresh.+ assert all(row["issue_key"] not in output for row in html_rows)++ def test_top_issues_cover_every_tier_while_the_html_panel_lists_tier1(+ self, success_result: ValidationResult+ ) -> None:+ results = [success_result, _similarity_result(severity=Severity.HIGH)]++ data = json.loads(JSONReporter(include_timestamp=False).render_all(results))+ html = HTMLReporter(include_timestamp=False).render_all(results)++ assert [row["message"] for row in data["top_issues"]] == ["same-name similarity match"]+ assert data["top_issues"] == [+ {key: value for key, value in row.items() if key != "issue_key"}+ for row in _html_report_data(html)["top_issues"]+ ]+ # The HTML summary panel lists the Tier 1 rows only.+ assert "same-name similarity match" not in _html_tab(html, "tier1")++ def test_gating_tiers_record_each_validator_policy(self) -> None:+ schema = ValidationResult(validator_name="Schema & Repository Governance")+ schema.metadata["gating"] = {"tier": 1, "blocking": True}+ blocking_dedup = ValidationResult(validator_name="Context Deduplication")+ blocking_dedup.metadata["gating"] = {"tier": 2, "blocking": True}+ advisory_dedup = ValidationResult(validator_name="Inter-Skill Deduplication")+ advisory_dedup.metadata["gating"] = {"tier": 2, "blocking": False}++ data = json.loads(JSONReporter(include_timestamp=False).render_all([schema, blocking_dedup, advisory_dedup]))++ assert data["gating"]["tiers"] == {+ "1": {+ "blocking": True,+ "validators": ["Schema & Repository Governance"],+ "validator_policies": [{"validator": "Schema & Repository Governance", "blocking": True}],+ },+ "2": {+ "blocking": True,+ "validators": ["Context Deduplication", "Inter-Skill Deduplication"],+ "validator_policies": [+ {"validator": "Context Deduplication", "blocking": True},+ {"validator": "Inter-Skill Deduplication", "blocking": False},+ ],+ },+ }+def test_render_all_persists_typed_rubric_execution_metadata(self) -> None:result = ValidationResult(validator_name="RUBRIC_EVAL",@@ -1158,6 +1268,40 @@ def test_folded_pii_finding_in_a_folder_run_counts_each_location_once(tmp_path)assert re.search(rf"\| `{finding.finding_id}` \| HIGH \| [^|]+ \| 2 \| PII Scan \|", markdown)+def test_top_issues_count_a_folded_pii_finding_as_one_record(tmp_path) -> None:+ from skillevaluator.validators.security import SecurityValidator++ skill = tmp_path / "simple"+ skill.mkdir()+ (skill / "SKILL.md").write_text(+ "---\nname: simple\ndescription: A sample skill.\n---\n# Simple\n\n\n"+ "See /Users/johndoe/a.\nSee /Users/johndoe/b.\nSee /Users/johndoe/c.\n",+ encoding="utf-8",+ )+ result = SecurityValidator(use_llm=False).validate_pii_only(skill)+ result.validator_name = "PII Scan"+ [finding] = [finding for finding in result.findings if "/Users/johndoe/" in finding.message]+ assert finding.metadata["occurrence_count"] == 3++ data = json.loads(JSONReporter(include_timestamp=False, target_path=str(skill)).render_all([result]))++ # finding_groups lists each folded location; top_issues counts raw finding+ # records, the unit of skills[].issue_count, so the folded finding counts once.+ [group] = [group for group in data["finding_groups"] if group["finding_id"] == finding.finding_id]+ assert len(group["occurrences"]) == 3+ [issue] = [issue for issue in data["top_issues"] if "/Users/johndoe/" in issue["message"]]+ assert issue["total_count"] == 1+ assert data["skills"] == [{"name": "simple", "passed": False, "issue_count": len(result.findings)}]+ assert sum(issue["total_count"] for issue in data["top_issues"]) == data["skills"][0]["issue_count"]++ # The JSON field table in the CI docs states that unit.+ docs = (Path(__file__).resolve().parents[2] / "docs" / "ci-integration.mdx").read_text(encoding="utf-8")+ rows = {line.split(" | ", 1)[0]: line for line in docs.splitlines() if line.startswith("| `")}+ assert "`total_count` of raw finding records" in rows["| `top_issues`"]+ assert "raw occurrences" not in rows["| `top_issues`"]+ assert "`issue_count` counts raw finding records" in rows["| `skills`"]++def test_finding_group_occurrences_do_not_copy_the_source_line() -> None:result = ValidationResult(validator_name="Code Risk")for line_number in (3, 9):diff --git a/tests/test_commands.py b/tests/test_commands.pyindex 9846142f..40e872b4 100644--- a/tests/test_commands.py+++ b/tests/test_commands.py@@ -6,6 +6,7 @@ from __future__ import annotationsimport contextvarsimport jsonimport shutil+import subprocessimport threadingfrom collections.abc import Callablefrom datetime import timedelta@@ -488,6 +489,93 @@ def test_validate_quiet_always_writes_html_and_json_reports() -> None:assert any(p.suffix == ".json" for p in reports), reports[email protected]("in_git_checkout", [False, True], ids=("plain-folder", "git-checkout"))+def test_validate_json_names_a_clean_single_skill_and_records_the_evaluator(+ tmp_path: Path,+ in_git_checkout: bool,+) -> None:+ """A clean skill run with only the schema check has no finding path or+ quality score to name the skill by, and inside a Git checkout the report+ target is the remote URL. The JSON must still list the one skill."""+ from skillevaluator import __version__++ if in_git_checkout and shutil.which("git") is None:+ pytest.skip("git is not installed")+ repo = tmp_path / "repo"+ skill = repo / "skills" / "simple"+ shutil.copytree(FIXTURE, skill)+ if in_git_checkout:+ subprocess.run(["git", "init", "-q", str(repo)], check=True)+ subprocess.run(+ ["git", "-C", str(repo), "remote", "add", "origin", "https://github.com/example/skills.git"],+ check=True,+ )+ output_dir = tmp_path / "reports"++ result = CliRunner().invoke(+ cli,+ [+ "validate",+ str(skill),+ "--tiers",+ "1",+ "--checks",+ "schema",+ "--no-llm",+ "--report",+ "json",+ "--output-dir",+ str(output_dir),+ ],+ )++ assert result.exit_code == 0, result.output+ report = json.loads(next(output_dir.glob("skillevaluator-output-*.json")).read_text(encoding="utf-8"))+ assert report["skills"] == [{"name": "simple", "passed": True, "issue_count": 0}]+ assert [item["name"] for contributor in report["contributors"] for item in contributor["items"]] == ["simple"]+ assert report["version"] == __version__+ if in_git_checkout:+ assert report["target_path"].startswith("https://github.com/example/skills/")+ else:+ assert report["target_path"] == str(skill.resolve())+ assert report["top_issues"] == []+ assert report["gating"]["tiers"]["1"]["validator_policies"] == [+ {"validator": "Schema & Repository Governance", "blocking": True}+ ]+ benchmark = (output_dir / "BENCHMARK.md").read_text(encoding="utf-8")+ assert "- Skill: `simple`" in benchmark+ assert f"- Evaluator version: `{__version__}`" in benchmark+++def test_validate_json_keeps_one_skills_entry_per_rule_of_a_rules_folder(team_rule_dir: Path, tmp_path: Path) -> None:+ """Only a skill directory names its one skill explicitly. A rules folder+ groups its results per rule file, so naming the folder would fold both+ rules into one entry."""+ output_dir = tmp_path / "reports"++ result = CliRunner().invoke(+ cli,+ [+ "validate",+ str(team_rule_dir),+ "--tiers",+ "1",+ "--checks",+ "schema",+ "--no-llm",+ "--report",+ "json",+ "--output-dir",+ str(output_dir),+ ],+ )++ assert result.exit_code == 0, result.output+ report = json.loads(next(output_dir.glob("skillevaluator-output-*.json")).read_text(encoding="utf-8"))+ assert sorted(entry["name"] for entry in report["skills"]) == ["error-handling", "testing"]+ assert not (output_dir / "BENCHMARK.md").exists()++def test_validate_default_reports_do_not_write_into_authored_sibling_skill(monkeypatch: pytest.MonkeyPatch,tmp_path: Path,