Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier3): show which Tier 3 source checks failed (#290)
Narendran Raghavand69c438d6ec7
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 292b1c25..9137700d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -276,6 +276,21 @@ All notable changes to SkillEvaluator are documented in this file.- CI quality thresholds apply to every parallel scan worker. Report overviewsand detailed reports surface failed quality gates ahead of advisory findings,with links to the check that needs attention.+- When `validate` finds an unusable Tier 3 source, the terminal and HTML+ report now say which entries or files failed and why, instead of only+ "Create or fix the Tier 3 source ... and rerun validation." The terminal+ lists the errors under the Tier 3 finding when the full CLI report prints+ (`-r cli` or `--verbose`) and otherwise in the compact Tier 3 panel. When+ the run also writes an HTML or JSON report, the terminal shows the first+ five and names that report for the rest; when it writes neither (for example+ `-r cli` alone, or `--verbose` without `-r`), the terminal lists every+ error. The HTML dashboard shows a single error in full or the error count,+ and Full Insights lists every error under Source Validation Errors. The+ errors come from the skill's eval files, so they are printed as plain text+ with terminal control sequences removed. Passing checks and warnings are not+ listed. The JSON report still carries every error under+ `tier3.provenance.source_validation`. Validation rules, gate decisions, and+ the finding's suggestion text are unchanged.- Tier 2 caps every intra-skill embedding chunk at 3,000 characters, preservesmeaningful short tails and actual Python docstring source ranges, and avoidscounting pieces of one original chunk as duplicates of each other. Oversizeddiff --git a/docs/eval-datasets.mdx b/docs/eval-datasets.mdxindex 539eaee6..12000b0c 100644--- a/docs/eval-datasets.mdx+++ b/docs/eval-datasets.mdx@@ -347,6 +347,24 @@ skillevaluator tier3 validate ./my-skillRun it in CI or before a long evaluation: it catches an empty dataset, missingrequired fields, duplicate case IDs, and invalid config in seconds.+`validate PATH` also checks the selected Tier 3 source (dataset entries and+case IDs, `config.yml`, or the native Harbor task layout) before any live run.+When that check fails, each error names the entry or file and what is wrong,+for example `entry case-3: Missing required fields: question`:++- The terminal lists the errors under the Tier 3 finding when the full CLI+ report prints (`-r cli` or `--verbose`), otherwise in the compact Tier 3+ panel. When the run also writes an HTML or JSON report, the terminal shows+ the first five and names that report for the rest. When it writes neither+ (for example `-r cli` alone, or `--verbose` without `-r`), the terminal lists+ every error.+- The HTML report shows the exact error on the Tier 3 dashboard when there is+ one, or the error count when there are several. **Full Insights → Source+ Validation Errors** lists every error in order.+- The JSON report keeps every error under `tier3.provenance.source_validation`.++Passing checks and warnings are not listed as source errors.+## Next steps<CardGroup cols={2}>diff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.pyindex 80d27425..4c605164 100644--- a/src/skillevaluator/cli.py+++ b/src/skillevaluator/cli.py@@ -2425,7 +2425,14 @@ def validate(repo_context_exclude_paths=tuple(repo_context_exclude_paths),)results.append(tier3_result)- tier3_ran, tier3_ok, tier3_rows, tier3_skip = summarize_tier3(tier3_result)+ # The panel follows the formats this run emits: with cli the full report+ # lists the source errors under the Tier 3 finding, so the panel keeps+ # the count; it shortens its own list only when an HTML or JSON report+ # keeps every error.+ tier3_ran, tier3_ok, tier3_rows, tier3_skip = summarize_tier3(+ tier3_result,+ report_formats=_effective_report_formats(report_formats, quiet=quiet),+ )if autopilot_error and not tier3_ran:from skillevaluator.reporting.base import get_skip_reasondiff --git a/src/skillevaluator/constants.py b/src/skillevaluator/constants.pyindex 207217c4..a7a73c21 100644--- a/src/skillevaluator/constants.py+++ b/src/skillevaluator/constants.py@@ -567,6 +567,11 @@ AGENT_EVAL_DATASET_REQUIRED_FIELDS = [AGENT_EVAL_DATASET_FORMATS = {".json", ".jsonl", ".yaml", ".yml"}+# Terminal views preview this many failed Tier 3 source checks when the run+# also writes an HTML or JSON report, which keeps all of them; otherwise they+# print every failed check.+SOURCE_VALIDATION_CLI_LIMIT = 5+# =============================================================================# TIER 3 DIMENSION MAPPING (BENCHMARK.md / reporting)diff --git a/src/skillevaluator/evaluation/tier3_report.py b/src/skillevaluator/evaluation/tier3_report.pyindex ec17bda9..695851aa 100644--- a/src/skillevaluator/evaluation/tier3_report.py+++ b/src/skillevaluator/evaluation/tier3_report.py@@ -382,6 +382,9 @@ def dataset_required_result(applicability = "not_required"execution_status = "not_applicable"+ # Reporters list these as source errors, so passing checks and warnings stay+ # out. The detail lives in metadata rather than the suggestion, which keeps+ # the finding text stable while the author fixes entries one by one.serialized_checks = [{"path": str(getattr(check, "path", "")),@@ -389,6 +392,7 @@ def dataset_required_result("message": str(getattr(check, "message", check)),}for check in checks+ if str(getattr(check, "status", "error")) in {"missing", "error"}]result = ValidationResult(validator_name=_AGENT_EVAL_VALIDATOR,diff --git a/src/skillevaluator/reporting/base.py b/src/skillevaluator/reporting/base.pyindex a03536d5..10d5a79b 100644--- a/src/skillevaluator/reporting/base.py+++ b/src/skillevaluator/reporting/base.py@@ -36,6 +36,7 @@ from skillevaluator.constants import (DIMENSION_MAPPING,DIMENSION_VERDICT_NEUTRAL_THRESHOLD,DIMENSION_VERDICT_PASS_THRESHOLD,+ SOURCE_VALIDATION_CLI_LIMIT,)from skillevaluator.publication_evidence import (TIER1_REQUIRED_CHECKS_KEY,@@ -55,6 +56,8 @@ from skillevaluator.utils.rich_markup import strip_terminal_controls_AGENT_EVAL_MAX_FUTURE_CLOCK_SKEW = timedelta(minutes=5)if TYPE_CHECKING:+ from collections.abc import Iterable+from skillevaluator.models import ValidationResult@@ -2501,6 +2504,76 @@ def additional_warnings(result: ValidationResult) -> list[str]:return [warning for warning in result.warnings if warning not in represented]+_SOURCE_VALIDATION_ERROR_STATUSES = frozenset({"missing", "error"})+++def source_validation_errors(metadata: object) -> list[dict[str, str]]:+ """Return the failed Tier 3 source checks recorded in ``metadata``, in order.++ Tier 3 source preflight stores its checks under ``source_validation`` as+ ``{"path", "status", "message"}`` items. Paths and messages come from the+ skill's own eval files (entry IDs, parser errors), so callers still escape+ them for their medium. Malformed items and checks that passed or only+ warned are left out.+ """+ raw_checks = metadata.get("source_validation") if isinstance(metadata, dict) else None+ if not isinstance(raw_checks, list):+ return []+ errors: list[dict[str, str]] = []+ for check in raw_checks:+ if not isinstance(check, dict):+ continue+ path = check.get("path", "")+ status = check.get("status", "error")+ message = check.get("message")+ if (+ isinstance(path, str)+ and isinstance(status, str)+ and status in _SOURCE_VALIDATION_ERROR_STATUSES+ and isinstance(message, str)+ and message.strip()+ ):+ errors.append({"path": path, "status": status, "message": message})+ return errors+++def source_validation_full_report(report_formats: Iterable[str]) -> str | None:+ """Name the report file of this run that lists every failed Tier 3 source check.++ ``report_formats`` are the formats the run actually emits. HTML Full+ Insights and the JSON report's ``tier3.provenance.source_validation`` list+ every error. ``cli`` writes no file, Markdown has no section for them, and+ SARIF keeps them only as raw result properties, so ``None`` means no report+ of this run is a place to send the reader.+ """+ formats = set(report_formats)+ if "html" in formats:+ return "Full Insights in the HTML report"+ if "json" in formats:+ return "tier3.provenance.source_validation in the JSON report"+ return None+++def source_validation_preview(+ errors: list[dict[str, str]],+ report_formats: Iterable[str],+) -> tuple[list[dict[str, str]], str | None]:+ """Return the failed Tier 3 source checks a terminal view prints, and the note after them.++ A terminal view previews the first ``SOURCE_VALIDATION_CLI_LIMIT`` errors+ and names the report with the rest only when this run writes one (see+ :func:`source_validation_full_report`). Otherwise, for example with an+ explicit ``-r cli``, it prints every error and no note, because the rest+ would not be reachable anywhere else.+ """+ full_report = source_validation_full_report(report_formats)+ hidden = len(errors) - SOURCE_VALIDATION_CLI_LIMIT+ if full_report is None or hidden <= 0:+ return list(errors), None+ noun = "error" if hidden == 1 else "errors"+ return errors[:SOURCE_VALIDATION_CLI_LIMIT], f"{hidden} more {noun}: see {full_report}."++class ReporterBase(ABC):"""Abstract base class for validation result reporters.diff --git a/src/skillevaluator/reporting/cli.py b/src/skillevaluator/reporting/cli.pyindex e52cf543..1ceb2219 100644--- a/src/skillevaluator/reporting/cli.py+++ b/src/skillevaluator/reporting/cli.py@@ -39,6 +39,8 @@ from skillevaluator.reporting.base import (get_skip_reason,is_advisory_agent_eval_skip,passes_required_gate,+ source_validation_errors,+ source_validation_preview,)from skillevaluator.reporting.harbor_viewer import (harbor_evidence_link_text,@@ -49,10 +51,13 @@ from skillevaluator.reporting.run_status import Status, resolve_run_status, resufrom skillevaluator.utils.rich_markup import escape_markupif TYPE_CHECKING:+ from collections.abc import Iterable+from skillevaluator.models import Finding, ValidationResult_SKIP_REASON_MAX_CHARS = 1024+_SOURCE_ERROR_MAX_CHARS = 1024def _related_paths(finding: Finding) -> list[str]:@@ -68,6 +73,15 @@ def _related_paths(finding: Finding) -> list[str]:return paths+def _rich_source_error(error: dict[str, str]) -> str:+ """Return one failed Tier 3 source check as a bounded line inert to Rich markup."""+ text = f"{error['path']}: {error['message']}" if error["path"] else error["message"]+ text = " ".join(text.split())+ if len(text) > _SOURCE_ERROR_MAX_CHARS:+ text = text[: _SOURCE_ERROR_MAX_CHARS - 1] + "…"+ return escape_markup(text)++def _rich_skip_reason(result: ValidationResult) -> str:"""Return one bounded, single-line skip reason inert to Rich markup."""reason = " ".join(get_skip_reason(result).replace("\r\n", "\n").replace("\r", "\n").split())@@ -83,13 +97,17 @@ class CLIReporter(ReporterBase):Supports both direct printing and string capture for testing."""- def __init__(self, console: Console | None = None) -> None:+ def __init__(self, console: Console | None = None, *, report_formats: Iterable[str] = ()) -> None:"""Initialize CLI reporter.Args:console: Rich Console instance (creates new one if not provided)+ report_formats: Every format the run emits alongside this report.+ Failed Tier 3 source checks are cut to a preview only when one+ of these files lists all of them."""self._console = console+ self._report_formats = tuple(report_formats)@propertydef console(self) -> Console:@@ -574,6 +592,14 @@ class CLIReporter(ReporterBase):"Recommendation" if finding.category in ("WHOLE_DUPLICATE", "PARTIAL_OVERLAP", "DUPLICATE") else "Fix")console.print(f" [dim]{label}:[/dim] {escape_markup(finding.suggestion)}")+ source_errors = source_validation_errors(finding.metadata)+ if source_errors:+ shown, overflow_note = source_validation_preview(source_errors, self._report_formats)+ console.print(" [dim]Source validation:[/dim]")+ for error in shown:+ console.print(f" {_rich_source_error(error)}")+ if overflow_note:+ console.print(f" [dim]{escape_markup(overflow_note)}[/dim]")console.print()def _print_summary_table(self, results: list[ValidationResult], console: Console) -> None:diff --git a/src/skillevaluator/reporting/console_ui.py b/src/skillevaluator/reporting/console_ui.pyindex df9625f9..12723662 100644--- a/src/skillevaluator/reporting/console_ui.py+++ b/src/skillevaluator/reporting/console_ui.py@@ -45,10 +45,16 @@ from rich.panel import Panelfrom rich.text import Textfrom skillevaluator.constants import CONTENT_DEDUP_LLM_REFUSED_CHECK-from skillevaluator.reporting.base import get_skip_reason+from skillevaluator.reporting.base import (+ get_skip_reason,+ source_validation_errors,+ source_validation_preview,+)from skillevaluator.utils.rich_markup import strip_terminal_controlsif TYPE_CHECKING:+ from collections.abc import Iterable+from skillevaluator.models.result import ValidationResultGREEN = "#76B900" # NVIDIA green: success + identity moments only@@ -1195,6 +1201,27 @@ def _tier2_finding_is_scan_failure(finding: object) -> bool:return check_name in _TIER2_SCAN_FAILURE_CHECKS or check_name.endswith("_limit") or check_name.endswith("_error")+def _tier3_source_validation_rows(result: ValidationResult, *, report_formats: tuple[str, ...]) -> list[TierRow]:+ """Count the failed Tier 3 source checks and, unless the CLI report lists them, show them."""+ errors = [error for finding in result.findings for error in source_validation_errors(finding.metadata)]+ if not errors:+ return []+ count = len(errors)+ summary = f"{count} source validation {'error' if count == 1 else 'errors'}"+ if "cli" in report_formats:+ return [detail_row("validation", summary)]+ shown, overflow_note = source_validation_preview(errors, report_formats)+ if len(shown) < count:+ summary += f" (showing first {len(shown)})"+ rows = [detail_row("validation", summary)]+ for error in shown:+ text = f"{error['path']}: {error['message']}" if error["path"] else error["message"]+ rows.append(detail_row("source", sanitize_tier3_console_text(text)))+ if overflow_note:+ rows.append(detail_row("", overflow_note))+ return rows++def _tier3_run_dir(payload: dict) -> Path | None:provenance = payload.get("provenance")raw = provenance.get("run_dir") if isinstance(provenance, dict) else None@@ -1224,8 +1251,19 @@ def _tier2_finding_is_advisory_refusal(finding: object) -> bool:return str(getattr(finding, "severity", "")).casefold() not in {"critical", "high"}-def summarize_tier3(result: ValidationResult) -> tuple[bool, bool, list[TierRow], str]:- """Return (ran, passed, rows, skip_reason) for the agent-eval result."""+def summarize_tier3(+ result: ValidationResult,+ *,+ report_formats: Iterable[str] = (),+) -> tuple[bool, bool, list[TierRow], str]:+ """Return (ran, passed, rows, skip_reason) for the agent-eval result.++ Failed Tier 3 source checks add a count row. ``report_formats`` are the+ formats the run emits. When they include ``cli``, the full CLI report lists+ the errors under the finding, so the panel keeps only the count. Otherwise+ the panel lists them, and shortens the list only when an HTML or JSON+ report written by the run keeps every error.+ """metadata = result.metadata if isinstance(result.metadata, dict) else {}raw_payload = metadata.get("agent_eval")payload = raw_payload if isinstance(raw_payload, dict) else {}@@ -1280,6 +1318,7 @@ def summarize_tier3(result: ValidationResult) -> tuple[bool, bool, list[TierRow]glyph_style=f"bold {RED}",))+ rows.extend(_tier3_source_validation_rows(result, report_formats=tuple(report_formats)))return True, ok, rows + fix_rows, ""diff --git a/src/skillevaluator/reporting/html.py b/src/skillevaluator/reporting/html.pyindex c3d838a9..ce8e7b37 100644--- a/src/skillevaluator/reporting/html.py+++ b/src/skillevaluator/reporting/html.py@@ -67,6 +67,7 @@ from skillevaluator.reporting.base import (result_publication_target_conflict_marker,result_publication_target_dict,select_agent_eval_candidate,+ source_validation_errors,)from skillevaluator.reporting.base import (is_tier2_validator_name as _is_tier2_validator_name,@@ -984,6 +985,8 @@ def _sanitize_tier3_display_payload(value: dict[str, Any]) -> dict[str, Any]:payload["dimension_hints"] = _tier3_text_mapping(payload.get("dimension_hints"))evaluator_paths = payload["provenance"].get("evaluator_paths")payload["provenance"]["evaluator_paths"] = evaluator_paths if isinstance(evaluator_paths, dict) else {}+ if "source_validation" in payload["provenance"]:+ payload["provenance"]["source_validation"] = source_validation_errors(payload["provenance"])harbor_viewer = _sanitize_tier3_harbor_viewer(payload.get("harbor_viewer"))if harbor_viewer:payload["harbor_viewer"] = harbor_viewerdiff --git a/src/skillevaluator/reporting/templates/report.html.j2 b/src/skillevaluator/reporting/templates/report.html.j2index f8452197..3e38f926 100644--- a/src/skillevaluator/reporting/templates/report.html.j2+++ b/src/skillevaluator/reporting/templates/report.html.j2@@ -2158,6 +2158,7 @@{% set t3_max_attempts = (t3_attempt_policy.max_attempts or 1) | int %}{% set t3_dataset = tier3.dataset or [] %}{% set t3_provenance = tier3.provenance or {} %}+ {% set t3_source_errors = t3_provenance.get('source_validation') or [] %}{% set t3_harbor = tier3.get('harbor_viewer') or t3_summary.get('harbor_viewer') or {} %}{% set t3_harbor_job_url = t3_harbor.get('job_url') %}{% set t3_harbor_analysis_url = t3_harbor.get('analysis_url') %}@@ -2575,6 +2576,15 @@<p class="t3-next-steps-conclusion">No recommendations were generated for this evaluation.</p>{% endif %}{% endif %}+ {% if (t3_source_errors | length) == 1 %}+ <p class="t3-next-steps-conclusion">+ Source validation error: {% if t3_source_errors[0].path %}{{ t3_source_errors[0].path }}: {% endif %}{{ t3_source_errors[0].message }}+ </p>+ {% elif (t3_source_errors | length) > 1 %}+ <p class="t3-next-steps-conclusion">+ {{ t3_source_errors | length }} source validation errors found. Open Full Insights to review the details.+ </p>+ {% endif %}</div>{% if tier3.dimensions %}@@ -3492,6 +3502,18 @@{% set t3_recommendations = tier3.recommendations or [] %}{% set t3_has_suggestions = (t3_recommendations | length) > 0 or (tier3.suggestions and (tier3.suggestions | length) > 0) %}+ {% if t3_source_errors %}+ <details class="t3-details" id="tier3-source-validation-errors" open>+ <summary>Source Validation Errors ({{ t3_source_errors | length }})</summary>+ <div class="t3-details-body">+ <ol class="t3-next-steps-list">+ {% for error in t3_source_errors %}+ <li>{% if error.path %}{{ error.path }}: {% endif %}{{ error.message }}</li>+ {% endfor %}+ </ol>+ </div>+ </details>+ {% endif %}<p style="color:var(--text-secondary);font-size:0.85rem;margin:0 0 0.85rem;">Insights combine deterministic baselines (best performing agent, weakest dimension, coverage gaps)with additional contextual observations and action items produced by an LLM-as-Judge.@@ -3601,7 +3623,7 @@</details>{% endif %}- {% if not t3_has_conclusions and not t3_has_suggestions %}+ {% if not t3_source_errors and not t3_has_conclusions and not t3_has_suggestions %}<p style="color:var(--text-secondary);">No evaluation insights were generated.</p>{% endif %}</section>diff --git a/src/skillevaluator/tier1/commands.py b/src/skillevaluator/tier1/commands.pyindex 3451569b..11374341 100644--- a/src/skillevaluator/tier1/commands.py+++ b/src/skillevaluator/tier1/commands.py@@ -501,7 +501,8 @@ def emit_reports(if policy is not None:apply_policy(results, policy)if "cli" in report_formats:- CLIReporter(console=console).print_all(results)+ # The CLI report previews failed Tier 3 source checks only when a file written below lists them all.+ CLIReporter(console=console, report_formats=report_formats).print_all(results)html_tabs = _derive_html_tabs(results)for fmt in report_formats:diff --git a/tests/reporting/test_tier3_source_validation_reporting.py b/tests/reporting/test_tier3_source_validation_reporting.pynew file mode 100644index 00000000..692b191a--- /dev/null+++ b/tests/reporting/test_tier3_source_validation_reporting.py@@ -0,0 +1,538 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Failed Tier 3 source checks are listed in every report, not only in JSON."""++from __future__ import annotations++import json+import logging+import re+from pathlib import Path++import pytest+from click.testing import CliRunner+from jinja2 import make_logging_undefined+from markupsafe import escape++from skillevaluator.cli import cli+from skillevaluator.constants import SOURCE_VALIDATION_CLI_LIMIT+from skillevaluator.evaluation.tier3_report import advisory_skip_result, dataset_required_result+from skillevaluator.models import Finding, ValidationResult+from skillevaluator.reporting import CLIReporter, HTMLReporter, JSONReporter+from skillevaluator.reporting.base import (+ source_validation_errors,+ source_validation_full_report,+ source_validation_preview,+)+from skillevaluator.reporting.console_ui import summarize_tier3+from skillevaluator.reporting.html import _sanitize_tier3_display_payload+from skillevaluator.tier3.evals_spec import CheckResult++_MESSAGE = "Missing required fields: question"+_HTML_REPORT = "Full Insights in the HTML report"+_JSON_REPORT = "tier3.provenance.source_validation in the JSON report"+_OVERFLOW = f"see {_HTML_REPORT}."+_JSON_OVERFLOW = f"see {_JSON_REPORT}."+# Format combinations a run can emit: with and without the CLI report, and+# with HTML, JSON, both, or neither written as a file.+_FORMAT_SETS = [+ (),+ ("cli",),+ ("cli", "markdown", "sarif"),+ ("cli", "json"),+ ("cli", "html"),+ ("cli", "json", "html"),+ ("markdown",),+ ("json",),+ ("html", "json"),+]+++def _written_report(report_formats: tuple[str, ...]) -> str | None:+ """The written report that lists every source error, or None when the run writes none."""+ if "html" in report_formats:+ return _HTML_REPORT+ if "json" in report_formats:+ return _JSON_REPORT+ return None+++def _overflow_note(hidden: int, report: str | None) -> str:+ return f"{hidden} more {'error' if hidden == 1 else 'errors'}: see {report}."+++def _plain(output: str) -> str:+ """Strip ANSI styling and Rich's line wrapping."""+ return " ".join(re.sub(r"\x1b\[[0-9;]*m", "", output).split())+++def _source_result(tmp_path: Path, checks: list[CheckResult], *, blocking: bool = True) -> ValidationResult:+ return dataset_required_result(+ tmp_path / "evals" / "evals.json",+ checks,+ blocking=blocking,+ skill_name="example-skill",+ source_kind="evals_json",+ )+++def _entry_errors(count: int, message: str = _MESSAGE) -> list[CheckResult]:+ return [CheckResult(f"entry case-{index}", "error", message) for index in range(1, count + 1)]+++def _html_section(html: str, section_id: str) -> str:+ start = html.index(f'id="{section_id}"')+ return html[start : html.index("</section>", start)]++[email protected]("blocking", [False, True])+def test_dataset_required_result_records_only_failed_checks(tmp_path: Path, blocking: bool) -> None:+ result = _source_result(+ tmp_path,+ [+ CheckResult("evals/evals.json", "ok", "Tier 3 eval dataset found."),+ CheckResult("entry case-1", "error", _MESSAGE),+ CheckResult("entry case-1", "warning", "Missing recommended fields: expected_skill"),+ CheckResult("evals/config.yml", "missing", "Required config is absent."),+ ],+ blocking=blocking,+ )++ expected = [+ {"path": "entry case-1", "status": "error", "message": _MESSAGE},+ {"path": "evals/config.yml", "status": "missing", "message": "Required config is absent."},+ ]+ assert result.findings[0].metadata["source_validation"] == expected+ assert result.metadata["agent_eval"]["provenance"]["source_validation"] == expected+ json_report = json.loads(JSONReporter(include_timestamp=False).render_all([result]))+ assert json_report["tier3"]["provenance"]["source_validation"] == expected+++def test_source_errors_stay_out_of_the_finding_text(tmp_path: Path) -> None:+ # Report grouping and SARIF results key on the finding text, so fixing one+ # entry at a time must not turn the remaining errors into a "new" finding.+ before = _source_result(tmp_path, _entry_errors(13))+ after = _source_result(tmp_path, _entry_errors(13)[3:])+ suggestion = f"Create or fix the Tier 3 source under {tmp_path / 'evals'} and rerun validation."++ assert before.findings[0].suggestion == after.findings[0].suggestion == suggestion+ assert before.findings[0].to_legacy_string() == after.findings[0].to_legacy_string()+ assert before.metadata["agent_eval"]["suggestions"] == after.metadata["agent_eval"]["suggestions"] == [suggestion]+ assert len(after.findings[0].metadata["source_validation"]) == 10++[email protected]("report_formats", [fmts for fmts in _FORMAT_SETS if "cli" in fmts])[email protected]("error_count", [0, 1, 5, 6, 13, 100])+def test_cli_report_previews_source_errors_in_order(+ tmp_path: Path,+ error_count: int,+ report_formats: tuple[str, ...],+) -> None:+ result = _source_result(tmp_path, _entry_errors(error_count))+ output = _plain(CLIReporter(report_formats=report_formats).render(result))++ report = _written_report(report_formats)+ # Without an HTML or JSON report the terminal is the only place the errors+ # appear, so it lists every one of them and names no report.+ shown = min(error_count, SOURCE_VALIDATION_CLI_LIMIT) if report else error_count+ assert re.findall(r"entry case-\d+:", output) == [f"entry case-{index}:" for index in range(1, shown + 1)]+ assert "Create or fix the Tier 3 source" in output+ assert ("Source validation:" in output) == (error_count > 0)+ if error_count:+ assert output.index("Source validation:") > output.index("Create or fix the Tier 3 source")+ hidden = error_count - shown+ if hidden:+ assert _overflow_note(hidden, report) in output+ else:+ assert "more error" not in output+ for other_report in {_HTML_REPORT, _JSON_REPORT} - {report}:+ assert other_report not in output+++def test_cli_reporter_without_report_formats_lists_every_source_error(tmp_path: Path) -> None:+ # A reporter that is not told about any written report must not point at one.+ output = _plain(CLIReporter().render(_source_result(tmp_path, _entry_errors(13))))++ assert re.findall(r"entry case-\d+:", output) == [f"entry case-{index}:" for index in range(1, 14)]+ assert "more error" not in output+ assert _HTML_REPORT not in output+ assert _JSON_REPORT not in output++[email protected]("report_formats", _FORMAT_SETS)[email protected]("error_count", [0, 5, 6, 13])+def test_source_validation_preview_names_only_written_reports(+ error_count: int,+ report_formats: tuple[str, ...],+) -> None:+ errors = [{"path": f"entry case-{index}", "status": "error", "message": _MESSAGE} for index in range(error_count)]+ report = _written_report(report_formats)++ shown, note = source_validation_preview(errors, report_formats)++ assert source_validation_full_report(report_formats) == report+ if report and error_count > SOURCE_VALIDATION_CLI_LIMIT:+ assert shown == errors[:SOURCE_VALIDATION_CLI_LIMIT]+ assert note == _overflow_note(error_count - SOURCE_VALIDATION_CLI_LIMIT, report)+ else:+ assert shown == errors+ assert note is None+++def test_cli_report_prints_source_errors_as_inert_text(tmp_path: Path) -> None:+ result = _source_result(+ tmp_path,+ [+ CheckResult(+ "entry [bold]x[/bold]\x1b[2J\x1b]8;;https://evil.test/\x07", "error", "Expected <field>\n& more."+ ),+ CheckResult("", "error", "Cannot parse dataset: [/x] at C:\\evals\\"),+ ],+ )++ text = re.sub(r"\x1b\[[0-9;]*m", "", CLIReporter().render(result))++ # Rich's own styling aside, no escape sequence from the eval file reaches the terminal.+ assert re.search(r"[\x00-\x09\x0b-\x1f\x7f-\x9f]", text) is None, repr(text)+ output = " ".join(text.split())+ assert "Source validation: entry [bold]x[/bold]: Expected <field> & more." in output+ assert "more. Cannot parse dataset: [/x] at C:\\evals\\" in output+ assert "evil.test" not in output+++def test_cli_report_ignores_malformed_source_validation_metadata() -> None:+ result = ValidationResult(validator_name="AGENT_EVAL")+ result.add_finding(+ Finding(+ category="AGENT_EVAL",+ severity="high",+ check_name="eval_dataset_required",+ message="No valid Tier 3 source.",+ file_path="evals/evals.json",+ suggestion="Keep the existing recommendation.",+ metadata={+ "source_validation": [+ "entry text-only",+ {"path": "entry ok", "status": "ok", "message": "Loaded."},+ {"path": "entry warn", "status": "warning", "message": "Optional field."},+ {"path": 7, "status": "error", "message": "Non-string path."},+ {"path": "entry list", "status": ["error"], "message": "List status."},+ {"path": "entry dict", "status": {"error": True}, "message": "Mapping status."},+ {"path": "entry blank", "status": "error", "message": " "},+ {"path": "entry kept", "status": "error", "message": "Invalid entry."},+ ]+ },+ )+ )++ output = _plain(CLIReporter().render(result))++ assert "Keep the existing recommendation." in output+ assert "Source validation: entry kept: Invalid entry." in output+ for dropped in (+ "text-only",+ "Loaded.",+ "Optional field.",+ "Non-string path.",+ "List status.",+ "Mapping status.",+ "entry blank",+ ):+ assert dropped not in output++[email protected]("metadata", [{}, {"source_validation": []}, {"source_validation": "entry x: bad"}])+def test_cli_report_omits_source_section_without_errors(metadata: dict) -> None:+ result = ValidationResult(validator_name="SCHEMA")+ result.add_finding(+ Finding(+ category="SCHEMA",+ severity="high",+ check_name="schema-check",+ message="A validation finding.",+ file_path="SKILL.md",+ suggestion="Keep the existing recommendation.",+ metadata=metadata,+ )+ )++ output = _plain(CLIReporter().render(result))++ assert "Keep the existing recommendation." in output+ assert "Source validation:" not in output++[email protected]("blocking", [False, True])[email protected]("error_count", [0, 1, 5, 6, 13, 100])[email protected]("report_formats", _FORMAT_SETS)+def test_compact_tier3_panel_counts_and_previews_source_errors(+ tmp_path: Path,+ blocking: bool,+ error_count: int,+ report_formats: tuple[str, ...],+) -> None:+ result = _source_result(tmp_path, _entry_errors(error_count), blocking=blocking)+ finding = result.findings[0]+ original_suggestion = finding.suggestion+ original_metadata = json.dumps(result.metadata, sort_keys=True)++ ran, _passed, rows, skip_reason = summarize_tier3(result, report_formats=report_formats)++ assert ran is True+ assert skip_reason == ""+ report = _written_report(report_formats)+ # The full CLI report lists the errors itself; otherwise the panel lists all+ # of them unless an HTML or JSON report written by the run keeps the rest.+ if "cli" in report_formats:+ shown = 0+ elif report:+ shown = min(error_count, SOURCE_VALIDATION_CLI_LIMIT)+ else:+ shown = error_count+ assert [row.segments[0][0] for row in rows if row.label == "source"] == [+ f"entry case-{index}: {_MESSAGE}" for index in range(1, shown + 1)+ ]+ counts = [row.segments[0][0] for row in rows if row.label == "validation"]+ truncated = "cli" not in report_formats and shown < error_count+ if error_count:+ expected = f"{error_count} source validation {'error' if error_count == 1 else 'errors'}"+ if truncated:+ expected += f" (showing first {SOURCE_VALIDATION_CLI_LIMIT})"+ assert counts == [expected]+ else:+ assert counts == []+ rendered = " ".join(text for row in rows for text, _style in row.segments)+ if truncated:+ assert _overflow_note(error_count - SOURCE_VALIDATION_CLI_LIMIT, report) in rendered+ else:+ assert "more error" not in rendered+ assert finding.suggestion == original_suggestion+ assert json.dumps(result.metadata, sort_keys=True) == original_metadata+++def test_compact_tier3_panel_projects_source_errors_to_one_safe_line(tmp_path: Path) -> None:+ result = _source_result(+ tmp_path,+ [CheckResult("entry \x1b[2Jcase\x07-1", "error", "Expected\na value [bold]here[/bold]. " + "x" * 300)],+ )++ _ran, _passed, rows, _skip_reason = summarize_tier3(result)++ (source_row,) = [row for row in rows if row.label == "source"]+ text = source_row.segments[0][0]+ assert text.startswith("entry [2Jcase -1: Expected a value [bold]here[/bold]. x")+ assert text.endswith("…")+ assert len(text) <= 110+ assert not any(ord(character) < 32 for character in text)++[email protected]("error_count", [0, 1, 2, 13, 100])+def test_html_report_shows_source_errors_in_preview_and_full_insights(tmp_path: Path, error_count: int) -> None:+ result = _source_result(tmp_path, _entry_errors(error_count))+ html = HTMLReporter(include_timestamp=False).render_all([result])+ preview = _html_section(html, "tier3-next-steps-preview")+ insights = _html_section(html, "tier3-page-suggestions")++ assert "Open Full Insights</button>" in preview+ assert "Create or fix the Tier 3 source" in preview+ assert "Create or fix the Tier 3 source" in insights+ if error_count == 1:+ assert f"Source validation error: entry case-1: {_MESSAGE}" in preview+ elif error_count > 1:+ assert f"{error_count} source validation errors found. Open Full Insights to review the details." in preview+ assert "entry case-" not in preview+ else:+ assert "Source validation error" not in preview+ assert "source validation errors found" not in preview++ if error_count:+ assert f"Source Validation Errors ({error_count})" in insights+ items = re.findall(r"<li>(entry case-\d+): ([^<]*)</li>", insights)+ assert items == [(f"entry case-{index}", _MESSAGE) for index in range(1, error_count + 1)]+ else:+ assert 'id="tier3-source-validation-errors"' not in insights+++def test_html_report_escapes_source_errors(tmp_path: Path) -> None:+ result = _source_result(tmp_path, [CheckResult("entry <b>&", "error", "Expected <script>alert(1)</script>.")])+ html = HTMLReporter(include_timestamp=False).render_all([result])+ preview = _html_section(html, "tier3-next-steps-preview")+ insights = _html_section(html, "tier3-page-suggestions")+ error_text = escape("entry <b>&: Expected <script>alert(1)</script>.")++ assert f"Source validation error: {error_text}" in preview+ assert f"<li>{error_text}</li>" in insights+ assert "<script>alert(1)" not in html++[email protected]("has_source_errors", [False, True])+def test_html_source_errors_keep_existing_insights(tmp_path: Path, has_source_errors: bool) -> None:+ result = _source_result(tmp_path, _entry_errors(1 if has_source_errors else 0))+ payload = result.metadata["agent_eval"]+ payload["conclusions"] = [{"title": "Coverage gap", "message": "Error handling needs more coverage."}]+ payload["recommendations"] = [{"title": "Improve coverage", "message": "Add an error-handling test."}]++ insights = _html_section(HTMLReporter(include_timestamp=False).render_all([result]), "tier3-page-suggestions")++ assert "<summary>Conclusions</summary>" in insights+ assert "Error handling needs more coverage." in insights+ assert "<summary>Recommendations</summary>" in insights+ assert "Add an error-handling test." in insights+ assert ('id="tier3-source-validation-errors"' in insights) == has_source_errors++[email protected]("has_source_errors", [False, True])+def test_html_empty_insights_message_requires_no_source_errors(tmp_path: Path, has_source_errors: bool) -> None:+ payload = _source_result(tmp_path, _entry_errors(1 if has_source_errors else 0)).metadata["agent_eval"]+ payload["suggestions"] = []+ # A bare payload carries no finding diagnostics that would become recommendations.+ result = ValidationResult(validator_name="AGENT_EVAL")+ result.passed = False+ result.metadata["agent_eval"] = payload++ insights = _html_section(HTMLReporter(include_timestamp=False).render_all([result]), "tier3-page-suggestions")++ assert ("No evaluation insights were generated." in insights) == (not has_source_errors)+ assert ('id="tier3-source-validation-errors"' in insights) == has_source_errors+++def test_skipped_tier3_html_defines_source_errors(caplog: pytest.LogCaptureFixture) -> None:+ reporter = HTMLReporter(include_timestamp=False, expected_skill_name="example-skill")+ reporter._env.undefined = make_logging_undefined(logger=logging.getLogger(__name__))++ with caplog.at_level(logging.WARNING):+ html = reporter.render_all([advisory_skip_result("Harbor runtime unavailable", skill_name="example-skill")])++ assert "Harbor runtime unavailable" in html+ assert 'id="tier3-source-validation-errors"' not in html+ assert "t3_source_errors" not in caplog.text+++def test_html_payload_keeps_only_well_formed_source_errors() -> None:+ payload = _sanitize_tier3_display_payload(+ {+ "provenance": {+ "source_validation": [+ {"path": "entry a", "status": "error", "message": "Invalid entry.", "extra": "dropped"},+ {"path": "entry b", "status": "ok", "message": "Loaded."},+ {"path": None, "status": "error", "message": "Bad path."},+ {"path": "entry d", "status": ["error"], "message": "Unhashable status."},+ ["entry c", "error", "Not a mapping."],+ {"message": "No path or status recorded."},+ ]+ }+ }+ )++ assert payload["provenance"]["source_validation"] == [+ {"path": "entry a", "status": "error", "message": "Invalid entry."},+ {"path": "", "status": "error", "message": "No path or status recorded."},+ ]+ assert source_validation_errors({"source_validation": {"path": "entry a"}}) == []+++def _write_invalid_dataset(skill: Path, count: int) -> None:+ (skill / "evals").mkdir()+ (skill / "evals" / "evals.json").write_text(+ json.dumps([{"id": f"case-{index}", "expected_behavior": ["Done."]} for index in range(1, count + 1)]),+ encoding="utf-8",+ )+++def _invoke_validate(+ monkeypatch: pytest.MonkeyPatch,+ tmp_path: Path,+ extra_args: list[str],+ *,+ blocking: bool,+) -> tuple[str, Path]:+ """Run ``validate`` on a skill with 13 invalid eval entries; return its plain output and report directory."""+ from skillevaluator import cli as cli_module+ from skillevaluator.evaluation import EvaluationService++ # A forced terminal would animate the pipeline view and repeat each frame.+ for name in ("FORCE_COLOR", "TTY_COMPATIBLE"):+ monkeypatch.delenv(name, raising=False)+ monkeypatch.setenv("COLUMNS", "120")+ skill = tmp_path / "example-skill"+ skill.mkdir()+ (skill / "SKILL.md").write_text("---\nname: example-skill\ndescription: test\n---\n", encoding="utf-8")+ _write_invalid_dataset(skill, 13)+ tier1 = ValidationResult(validator_name="SCHEMA")+ tier1.add_success("schema", "ok")+ monkeypatch.setattr(cli_module, "run_validation", lambda *_args, **_kwargs: [tier1])+ monkeypatch.setattr(+ EvaluationService,+ "evaluate",+ lambda *_args, **_kwargs: pytest.fail("source preflight must run before live evaluation"),+ )+ output_dir = tmp_path / "reports"+ args = ["validate", "--no-autopilot", str(skill), "--no-tier2", "--tier3", "--output-dir", str(output_dir)]+ args.extend(extra_args)+ if blocking:+ args.append("--block-on-agent-eval")++ command = CliRunner().invoke(cli, args)++ assert command.exit_code == (1 if blocking else 0), command.output+ return _plain(command.output), output_dir++[email protected]("blocking", [False, True])[email protected]("report_formats", [None, "cli", "json", "cli,json", "cli,json,html", "json,html"])+def test_validate_shows_source_errors_once(+ monkeypatch: pytest.MonkeyPatch,+ tmp_path: Path,+ blocking: bool,+ report_formats: str | None,+) -> None:+ extra_args = ["--report", report_formats] if report_formats else []++ output, output_dir = _invoke_validate(monkeypatch, tmp_path, extra_args, blocking=blocking)++ # The quiet default writes HTML and JSON; an explicit -r is honoured exactly.+ emitted = tuple(report_formats.split(",")) if report_formats else ("html", "json")+ full_cli_report = "cli" in emitted+ report = _written_report(emitted)+ shown = SOURCE_VALIDATION_CLI_LIMIT if report else 13+ assert "13 source validation errors" in output+ assert ("(showing first 5)" in output) == (not full_cli_report and report is not None)+ assert ("Source validation:" in output) == full_cli_report+ assert re.findall(r"entry case-\d+:", output) == [f"entry case-{index}:" for index in range(1, shown + 1)]+ if report:+ assert output.count(_overflow_note(13 - shown, report)) == 1+ else:+ assert "more error" not in output+ for other_report in {_HTML_REPORT, _JSON_REPORT} - {report}:+ assert other_report not in output+ json_reports = list(output_dir.glob("*.json"))+ assert len(json_reports) == ("json" in emitted)+ if json_reports:+ json_report = json.loads(json_reports[0].read_text(encoding="utf-8"))+ assert len(json_report["tier3"]["provenance"]["source_validation"]) == 13+ html_reports = list(output_dir.glob("*.html"))+ assert len(html_reports) == ("html" in emitted)+ if html_reports:+ html = html_reports[0].read_text(encoding="utf-8")+ assert "Source Validation Errors (13)" in html+ assert f"<li>entry case-13: {_MESSAGE}</li>" in html++[email protected]("blocking", [False, True])+def test_verbose_validate_without_file_reports_lists_every_source_error(+ monkeypatch: pytest.MonkeyPatch,+ tmp_path: Path,+ blocking: bool,+) -> None:+ # --verbose prints the full CLI report and, without -r, writes no HTML or+ # JSON report, so every error must be in the terminal output.+ output, output_dir = _invoke_validate(monkeypatch, tmp_path, ["--verbose"], blocking=blocking)++ assert "Source validation:" in output+ assert re.findall(r"entry case-\d+:", output) == [f"entry case-{index}:" for index in range(1, 14)]+ assert "more error" not in output+ assert _HTML_REPORT not in output+ assert _JSON_REPORT not in output+ assert list(output_dir.glob("*.json")) == []+ assert list(output_dir.glob("*.html")) == []