Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

fix(reporting): align scoped verdicts and trial accounting (#351)

Narendran Raghavandd274121e875
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 71b48c84..963f987c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,11 @@ All notable changes to SkillEvaluator are documented in this file.
- When CI redaction masks a credential inside a reported PII value, remove the
finding's digest as well.
+- Align final terminal and HTML command outcomes: Passed required checks,
+ Changes required, or Incomplete. Keep publication eligibility and Best agent
+ score separate. Tier counts exclude explicit N/A checks from execution;
+ Tier 3 trial summaries use recorded condition counters before retained rows.
+
## 0.8.0 - 2026-11-05
### Added
diff --git a/docs/reports.mdx b/docs/reports.mdx
index 77095985..93a20b2b 100644
--- a/docs/reports.mdx
+++ b/docs/reports.mdx
@@ -63,6 +63,35 @@ skillevaluator validate ./my-skill -r json,html,markdown,sarif -o reports
For gating a pipeline on the JSON output, see [Gate Your CI](ci-integration.mdx).
+## Reading the final status
+
+The compact terminal summary, full terminal report, and HTML header describe the
+same command outcome: **Passed required checks**, **Changes required**, or
+**Incomplete**. A confirmed failure takes precedence over missing evidence.
+Advisory tiers retain their own findings and execution status without changing
+the required gate or process exit code.
+
+**Publication eligibility** is a separate, stricter assessment of the recorded
+policy, provenance, and evidence coverage. A passed required gate does not imply
+publication eligibility. Standalone Tier 3 reports show **Not assessed (Tier 3
+only)**. **Best agent score** describes one evaluated agent, rather than approval
+of the whole validation run.
+
+Each tier counts its own passed, failed, incomplete, skipped, and N/A results.
+Skipped and explicitly inapplicable checks are excluded from executed-check
+denominators. A tier that produced no results is **Not run**; presentation of an
+unrequested tier does not create evidence that it ran.
+
+Tier 3 summaries prefer recorded per-condition counters: for example,
+**188 scored trials: 101 with skill, 87 baseline**. Condition details retain
+expected counts and execution status, so partial execution and early stopping
+remain distinguishable. Missing or malformed counters are **unavailable**, and
+explicitly skipped arms remain **skipped**. Legacy reports with trial rows but
+no counters show **retained rows**, which are observations rather than complete
+scored totals. Preview limits and physical retry rows do not replace recorded
+attempt counts. These are display projections; canonical JSON and evaluation
+payloads are unchanged.
+
## File naming
Report filenames follow two patterns, both under the output directory:
diff --git a/docs/superpowers/plans/2026-11-05-professional-reporting.md b/docs/superpowers/plans/2026-11-05-professional-reporting.md
new file mode 100644
index 00000000..7e3a4770
--- /dev/null
+++ b/docs/superpowers/plans/2026-11-05-professional-reporting.md
@@ -0,0 +1,94 @@
+# Professional NVIDIA Reporting Implementation Plan
+
+> For agentic workers: implement task by task using subagent-driven development or executing-plans. Each task requires self-review, an independent review, and fixes for confirmed findings before its PR is ready.
+
+**Goal:** Make GitHub SkillEvaluator's CLI and standalone HTML reports consistent, understandable, accessible, and professional with NVIDIA styling.
+
+**Architecture:** Extend existing public status, presentation, provenance, resource, and Harbor-link abstractions. Keep canonical evaluation data and decision policies unchanged; render shared presentation projections into HTML and terminal views.
+
+**Tech stack:** Python, Rich, Jinja2, vanilla browser JavaScript/CSS, packaged Chart.js 4.5.1, pytest, Ruff, Fern.
+
+**Spec:** The user-approved Professional NVIDIA Reporting plan, reproduced as executable tasks below. GitHub repository only; no internal source or integrations are imported.
+
+## Global Constraints
+
+- Preserve scoring formulas, thresholds, publication policy, exit codes, CLI flags/defaults, artifact filenames, canonical JSON schemas, redaction, and report-size limits.
+- Use latest GitHub main in isolated worktrees; preserve the existing checkout and its untracked duplicate directories.
+- Extend existing shared status logic; no duplicate verdict resolver, fabricated measurements, inferred missing baselines, or mutated canonical payloads.
+- Keep report_scope's existing decision semantics separate from requested/displayed tier context. Optional keyword-only presentation arguments must preserve existing callers and emit_reports()'s boolean return.
+- Keep private integrations and internal plugin-specific code out of the public changes. Use only existing public recorded evidence and public upstream assets.
+- Each PR includes focused behavioral tests, documentation/changelog, signed-off commits, root self-review, independent review, and fixes for confirmed findings.
+- Publish five focused PRs; keep drafts during verification, mark ready when verified, and leave them open without merging main.
+
+## Review Focus
+
+- Confirmed failures plus missing evidence: command status must agree across terminal and HTML while retaining stricter publication assessment separately.
+- Missing/N/A/skipped/baseline-only/retained/truncated evidence: unknown is not zero, N/A is not execution, and preview caps cannot change totals.
+- Long hostile or Unicode names/paths: escape markup/HTML/control characters, preserve safe links, and fit narrow displays without losing numeric values.
+- Offline and installed-wheel reports: all chart assets/licenses are packaged; generated reports make no runtime network requests for charts.
+- Non-TTY interruption/closed streams: durable progress must stop cleanly without affecting evaluation, duplicating events, or leaking workers.
+
+### Task 1: Consistent status, scope, and execution accounting
+
+Branch: naren/reporting-status-accounting, based on current main.
+
+- [ ] Extend existing run_status and presentation helpers with an immutable final-render context: command outcome, separate publication assessment, tier status/counts, score scope, and per-condition coverage. Build once per final render and share across compact CLI, full CLI, HTML header/hero/tiles.
+- [ ] Preserve failure precedence and advisory gating. Use primary labels Passed required checks / Changes required / Incomplete. Keep publication eligibility separate; standalone Tier 3 publication remains Not assessed.
+- [ ] Label agent scores explicitly (Best agent score); they are not whole-validation approval.
+- [ ] Count each tier's own results; distinguish passed, failed, incomplete, skipped, not run and N/A, excluding N/A from executed denominators.
+- [ ] Prefer recorded per-condition scored/expected counters before previews. Display N scored trials: X with skill, Y baseline. Missing/malformed remains unavailable, explicit skip stays skipped, legacy row counts are observed/retained rows, not complete totals. Handle retries, early stopping, skill-only/baseline-only, partial/failed arms and >64 preview rows.
+- [ ] Add optional keyword-only requested/displayed presentation context where necessary in emit_reports and HTMLReporter, preserving defaults, decision report_scope, return values and JSON schemas. Do not synthesize ValidationResult evidence for missing tiles.
+- [ ] Add meaningful failing tests first, implement, run focused reporting/status/CLI tests; prove payload immutability and status/count parity. Update user docs and changelog; signed-off commit; self-review and independent review.
+
+### Task 2: Truly self-contained HTML charts
+
+Branch: naren/reporting-offline-charts, independently based on current main.
+
+- [ ] Vendor public Chart.js 4.5.1 UMD from public upstream; verify its original bytes using the existing SRI, record source/version/digests, retain applicable Chart.js/bundled dependency licenses in resources, notices and generated reports.
+- [ ] Load fixed package resources with a cached loader and embed only when Tier 3 requires charts. Remove CDN/source-map dependencies. No new npm/Python runtime dependencies or lockfile churn.
+- [ ] Preserve chart data, theme changes, print redraw and number formatting. Add accessible descriptions/table references and visible initialization-failure fallback instead of blank space.
+- [ ] Package assets/licenses in wheel and sdist. Replace CDN tests with embedded-runtime/version/digest/license tests; confirm Tier 1 avoids asset bytes and installed-wheel render works without network.
+- [ ] Update docs/changelog, run focused tests/build/package checks; signed-off commit; self-review and independent review.
+
+### Task 3: Persistent HTML navigation and accessible NVIDIA styling
+
+Branch: naren/reporting-workspace-navigation, based on reviewed Task 1.
+
+- [ ] Retain existing logo, public dark sidebar and restrained surfaces. Compact hero to skill, overall decision/reason, labeled agent result; move environment/metrics/provenance details into existing disclosures.
+- [ ] Comprehensive validate displays all 3 tier tiles with truthful absent/unrequested states; targeted reports keep focused scope. Sidebar has Summary, Tier 1/2/3 and Run details; mobile has compact sticky tier navigation.
+- [ ] One activation function synchronizes sidebar, tiles, active panel, selected state and focus. Place evidence destination outside hidden tier panels; preserve exact finding links, clearing filters and highlighting/focus. Current sidebar follows real section navigation.
+- [ ] Retain Overview/Insights/Agent Details/Trials/Dataset/Diagnostics. Associate tabs/panels and aria state; roving tabindex and Left/Right/Home/End/Enter/Space support for subtabs and agent tabs, including unusual names.
+- [ ] Remove expanded height caps that hide findings or tables. One main page scroll, horizontal overflow only for wide tables, bounded nested vertical scrolling only for raw logs/JSON. Native theme-aware scrollbars, stable gutters, forced-colors fallback; no simulated scrollbars.
+- [ ] Add System/Light/Dark control; persist explicit preferences, follow system by default. Respect reduced motion and print/no-JS evidence access and expansion restoration. Offset sticky mobile/local navigation without overlap.
+- [ ] Use lime #76B900 for accents, dark green #3E6500 for light-theme text/actions, charcoal navigation, restrained borders/shadows, red/amber semantic states. Normal text 14px, secondary labels >=11px, WCAG AA text/controls.
+- [ ] Add behavioral tests plus browser checks at 1440/760/390px, both themes, keyboard, long reports, reduced motion, no JS, and print. Update docs/changelog; signed-off commit; self-review and independent review.
+
+### Task 4: Clearer Tier 2 summaries and complete evidence navigation
+
+Branch: naren/reporting-evidence-views, based on reviewed Task 3.
+
+- [ ] Split Tier 2 into Within this skill / Across skills using existing typed workflow and similarity metadata. KPIs: Checks completed, Duplicate findings, Affected files/skills, Catalog entries compared; explicit scope and unavailable values.
+- [ ] Completed zero matches differs from skipped/unavailable comparison. Explain target-vs-catalog vs pairwise collections; retain self-exclusion and duplicate accounting. Context/token savings only if recorded.
+- [ ] Expose every existing normalized Harbor job in HTML, CLI and Markdown, with Results and available Analysis. Preserve names or Job N fallback, legacy single-job/analysis-only, deduplication, URL safety and escaping; no inferred attribution.
+- [ ] Surface existing sanitized scanner/model/evaluator/dataset/source provenance in appropriate disclosures. Retain both-arm resource comparisons, nulls, omission counts and full downloads. Explicit with-skill-only chart labels; no cached-token double counting.
+- [ ] Test missing/skipped/zero-match workflows, 0/1/multiple/duplicate/unsafe job links, retained reports, nullable telemetry, limits and immutability. Update docs/changelog; signed-off commit; self-review and independent review.
+
+### Task 5: Consistent responsive CLI and durable CI progress
+
+Branch: naren/reporting-cli-polish, based on reviewed Task 1.
+
+- [ ] Share lightweight semantic Rich styles across compact/full/direct Tier 3 reports. Preserve green/gray pipeline and amber numbered fixes; remove inconsistent decorative cyan/magenta in report views.
+- [ ] Reuse grade mapping A/B green, C amber, D/F red with failed/incomplete/unavailable overrides. Displayed zero lift is neutral; preserve unavailability/inconclusive labels and all scoring thresholds.
+- [ ] Keep each condition label/number/complete bar atomic; stack at narrow widths and preserve all numbers. Test 98/80/60 columns and long/wide Unicode labels.
+- [ ] Add concise successful quality/lift evidence to existing full-report Details cells; do not expand every successful panel.
+- [ ] Use actual producer-emitted paths/basenames, not hardcoded report.html. Safe hyperlinks only for capable interactive output; readable paths with no newly added OSC8 in plain/no-color/redirected/string-render APIs. Preserve escaping and control stripping.
+- [ ] Final panels stdout; durable noninteractive progress stderr: tier/check starts, stage changes, measured completion milestones, completion. Deduplicate unchanged events; heartbeat after 60 seconds without progress. Reuse progress lifecycle, no invented percentages, clean interruption/closed-stream handling and no leaked workers.
+- [ ] Fake-clock progress tests, artifact/escaping/color/layout/status parity tests, unchanged CLI surface and JSON/exit behavior. Update docs/changelog; signed-off commit; self-review and independent review.
+
+## Validation and delivery
+
+- Focused regressions per task, then repository-required make lint / make test / make build, Fern validation, OSS source/archive boundary, packaging and relevant compatibility tests. Record baseline environment failures separately from regressions; never claim unrun checks passed.
+- Review identical synthetic/retained fixtures at desktop/mobile and terminal widths in both themes, no-color/pipes, offline, keyboard/reduced-motion/no-JS/print. No live provider evaluation needed for reporting changes.
+- Optimize classify-once projections, immutable asset cache and repeated traversals only where justified; avoid broad rewrites and unnecessary polling/workers.
+- PR dependencies: 1 first; 2 independent; 3 and 5 follow 1; 4 follows 3. Test an isolated assembled integration checkout with all five changes. Do not merge main.
+- Deliver five PR links, CI statuses, before/after previews, and concise review/fix records. All commits signed off; PR template and changelog/docs updated.
diff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.py
index e75b9a06..3d705ff5 100644
--- a/src/skillevaluator/cli.py
+++ b/src/skillevaluator/cli.py
@@ -13,7 +13,7 @@ from concurrent.futures import ProcessPoolExecutor, as_completed
from contextvars import ContextVar
from datetime import UTC, datetime
from pathlib import Path
-from typing import Any
+from typing import TYPE_CHECKING, Any
import click
@@ -77,6 +77,10 @@ from skillevaluator.validators.pii_waivers import (
today_utc,
)
+if TYPE_CHECKING:
+ from skillevaluator.reporting.presentation import FinalRenderContext
+
+
CONTEXT_SETTINGS = {"help_option_names": ["-h", "--help"]}
@@ -1635,9 +1639,23 @@ def _finish_pipeline_view(
report_formats: tuple[str, ...],
target_path: Path,
agent_eval: bool,
+ presentation_context: FinalRenderContext | None = None,
) -> None:
"""Render the quiet-mode verdict panel and report footer."""
from skillevaluator.reporting.console_ui import MUTED, Verdict, _is_skipped, first_fix
+ from skillevaluator.reporting.presentation import build_final_render_context
+ from skillevaluator.reporting.run_status import Status
+
+ context = presentation_context or build_final_render_context(
+ [
+ *tier_gate_results,
+ *([tier3_result] if tier3_result is not None and tier3_result not in tier_gate_results else []),
+ ]
+ )
+ details = tuple(f"Tier {number}: {tier.summary}" for number, tier in context.tiers.items() if tier.displayed)
+ if context.tiers[3].status != Status.NOT_RUN:
+ details += (context.coverage.text, *context.coverage.details)
+ details += (f"Publication eligibility: {context.publication_label}",)
ext = {"html": ".html", "json": ".json", "markdown": ".md", "sarif": ".sarif.json"}
links: list[tuple[str, str]] = [
@@ -1649,21 +1667,13 @@ def _finish_pipeline_view(
summary = payload.get("summary") or payload
if not gate_failed:
- ran = sum(1 for block in view.blocks if block.status not in ("pending", "skip"))
advisory_failed = any(block.status == "fail" and block.number in {2, 3} for block in view.blocks)
if advisory_failed:
- headline = "gating tiers passed · advisory tier reported findings (see report)"
- else:
- headline = f"all {ran} tier{'s' if ran != 1 else ''} passed"
- skipped = sum(
- 1 for r in tier_gate_results if isinstance(r.metadata, dict) and r.metadata.get("skipped_by_user")
- )
- if skipped:
- headline += f" · {skipped} check{'s' if skipped != 1 else ''} skipped"
- lift = summary.get("overall_lift")
- if agent_eval and isinstance(lift, (int, float)):
- headline += f" · skill lift {lift:+.2f}"
- view.finish(Verdict(passed=True, headline=headline), links)
+ details += ("advisory tier reported findings (see report)",)
+ lift = summary.get("overall_lift")
+ if agent_eval and isinstance(lift, int | float) and not isinstance(lift, bool) and math.isfinite(lift):
+ details += (f"skill lift {lift:+.2f}",)
+ view.finish(Verdict(passed=True, headline=context.command_label, details=details), links)
return
failed = [result for result in tier_gate_results if not result.passed and not _is_skipped(result)]
@@ -1689,7 +1699,17 @@ def _finish_pipeline_view(
headline = " · ".join(parts) or "validation failed"
rerun = _rerun_hint(target_path, agent_eval)
fix = [("see what to fix above, then rerun", MUTED)] if failed else first_fix(tier_gate_results)
- view.finish(Verdict(passed=False, headline=headline, fix=fix, rerun=rerun), links)
+ view.finish(
+ Verdict(
+ passed=False,
+ headline=f"{context.command_label}: {headline}",
+ fix=fix,
+ rerun=rerun,
+ incomplete=context.command_status == Status.INCOMPLETE,
+ details=details,
+ ),
+ links,
+ )
def _print_tier_banner(title: str) -> None:
@@ -2553,8 +2573,17 @@ def validate(
# "cli", which renders the full Rich report below the pipeline view.
effective_formats = _effective_report_formats(report_formats, quiet=quiet)
report_basename_value = make_timestamped_basename(f"{REPORT_PREFIX}-output")
+ from skillevaluator.reporting.presentation import build_final_render_context
+
+ presentation_context = build_final_render_context(
+ results,
+ requested_tiers=tuple(number for number, enabled in ((1, True), (2, dedup), (3, agent_eval)) if enabled),
+ displayed_tiers=(1, 2, 3),
+ expected_skill_name=publication_name,
+ )
emit_reports(
results,
+ presentation_context=presentation_context,
report_formats=effective_formats,
output_dir=output_dir,
basename=report_basename_value,
@@ -2611,6 +2640,7 @@ def validate(
report_formats=effective_formats,
target_path=target_path,
agent_eval=agent_eval,
+ presentation_context=presentation_context,
)
if gate_failed:
raise click.ClickException("validation failed")
diff --git a/src/skillevaluator/reporting/cli.py b/src/skillevaluator/reporting/cli.py
index 1ceb2219..ef3b26c8 100644
--- a/src/skillevaluator/reporting/cli.py
+++ b/src/skillevaluator/reporting/cli.py
@@ -38,6 +38,7 @@ from skillevaluator.reporting.base import (
additional_errors,
get_skip_reason,
is_advisory_agent_eval_skip,
+ is_cleanly_skipped,
passes_required_gate,
source_validation_errors,
source_validation_preview,
@@ -47,7 +48,8 @@ from skillevaluator.reporting.harbor_viewer import (
normalize_harbor_viewer_for_display,
safe_url,
)
-from skillevaluator.reporting.run_status import Status, resolve_run_status, result_status
+from skillevaluator.reporting.presentation import FinalRenderContext, build_final_render_context, trial_coverage
+from skillevaluator.reporting.run_status import Status, result_status
from skillevaluator.utils.rich_markup import escape_markup
if TYPE_CHECKING:
@@ -82,12 +84,17 @@ def _rich_source_error(error: dict[str, str]) -> str:
return escape_markup(text)
+def _rich_line(text: str) -> str:
+ """Return one bounded, single-line message inert to Rich markup."""
+ line = " ".join(text.replace("\r\n", "\n").replace("\r", "\n").split())
+ if len(line) > _SKIP_REASON_MAX_CHARS:
+ line = line[: _SKIP_REASON_MAX_CHARS - 1] + "…"
+ return escape_markup(line)
+
+
def _rich_skip_reason(result: ValidationResult) -> str:
"""Return one bounded, single-line skip reason inert to Rich markup."""
- reason = " ".join(get_skip_reason(result).replace("\r\n", "\n").replace("\r", "\n").split())
- if len(reason) > _SKIP_REASON_MAX_CHARS:
- reason = reason[: _SKIP_REASON_MAX_CHARS - 1] + "…"
- return escape_markup(reason)
+ return _rich_line(get_skip_reason(result))
class CLIReporter(ReporterBase):
@@ -97,7 +104,13 @@ class CLIReporter(ReporterBase):
Supports both direct printing and string capture for testing.
"""
- def __init__(self, console: Console | None = None, *, report_formats: Iterable[str] = ()) -> None:
+ def __init__(
+ self,
+ console: Console | None = None,
+ *,
+ report_formats: Iterable[str] = (),
+ presentation_context: FinalRenderContext | None = None,
+ ) -> None:
"""Initialize CLI reporter.
Args:
@@ -108,6 +121,7 @@ class CLIReporter(ReporterBase):
"""
self._console = console
self._report_formats = tuple(report_formats)
+ self._presentation_context = presentation_context
@property
def console(self) -> Console:
@@ -159,8 +173,9 @@ class CLIReporter(ReporterBase):
def _render_all_results(self, results: list[ValidationResult], console: Console) -> None:
"""Render all results with summary table and failure details."""
+ context = self._presentation_context or build_final_render_context(results)
# Print summary table
- self._print_summary_table(results, console)
+ self._print_summary_table(results, console, presentation=context)
console.print()
# Print detailed results for failures
@@ -188,22 +203,17 @@ class CLIReporter(ReporterBase):
for result in non_blocking:
self.render_result(result, console)
- # Print overall status
- advisory_skips = [result for result in results if self._is_advisory_agent_eval_skip(result)]
- run_status = resolve_run_status(results)
- required_passed = run_status.headline == Status.PASSED
- if run_status.headline == Status.INCOMPLETE:
- console.print("\n[bold yellow][INCOMPLETE] Validation evidence is incomplete[/bold yellow]\n")
- elif required_passed:
- if advisory_skips:
- console.print(
- "\n[bold green][PASS] Required validations passed[/bold green] "
- f"[yellow]({len(advisory_skips)} live evaluation skipped)[/yellow]\n"
- )
- else:
- console.print("\n[bold green][PASS] All validations passed[/bold green]\n")
- else:
- console.print("\n[bold red][FAIL] Validation failed[/bold red]\n")
+ color = {Status.PASSED: "green", Status.FAILED: "red"}.get(context.command_status, "yellow")
+ console.print(f"\n[bold {color}][{context.command_status.verdict}] {context.command_label}[/bold {color}]")
+ for number, tier in context.tiers.items():
+ if not tier.displayed:
+ continue
+ console.print(f"Tier {number}: {tier.summary}", highlight=False, soft_wrap=True)
+ if context.tiers[3].status not in (Status.NOT_RUN, Status.SKIPPED):
+ console.print(context.coverage.text, highlight=False, soft_wrap=True)
+ for line in context.coverage.details:
+ console.print(line, highlight=False, soft_wrap=True)
+ console.print(f"Publication eligibility: {context.publication_label}\n")
def render_result(self, result: ValidationResult, console: Console) -> None:
"""Render a single validation result."""
@@ -388,6 +398,7 @@ class CLIReporter(ReporterBase):
f" [dim]Harbor analysis:[/dim] [cyan]{escape_markup(harbor_viewer['analysis_url'])}[/cyan]",
soft_wrap=True,
)
+ console.print(f" {trial_coverage(agent_eval).text}")
console.print()
evaluators = agent_eval.get("evaluators", {})
@@ -602,7 +613,9 @@ class CLIReporter(ReporterBase):
console.print(f" [dim]{escape_markup(overflow_note)}[/dim]")
console.print()
- def _print_summary_table(self, results: list[ValidationResult], console: Console) -> None:
+ def _print_summary_table(
+ self, results: list[ValidationResult], console: Console, *, presentation: FinalRenderContext | None = None
+ ) -> None:
"""Print summary table for all validators."""
table = Table(
title="SkillEvaluator Validation Results",
@@ -616,22 +629,24 @@ class CLIReporter(ReporterBase):
table.add_column("Details")
for result in results:
- advisory_skip = result_status(result) == Status.SKIPPED
- status = (
- "[yellow]SKIP[/yellow]"
- if advisory_skip
- else "[bold yellow]INCOMPLETE[/bold yellow]"
- if result_status(result) == Status.INCOMPLETE
- else "[green]PASS[/green]"
- if result.passed
- else "[red]FAIL[/red]"
- )
+ outcome = presentation.result_statuses[id(result)] if presentation else result_status(result).value
+ advisory_skip = outcome == "skipped"
+ status = {
+ "skipped": "[yellow]SKIP[/yellow]",
+ "not_applicable": "[dim]N/A[/dim]",
+ "incomplete": "[bold yellow]INCOMPLETE[/bold yellow]",
+ "passed": "[green]PASS[/green]",
+ "pass": "[green]PASS[/green]",
+ }.get(outcome, "[red]FAIL[/red]")
s = result.summary
static_test_evidence = self._static_test_evidence_message(result)
- if advisory_skip:
+ if advisory_skip or (outcome == "not_applicable" and is_cleanly_skipped(result)):
details = _rich_skip_reason(result)
- elif result_status(result) == Status.INCOMPLETE:
+ elif outcome == "not_applicable":
+ # Keep the recorded reason; "N/A │ 1 checks passed" contradicts itself.
+ details = _rich_line(result.success_details[0].message) if result.success_details else "Not applicable"
+ elif outcome == "incomplete":
scanners = escape_markup(", ".join(result.incomplete_scans))
details = f"[bold yellow]{scanners} did not complete[/bold yellow]"
counts = []
diff --git a/src/skillevaluator/reporting/console_ui.py b/src/skillevaluator/reporting/console_ui.py
index 61f00bd1..a12ccc43 100644
--- a/src/skillevaluator/reporting/console_ui.py
+++ b/src/skillevaluator/reporting/console_ui.py
@@ -487,6 +487,8 @@ class Verdict:
headline: str
fix: list[tuple[str, str]] | None = None
rerun: str | None = None
+ incomplete: bool = False
+ details: tuple[str, ...] = ()
class ValidateView:
@@ -729,11 +731,18 @@ class ValidateView:
body = _pill("✓ PASS", GREEN) + Text.assemble(
(" ", ""), (headline, f"bold {TEXT}"), (" · exit 0", MUTED)
)
- return Panel(body, box=box.ROUNDED, border_style=GREEN, width=self.width, padding=(0, 2))
- line1 = _pill("✗ FAIL", RED, ink="#1C0605") + Text.assemble(
+ return Panel(
+ Group(body, *(Text(strip_terminal_controls(line), style=MUTED) for line in verdict.details)),
+ box=box.ROUNDED,
+ border_style=GREEN,
+ width=self.width,
+ padding=(0, 2),
+ )
+ color = AMBER if verdict.incomplete else RED
+ line1 = _pill("INCOMPLETE" if verdict.incomplete else "✗ FAIL", color, ink="#1C0605") + Text.assemble(
(" ", ""), (headline, f"bold {TEXT}"), (" · exit 1", MUTED)
)
- parts: list = [line1]
+ parts: list = [line1, *(Text(strip_terminal_controls(line), style=MUTED) for line in verdict.details)]
if verdict.fix:
fix = Text.assemble(("fix ", f"bold {MUTED}"))
for chunk, style in verdict.fix:
@@ -741,7 +750,7 @@ class ValidateView:
parts.extend([Text(), fix])
if verdict.rerun:
parts.append(Text.assemble((" ", ""), (strip_terminal_controls(verdict.rerun), GREEN)))
- return Panel(Group(*parts), box=box.ROUNDED, border_style=RED, width=self.width, padding=(0, 2))
+ return Panel(Group(*parts), box=box.ROUNDED, border_style=color, width=self.width, padding=(0, 2))
def _footer_lines(self):
for label, target in self.links:
diff --git a/src/skillevaluator/reporting/html.py b/src/skillevaluator/reporting/html.py
index ae56d947..78b7aba7 100644
--- a/src/skillevaluator/reporting/html.py
+++ b/src/skillevaluator/reporting/html.py
@@ -56,8 +56,6 @@ from skillevaluator.reporting.base import (
get_skip_reason,
is_advisory_agent_eval_skip,
is_cleanly_skipped,
- is_tier2_result,
- is_tier3_result,
passes_required_gate,
publication_free_text,
publication_identity_present,
@@ -80,8 +78,14 @@ from skillevaluator.reporting.finding_summary import (
total_issue_count,
)
from skillevaluator.reporting.harbor_viewer import normalize_agent_eval_harbor_links
-from skillevaluator.reporting.presentation import success_is_not_applicable
-from skillevaluator.reporting.run_status import Status, resolve_run_status, result_status
+from skillevaluator.reporting.presentation import (
+ FinalRenderContext,
+ TierPresentation,
+ build_final_render_context,
+ result_execution_status,
+ success_is_not_applicable,
+)
+from skillevaluator.reporting.run_status import Status, result_status, result_tier
from skillevaluator.reporting.tier3_scope import completed_agent_result, validate_report_scope
from skillevaluator.reporting.trial_charts import steps_by_case_chart
from skillevaluator.skillspector_provenance import skillspector_provenance_records
@@ -1605,21 +1609,7 @@ def _detail_display_status(detail: Any) -> str:
def _result_display_status(result: ValidationResult) -> str:
"""Describe execution separately from the preserved gate boolean."""
- not_applicable = (
- bool(result.success_details)
- and not result.findings
- and all(_detail_display_status(detail) == "not_applicable" for detail in result.success_details)
- )
- outcome = result_status(result)
- if outcome == Status.FAILED:
- return "fail"
- if outcome == Status.INCOMPLETE:
- return "incomplete"
- if is_cleanly_skipped(result):
- return "not_applicable" if "not applicable" in get_skip_reason(result).casefold() else "skipped"
- if result.passed and not_applicable:
- return "not_applicable"
- return "pass" if result.passed else "fail"
+ return result_execution_status(result)
def _validated_ci_context(context: dict[str, str] | None) -> dict[str, str]:
@@ -1677,6 +1667,9 @@ class HTMLReporter(ReporterBase):
ci_context: dict[str, str] | None = None,
report_scope: str = "publication",
scan_root: str | None = None,
+ requested_tiers: tuple[int, ...] | None = None,
+ displayed_tiers: tuple[int, ...] | None = None,
+ presentation_context: FinalRenderContext | None = None,
) -> None:
self.include_timestamp = include_timestamp
self.title = title or "SkillEvaluator Validation Report"
@@ -1696,6 +1689,9 @@ class HTMLReporter(ReporterBase):
# The folder the scanners ran on, used to show finding locations
# relative to the skill in "Fix first".
self.scan_root = scan_root
+ self.requested_tiers = requested_tiers
+ self.displayed_tiers = displayed_tiers
+ self.presentation_context = presentation_context
self._env = self._create_environment()
@staticmethod
@@ -2403,20 +2399,15 @@ class HTMLReporter(ReporterBase):
name rather than relying on list slicing (which would silently break
if a tier ever produced zero results).
"""
- tier1: list[ValidationResult] = []
- tier2: list[ValidationResult] = []
- tier3: list[ValidationResult] = []
- for r in results:
- if is_tier3_result(r):
- tier3.append(r)
- elif is_tier2_result(r):
- tier2.append(r)
- else:
- tier1.append(r)
- return tier1, tier2, tier3
+ buckets: dict[int, list[ValidationResult]] = {1: [], 2: [], 3: []}
+ for result in results:
+ buckets[result_tier(result)].append(result)
+ return buckets[1], buckets[2], buckets[3]
@staticmethod
- def _compute_tier_summary(results: list[ValidationResult]) -> dict[str, Any]:
+ def _compute_tier_summary(
+ results: list[ValidationResult], *, presentation: TierPresentation | None = None
+ ) -> dict[str, Any]:
"""Compact stats for a single tier — what the hero chip displays.
Returns ``passed`` (boolean), executed/pass/skip counts,
@@ -2430,12 +2421,14 @@ class HTMLReporter(ReporterBase):
extra "did this tier run?" predicate.
"""
total = len(results)
- skipped_count = sum(1 for r in results if is_cleanly_skipped(r))
- executed_count = total - skipped_count
- passed_count = sum(1 for r in results if r.passed and not r.is_incomplete and not is_cleanly_skipped(r))
+ statuses = [result_execution_status(r) for r in results] if presentation is None else []
+ skipped_count = presentation.skipped if presentation else statuses.count("skipped")
+ not_applicable_count = presentation.not_applicable if presentation else statuses.count("not_applicable")
+ passed_count = presentation.passed if presentation else statuses.count("pass")
+ failed_count = presentation.failed if presentation else statuses.count("fail")
+ incomplete_count = presentation.incomplete if presentation else statuses.count("incomplete")
+ executed_count = passed_count + failed_count + incomplete_count
advisory_skipped_count = sum(1 for r in results if is_advisory_agent_eval_skip(r))
- failed_count = sum(1 for r in results if result_status(r) == Status.FAILED)
- incomplete_count = sum(1 for r in results if result_status(r) == Status.INCOMPLETE)
findings = summarize_findings(results)
messages = summarize_messages(results)
unique_by_severity = findings["unique_by_severity"]
@@ -2450,6 +2443,7 @@ class HTMLReporter(ReporterBase):
"executed_count": executed_count,
"passed_count": passed_count,
"skipped_count": skipped_count,
+ "not_applicable_count": not_applicable_count,
"advisory_skipped_count": advisory_skipped_count,
"failed_count": failed_count,
"issue_count": findings["unique_findings"],
@@ -2537,7 +2531,9 @@ class HTMLReporter(ReporterBase):
return output
@staticmethod
- def _tier3_report_data(results: list[ValidationResult]) -> dict[str, Any] | None:
+ def _tier3_report_data(
+ results: list[ValidationResult], *, selected: tuple[ValidationResult, dict] | None = None
+ ) -> dict[str, Any] | None:
"""Return safe Tier 3 display data without certifying partial evidence."""
if not results:
return None
@@ -2672,7 +2668,7 @@ class HTMLReporter(ReporterBase):
return payload_has_agent_evidence(candidate)
return not execution_status and result_has_execution_evidence(result)
- selected = select_agent_eval_candidate(results)
+ selected = selected if selected is not None else select_agent_eval_candidate(results)
payload_result, payload = selected if selected is not None else (None, None)
serialization_issue = agent_eval_report_serialization_issue(payload) if payload is not None else None
rejected_assessment = (
@@ -2875,12 +2871,14 @@ class HTMLReporter(ReporterBase):
def render_all(self, results: list[ValidationResult]) -> str:
validate_report_scope(self.report_scope, results)
tier1_results, tier2_results, tier3_results = self._split_results_by_tier(results)
- tier3_data = self._tier3_report_data(tier3_results)
+ selected = select_agent_eval_candidate(tier3_results)
+ raw_payload = selected[1] if selected is not None else None
+ tier3_data = self._tier3_report_data(tier3_results, selected=selected)
completed_agent = completed_agent_result(tier3_data)
# Preserve the raw policy/provenance assessment, then let the emitted
# payload make the decision only more conservative. Display shaping
# must never invent a successful run or erase a persisted waiver.
- raw_publication = assess_publication(results, expected_skill_name=self.expected_skill_name)
+ raw_publication = assess_publication(results, raw_payload, expected_skill_name=self.expected_skill_name)
# Without a persisted payload, tier3_data is a synthesized display
# fallback rather than evidence, so only the raw assessment applies.
emitted_publication = (
@@ -2889,7 +2887,7 @@ class HTMLReporter(ReporterBase):
tier3_data,
expected_skill_name=self.expected_skill_name,
)
- if select_agent_eval_candidate(tier3_results) is not None
+ if selected is not None
else raw_publication
)
publication = max(
@@ -2900,14 +2898,22 @@ class HTMLReporter(ReporterBase):
),
)
- run_status = resolve_run_status(results)
- all_passed = run_status.headline == Status.PASSED
- has_incomplete = run_status.headline == Status.INCOMPLETE
- overall_status = run_status.headline.value
+ context = self.presentation_context or build_final_render_context(
+ results,
+ requested_tiers=self.requested_tiers,
+ displayed_tiers=self.displayed_tiers,
+ report_scope=self.report_scope,
+ expected_skill_name=self.expected_skill_name,
+ agent_eval=raw_payload,
+ publication_assessment=publication,
+ )
+ all_passed = context.command_status == Status.PASSED
+ has_incomplete = context.command_status == Status.INCOMPLETE
+ overall_status = context.command_status.value
finding_summary = summarize_findings(results)
message_summary = summarize_messages(results)
skipped_count = sum(1 for r in results if is_cleanly_skipped(r))
- passed_count = sum(1 for r in results if r.passed and not r.is_incomplete and not is_cleanly_skipped(r))
+ passed_count = sum(tier.passed for tier in context.tiers.values())
advisory_skipped_count = sum(1 for r in results if is_advisory_agent_eval_skip(r))
failed_count = sum(1 for r in results if not passes_required_gate(r))
# Count unique findings rather than raw validator records that repeat
@@ -2915,7 +2921,7 @@ class HTMLReporter(ReporterBase):
# (failed gates, scanner and execution errors), as Markdown does.
total_issues = total_issue_count(finding_summary, message_summary)
total_validators = len(results)
- executed_count = total_validators - skipped_count
+ executed_count = sum(tier.executed for tier in context.tiers.values())
pass_percentage = round((passed_count / executed_count * 100) if executed_count > 0 else 0, 1)
timestamp = ""
@@ -2964,11 +2970,11 @@ class HTMLReporter(ReporterBase):
# Per-tier summaries so the hero card chips can show "Tier 1: 6/7
# passed (0 critical)" etc. without leaking Tier 2 / Tier 3 stats
# from the global aggregate counters.
- tier1_summary = self._compute_tier_summary(tier1_results)
+ tier1_summary = self._compute_tier_summary(tier1_results, presentation=context.tiers[1])
tier1_skillspector_versions = _skillspector_versions(tier1_results)
tier1_skillspector_provenance = _skillspector_provenance(tier1_results)
- tier2_summary = self._compute_tier_summary(tier2_results)
- tier3_summary = self._compute_tier_summary(tier3_results)
+ tier2_summary = self._compute_tier_summary(tier2_results, presentation=context.tiers[2])
+ tier3_summary = self._compute_tier_summary(tier3_results, presentation=context.tiers[3])
tier3_preview, tier3_preview_notice = _bounded_tier3_preview(tier3_data)
tier3_canonical_data, tier3_canonical_encoding = _canonical_tier3_embed(tier3_data)
tier3_truncation = tier3_data.get("report_truncation", {}) if isinstance(tier3_data, dict) else {}
@@ -2981,15 +2987,9 @@ class HTMLReporter(ReporterBase):
tier1_top_issues = self._compute_top_issues(tier1_skills)
tier1_contributors = self._extract_contributors(tier1_skills, tier1_display_results)
tier1_display_skipped = sum(1 for result in tier1_display_results if is_cleanly_skipped(result))
- tier1_display_total = len(tier1_display_results) - tier1_display_skipped
- tier1_display_passed = sum(
- 1
- for result in tier1_display_results
- if result.passed and not result.is_incomplete and not is_cleanly_skipped(result)
- )
- tier1_display_failed = sum(
- 1 for result in tier1_display_results if not result.passed and not is_cleanly_skipped(result)
- )
+ tier1_display_total = context.tiers[1].executed
+ tier1_display_passed = context.tiers[1].passed
+ tier1_display_failed = context.tiers[1].failed
tier1_display_total_skills = len(tier1_skills)
tier1_display_passed_skills = sum(1 for skill in tier1_skills.values() if skill["passed"])
tier1_display_summary = {
@@ -3255,12 +3255,13 @@ class HTMLReporter(ReporterBase):
cl = self.content_label
return template.render(
report_scope=self.report_scope,
+ presentation=context,
completed_agent=completed_agent,
title=self.title,
ci_context=self.ci_context,
explicit_skill_name=self.explicit_skill_name,
tier1_execution_summary={
- status: sum(_result_display_status(r) == status for r in tier1_results)
+ status: sum(context.result_statuses[id(r)] == status for r in tier1_results)
for status in ("pass", "fail", "incomplete", "skipped", "not_applicable")
},
timestamp=timestamp,
diff --git a/src/skillevaluator/reporting/presentation.py b/src/skillevaluator/reporting/presentation.py
index ccbf5de2..abdca11b 100644
--- a/src/skillevaluator/reporting/presentation.py
+++ b/src/skillevaluator/reporting/presentation.py
@@ -3,6 +3,20 @@
from __future__ import annotations
+from collections.abc import Iterable, Mapping
+from dataclasses import dataclass
+from types import MappingProxyType
+from typing import TYPE_CHECKING
+
+from skillevaluator.reporting import base as reporting_base
+from skillevaluator.reporting.base import get_skip_reason, select_agent_eval_payload
+from skillevaluator.reporting.run_status import Status, resolve_run_status, result_status
+from skillevaluator.reporting.tier3_scope import validate_report_scope
+
+if TYPE_CHECKING:
+ from skillevaluator.models import ValidationResult
+ from skillevaluator.reporting.base import PublicationAssessment
+
def success_is_not_applicable(check: object, message: object, metadata: object = None) -> bool:
"""Use typed producer metadata, with exact legacy evidence for retained reports.
@@ -22,3 +36,336 @@ def success_is_not_applicable(check: object, message: object, metadata: object =
return False
expected = evidence[check]
return expected is None or message == expected
+
+
+# These projections contain only immutable display values. They never rewrite
+# ValidationResult or the canonical agent payload (including retained reports).
+
+
+@dataclass(frozen=True)
+class TierPresentation:
+ status: Status
+ requested: bool
+ displayed: bool
+ passed: int = 0
+ failed: int = 0
+ incomplete: int = 0
+ skipped: int = 0
+ not_applicable: int = 0
+ assessment_status: str | None = None
+
+ @property
+ def display_status(self) -> str:
+ if self.assessment_status is not None:
+ return {"pass": "passed", "fail": "failed"}.get(self.assessment_status, self.assessment_status)
+ if not self.executed and self.not_applicable and not self.skipped:
+ return "not_applicable"
+ return self.status.value
+
+ @property
+ def summary(self) -> str:
+ if self.status == Status.NOT_RUN:
+ return "Not run" if self.requested else "Not run (not requested)"
+ if not self.executed:
+ return f"{self.skipped} skipped, {self.not_applicable} N/A; no checks executed"
+ assessment = (
+ f"{self.display_status.capitalize()} evidence; "
+ if self.assessment_status is not None and self.display_status != self.status.value
+ else ""
+ )
+ return (
+ assessment + f"{self.passed} of {self.executed} checks passed; "
+ f"{self.failed} failed, {self.incomplete} incomplete, "
+ f"{self.skipped} skipped, {self.not_applicable} N/A"
+ )
+
+ @property
+ def executed(self) -> int:
+ return self.passed + self.failed + self.incomplete
+
+
+@dataclass(frozen=True)
+class ConditionCoverage:
+ status: str = "unavailable"
+ scored: int | None = None
+ expected: int | None = None
+ observed: int | None = None
+
+
+@dataclass(frozen=True)
+class TrialCoverage:
+ with_skill: ConditionCoverage
+ baseline: ConditionCoverage
+ agents: Mapping[str, Mapping[str, ConditionCoverage]]
+
+ @property
+ def details(self) -> tuple[str, ...]:
+ lines = []
+ for label, arm in (("With skill", self.with_skill), ("Baseline", self.baseline)):
+ if arm.status == "skipped":
+ lines.append(f"{label}: skipped")
+ elif arm.scored is not None and arm.expected is not None:
+ lines.append(f"{label}: {arm.scored} of {arm.expected} scored ({arm.status})")
+ return tuple(lines)
+
+ @property
+ def text(self) -> str:
+ arms = (self.with_skill, self.baseline)
+ if all(arm.scored is not None or arm.status == "skipped" for arm in arms):
+ total = sum(arm.scored or 0 for arm in arms)
+ with_text = "with skill skipped" if arms[0].status == "skipped" else f"{arms[0].scored} with skill"
+ baseline_text = "baseline skipped" if arms[1].status == "skipped" else f"{arms[1].scored} baseline"
+ return f"{total} scored trials: {with_text}, {baseline_text}"
+ parts = []
+ for label, arm in zip(("with skill", "baseline"), arms, strict=True):
+ if arm.status == "skipped":
+ parts.append(f"{label} skipped")
+ elif arm.scored is not None:
+ parts.append(f"{arm.scored} {label} scored trials")
+ else:
+ parts.append(f"{label} scored trials unavailable")
+ for label, arm in zip(("with-skill", "baseline"), arms, strict=True):
+ if arm.scored is None and arm.observed is not None and arm.status != "skipped":
+ parts.append(f"{arm.observed} retained {label} rows")
+ return "; ".join(parts)
+
+
+@dataclass(frozen=True)
+class FinalRenderContext:
+ """Immutable human-facing outcomes, separate from publication approval.
+
+ required_failed includes every gate-rejected result except classified
+ incomplete, including pass-shaped evidence rejected by the required gate.
+ required_incomplete counts classified incomplete results rejected by that
+ gate. advisory_unfavorable counts failed/incomplete results that permit it.
+ Standalone Tier 3 adverse results follow the existing implicit-gate headline.
+ """
+
+ command_status: Status
+ command_label: str
+ publication_status: Status | None
+ publication_label: str
+ tiers: Mapping[int, TierPresentation]
+ result_statuses: Mapping[int, str]
+ score_scope: str
+ coverage: TrialCoverage
+ required_failed: int = 0
+ required_incomplete: int = 0
+ advisory_unfavorable: int = 0
+
+
+def result_execution_status(result: ValidationResult, outcome: Status | None = None) -> str:
+ """Display explicit N/A evidence without changing gate or JSON status."""
+ outcome = result_status(result) if outcome is None else outcome
+ if outcome in (Status.FAILED, Status.INCOMPLETE):
+ return "fail" if outcome == Status.FAILED else "incomplete"
+ if outcome == Status.SKIPPED:
+ return "not_applicable" if "not applicable" in get_skip_reason(result).casefold() else "skipped"
+ if (
+ result.success_details
+ and not result.findings
+ and all(
+ success_is_not_applicable(detail.check_name, detail.message, detail.metadata)
+ for detail in result.success_details
+ )
+ ):
+ return "not_applicable"
+ return "pass"
+
+
+def _count(value: object) -> int | None:
+ return value if type(value) is int and 0 <= value <= 2**63 - 1 else None
+
+
+def trial_coverage(payload: object) -> TrialCoverage:
+ """Prefer per-condition counters; rows are only retained observations.
+
+ Physical retry rows, preview limits and early-stop policy cannot establish
+ the number of scored attempts. Missing/malformed counters stay unknown.
+ """
+ payload = payload if isinstance(payload, dict) else {}
+ raw_agents = payload.get("agents")
+ raw_agents = raw_agents if isinstance(raw_agents, dict) else {}
+ # Reserve an agent record and its two condition records before reading any
+ # values. Retained rows in both arms consume the same remaining allowance.
+ remaining = reporting_base.AGENT_EVAL_REPORT_MAX_NODES
+ agent_nodes = 3 * len(raw_agents)
+ if agent_nodes > remaining:
+ return TrialCoverage(ConditionCoverage(), ConditionCoverage(), MappingProxyType({}))
+ remaining -= agent_nodes
+ # The producer marks deliberately unexecuted arms at payload level: an
+ # advisory skip sets execution_status="skipped" with no agents, and
+ # --skip-baseline omits the baseline condition (mode "with_skill_only").
+ if payload.get("execution_status") == "skipped":
+ skipped_arms = {"with_skill", "without_skill"}
+ else:
+ raw_mode = payload.get("mode")
+ mode = raw_mode if isinstance(raw_mode, str) else None
+ skipped_arms = {"with_skill_only": {"without_skill"}, "baseline_only": {"with_skill"}}.get(mode, set())
+
+ def retained_rows(value: object) -> int | None:
+ nonlocal remaining
+ if not isinstance(value, list):
+ return None
+ nodes = 1 + len(value)
+ if nodes > remaining:
+ remaining = 0
+ return None
+ remaining -= nodes
+ return sum(isinstance(row, dict) for row in value)
+
+ agents = {}
+ for name, raw in raw_agents.items():
+ if not isinstance(name, str) or not isinstance(raw, dict):
+ continue
+ conditions = raw.get("conditions")
+ conditions = conditions if isinstance(conditions, dict) else {}
+ coverage = {}
+ for arm in ("with_skill", "without_skill"):
+ evidence = conditions.get(arm)
+ if evidence is None and arm in skipped_arms:
+ evidence = {"execution_status": "skipped"}
+ evidence = evidence if isinstance(evidence, dict) else {}
+ status = evidence.get("execution_status")
+ status = status if status in ("succeeded", "failed", "incomplete", "skipped") else "unavailable"
+ coverage[arm] = ConditionCoverage(
+ status, _count(evidence.get("scored_attempts")), _count(evidence.get("expected_attempts"))
+ )
+ agents[name] = MappingProxyType(coverage)
+
+ def combine(arm: str) -> ConditionCoverage:
+ members = [agent[arm] for agent in agents.values()]
+ if not members and arm in skipped_arms:
+ return ConditionCoverage("skipped")
+ status = next(
+ (
+ s
+ for s in ("failed", "incomplete", "unavailable", "succeeded", "skipped")
+ if any(c.status == s for c in members)
+ ),
+ "unavailable",
+ )
+
+ def total(field: str) -> int | None:
+ values = [getattr(c, field) if c.status != "skipped" else 0 for c in members]
+ return sum(values) if values and all(value is not None for value in values) else None
+
+ scored = total("scored")
+ observed = None
+ if scored is None:
+ if arm == "with_skill":
+ observed = retained_rows(payload.get("trials"))
+ else:
+ for raw in raw_agents.values():
+ if not isinstance(raw, dict):
+ continue
+ rows = retained_rows(raw.get("trials_baseline"))
+ if rows is None:
+ observed = None
+ break
+ observed = (observed or 0) + rows
+ return ConditionCoverage(status, scored, total("expected"), observed)
+
+ return TrialCoverage(combine("with_skill"), combine("without_skill"), MappingProxyType(agents))
+
+
+def build_final_render_context(
+ results: Iterable[ValidationResult],
+ *,
+ requested_tiers: Iterable[int] | None = None,
+ displayed_tiers: Iterable[int] | None = None,
+ report_scope: str = "publication",
+ expected_skill_name: str | None = None,
+ agent_eval: dict | None = None,
+ publication_assessment: PublicationAssessment | None = None,
+) -> FinalRenderContext:
+ """Classify once for final human renders; decision scope stays independent."""
+ results = list(results)
+ validate_report_scope(report_scope, results)
+ payload = agent_eval if agent_eval is not None else select_agent_eval_payload(results)
+ run = resolve_run_status(results, agent_eval=payload, expected_skill_name=expected_skill_name)
+ present = tuple(n for n, tier in run.tiers.items() if tier.results)
+ requested = frozenset(present if requested_tiers is None else requested_tiers)
+ displayed = frozenset(requested if displayed_tiers is None else displayed_tiers)
+ if any(type(n) is not int or n not in (1, 2, 3) for n in requested | displayed):
+ raise ValueError("presentation tiers must be 1, 2 or 3")
+ outcomes = {id(r): result_execution_status(r, run.of(r)) for r in results}
+ assessment = publication_assessment or run.publication_assessment
+ tier3 = run.tiers[3]
+ standalone_tier3_gate = (
+ report_scope == "tier3" and tier3.blocking is None and tier3.status in (Status.FAILED, Status.INCOMPLETE)
+ )
+ required_failed = required_incomplete = advisory_unfavorable = 0
+ tiers = {}
+ for n, tier in run.tiers.items():
+ counts = {
+ status: sum(outcomes[id(r)] == status for r in tier.results)
+ for status in ("pass", "fail", "incomplete", "skipped", "not_applicable")
+ }
+ tiers[n] = TierPresentation(
+ tier.status,
+ n in requested,
+ n in displayed,
+ counts["pass"],
+ counts["fail"],
+ counts["incomplete"],
+ counts["skipped"],
+ counts["not_applicable"],
+ (
+ "fail"
+ if tier.status == Status.FAILED
+ else "incomplete"
+ if tier.status == Status.INCOMPLETE
+ else "skipped"
+ if tier.status == Status.SKIPPED
+ else assessment.tier3.status
+ )
+ if n == 3 and tier.results
+ else None,
+ )
+ for item in tier.results:
+ outcome = outcomes[id(item)]
+ required = not reporting_base.passes_required_gate(item)
+ if standalone_tier3_gate and n == 3 and outcome in {"fail", "incomplete"}:
+ required = True
+ if required:
+ if outcome == "incomplete":
+ required_incomplete += 1
+ else:
+ required_failed += 1
+ elif outcome in {"fail", "incomplete"}:
+ advisory_unfavorable += 1
+ publication = (
+ None
+ if report_scope == "tier3"
+ else {"pass": Status.PASSED, "fail": Status.FAILED, "neutral": Status.NEUTRAL, "incomplete": Status.INCOMPLETE}[
+ assessment.status
+ ]
+ )
+ headline = run.headline
+ if standalone_tier3_gate:
+ # A standalone Tier 3 report carries no gating metadata, so Tier 3 is
+ # the command's only gate; its verdict must drive the header too.
+ headline = tier3.status
+ return FinalRenderContext(
+ headline,
+ {Status.PASSED: "Passed required checks", Status.FAILED: "Changes required", Status.INCOMPLETE: "Incomplete"}[
+ headline
+ ],
+ publication,
+ "Not assessed (Tier 3 only)"
+ if publication is None
+ else {
+ Status.PASSED: "Eligible",
+ Status.FAILED: "Not eligible",
+ Status.INCOMPLETE: "Incomplete",
+ Status.NEUTRAL: "Neutral",
+ }[publication],
+ MappingProxyType(tiers),
+ MappingProxyType(outcomes),
+ "Best agent score",
+ trial_coverage(payload),
+ required_failed=required_failed,
+ required_incomplete=required_incomplete,
+ advisory_unfavorable=advisory_unfavorable,
+ )
diff --git a/src/skillevaluator/reporting/run_status.py b/src/skillevaluator/reporting/run_status.py
index 91f2bd39..6cb96a7e 100644
--- a/src/skillevaluator/reporting/run_status.py
+++ b/src/skillevaluator/reporting/run_status.py
@@ -19,6 +19,7 @@ from typing import TYPE_CHECKING
if TYPE_CHECKING:
from skillevaluator.models.result import ValidationResult
+ from skillevaluator.reporting.base import PublicationAssessment
from skillevaluator.reporting.base import (
assess_publication,
is_cleanly_skipped,
@@ -56,12 +57,22 @@ class RunStatus:
missing: tuple[str, ...]
advisory: tuple[str, ...]
_outcomes: Mapping[int, Status] = field(repr=False)
- _publication: Callable[[], Status] = field(repr=False, compare=False)
+ _publication: Callable[[], PublicationAssessment] = field(repr=False, compare=False)
+
+ @cached_property
+ def publication_assessment(self) -> PublicationAssessment:
+ """Cache the existing assessment for final presentation consumers."""
+ return self._publication()
@cached_property
def publication(self) -> Status:
"""Assess publication once, only on surfaces that need its stricter verdict."""
- return self._publication()
+ return {
+ "pass": Status.PASSED,
+ "neutral": Status.NEUTRAL,
+ "fail": Status.FAILED,
+ "incomplete": Status.INCOMPLETE,
+ }[self.publication_assessment.status]
def of(self, result: ValidationResult) -> Status:
outcome = self._outcomes.get(id(result))
@@ -104,7 +115,7 @@ def result_status(result: ValidationResult) -> Status:
return Status.PASSED if result.passed else Status.FAILED
-def _tier(result: ValidationResult) -> int:
+def result_tier(result: ValidationResult) -> int:
gate = result.metadata.get("gating")
if isinstance(gate, dict) and type(gate.get("tier")) is int and gate["tier"] in (1, 2, 3):
return gate["tier"]
@@ -136,7 +147,7 @@ def resolve_run_status(results: Iterable[ValidationResult], *, agent_eval=None,
)
tiers = {}
for number in (1, 2, 3):
- members = tuple(result for result in results if _tier(result) == number)
+ members = tuple(result for result in results if result_tier(result) == number)
blocking = [
r.metadata["gating"].get("blocking", True) for r in members if isinstance(r.metadata.get("gating"), dict)
]
@@ -144,14 +155,8 @@ def resolve_run_status(results: Iterable[ValidationResult], *, agent_eval=None,
number, _combined(outcomes[id(r)] for r in members), any(blocking) if blocking else None, members
)
- def publication_status() -> Status:
- assessment = assess_publication(list(results), agent_eval, expected_skill_name=expected_skill_name)
- return {
- "pass": Status.PASSED,
- "neutral": Status.NEUTRAL,
- "fail": Status.FAILED,
- "incomplete": Status.INCOMPLETE,
- }[assessment.status]
+ def publication_assessment() -> PublicationAssessment:
+ return assess_publication(list(results), agent_eval, expected_skill_name=expected_skill_name)
return RunStatus(
headline,
@@ -165,5 +170,5 @@ def resolve_run_status(results: Iterable[ValidationResult], *, agent_eval=None,
and outcomes[id(r)] != Status.PASSED
),
MappingProxyType(outcomes),
- publication_status,
+ publication_assessment,
)
diff --git a/src/skillevaluator/reporting/templates/report.html.j2 b/src/skillevaluator/reporting/templates/report.html.j2
index 11e731e5..f885b25b 100644
--- a/src/skillevaluator/reporting/templates/report.html.j2
+++ b/src/skillevaluator/reporting/templates/report.html.j2
@@ -1242,17 +1242,9 @@
{#- A failing tier whose gate does not block is "Failed (advisory)", never "Changes required". -#}
{% set tier3_advisory_fail = tier3_verdict == 'fail' and 'tier3' in nonblocking_tiers %}
{% set has_t12_tab = tabs | selectattr('id', 'in', ['tier1', 'tier2']) | list | length > 0 %}
- {% if has_t12_tab %}
- {#- The summary's rule: when no Tier 1 or Tier 2 check ran, the verdict is Skipped, not Passed. -#}
- {% set t12_executed = tier1_summary.executed_count + tier2_summary.executed_count %}
- {% set verdict_key = 'incomplete' if has_incomplete else ('fail' if not all_passed else ('pass' if t12_executed else 'skipped')) %}
- {% elif has_tier3 %}
- {% set verdict_key = 'skipped' if tier3_is_skipped else (tier3_verdict if tier3_verdict in ['pass', 'fail', 'neutral'] else 'incomplete') %}
- {% else %}
- {% set verdict_key = 'incomplete' %}
- {% endif %}
+ {% set verdict_key = {'passed': 'pass', 'failed': 'fail'}.get(presentation.command_status.value, 'incomplete') %}
{% set verdict_class = {'pass': 'pass', 'fail': 'fail'}.get(verdict_key, 'warn') %}
- {% set verdict_label = advisory_fail_word if not has_t12_tab and verdict_key == 'fail' and tier3_advisory_fail else verdict_words[verdict_key] %}
+ {% set verdict_label = presentation.command_label %}
{% set check_words = {'pass': 'Passed', 'fail': 'Failed', 'incomplete': 'Incomplete', 'skipped': 'Skipped', 'not_applicable': 'N/A'} %}
<header class="main-header">
<div class="header-content">
@@ -1304,7 +1296,7 @@
-#}
{% set hero_title = explicit_skill_name | default(none) or friendly_label or (tier3.skill_name if has_tier3 and tier3.skill_name else (target_display or title)) %}
{% set hero_score = completed_agent.score if completed_agent else (tier3.overall_score if has_tier3 and tier3.overall_score is not none else none) %}
- {% set hero_trials = ((tier3.trials or []) | length) if has_tier3 else 0 %}
+ {% set hero_trials = presentation.coverage.text %}
{% set hero_metric_count = ((tier3.metric_ids or tier3.evaluators or {}) | length) if has_tier3 else 0 %}
{% set hero_harbor = (tier3.get('harbor_viewer') or tier3.get('summary', {}).get('harbor_viewer') or {}) if has_tier3 else {} %}
{% set hero_harbor_job_url = hero_harbor.get('job_url') %}
@@ -1314,6 +1306,7 @@
<div class="agent-hero-title-block">
<span class="agent-hero-eyebrow">{{ content_label }} evaluation</span>
<h1 class="agent-hero-title" id="report-title">{{ hero_title }}</h1>
+ <p class="hero-command-status {{ verdict_class }}">{{ presentation.command_label }}</p>
{#-
Show the *full* path/URL only when it actually adds
information (i.e. it differs from the friendly label).
@@ -1348,13 +1341,13 @@
{% set t3_key = tier3_verdict if tier3_verdict in ['pass', 'fail', 'neutral'] else 'incomplete' %}
<span class="verdict-chip {{ {'pass': 'pass', 'fail': 'fail'}.get(t3_key, 'warn') }}"><span class="sr-only">Tier 3 verdict: </span>{{ advisory_fail_word if tier3_advisory_fail else verdict_words[t3_key] }}</span>
{% if t3_key == 'incomplete' and tier3_status_reason %}<p class="hero-reason">{{ tier3_status_reason }}</p>{% endif %}
- <div class="agent-hero-score {{ 'score-pass' if hero_score is not none and hero_score >= 0.8 else ('score-warn' if hero_score is not none and hero_score >= 0.6 else 'score-fail') }}">{{ "%.2f" | format(hero_score) if hero_score is not none else "N/A" }}<span class="hero-score-label">{{ 'baseline score' if tier3.get('mode') == 'baseline_only' else 'overall score' }}</span></div>
+ <div class="agent-hero-score {{ 'score-pass' if hero_score is not none and hero_score >= 0.8 else ('score-warn' if hero_score is not none and hero_score >= 0.6 else 'score-fail') }}">{{ "%.2f" | format(hero_score) if hero_score is not none else "N/A" }}<span class="hero-score-label">{{ presentation.score_scope }}{{ ' (baseline)' if tier3.get('mode') == 'baseline_only' else '' }}</span></div>
<div class="agent-hero-detail">
<div><strong>Best performing agent:</strong> {{ tier3.best_agent or 'N/A' }}</div>
{% if tier3.get('mode') == 'baseline_only' %}<div>Baseline-only run (<code>--skip-with-skill</code>): no with-skill arm, so no skill lift was measured.</div>{% endif %}
{% if completed_agent and completed_agent.needs_attention %}<div class="score-warn">Completed-agent result. Other execution failures need attention; see agent details.</div>{% endif %}
{% if completed_agent and completed_agent.recovered_attempts %}<div>{{ completed_agent.recovered_attempts }} timeout attempt(s) recovered; failed history retained.</div>{% endif %}
- <div><strong>Trials:</strong> {{ hero_trials }} across {{ (tier3.agents_run or []) | length }} agent(s)</div>
+ <div class="trial-coverage">{{ hero_trials }}</div>{% for detail in presentation.coverage.details %}<div>{{ detail }}</div>{% endfor %}
<div><strong>Metrics:</strong> {{ hero_metric_count }} evaluators</div>
{% if tier3_environment %}<div><strong>Environment:</strong> <code>{{ tier3_environment }}</code></div>{% endif %}
{% if tier3.get('overall_lift') is not none %}<div><strong>Best-agent lift:</strong> {{ "%+.2f" | format(tier3.get('overall_lift')) }}</div>{% endif %}
@@ -1388,7 +1381,7 @@
{% set single_tier = tabs | length < 2 %}
<div class="tier-cards{{ ' single-tier' if single_tier else '' }}{{ ' tier-line' if single_tier and has_tier1 else '' }}">
{% if has_tier1 %}
- {% set t1_class = 'warn' if tier1_summary.incomplete_count else ('skipped' if tier1_summary.executed_count == 0 else ('pass' if tier1_summary.all_passed else 'fail')) %}
+ {% set t1_class = 'fail' if tier1_summary.failed_count else ('warn' if tier1_summary.incomplete_count else ('skipped' if tier1_summary.executed_count == 0 else 'pass')) %}
<div class="tier-card {{ t1_class }}{% if initial_tab == 'tier1' %} active{% endif %}"
role="button"
tabindex="0"
@@ -1399,7 +1392,7 @@
title="Open Tier 1: Security and Static Validation">
<div class="tier-card-head">
<span class="tier-card-label">Tier 1</span>
- {% if not single_tier %}<span class="tier-card-verdict">{{ verdict_words[{'pass': 'pass', 'fail': 'fail', 'warn': 'incomplete'}.get(t1_class, 'skipped')] }}</span>{% endif %}
+ {% if not single_tier %}<span class="tier-card-verdict">{{ 'N/A' if presentation.tiers[1].display_status == 'not_applicable' else verdict_words[{'pass': 'pass', 'fail': 'fail', 'warn': 'incomplete'}.get(t1_class, 'skipped')] }}</span>{% endif %}
</div>
<p class="tier-card-title">Security &amp; Static Validation</p>
<div class="tier-card-stats">
@@ -1421,7 +1414,7 @@
</div>
{% endif %}
{% if has_tier2 %}
- {% set t2_class = 'warn' if tier2_summary.incomplete_count else ('skipped' if tier2_summary.executed_count == 0 else ('pass' if tier2_summary.all_passed else 'fail')) %}
+ {% set t2_class = 'fail' if tier2_summary.failed_count else ('warn' if tier2_summary.incomplete_count else ('skipped' if tier2_summary.executed_count == 0 else 'pass')) %}
<div class="tier-card {{ t2_class }}{% if initial_tab == 'tier2' %} active{% endif %}"
role="button"
tabindex="0"
@@ -1432,19 +1425,21 @@
title="Open Tier 2: Deduplication">
<div class="tier-card-head">
<span class="tier-card-label">Tier 2</span>
- <span class="tier-card-verdict">{{ advisory_fail_word if t2_class == 'fail' and 'tier2' in nonblocking_tiers else verdict_words[{'pass': 'pass', 'fail': 'fail', 'warn': 'incomplete'}.get(t2_class, 'skipped')] }}</span>
+ <span class="tier-card-verdict">{{ 'N/A' if presentation.tiers[2].display_status == 'not_applicable' else (advisory_fail_word if t2_class == 'fail' and 'tier2' in nonblocking_tiers else verdict_words[{'pass': 'pass', 'fail': 'fail', 'warn': 'incomplete'}.get(t2_class, 'skipped')]) }}</span>
</div>
<p class="tier-card-title">Deduplication</p>
<div class="tier-card-stats">
<span><strong>{{ tier2_summary.passed_count }}</strong> of {{ tier2_summary.executed_count }} checks passed</span>
+ {% if tier2_summary.not_applicable_count %}<span><strong>{{ tier2_summary.not_applicable_count }}</strong> not applicable</span>{% endif %}
{% if tier2_summary.skipped_count %}<span><strong>{{ tier2_summary.skipped_count }}</strong> skipped</span>{% endif %}
{% if tier2_summary.issue_count %}<span><strong>{{ tier2_summary.issue_count }}</strong> duplicate(s)</span>{% endif %}
</div>
</div>
{% endif %}
{% if has_tier3 %}
- {% set tier3_display_verdict = verdict_words['skipped'] if tier3_is_skipped else (advisory_fail_word if tier3_advisory_fail else verdict_words[tier3_verdict if tier3_verdict in ['pass', 'fail', 'neutral'] else 'incomplete']) %}
- {% set t3_class = 'skipped' if tier3_is_skipped else ('warn' if completed_agent and completed_agent.needs_attention else ('pass' if tier3_verdict == 'pass' else ('fail' if tier3_verdict == 'fail' else 'warn'))) %}
+ {% set t3_tile_verdict = presentation.tiers[3].assessment_status or 'incomplete' %}
+ {% set tier3_display_verdict = verdict_words['skipped'] if tier3_is_skipped else (advisory_fail_word if t3_tile_verdict == 'fail' and 'tier3' in nonblocking_tiers else verdict_words[t3_tile_verdict if t3_tile_verdict in ['pass', 'fail', 'neutral'] else 'incomplete']) %}
+ {% set t3_class = 'fail' if t3_tile_verdict == 'fail' else ('skipped' if tier3_is_skipped else ('warn' if completed_agent and completed_agent.needs_attention else ('pass' if t3_tile_verdict == 'pass' else 'warn'))) %}
<div class="tier-card {{ t3_class }}{% if initial_tab == 'tier3' %} active{% endif %}"
role="button"
tabindex="0"
@@ -1455,16 +1450,16 @@
title="Open Tier 3: Live Agent Evaluation">
<div class="tier-card-head">
<span class="tier-card-label">Tier 3</span>
- <span class="tier-card-verdict">{{ tier3_display_verdict }}{% if report_scope == 'tier3' and completed_agent %} · Quality{% if completed_agent.needs_attention %}; needs attention{% endif %}{% endif %}</span>
+ <span class="tier-card-verdict">{{ tier3_display_verdict }}</span>
</div>
<p class="tier-card-title">Live Agent Evaluation</p>
<div class="tier-card-stats">
{% if tier3_is_skipped %}
<span>{{ tier3_skip_message }}</span>
{% else %}
- <span>score <strong>{{ "%.2f" | format(hero_score) if hero_score is not none else "N/A" }}</strong></span>
+ <span>Best agent score <strong>{{ "%.2f" | format(hero_score) if hero_score is not none else "N/A" }}</strong></span>
{% if tier3.get('overall_lift') is not none %}<span>lift <strong>{{ "%+.2f" | format(tier3.get('overall_lift')) }}</strong></span>{% endif %}
- {% if hero_trials %}<span><strong>{{ hero_trials }}</strong> trial(s)</span>{% endif %}
+ <span class="trial-coverage">{{ hero_trials }}</span>
{% endif %}
</div>
{% if hero_harbor_job_url or hero_harbor_analysis_url %}
@@ -1497,7 +1492,7 @@
{% set incomplete_skills = skills.values() | selectattr('status', 'defined') | selectattr('status', 'equalto', 'incomplete') | list | length %}
{% set failed_skills = [tier1_display_summary.failed_skills - incomplete_skills, 0] | max %}
{# ==================== EXECUTIVE SUMMARY ==================== #}
- {% set t1_key = 'incomplete' if tier1_summary.incomplete_count else ('skipped' if tier1_summary.executed_count == 0 else ('pass' if tier1_summary.all_passed else 'fail')) %}
+ {% set t1_key = 'fail' if tier1_summary.failed_count else ('incomplete' if tier1_summary.incomplete_count else ('skipped' if tier1_summary.executed_count == 0 else 'pass')) %}
{% set t1_class = {'pass': 'pass', 'fail': 'fail'}.get(t1_key, 'warn') %}
{% set t1_all_checks = total_validators + ui_t1_na + (skipped_count or 0) %}
{% set qualities = skills.values() | selectattr('quality', 'defined') | map(attribute='quality') | selectattr('overall_score', 'defined') | list %}
@@ -1512,7 +1507,7 @@
{%- elif t1_key == 'skipped' -%}
No Tier 1 check ran, so there is no result to report.
{%- else -%}
- All {{ total_validators }} checks that apply passed.{% if tier1_summary.issue_count %} {{ tier1_summary.issue_count }} advisory finding{{ 's do' if tier1_summary.issue_count != 1 else ' does' }} not block.{% endif %}
+ All {{ tier1_summary.executed_count }} checks that apply passed.{% if tier1_summary.issue_count %} {{ tier1_summary.issue_count }} advisory finding{{ 's do' if tier1_summary.issue_count != 1 else ' does' }} not block.{% endif %}
{%- endif -%}
{%- if tier1_incomplete_scans -%}
{%- if tier1_scanners_missing %} Install or enable the scanner and run the evaluation again.
diff --git a/src/skillevaluator/tier1/commands.py b/src/skillevaluator/tier1/commands.py
index 6becf261..dca05fee 100644
--- a/src/skillevaluator/tier1/commands.py
+++ b/src/skillevaluator/tier1/commands.py
@@ -32,8 +32,9 @@ from skillevaluator.publication_identity import (
publication_target_from_path,
)
from skillevaluator.reporting import CLIReporter, HTMLReporter, JSONReporter, MarkdownReporter, SARIFReporter
-from skillevaluator.reporting.html import is_tier2_validator_name
from skillevaluator.reporting.naming import DEFAULT_REPORT_BASENAME
+from skillevaluator.reporting.presentation import FinalRenderContext, build_final_render_context
+from skillevaluator.reporting.run_status import result_tier
from skillevaluator.utils.rich_markup import escape_markup
from skillevaluator.validators.base import continue_on_failure_scope
from skillevaluator.validators.code_risk import CodeRiskValidator
@@ -476,26 +477,15 @@ def run_lint_scripts(target_path: Path) -> list[ValidationResult]:
]
-def _is_dedup_result(result: ValidationResult) -> bool:
- """Return True when a result came from a Tier 2 deduplication validator."""
- return is_tier2_validator_name(result.validator_name)
-
-
def _derive_html_tabs(results: list[ValidationResult]) -> list[dict[str, str]]:
- """Build the HTML navigation tabs from the tiers present in *results*.
-
- Tier 1 is included only when a non-Tier-2 result is present; this keeps
- standalone similarity and deduplication reports in their actual tier.
- Tier 3 is added when a live agent-evaluation payload is attached.
- """
- tabs: list[dict[str, str]] = []
- if not results or any(not _is_dedup_result(result) for result in results):
- tabs.append({"id": "tier1", "label": "Tier 1: Security and Static Validation"})
- if any(_is_dedup_result(r) for r in results):
- tabs.append({"id": "tier2", "label": "Tier 2: Deduplication"})
- if any((getattr(r, "metadata", None) or {}).get("agent_eval") for r in results):
- tabs.append({"id": "tier3", "label": "Tier 3: Live Agent Evaluation"})
- return tabs
+ """Build populated tabs with the same tier classifier as the HTML evidence."""
+ tiers = {result_tier(result) for result in results} if results else {1}
+ labels = {
+ 1: "Tier 1: Security and Static Validation",
+ 2: "Tier 2: Deduplication",
+ 3: "Tier 3: Live Agent Evaluation",
+ }
+ return [{"id": f"tier{tier}", "label": labels[tier]} for tier in sorted(tiers)]
def emit_reports(
@@ -511,6 +501,9 @@ def emit_reports(
announce_paths: bool = True,
sarif_scan_root: Path | None = None,
sarif_repository_root: Path | None = None,
+ requested_tiers: tuple[int, ...] | None = None,
+ displayed_tiers: tuple[int, ...] | None = None,
+ presentation_context: FinalRenderContext | None = None,
) -> bool:
"""Render reports and return whether every result passed.
@@ -525,9 +518,15 @@ def emit_reports(
"""
if policy is not None:
apply_policy(results, policy)
+ context = presentation_context or build_final_render_context(
+ results,
+ requested_tiers=requested_tiers,
+ displayed_tiers=displayed_tiers,
+ expected_skill_name=expected_skill_name,
+ )
if "cli" in report_formats:
# The CLI report previews failed Tier 3 source checks only when a file written below lists them all.
- CLIReporter(console=console, report_formats=report_formats).print_all(results)
+ CLIReporter(console=console, report_formats=report_formats, presentation_context=context).print_all(results)
html_tabs = _derive_html_tabs(results)
for fmt in report_formats:
@@ -539,6 +538,9 @@ def emit_reports(
target_path=target_path,
content_label=content_label,
tabs=html_tabs,
+ presentation_context=context,
+ requested_tiers=requested_tiers,
+ displayed_tiers=displayed_tiers,
expected_skill_name=expected_skill_name,
)
elif fmt == "sarif":
diff --git a/tests/reporting/test_final_render_context.py b/tests/reporting/test_final_render_context.py
new file mode 100644
index 00000000..2d048831
--- /dev/null
+++ b/tests/reporting/test_final_render_context.py
@@ -0,0 +1,729 @@
+# SPDX-License-Identifier: Apache-2.0
+"""Human reports retain recorded coverage and distinguish it from approval."""
+
+from __future__ import annotations
+
+import copy
+import re
+from dataclasses import FrozenInstanceError
+
+import pytest
+
+from skillevaluator.models import ValidationResult
+from skillevaluator.reporting import presentation
+from skillevaluator.reporting.cli import CLIReporter
+from skillevaluator.reporting.html import HTMLReporter
+from skillevaluator.reporting.run_status import Status
+
+
+def context(results, **kwargs):
+ builder = getattr(presentation, "build_final_render_context", None)
+ assert callable(builder), "final reports need one shared immutable presentation context"
+ return builder(results, **kwargs)
+
+
+def result(tier=1, **metadata):
+ return ValidationResult(
+ validator_name="AGENT_EVAL" if tier == 3 else f"Tier{tier}",
+ metadata={
+ "gating": {"tier": tier, "blocking": True},
+ **metadata,
+ },
+ )
+
+
+def test_tier_counts_exclude_na_and_do_not_leak_other_tiers():
+ na = result()
+ na.add_success("code_file_discovery", "No code files found - code risk analysis is not applicable")
+ failed = result(2)
+ failed.add_error("duplicate")
+ incomplete = result(2)
+ incomplete.mark_scan_incomplete("catalog")
+ ctx = context(
+ [result(), na, result(skipped=True), failed, incomplete], requested_tiers=(1, 2), displayed_tiers=(1, 2, 3)
+ )
+ assert ctx.command_status == Status.FAILED
+ assert ctx.command_label == "Changes required"
+ assert (ctx.tiers[1].passed, ctx.tiers[1].executed, ctx.tiers[1].not_applicable, ctx.tiers[1].skipped) == (
+ 1,
+ 1,
+ 1,
+ 1,
+ )
+ assert (ctx.tiers[2].passed, ctx.tiers[2].failed, ctx.tiers[2].incomplete, ctx.tiers[2].executed) == (0, 1, 1, 2)
+ assert ctx.tiers[2].status == Status.FAILED
+ assert ctx.tiers[3].status == Status.NOT_RUN
+ assert not ctx.tiers[3].requested
+ with pytest.raises((FrozenInstanceError, AttributeError)):
+ ctx.command_label = "changed"
+ with pytest.raises(TypeError):
+ ctx.tiers[1] = ctx.tiers[2]
+
+
+def test_recorded_condition_coverage_outlives_preview_and_retry_rows():
+ payload = {
+ "agents": {
+ "agent": {
+ "conditions": {
+ "with_skill": {"execution_status": "succeeded", "scored_attempts": 101, "expected_attempts": 120},
+ "without_skill": {"execution_status": "failed", "scored_attempts": 87, "expected_attempts": 120},
+ },
+ "trials_baseline": [{}] * 64,
+ }
+ },
+ "trials": [{}] * 64,
+ }
+ source = result(3, agent_eval=payload)
+ original = copy.deepcopy(source)
+ ctx = context([source], report_scope="tier3")
+ assert ctx.coverage.text == "188 scored trials: 101 with skill, 87 baseline"
+ assert ctx.coverage.with_skill.expected == 120
+ assert ctx.coverage.baseline.status == "failed"
+ assert ctx.publication_status is None
+ assert ctx.publication_label == "Not assessed (Tier 3 only)"
+ assert source == original
+
+
[email protected]("bad", [None, True, -1, "3", 1.5])
+def test_missing_or_malformed_counters_never_become_zero(bad):
+ payload = {
+ "agents": {
+ "a": {
+ "conditions": {
+ "with_skill": {"execution_status": "incomplete", "scored_attempts": bad},
+ "without_skill": {"execution_status": "skipped", "scored_attempts": 0, "expected_attempts": 0},
+ }
+ }
+ },
+ "trials": [{"overall": 0.8}],
+ }
+ coverage = context([result(3, agent_eval=payload)]).coverage
+ assert coverage.with_skill.scored is None
+ assert coverage.baseline.status == "skipped"
+ assert "unavailable" in coverage.text
+ assert "baseline skipped" in coverage.text
+ assert "0 scored trials" not in coverage.text
+
+
+def test_legacy_rows_are_observed_evidence_not_complete_totals():
+ coverage = context(
+ [
+ result(
+ 3,
+ agent_eval={
+ "trials": [{"overall": 0.8}] * 64,
+ "agents": {"a": {"trials_baseline": [{"overall": 0.2}] * 3}},
+ },
+ )
+ ]
+ ).coverage
+ assert coverage.with_skill.scored is None
+ assert coverage.with_skill.observed == 64
+ assert coverage.baseline.observed == 3
+ assert "64 retained with-skill rows" in coverage.text
+ assert "3 retained baseline rows" in coverage.text
+ assert "67 scored" not in coverage.text
+
+
+def test_failed_result_outranks_incomplete_in_html_tile_and_header():
+ failed = result()
+ failed.add_error("unsafe")
+ missing = result()
+ missing.mark_scan_incomplete("scanner")
+ html = HTMLReporter(tabs=[{"id": "tier1", "label": "Tier 1"}, {"id": "tier2", "label": "Tier 2"}]).render_all(
+ [failed, missing]
+ )
+ tile = re.search(r'<div class="tier-card ([^"]+)"[^>]+data-target-tab="tier1"', html)
+ assert tile and "fail" in tile[1]
+ assert 'data-verdict="fail"' in html
+ assert "Changes required" in html
+
+
+def test_full_cli_and_html_distinguish_required_pass_from_publication():
+ source = result()
+ terminal = re.sub(r"\x1b\[[0-9;]*m", "", CLIReporter().render_all([source]))
+ html = HTMLReporter().render_all([source])
+ assert "Passed required checks" in terminal
+ assert "Passed required checks" in html
+ assert "Publication eligibility" in terminal
+ assert "Publication eligibility" in html
+
+
+def test_compact_panel_shares_incomplete_status_and_publication(tmp_path):
+ from io import StringIO
+
+ from rich.console import Console
+
+ from skillevaluator.cli import _finish_pipeline_view
+ from skillevaluator.reporting.console_ui import ValidateView
+
+ source = result()
+ source.mark_scan_incomplete("scanner")
+ output = StringIO()
+ view = ValidateView("demo", [(1, "Tier 1", "")], console=Console(file=output, width=120, color_system=None))
+ _finish_pipeline_view(
+ view,
+ tier_gate_results=[source],
+ tier3_result=None,
+ gate_failed=True,
+ output_dir=tmp_path,
+ basename="report",
+ report_formats=(),
+ target_path=tmp_path,
+ agent_eval=False,
+ )
+ assert "Incomplete" in output.getvalue()
+ assert "INCOMPLETE" in output.getvalue()
+ assert "Publication eligibility: Incomplete" in output.getvalue()
+ assert "✗ FAIL" not in output.getvalue()
+
+
+def test_html_coverage_uses_full_raw_counts_and_preserves_payload():
+ source = result(
+ 3,
+ agent_eval={
+ "agents": {
+ "a": {
+ "conditions": {
+ "with_skill": {
+ "execution_status": "succeeded",
+ "scored_attempts": 101,
+ "expected_attempts": 120,
+ },
+ "without_skill": {
+ "execution_status": "failed",
+ "scored_attempts": 87,
+ "expected_attempts": 120,
+ },
+ }
+ }
+ },
+ "trials": [{"overall": 0.8}] * 100,
+ },
+ )
+ original = copy.deepcopy(source)
+ html = HTMLReporter().render(source)
+ assert "188 scored trials: 101 with skill, 87 baseline" in html
+ assert "Best agent score" in html
+ assert source == original
+
+
+def test_required_failure_cannot_be_replaced_by_completed_quality_verdict():
+ source = result(3, agent_eval={"verdict": "pass", "overall_score": 0.8})
+ source.add_error("agent setup failed")
+ html = HTMLReporter(report_scope="tier3").render(source)
+ assert re.search(r'header-status-badge[^>]+data-verdict="fail"', html)
+ assert 'data-publication-status="not-assessed"' in html
+
+
+def test_missing_requested_tier_is_never_reported_as_zero_checks_passed():
+ source = result()
+ ctx = context([source], requested_tiers=(1,), displayed_tiers=(1, 2, 3))
+ output = re.sub(r"\x1b\[[0-9;]*m", "", CLIReporter(presentation_context=ctx).render_all([source]))
+ assert "Tier 2: Not run (not requested)" in output
+ assert "Tier 3: Not run (not requested)" in output
+
+
+def test_partial_expected_attempt_counts_remain_visible():
+ source = result(
+ 3,
+ agent_eval={
+ "agents": {
+ "a": {
+ "conditions": {
+ "with_skill": {"execution_status": "incomplete", "scored_attempts": 4, "expected_attempts": 10},
+ "without_skill": {"execution_status": "skipped", "scored_attempts": 0, "expected_attempts": 0},
+ }
+ }
+ }
+ },
+ )
+ assert "With skill: 4 of 10 scored (incomplete)" in HTMLReporter().render(source)
+
+
+def test_confirmed_failure_never_gets_green_pass_in_full_cli_details():
+ source = result(confirmed_failure=True)
+ source.mark_scan_incomplete("scanner")
+ output = re.sub(r"\x1b\[[0-9;]*m", "", CLIReporter().render_all([source]))
+ assert "[PASS] Validation passed" not in output
+ assert "[INCOMPLETE] scanner did not complete" not in output
+ assert "[FAIL] Validation failed" in output
+
+
+def test_all_na_tier_remains_na_without_claiming_execution():
+ source = result()
+ source.add_success("lint", "No scripts/ or tools/ directory found")
+ ctx = context([source], displayed_tiers=(1, 2))
+ assert getattr(ctx.tiers[1], "display_status", None) == "not_applicable"
+ html = HTMLReporter(tabs=[{"id": "tier1", "label": "Tier 1"}, {"id": "tier2", "label": "Tier 2"}]).render(source)
+ assert re.search(r'tier-card-label">Tier 1</span>\s*<span class="tier-card-verdict">N/A</span>', html)
+
+
[email protected]("skipped_arm", ["with_skill", "without_skill"])
+def test_one_arm_runs_preserve_explicit_skip_and_do_not_infer_baseline(skipped_arm):
+ active = "without_skill" if skipped_arm == "with_skill" else "with_skill"
+ source = result(
+ 3,
+ agent_eval={
+ "agents": {
+ "a": {
+ "conditions": {
+ active: {"execution_status": "succeeded", "scored_attempts": 7, "expected_attempts": 7},
+ skipped_arm: {"execution_status": "skipped", "scored_attempts": 0, "expected_attempts": 0},
+ }
+ }
+ }
+ },
+ )
+ coverage = context([source]).coverage
+ assert coverage.text == (
+ "7 scored trials: with skill skipped, 7 baseline"
+ if skipped_arm == "with_skill"
+ else "7 scored trials: 7 with skill, baseline skipped"
+ )
+
+
+def test_emitting_human_reports_preserves_canonical_json_and_return_value(tmp_path, monkeypatch):
+ from io import StringIO
+
+ from rich.console import Console
+
+ from skillevaluator.reporting.json_reporter import JSONReporter
+ from skillevaluator.tier1 import commands
+
+ source = result(
+ 3,
+ agent_eval={
+ "agents": {
+ "a": {
+ "conditions": {
+ "with_skill": {"execution_status": "failed", "scored_attempts": 3, "expected_attempts": 8},
+ "without_skill": {"execution_status": "skipped", "scored_attempts": 0, "expected_attempts": 0},
+ }
+ }
+ }
+ },
+ )
+ source.add_error("execution failed")
+ source.metadata["gating"]["blocking"] = False
+ before = JSONReporter(include_timestamp=False).render(source)
+ snapshot = copy.deepcopy(source)
+ output = StringIO()
+ monkeypatch.setattr(commands, "console", Console(file=output, color_system=None, width=160))
+ assert (
+ commands.emit_reports([source], report_formats=("cli", "html"), output_dir=tmp_path, basename="projection")
+ is False
+ )
+ assert "Passed required checks" in output.getvalue()
+ assert "3 scored trials: 3 with skill, baseline skipped" in (tmp_path / "projection.html").read_text(
+ encoding="utf-8"
+ )
+ assert source == snapshot
+ assert JSONReporter(include_timestamp=False).render(source) == before
+
+
+def test_tier3_failure_tile_and_hero_keep_command_outcome_separate_from_quality():
+ from skillevaluator.evaluation.tier3_report import build_agent_eval_payload
+ from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS
+
+ conditions = {
+ arm: {"execution_status": "succeeded", "execution_errors": [], "expected_attempts": 1, "scored_attempts": 1}
+ for arm in ("with_skill", "without_skill")
+ }
+ payload = build_agent_eval_payload(
+ "demo",
+ {
+ "good": {
+ "with_skill": dict.fromkeys(DEFAULT_METRICS, 0.8),
+ "without_skill": dict.fromkeys(DEFAULT_METRICS, 0.4),
+ "execution_status": "succeeded",
+ "execution_errors": [],
+ "expected_attempts": 2,
+ "scored_attempts": 2,
+ "conditions": conditions,
+ "num_trials": 1,
+ "num_trials_baseline": 1,
+ },
+ "failed": {
+ "execution_status": "failed",
+ "execution_errors": ["agent setup failed"],
+ "with_skill": {},
+ "without_skill": {},
+ },
+ },
+ dataset=[{"id": "case-a"}],
+ attempt_policy={"max_attempts": 1, "stop_on_pass": False, "pass_threshold": 0.5},
+ use_llm_judge=False,
+ )
+ source = result(3, agent_eval=payload)
+ source.add_error("agent setup failed")
+ html = HTMLReporter(report_scope="tier3").render(source)
+ hero = html.split('<section class="agent-hero"', 1)[1].split("</section>", 1)[0]
+ assert "Changes required" in hero
+ assert re.search(r'<div class="tier-card fail[^\"]*"[^>]+data-target-tab="tier3"', hero)
+
+
+def test_partial_tier3_evidence_assessment_is_shared_without_changing_command_gate():
+ source = result(3, agent_eval={"verdict": "pass"})
+ source.metadata["gating"]["blocking"] = False
+ ctx = context([source])
+ assert ctx.command_status == Status.PASSED
+ assert ctx.tiers[3].display_status == "incomplete"
+ output = re.sub(r"\x1b\[[0-9;]*m", "", CLIReporter(presentation_context=ctx).render_all([source]))
+ assert "Tier 3: Incomplete evidence" in output
+ html = HTMLReporter(presentation_context=ctx).render(source)
+ assert re.search(r'tier-card-label">Tier 3</span>\s*<span class="tier-card-verdict">Incomplete</span>', html)
+
+
+def test_explicit_tier3_skip_is_shared_even_when_publication_is_incomplete():
+ source = result(
+ 3,
+ agent_eval={
+ "execution_status": "skipped",
+ "verdict": "neutral",
+ "provenance": {
+ "advisory": True,
+ "reason": "skipped",
+ "message": "provider unavailable",
+ },
+ },
+ )
+ source.metadata["gating"]["blocking"] = False
+ ctx = context([source])
+ assert ctx.tiers[3].status == Status.SKIPPED
+ assert ctx.tiers[3].display_status == "skipped"
+ assert ctx.publication_status == Status.INCOMPLETE
+
+
+def test_retained_row_observations_share_one_projection_budget(monkeypatch):
+ from skillevaluator.reporting import base
+
+ visited = []
+
+ class CountedRows(list):
+ def __iter__(self):
+ for row in super().__iter__():
+ visited.append(row)
+ yield row
+
+ monkeypatch.setattr(base, "AGENT_EVAL_REPORT_MAX_NODES", 100)
+ rows = CountedRows([{}] * 10)
+ coverage = presentation.trial_coverage(
+ {
+ "trials": rows,
+ "agents": {f"agent-{i}": {"trials_baseline": rows} for i in range(12)},
+ }
+ )
+ assert len(visited) <= 100
+ assert coverage.with_skill.observed == 10
+ assert coverage.baseline.observed is None
+ assert coverage.baseline.scored is None
+ assert "baseline scored trials unavailable" in coverage.text
+ assert "0 retained baseline rows" not in coverage.text
+
+
+def test_recorded_scored_counters_avoid_all_retained_row_iteration():
+ visited = []
+
+ class CountedRows(list):
+ def __iter__(self):
+ for row in super().__iter__():
+ visited.append(row)
+ yield row
+
+ rows = CountedRows([{}] * 1000)
+ conditions = {
+ arm: {"execution_status": "succeeded", "scored_attempts": 9, "expected_attempts": 12}
+ for arm in ("with_skill", "without_skill")
+ }
+ coverage = presentation.trial_coverage(
+ {
+ "trials": rows,
+ "agents": {f"agent-{i}": {"conditions": conditions, "trials_baseline": rows} for i in range(120)},
+ }
+ )
+ assert visited == []
+ assert coverage.text == "2160 scored trials: 1080 with skill, 1080 baseline"
+ assert coverage.with_skill.expected == 1440
+ assert coverage.baseline.expected == 1440
+ assert coverage.with_skill.observed is None
+ assert coverage.baseline.observed is None
+
+
+def test_exhausted_projection_budget_does_not_turn_other_arm_into_observed_zero(monkeypatch):
+ from skillevaluator.reporting import base
+
+ monkeypatch.setattr(base, "AGENT_EVAL_REPORT_MAX_NODES", 10)
+ coverage = presentation.trial_coverage({"trials": [{}] * 11, "agents": {"a": {"trials_baseline": []}}})
+ assert coverage.with_skill.observed is None
+ assert coverage.baseline.observed is None
+ assert "0 retained baseline rows" not in coverage.text
+
+
[email protected]("width", [60, 80, 98])
[email protected](("skipped", "not_applicable"), [(1, 0), (0, 2), (2, 3)])
+def test_zero_execution_counts_stay_explicit_at_terminal_widths(monkeypatch, width, skipped, not_applicable):
+ monkeypatch.setenv("COLUMNS", str(width))
+ monkeypatch.setenv("TERM", "xterm-256color")
+ monkeypatch.setenv("CI", "true")
+ sources = [result(skipped=True) for _ in range(skipped)]
+ for _ in range(not_applicable):
+ source = result()
+ source.add_success("lint", "No scripts/ or tools/ directory found")
+ sources.append(source)
+ output = CLIReporter().render_all(sources)
+ assert f"Tier 1: {skipped} skipped, {not_applicable} N/A; no checks executed" in output
+ assert "0 of 0 checks passed" not in output
+ assert "Passed required checks" in output
+ assert "Publication eligibility: Incomplete" in output
+
+
+ ("mode", "present", "expected"),
+ [
+ ("with_skill_only", "with_skill", "3 scored trials: 3 with skill, baseline skipped"),
+ ("baseline_only", "without_skill", "3 scored trials: with skill skipped, 3 baseline"),
+ ],
+)
+def test_producer_one_arm_mode_marks_the_missing_arm_skipped(mode, present, expected):
+ # --skip-baseline / --skip-with-skill omit the unexecuted condition; the
+ # payload-level mode is the producer's explicit skip marker.
+ counts = {"execution_status": "succeeded", "scored_attempts": 3, "expected_attempts": 3}
+ coverage = presentation.trial_coverage({"mode": mode, "agents": {"a": {"conditions": {present: counts}}}})
+ assert coverage.text == expected
+ assert "unavailable" not in coverage.text
+
+
+def test_advisory_tier3_skip_reports_skipped_arms_not_unavailable_counts():
+ coverage = presentation.trial_coverage({"execution_status": "skipped", "agents": {}})
+ assert (coverage.with_skill.status, coverage.baseline.status) == ("skipped", "skipped")
+ assert "unavailable" not in coverage.text
+ source = result(3, skipped=True, agent_eval={"execution_status": "skipped", "agents": {}})
+ output = CLIReporter().render_all([source])
+ assert "scored trials unavailable" not in output
+
+
+def test_standalone_tier3_report_header_follows_failing_tier3_verdict():
+ from skillevaluator.evaluation.tier3_report import _validation_result_from_payload, build_agent_eval_payload
+ from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS
+
+ def arm(n):
+ return {"execution_status": "succeeded", "execution_errors": [], "expected_attempts": n, "scored_attempts": n}
+
+ payload = build_agent_eval_payload(
+ "demo",
+ {
+ "agent": {
+ "with_skill": dict.fromkeys(DEFAULT_METRICS, 0.2),
+ "without_skill": dict.fromkeys(DEFAULT_METRICS, 0.9),
+ "execution_status": "succeeded",
+ "execution_errors": [],
+ "expected_attempts": 2,
+ "scored_attempts": 2,
+ "conditions": {"with_skill": arm(1), "without_skill": arm(1)},
+ "num_trials": 1,
+ "num_trials_baseline": 1,
+ }
+ },
+ dataset=[{"id": "case-a"}],
+ attempt_policy={"max_attempts": 1, "stop_on_pass": False, "pass_threshold": 0.5},
+ use_llm_judge=False,
+ )
+ source = _validation_result_from_payload(payload)
+ assert payload["verdict"] == "fail" and "gating" not in source.metadata
+ ctx = context([source], report_scope="tier3")
+ assert ctx.command_status == Status.FAILED
+ html = HTMLReporter(report_scope="tier3", tabs=[{"id": "tier3", "label": "Tier 3"}]).render_all([source])
+ assert re.search(r'header-status-badge[^>]*data-verdict="fail"', html)
+ assert "Passed required checks" not in html
+
+
+def test_not_applicable_rows_show_their_recorded_reason():
+ covered = ValidationResult(validator_name="Code Risk", metadata={"gating": {"tier": 1, "blocking": True}})
+ covered.add_success("code_file_discovery", "No code files found - code risk analysis is not applicable")
+ skipped = result(skipped=True, skip_reason="Lint is not applicable: no scripts directory")
+ output = re.sub(r"\s+", " ", CLIReporter().render_all([covered, skipped]))
+ assert "1 checks passed" not in output
+ assert "Skipped (see warnings)" not in output
+ assert "No code files found" in output
+ assert "Lint is not applicable: no scripts directory" in output
+
+
+def test_executed_tier_counts_stay_on_one_line_at_narrow_widths(monkeypatch):
+ monkeypatch.setenv("COLUMNS", "60")
+ monkeypatch.setenv("TERM", "xterm-256color")
+ monkeypatch.setenv("CI", "true")
+ passed = result()
+ passed.add_success("lint", "Lint passed")
+ failed = result()
+ failed.add_error("Lint failed")
+ output = CLIReporter().render_all([passed, failed, result(skipped=True)])
+ assert "Tier 1: 1 of 2 checks passed; 1 failed, 0 incomplete, 1 skipped, 0 N/A" in output
+
+
+ "mode",
+ [
+ pytest.param([], id="empty-list"),
+ pytest.param({}, id="empty-object"),
+ pytest.param(["with_skill_only"], id="list-mode"),
+ pytest.param({"baseline_only": True}, id="object-mode"),
+ pytest.param(None, id="null"),
+ pytest.param(1, id="number"),
+ pytest.param(True, id="boolean"),
+ pytest.param("unrecognized", id="unknown-string"),
+ ],
+)
[email protected]("recorded", [False, True], ids=["unavailable", "recorded-conditions"])
+def test_malformed_or_unknown_mode_preserves_coverage_evidence_and_payload(mode, recorded):
+ agents = {}
+ if recorded:
+ agents = {
+ "a": {
+ "conditions": {
+ "with_skill": {"execution_status": "succeeded", "scored_attempts": 2, "expected_attempts": 3},
+ "without_skill": {"execution_status": "incomplete", "scored_attempts": 1, "expected_attempts": 3},
+ }
+ }
+ }
+ payload = {"mode": mode, "agents": agents}
+ before = copy.deepcopy(payload)
+ coverage = presentation.trial_coverage(payload)
+ if recorded:
+ assert coverage.with_skill.status == "succeeded"
+ assert coverage.baseline.status == "incomplete"
+ assert coverage.with_skill.scored == 2
+ assert coverage.baseline.scored == 1
+ assert coverage.with_skill.expected == coverage.baseline.expected == 3
+ assert coverage.text == "3 scored trials: 2 with skill, 1 baseline"
+ else:
+ assert coverage.with_skill.status == coverage.baseline.status == "unavailable"
+ assert coverage.with_skill.scored is coverage.baseline.scored is None
+ assert coverage.with_skill.expected is coverage.baseline.expected is None
+ assert "with skill scored trials unavailable" in coverage.text
+ assert "baseline scored trials unavailable" in coverage.text
+ assert payload == before
+
+
+def passing_tier3_counts_payload():
+ from skillevaluator.evaluation.tier3_report import build_agent_eval_payload
+ from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS
+
+ arm = {"execution_status": "succeeded", "execution_errors": [], "expected_attempts": 1, "scored_attempts": 1}
+ return build_agent_eval_payload(
+ "demo",
+ {
+ "agent": {
+ "with_skill": dict.fromkeys(DEFAULT_METRICS, 0.8),
+ "without_skill": dict.fromkeys(DEFAULT_METRICS, 0.4),
+ "execution_status": "succeeded",
+ "execution_errors": [],
+ "expected_attempts": 2,
+ "scored_attempts": 2,
+ "conditions": {"with_skill": arm, "without_skill": arm},
+ "num_trials": 1,
+ "num_trials_baseline": 1,
+ }
+ },
+ dataset=[{"id": "case-a"}],
+ attempt_policy={"max_attempts": 1, "stop_on_pass": False, "pass_threshold": 0.5},
+ use_llm_judge=False,
+ )
+
+
[email protected]("tier", [1, 2, 3])
+ ("adverse", "blocking", "expected", "label"),
+ [
+ ("fail", True, (1, 0, 0), "Changes required"),
+ ("incomplete", True, (0, 1, 0), "Incomplete"),
+ ("fail", False, (0, 0, 1), "Passed required checks"),
+ ("incomplete", False, (0, 0, 1), "Passed required checks"),
+ ],
+)
+def test_per_result_adverse_counts_handle_mixed_tier_gates(
+ tier, adverse, blocking, expected, label, passing_tier3_counts_payload
+):
+ passing, unfavorable = result(tier), result(tier)
+ if tier == 3:
+ passing.metadata["agent_eval"] = copy.deepcopy(passing_tier3_counts_payload)
+ unfavorable.metadata["agent_eval"] = copy.deepcopy(passing_tier3_counts_payload)
+ unfavorable.metadata["gating"]["blocking"] = blocking
+ if adverse == "fail":
+ unfavorable.add_error("failed required or advisory check")
+ else:
+ unfavorable.mark_scan_incomplete("scanner")
+ sources = [passing, unfavorable]
+ before = copy.deepcopy(sources)
+ ctx = context(sources)
+ assert (ctx.required_failed, ctx.required_incomplete, ctx.advisory_unfavorable) == expected
+ assert ctx.command_label == label
+ assert sources == before
+
+
+ ("verdict", "advisory", "expected", "label"),
+ [
+ ("fail", False, (1, 0, 0), "Changes required"),
+ ("inconclusive", False, (0, 1, 0), "Incomplete"),
+ ("fail", True, (0, 0, 1), "Passed required checks"),
+ ("inconclusive", True, (0, 0, 1), "Passed required checks"),
+ ],
+)
+def test_standalone_adverse_counts_follow_existing_headline_override(verdict, advisory, expected, label):
+ source = ValidationResult(validator_name="AGENT_EVAL", passed=True, metadata={"agent_eval": {"verdict": verdict}})
+ if advisory:
+ source.metadata["gating"] = {"tier": 3, "blocking": False}
+ before = copy.deepcopy(source)
+ ctx = context([source], report_scope="tier3")
+ assert (ctx.required_failed, ctx.required_incomplete, ctx.advisory_unfavorable) == expected
+ assert ctx.command_label == label
+ assert source == before
+
+
+def test_gate_rejected_pass_shaped_tier3_counts_as_required_failure():
+ source = result(3, agent_eval={"verdict": "pass"})
+ before = copy.deepcopy(source)
+ ctx = context([source])
+ assert ctx.result_statuses[id(source)] == "pass"
+ assert (ctx.required_failed, ctx.required_incomplete, ctx.advisory_unfavorable) == (1, 0, 0)
+ assert ctx.command_label == "Changes required"
+ assert source == before
+
+
+def test_adverse_counts_default_for_existing_context_constructor_and_are_immutable():
+ ctx = context([])
+ legacy = presentation.FinalRenderContext(
+ ctx.command_status,
+ ctx.command_label,
+ ctx.publication_status,
+ ctx.publication_label,
+ ctx.tiers,
+ ctx.result_statuses,
+ ctx.score_scope,
+ ctx.coverage,
+ )
+ assert (legacy.required_failed, legacy.required_incomplete, legacy.advisory_unfavorable) == (0, 0, 0)
+ with pytest.raises(FrozenInstanceError):
+ legacy.required_failed = 1
+
+
+def test_adverse_counts_accumulate_across_required_and_advisory_results_in_all_tiers(passing_tier3_counts_payload):
+ failed = result(1)
+ failed.add_error("required failure")
+ missing = result(2)
+ missing.mark_scan_incomplete("required-scanner")
+ advisory_failed = result(2)
+ advisory_failed.metadata["gating"]["blocking"] = False
+ advisory_failed.add_error("advisory failure")
+ passed = result(3, agent_eval=copy.deepcopy(passing_tier3_counts_payload))
+ advisory_missing = result(3, agent_eval=copy.deepcopy(passing_tier3_counts_payload))
+ advisory_missing.metadata["gating"]["blocking"] = False
+ advisory_missing.mark_scan_incomplete("advisory-scanner")
+ sources = [result(1), failed, missing, advisory_failed, passed, advisory_missing]
+ before = copy.deepcopy(sources)
+ ctx = context(sources)
+ assert (ctx.required_failed, ctx.required_incomplete, ctx.advisory_unfavorable) == (1, 1, 2)
+ assert ctx.command_status == Status.FAILED
+ assert sources == before
diff --git a/tests/reporting/test_report_redesign.py b/tests/reporting/test_report_redesign.py
index cf9c2a5e..1ab68a33 100644
--- a/tests/reporting/test_report_redesign.py
+++ b/tests/reporting/test_report_redesign.py
@@ -275,7 +275,7 @@ def _tier_card_verdicts(html: str) -> list[tuple[str, str]]:
return re.findall(r'tier-card-label">(Tier \d)</span>\s*<span class="tier-card-verdict">([^<]+)</span>', html)
-def test_header_says_skipped_when_no_check_ran() -> None:
+def test_header_keeps_required_gate_separate_from_skipped_execution() -> None:
skipped = []
for name in ("Secrets Detection", "PII Scan"):
result = ValidationResult(validator_name=name)
@@ -285,10 +285,10 @@ def test_header_says_skipped_when_no_check_ran() -> None:
html = _render(_gated(skipped, 1))
summary = html.split('<section class="exec-summary', 1)[1].split("</section>", 1)[0]
- # The header and the summary give the same verdict: nothing ran, so nothing passed.
- assert _header_verdict(html) == [("skipped", "Skipped")]
+ # The command gate passed; the evidence summary still says nothing ran.
+ assert _header_verdict(html) == [("pass", "Passed required checks")]
assert '<span class="verdict-chip warn">Skipped</span>' in summary
- assert 'data-verdict="pass"' not in html
+ assert "0 of 0 checks passed" not in summary
def test_failing_advisory_tier_is_labelled_advisory() -> None:
@@ -303,7 +303,7 @@ def test_failing_advisory_tier_is_labelled_advisory() -> None:
blocking = _render(results(blocking=True))
# A failing tier whose gate does not block never says "Changes required".
- assert _header_verdict(advisory) == [("pass", "Passed")]
+ assert _header_verdict(advisory) == [("pass", "Passed required checks")]
assert _tier_card_verdicts(advisory) == [("Tier 1", "Passed"), ("Tier 2", "Failed (advisory)")]
assert _header_verdict(blocking) == [("fail", "Changes required")]
assert _tier_card_verdicts(blocking) == [("Tier 1", "Passed"), ("Tier 2", "Changes required")]
diff --git a/tests/reporting/test_reporters.py b/tests/reporting/test_reporters.py
index f144391e..fe091960 100644
--- a/tests/reporting/test_reporters.py
+++ b/tests/reporting/test_reporters.py
@@ -214,8 +214,8 @@ class TestCLIReporter:
output = CLIReporter().render_all([result])
assert "SKIP" in output
- assert "Required validations passed" in output
- assert "live evaluation skipped" in output
+ assert "Passed required checks" in output
+ assert "1 skipped" in output
@pytest.mark.parametrize(
("baseline", "lift", "expected_baseline", "expected_lift"),
@@ -380,7 +380,7 @@ class TestCLIReporter:
assert result.passed is True
assert "same-name similarity match" in output
- assert "All validations passed" in output
+ assert "Passed required checks" in output
def test_render_finding_shows_both_related_paths(self) -> None:
result = _similarity_result()
@@ -1716,7 +1716,7 @@ class TestTier3HarborCanonicalPayload:
cli_output = CLIReporter().render_all([result])
assert "SKIP" in cli_output
assert "Validation failed" not in cli_output
- assert "Required validations passed" in cli_output
+ assert "Passed required checks" in cli_output
def test_legacy_summary_without_execution_status_is_unknown(self, tmp_path: Path) -> None:
from skillevaluator.tier3.harbor.report_data import load_agent_data
@@ -1964,6 +1964,32 @@ class TestGatingSplit:
assert "same-name similarity match" in tier2
assert "Intra-skill check was not run" not in tier2
+ @pytest.mark.parametrize("tier", [2, 3])
+ @pytest.mark.parametrize("comprehensive", [False, True])
+ def test_emit_reports_uses_recorded_tier_for_custom_validator_names(
+ self, tmp_path: Path, tier: int, comprehensive: bool
+ ) -> None:
+ result = _similarity_result(severity=Severity.HIGH) if tier == 2 else _failed_result("Custom live evaluation")
+ result.validator_name = "Custom recorded check"
+ result.metadata["gating"] = {"tier": tier, "blocking": tier == 2}
+ before = JSONReporter(include_timestamp=False).render_all([result])
+ scope = {"requested_tiers": (tier,), "displayed_tiers": (1, 2, 3)} if comprehensive else {}
+
+ assert (
+ emit_reports([result], report_formats=("html",), output_dir=tmp_path, basename="recorded", **scope) is False
+ )
+
+ output = (tmp_path / "recorded.html").read_text(encoding="utf-8")
+ panel = _html_tab(output, f"tier{tier}")
+ assert "No check evidence was recorded for this tier." not in panel
+ if tier == 2:
+ assert "same-name similarity match" in panel
+ else:
+ assert 'class="tab-content active"' in panel
+ if not comprehensive:
+ assert 'id="tab-tier1"' not in output
+ assert JSONReporter(include_timestamp=False).render_all([result]) == before
+
def test_default_html_reporter_classifies_similarity_as_tier2(self) -> None:
output = HTMLReporter(include_timestamp=False).render_all([_similarity_result(severity=Severity.HIGH)])
diff --git a/tests/test_cli.py b/tests/test_cli.py
index 4d9059c3..53b8a5da 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -192,7 +192,7 @@ def test_similarity_cli_handles_pinned_openclaw_compatibility_layout(tmp_path: P
result = CliRunner().invoke(cli, ["similarity-check", str(collection), "--type", "skill"])
assert result.exit_code == 0, result.output
- assert "All validations passed" in result.output
+ assert "Passed required checks" in result.output
assert sorted(embedded_texts) == [
"autoreview: Review changes with multiple agents",
"second-skill: Exercise an unrelated workflow",
diff --git a/tests/test_commands.py b/tests/test_commands.py
index 7d3c6fb2..bf18805a 100644
--- a/tests/test_commands.py
+++ b/tests/test_commands.py
@@ -234,7 +234,7 @@ def test_validate_fixture_no_llm(tmp_path: Path) -> None:
)
assert result.exit_code == 0, result.output
- assert "All validations passed" in result.output
+ assert "Passed required checks" in result.output
def test_validate_reports_malformed_policy_without_traceback(tmp_path: Path) -> None:
diff --git a/tests/tier3/test_skip_with_skill.py b/tests/tier3/test_skip_with_skill.py
index 3db422a6..d0cfae42 100644
--- a/tests/tier3/test_skip_with_skill.py
+++ b/tests/tier3/test_skip_with_skill.py
@@ -561,7 +561,8 @@ def test_html_report_explains_the_baseline_only_run(tmp_path: Path, skill: Path)
assert "Baseline calibration (no with-skill arm)" in html
assert "No skill verdict: baseline-only run" in html
assert "Verdict at a glance:" not in html
- assert '<span class="hero-score-label">baseline score</span>' in html
+ assert '<span class="hero-score-label">Best agent score (baseline)</span>' in html
+ assert "no with-skill arm, so no skill lift was measured." in html
assert re.search(r"<td><strong>Goal Accuracy</strong></td>\s*<td>skipped</td>\s*<td>0\.67</td>", html)