Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

fix(reporting): explain each dimension, its evidence and baseline in reports (#376)

Christopher Kevin3fbb019479dd
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2be969c2..9fc28387 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -129,6 +129,15 @@ All notable changes to SkillEvaluator are documented in this file.
### Changed
+- HTML and `BENCHMARK.md` reports explain each Tier 3 dimension in plain
+ language with its evidence source and whether the run has a no-skill baseline
+ for it. Dimensions that could not be scored or do not apply stay listed with
+ the reason instead of disappearing or reading `Not available`, and baseline-only
+ cards show the baseline score. A short key in both reports separates the
+ verdict, dimension and overall scores, BENCHMARK Uplift, published Skill Lift,
+ pass@k, the Tier 1
+ Quality Score, and publication eligibility.
+
- Write Tier 3 accuracy deductions as plain-language reasons, identify findings
without trajectory evidence, and include expected/observed text only when
both values are recorded.
diff --git a/docs/reports.mdx b/docs/reports.mdx
index 05d7c8b9..1447887b 100644
--- a/docs/reports.mdx
+++ b/docs/reports.mdx
@@ -275,6 +275,22 @@ difference of the two displayed scores, so a cell such as
`58.4% → 94.8% (+36.4 points)` always adds up. A score outside 0–1 is shown as
`Not available`.
+A missing score is never shown as zero. When a standard signal was active for a
+successful agent but produced no with-skill score, the dimension reads `Not
+scored with the skill — baseline 50.0%; uplift unavailable` (or `Not scored — a
+signal this dimension needs produced no score` without a baseline). A dimension
+whose signals had nothing to judge in any eval case reads `Not applicable`; it
+has no score, so the agent's verdict stays NEUTRAL and cannot PASS. A
+baseline-only run shows `Baseline 67.0% — baseline-only run; no with-skill score
+or uplift`.
+
+The collapsed **Scoring Methodology** section lists, for every dimension, the
+question it answers in plain words, its weighted signals, where their evidence
+comes from (deterministic trace or trajectory checks, or an LLM judge), and
+whether this run has a no-skill baseline score for it. It ends with a short key
+that separates the overall verdict, dimension scores, the Overall row, uplift,
+pass@k, and the Tier 1 Quality Score.
+
The card carries one overall verdict — `PASS`, `NEUTRAL`, `FAIL`, or
`INCOMPLETE`. It fails closed: if a required scanner produced no trustworthy
evidence, the verdict is `INCOMPLETE` and the card explicitly says not to use
@@ -607,8 +623,10 @@ arm overall scores.
### Skill Lift
-Skill Lift compares Overall on the metrics and cases both arms scored, so it can differ from subtracting the displayed Overall scores. It uses the same score units as both arms and ranges from
--1.0 to +1.0. Because live agent runs are noisy, especially at low attempt
+Skill Lift compares Overall on the metrics and cases both arms scored, so it
+can differ from subtracting the displayed Overall scores. It uses the same
+score units as both arms and ranges from -1.0 to +1.0. Because live agent runs
+are noisy, especially at low attempt
counts, small deltas are kept neutral. The verdict bands:
| Lift | Verdict | Read it as |
@@ -629,6 +647,16 @@ arms; if you ran with `--skip-baseline`, there is no lift to report. In
`custom_only` grading, your own grader defines the score and lift — see
[Custom Graders & Tasks](/skills/skillevaluator/custom-graders).
+The HTML **Dimension Scores and Skill Lift** table shows each dimension's
+plain-language meaning and evidence source beside its score. For a successful
+best agent, a standard dimension whose signals were active but produced no score
+is listed as N/A with a **not scored with the skill** note and **No verdict**.
+A dimension whose signals had nothing to judge reads **Not applicable**; it has
+no score, so the agent's verdict stays NEUTRAL and cannot PASS. A
+collapsed **What each number means** panel under the Tier 3 overview cards
+separates the verdict, Overall Score, Skill Lift, pass@k, the Tier 1 Quality
+Score, and publication eligibility.
+
When the HTML report has no lift for an agent or dimension, its label says
why. **no baseline** means that row has no baseline score, for example after
`--skip-baseline`. **comparison unavailable** means a baseline score was
diff --git a/src/skillevaluator/constants.py b/src/skillevaluator/constants.py
index b72c6d37..d7ecdf68 100644
--- a/src/skillevaluator/constants.py
+++ b/src/skillevaluator/constants.py
@@ -629,6 +629,16 @@ DIMENSION_HINTS: dict[str, str] = {
"efficiency": "Uses the skill and tools productively without wasted work.",
}
+# Where each Tier 3 signal's evidence comes from, in plain words.
+SIGNAL_EVIDENCE_SOURCES: dict[str, str] = {
+ "security": "deterministic checks of the agent's commands and outputs",
+ "accuracy": "an LLM judge comparing the final answer with the reference answer",
+ "skill_execution": "deterministic checks of the agent's trajectory",
+ "goal_accuracy": "an LLM judge of whether the user's goal was met",
+ "behavior_check": "an LLM judge of the expected workflow",
+ "skill_efficiency": "deterministic checks of the agent's trajectory",
+}
+
# Token-efficiency evaluator: derived locally from ``trial.tokens`` (the
# per-trial trajectory token counters). Per-trial sub-score uses a
# soft asymptote ``1 / (1 + total / TOKEN_EFFICIENCY_HALF_LIFE)`` over the
diff --git a/src/skillevaluator/reporting/benchmark.py b/src/skillevaluator/reporting/benchmark.py
index f35b07da..0bb76eeb 100644
--- a/src/skillevaluator/reporting/benchmark.py
+++ b/src/skillevaluator/reporting/benchmark.py
@@ -19,6 +19,7 @@ from skillevaluator.constants import (
DIMENSION_MAPPING,
DIMENSION_VERDICT_NEUTRAL_THRESHOLD,
DIMENSION_VERDICT_PASS_THRESHOLD,
+ SIGNAL_EVIDENCE_SOURCES,
TIER3_LIFT_FAIL_THRESHOLD,
TIER3_LIFT_PASS_THRESHOLD,
)
@@ -485,6 +486,7 @@ class BenchmarkReporter(ReporterBase):
lines.append("| " + " | ".join(_md_cell(header, private_labels) for header in headers) + " |")
lines.append("|---|" + "|".join(["---:"] * len(agents)) + "|")
+ baseline_only = (ae or {}).get("mode") == "baseline_only"
overall_row = ["Overall"]
shared_basis_agents: dict[tuple[str, ...], list[str]] = {}
for name, agent in agents.items():
@@ -492,7 +494,8 @@ class BenchmarkReporter(ReporterBase):
baseline = agent.get("baseline")
lift_overall = _mapping(agent.get("lift_overall"))
if (
- _number(lift_overall.get("with_skill")) is not None
+ not baseline_only
+ and _number(lift_overall.get("with_skill")) is not None
and _number(lift_overall.get("baseline")) is not None
):
# Show the scores the published lift subtracts so the row matches it.
@@ -518,6 +521,7 @@ class BenchmarkReporter(ReporterBase):
baseline,
with_skill,
skill_never_activated=_mapping(agent.get("skill_activation")).get("status") == "never_activated",
+ baseline_only=baseline_only,
)
)
lines.append("| " + " | ".join(_md_cell(value, private_labels) for value in overall_row) + " |")
@@ -530,11 +534,13 @@ class BenchmarkReporter(ReporterBase):
if dimension and dimension.get("partial"):
has_partial = True
row.append(
- _score_transition(
- dimension,
+ _dimension_cell(
+ agent,
+ dim_id,
skill_never_activated=(
_mapping(agent.get("skill_activation")).get("status") == "never_activated"
),
+ baseline_only=baseline_only,
)
)
lines.append("| " + " | ".join(_md_cell(value, private_labels) for value in row) + " |")
@@ -554,6 +560,12 @@ class BenchmarkReporter(ReporterBase):
"45.0 percentage points above its 47.0% no-skill baseline."
),
"",
+ (
+ "**Not scored** means a signal the dimension needs produced no score, and **Not applicable** "
+ "means no eval case gave its signals anything to judge; neither is a score of zero. "
+ "Scoring Methodology below explains each dimension and where its evidence comes from."
+ ),
+ "",
]
)
if shared_basis_agents:
@@ -761,14 +773,29 @@ class BenchmarkReporter(ReporterBase):
"<details>",
"<summary>Show dimension definitions, source signals, and thresholds</summary>",
"",
- "| Dimension | Question | Scored signals |",
- "|---|---|---|",
+ "| Dimension | What it measures | Scored signals | Evidence source | Baseline in this run |",
+ "|---|---|---|---|---|",
]
)
+ agents = _agents(ae)
+ baseline_only = (ae or {}).get("mode") == "baseline_only"
for dim_id, config in DIMENSION_MAPPING.items():
signals = _weighted_signals(config)
- question = config.get("question") or DIMENSION_HINTS.get(dim_id, "")
- lines.append(f"| {dim_id.title()} | {_trusted_md_cell(question)} | {_trusted_md_cell(signals)} |")
+ meaning = " ".join(
+ part for part in (config.get("question"), DIMENSION_HINTS.get(dim_id)) if isinstance(part, str) and part
+ )
+ evidence = "; ".join(
+ dict.fromkeys(
+ SIGNAL_EVIDENCE_SOURCES[signal]
+ for signal in config.get("evaluators", [])
+ if signal in SIGNAL_EVIDENCE_SOURCES
+ )
+ )
+ baseline = _baseline_coverage(agents, dim_id, baseline_only=baseline_only)
+ lines.append(
+ f"| {dim_id.title()} | {_trusted_md_cell(meaning)} | {_trusted_md_cell(signals)} | "
+ f"{_trusted_md_cell(evidence[:1].upper() + evidence[1:])} | {_trusted_md_cell(baseline)} |"
+ )
lines.extend(
[
@@ -803,6 +830,36 @@ class BenchmarkReporter(ReporterBase):
"- Token efficiency is a separate report-only signal. It does not change a dimension "
"score or the overall verdict."
),
+ (
+ "- The baseline is the same eval cases run by the same agent without the skill. Every "
+ "dimension is compared with it when the baseline arm ran and scored that dimension."
+ ),
+ "",
+ "Which number answers which question:",
+ "",
+ (
+ "- **Overall verdict** (top of this card): whether the evidence supports publishing the skill. "
+ "It needs every required tier, not only Tier 3 scores."
+ ),
+ (
+ "- **Dimension score**: how well the agent did on one question above when it had the skill, "
+ "from 0% to 100%."
+ ),
+ "- **Overall** row: the equal-weight mean of the five dimension scores with the skill.",
+ (
+ "- **Uplift**: the displayed skill score minus the displayed no-skill baseline score. "
+ "Published Skill Lift compares Overall only on the metrics and cases both arms scored, "
+ "so it can differ. Uplift does not set the verdict."
+ ),
+ (
+ "- **pass@k** (HTML report and `pass_at_k_lift.json`): the share of eval cases with at least "
+ "one passing attempt out of k. It measures how repeatable success is, separately from the "
+ "dimension scores."
+ ),
+ (
+ "- **Quality Score** (Tier 1, out of 100): a static review of the skill's files. It is not an "
+ "input to any Tier 3 score or uplift."
+ ),
"",
]
)
@@ -1245,17 +1302,96 @@ def _agent_dimension(agent: dict[str, Any], dim_id: str) -> dict[str, Any] | Non
return None
-def _score_transition(dimension: dict[str, Any] | None, *, skill_never_activated: bool = False) -> str:
+def _score_transition(
+ dimension: dict[str, Any] | None,
+ *,
+ skill_never_activated: bool = False,
+ baseline_only: bool = False,
+) -> str:
if not dimension:
return "Not available"
return _score_transition_values(
dimension.get("baseline"),
dimension.get("with_skill", dimension.get("score")),
skill_never_activated=skill_never_activated,
+ baseline_only=baseline_only,
)
-def _score_transition_values(baseline: object, score: object, *, skill_never_activated: bool = False) -> str:
+def _single_score_label(score: float) -> str:
+ decimal_places = _score_display_decimal_places(score)
+ return _format_percentage(_percentage_point_units(score, decimal_places), decimal_places)
+
+
+def _agent_has_dimension_signals(agent: dict[str, Any], dim_id: str, *, baseline_only: bool = False) -> bool:
+ metrics = agent.get("metrics_without_skill" if baseline_only else "metrics_with_skill")
+ if not isinstance(metrics, list):
+ return False
+ config = DIMENSION_MAPPING.get(dim_id, {})
+ signals = {*config.get("evaluators", []), *config.get("fallback_evaluators", [])}
+ return any(isinstance(metric, str) and metric in signals for metric in metrics)
+
+
+def _baseline_coverage(agents: dict[str, dict[str, Any]], dim_id: str, *, baseline_only: bool) -> str:
+ """Say whether this run has a no-skill baseline score for one dimension."""
+ if not agents:
+ return "Compared when the baseline arm runs"
+ with_baseline = 0
+ eligible = 0
+ for agent in agents.values():
+ if agent.get("execution_status") == "failed":
+ continue
+ baseline = _number((_agent_dimension(agent, dim_id) or {}).get("baseline"))
+ has_baseline = baseline is not None and 0 <= baseline <= 1
+ has_signals = agent.get("execution_status") == "succeeded" and _agent_has_dimension_signals(
+ agent, dim_id, baseline_only=baseline_only
+ )
+ if not has_baseline and not has_signals:
+ continue
+ eligible += 1
+ if has_baseline:
+ with_baseline += 1
+ if baseline_only:
+ if not eligible:
+ return "Not available"
+ return "Baseline only; no with-skill arm ran" if with_baseline else "Not scored in this run"
+ if not eligible:
+ return "Not available"
+ if with_baseline == eligible:
+ return "Yes, a no-skill baseline was scored"
+ if with_baseline == 0:
+ return "No baseline score; uplift unavailable"
+ return f"Baseline for {with_baseline} of {eligible} agents"
+
+
+def _dimension_cell(
+ agent: dict[str, Any],
+ dim_id: str,
+ *,
+ skill_never_activated: bool = False,
+ baseline_only: bool = False,
+) -> str:
+ """Render one Results at a Glance dimension cell, explaining a missing score."""
+ dimension = _agent_dimension(agent, dim_id)
+ if dimension is not None:
+ return _score_transition(dimension, skill_never_activated=skill_never_activated, baseline_only=baseline_only)
+ not_applicable = agent.get("not_applicable_dimensions")
+ if isinstance(not_applicable, list) and any(
+ isinstance(entry, dict) and entry.get("id") == dim_id for entry in not_applicable
+ ):
+ return "Not applicable — no eval case gave its signals anything to judge"
+ if agent.get("execution_status") == "succeeded" and _agent_has_dimension_signals(agent, dim_id):
+ return "Not scored — a signal this dimension needs produced no score"
+ return "Not available"
+
+
+def _score_transition_values(
+ baseline: object,
+ score: object,
+ *,
+ skill_never_activated: bool = False,
+ baseline_only: bool = False,
+) -> str:
"""Render ``baseline → score (uplift)`` so the visible arithmetic is exact.
Scores are fractions in [0, 1]. A missing, non-finite, or out-of-range score,
@@ -1263,9 +1399,14 @@ def _score_transition_values(baseline: object, score: object, *, skill_never_act
rather than clamped or published.
"""
parsed_score = _number(score)
+ parsed_baseline = _number(baseline)
+ if score is None and parsed_baseline is not None and 0 <= parsed_baseline <= 1:
+ baseline_label = _single_score_label(parsed_baseline)
+ if baseline_only:
+ return f"Baseline {baseline_label} — baseline-only run; no with-skill score or uplift"
+ return f"Not scored with the skill — baseline {baseline_label}; uplift unavailable"
if parsed_score is None or not 0 <= parsed_score <= 1:
return "Not available"
- parsed_baseline = _number(baseline)
if skill_never_activated:
# No uplift is shown, so the baseline neither sets the precision nor withholds the score.
parsed_baseline = None
diff --git a/src/skillevaluator/reporting/html.py b/src/skillevaluator/reporting/html.py
index 9d5ee252..e28c324e 100644
--- a/src/skillevaluator/reporting/html.py
+++ b/src/skillevaluator/reporting/html.py
@@ -40,7 +40,16 @@ from urllib.parse import urlparse
from jinja2 import BaseLoader, Environment
from skillevaluator import __version__
-from skillevaluator.constants import QUALITY_GRADE_THRESHOLDS, TIER3_LIFT_FAIL_THRESHOLD, TIER3_LIFT_PASS_THRESHOLD
+from skillevaluator.constants import (
+ DIMENSION_HINTS,
+ DIMENSION_MAPPING,
+ DIMENSION_VERDICT_NEUTRAL_THRESHOLD,
+ DIMENSION_VERDICT_PASS_THRESHOLD,
+ QUALITY_GRADE_THRESHOLDS,
+ SIGNAL_EVIDENCE_SOURCES,
+ TIER3_LIFT_FAIL_THRESHOLD,
+ TIER3_LIFT_PASS_THRESHOLD,
+)
from skillevaluator.publication_evidence import result_publication_evidence_dict
from skillevaluator.reporting.base import (
AGENT_EVAL_EXACT_TEXT_KEYS,
@@ -896,6 +905,44 @@ def _sanitize_tier3_agent(value: dict[str, Any]) -> dict[str, Any]:
return agent
+def _tier3_html_dimensions(payload: dict[str, Any] | None) -> list[dict[str, Any]]:
+ """Add display-only missing-signal rows for a successful best agent."""
+ if not isinstance(payload, dict):
+ return []
+ dimensions = [dict(item) for item in _dict_list(payload.get("dimensions"))]
+ dimension_ids = {item.get("id") for item in dimensions}
+ dimension_ids.update(item.get("id") for item in _dict_list(payload.get("not_applicable_dimensions")))
+ agents = payload.get("agents")
+ summary = payload.get("summary")
+ summary = summary if isinstance(summary, dict) else {}
+ best_name = payload.get("best_agent") or summary.get("best_agent")
+ best = agents.get(best_name) if isinstance(agents, dict) and isinstance(best_name, str) else None
+ if not isinstance(best, dict) or best.get("execution_status") != "succeeded":
+ return dimensions
+ active_metrics = set(_tier3_identifier_list(best.get("metrics_with_skill")))
+ for dimension_id, config in DIMENSION_MAPPING.items():
+ if dimension_id in dimension_ids:
+ continue
+ signals = list(dict.fromkeys([*config.get("evaluators", []), *config.get("fallback_evaluators", [])]))
+ active_signals = [signal for signal in signals if signal in active_metrics]
+ if not active_signals:
+ continue
+ dimensions.append(
+ {
+ "id": dimension_id,
+ "score": None,
+ "with_skill": None,
+ "baseline": None,
+ "lift": None,
+ "verdict": None,
+ "evaluators": active_signals,
+ "reasoning_bullets": ["A signal this dimension needs produced no score."],
+ }
+ )
+ dimension_ids.add(dimension_id)
+ return dimensions
+
+
def _scrub_untrusted_tier3_agents(value: object) -> dict[str, dict[str, Any]]:
"""Retain agent identity/coverage while removing untrusted score evidence."""
agents: dict[str, dict[str, Any]] = {}
@@ -1135,7 +1182,8 @@ def _sanitize_tier3_display_payload(value: dict[str, Any]) -> dict[str, Any]:
if not isinstance(payload.get(key), dict):
payload[key] = {}
payload["metric_labels"] = _tier3_text_mapping(payload.get("metric_labels"))
- payload["dimension_hints"] = _tier3_text_mapping(payload.get("dimension_hints"))
+ payload["dimension_hints"] = _tier3_text_mapping(payload.get("dimension_hints")) or dict(DIMENSION_HINTS)
+ payload["signal_evidence"] = dict(SIGNAL_EVIDENCE_SOURCES)
evaluator_paths = payload["provenance"].get("evaluator_paths")
payload["provenance"]["evaluator_paths"] = evaluator_paths if isinstance(evaluator_paths, dict) else {}
if "source_validation" in payload["provenance"]:
@@ -1175,14 +1223,21 @@ def _sanitize_tier3_display_payload(value: dict[str, Any]) -> dict[str, Any]:
if (card := _sanitize_tier3_evaluator_card(item)) is not None
]
payload["cases"] = _dict_list(payload.get("cases"))
+ # Unscored dimensions stay listed so the report can explain them.
payload["dimensions"] = [
dimension
for item in _dict_list(payload.get("dimensions"))
- if (
- (dimension := _sanitize_tier3_dimension(item)).get("score") is not None
- or dimension.get("with_skill") is not None
+ if any(
+ (dimension := _sanitize_tier3_dimension(item)).get(key) is not None
+ for key in ("score", "with_skill", "baseline")
)
]
+ scored_dimension_ids = {dimension.get("id") for dimension in payload["dimensions"]}
+ payload["not_applicable_dimensions"] = [
+ {"id": dimension_id, "reason": reason if isinstance(reason := item.get("reason"), str) else ""}
+ for item in _dict_list(payload.get("not_applicable_dimensions"))
+ if (dimension_id := _tier3_display_identifier(item.get("id"))) and dimension_id not in scored_dimension_ids
+ ]
for key in ("conclusions", "recommendations", "suggestions_v2"):
payload[key] = _sanitize_tier3_insight_items(payload.get(key))
payload["dataset"] = [_sanitize_tier3_dataset_case(item) for item in _dict_list(payload.get("dataset"))]
@@ -3470,6 +3525,7 @@ class HTMLReporter(ReporterBase):
content_label_plural=cl + "s",
quality_scores=quality_scores_by_skill,
tier3=tier3_preview,
+ tier3_display_dimensions=_tier3_html_dimensions(tier3_data),
tier3_dataset_provenance_line=tier3_dataset_provenance_line,
tier3_dataset_provenance_warning=tier3_dataset_provenance_warning,
chart_assets=_load_chart_assets,
@@ -3501,6 +3557,8 @@ class HTMLReporter(ReporterBase):
evaluated_source=recorded_evaluated_source(result.metadata for result in results),
tier3_lift_pass_threshold=TIER3_LIFT_PASS_THRESHOLD,
tier3_lift_fail_threshold=TIER3_LIFT_FAIL_THRESHOLD,
+ tier3_dimension_pass_threshold=DIMENSION_VERDICT_PASS_THRESHOLD,
+ tier3_dimension_neutral_threshold=DIMENSION_VERDICT_NEUTRAL_THRESHOLD,
quality_min_score=quality_min_score,
quality_grade_legend=QUALITY_GRADE_LEGEND,
verdict_words=VERDICT_WORDS,
diff --git a/src/skillevaluator/reporting/templates/report.html.j2 b/src/skillevaluator/reporting/templates/report.html.j2
index 751e4b71..4af34ee5 100644
--- a/src/skillevaluator/reporting/templates/report.html.j2
+++ b/src/skillevaluator/reporting/templates/report.html.j2
@@ -2577,6 +2577,19 @@
<p class="dashboard-card-subtitle">Compared with baseline</p>
</div>
</div>
+ <details class="t3-details" id="tier3-number-key">
+ <summary>What each number means</summary>
+ <div class="t3-details-body">
+ <ul class="t3-reasoning-list">
+ <li><strong>Verdict</strong>: the live-evaluation result from the five dimensions. It passes when one agent scores at least {{ "%.2f" | format(tier3_dimension_pass_threshold) }} on every dimension with the skill.</li>
+ <li><strong>Overall Score</strong>: the equal-weight mean of the five dimension scores with the skill, from 0.00 to 1.00.</li>
+ <li><strong>Skill Lift</strong>: compares Overall on the metrics and cases both arms scored, so it can differ from subtracting the displayed Overall scores. It shows whether the skill helped and does not change the verdict; N/A means no comparable baseline, not zero.</li>
+ <li><strong>pass@k</strong> (Trials): the share of eval cases with at least one passing attempt out of k. It measures how repeatable success is.</li>
+ <li><strong>Quality Score</strong> (Tier 1, out of 100): a static review of the skill's files. It is not an input to any live-evaluation score.</li>
+ <li><strong>Publication eligibility</strong>: decided separately in BENCHMARK.md from every required tier's evidence, so a passing verdict here is not by itself a publication decision.</li>
+ </ul>
+ </div>
+ </details>
{% set t3_preview_conclusion = (tier3.conclusions[0] if tier3.conclusions and (tier3.conclusions | length) > 0 else None) %}
{% set t3_preview_recommendations = tier3.recommendations or [] %}
@@ -2641,12 +2654,20 @@
{% endif %}
</div>
- {% if tier3.dimensions %}
+ {% set t3_na_dimensions = tier3.not_applicable_dimensions or [] %}
+ {% if tier3_display_dimensions or t3_na_dimensions %}
+ {% set t3_baseline_only = tier3.get('mode') == 'baseline_only' %}
+ {% set t3_signal_evidence = tier3.signal_evidence or {} %}
<details class="t3-details" id="tier3-dimension-lift" open>
<summary>Dimension Scores and Skill Lift (Best Performing Agent: {{ t3_best_name or 'N/A' }})</summary>
<div class="t3-details-body">
- <p style="color:var(--text-secondary);font-size:0.85rem;margin-top:0;margin-bottom:0.75rem;">
- Consolidated evaluation dimensions. The Signals column shows which evaluator metrics feed each human-readable dimension.
+ <p class="t3-section-hint">
+ Each dimension answers one question about the skill, scored from 0.00 to 1.00.
+ Score is the agent's result with the skill; Baseline is the same eval cases run without it;
+ Skill Lift compares Overall on the metrics and cases both arms scored, so it can differ from subtracting the displayed Overall scores. A dimension passes at {{ "%.2f" | format(tier3_dimension_pass_threshold) }} or above
+ and fails below {{ "%.2f" | format(tier3_dimension_neutral_threshold) }}.
+ Signals are the evaluator metrics behind each dimension, and Evidence says where they come from.
+ N/A means no measurement, never a score of zero; the note under it says why.
</p>
<div class="table-scroll">
<table class="summary-table" id="tier3-dimension-table">
@@ -2658,29 +2679,41 @@
<tbody>
{% set t3_best_agent = t3_agents.get(t3_best_name, {}) %}
{% set baseline_na_metrics = t3_best_agent.get('not_applicable_metrics', {}).get('without_skill', []) %}
- {% for dim in tier3.dimensions %}
+ {% for dim in tier3_display_dimensions %}
{% set dim_score = dim.score if dim.score is not none else dim.with_skill %}
- {% set dim_color = 'var(--success-color)' if dim_score >= 0.7 else ('var(--warning-color)' if dim_score >= 0.4 else 'var(--danger-color)') %}
+ {% set dim_color = 'var(--text-secondary)' if dim_score is none else ('var(--success-color)' if dim_score >= 0.7 else ('var(--warning-color)' if dim_score >= 0.4 else 'var(--danger-color)')) %}
+ {% set dim_evidence = [] %}
+ {% for metric_id in dim.evaluators or [] %}{% if t3_signal_evidence.get(metric_id) and t3_signal_evidence.get(metric_id) not in dim_evidence %}{% set _ = dim_evidence.append(t3_signal_evidence.get(metric_id)) %}{% endif %}{% endfor %}
<tr>
<td>
<strong>{{ "Skill Efficiency" if dim.id == "efficiency" else (dim.id | title) }}</strong>
{% if t3_dim_hints.get(dim.id) %}<div class="t3-dim-hint">{{ t3_dim_hints.get(dim.id) }}</div>{% endif %}
+ {% if dim_evidence %}<div class="t3-dim-hint">Evidence: {{ dim_evidence | join('; ') }}.</div>{% endif %}
</td>
<td>
+ {% if dim_score is not none %}
<div class="metric-bar-cell">
<span class="metric-value" style="color:{{ dim_color }}">{{ "%.2f" | format(dim_score) }}</span>
<div class="mini-bar"><span style="width:{{ (dim_score * 100) | round(0) }}%; background:{{ dim_color }};"></span></div>
</div>
+ {% else %}
+ <span class="metric-value" style="color:{{ dim_color }}">N/A</span>
+ <div class="t3-dim-hint">{{ "with-skill arm not run" if t3_baseline_only else "not scored with the skill" }}</div>
+ {% endif %}
+ </td>
+ <td{% if dim.id == 'discoverability' and 'skill_execution' in baseline_na_metrics %} title="No-skill baseline: This arm has no skill under test installed, so there is no skill to discover or run"{% elif dim.id == 'efficiency' and 'skill_efficiency' in baseline_na_metrics %} title="No-skill baseline: This arm has no skill under test installed, so there is no skill to route to"{% endif %}>
+ {% if dim.baseline is not none %}{{ "%.2f" | format(dim.baseline) }}{% else %}N/A
+ <div class="t3-dim-hint">{{ "baseline not scored" if t3_best and t3_best.baseline is not none else "baseline not run" }}</div>{% endif %}
</td>
- <td{% if dim.id == 'discoverability' and 'skill_execution' in baseline_na_metrics %} title="No-skill baseline: This arm has no skill under test installed, so there is no skill to discover or run"{% elif dim.id == 'efficiency' and 'skill_efficiency' in baseline_na_metrics %} title="No-skill baseline: This arm has no skill under test installed, so there is no skill to route to"{% endif %}>{{ "%.2f" | format(dim.baseline) if dim.baseline is not none else "N/A" }}</td>
<td style="color:{{ 'var(--success-color)' if dim.lift is not none and dim.lift >= tier3_lift_pass_threshold else ('var(--danger-color)' if dim.lift is not none and dim.lift <= tier3_lift_fail_threshold else 'var(--text-secondary)') }};font-weight:600;">
- {{ "%+.2f" | format(dim.lift) if dim.lift is not none else "N/A" }}
+ {% if dim.lift is not none %}{{ "%+.2f" | format(dim.lift) }}{% else %}N/A
+ <div class="t3-dim-hint">{% if t3_baseline_only %}baseline-only run{% elif dim_score is none %}with-skill unavailable{% elif dim.baseline is not none %}comparison unavailable{% else %}no baseline{% endif %}</div>{% endif %}
</td>
{% set dim_regressed = dim.verdict == 'PASS' and dim.lift is not none and dim.lift <= tier3_lift_fail_threshold %}
- <td><span class="t3-pill {{ 'warning' if dim_regressed else ('ok' if dim.verdict == 'PASS' else ('critical' if dim.verdict == 'FAIL' else 'warning')) }}">{{ dim.verdict }}{{ ' · regressed' if dim_regressed else '' }}</span></td>
+ <td>{% if dim.verdict %}<span class="t3-pill {{ 'warning' if dim_regressed else ('ok' if dim.verdict == 'PASS' else ('critical' if dim.verdict == 'FAIL' else 'warning')) }}">{{ dim.verdict }}{{ ' · regressed' if dim_regressed else '' }}</span>{% else %}<span class="t3-pill warning">No verdict</span>{% endif %}</td>
<td>
{% for metric_id in dim.evaluators or [] %}
- <span class="metadata-tag">{{ t3_metric_labels.get(metric_id, metric_id | replace("_", " ") | title) }}</span>
+ <span class="metadata-tag" title="{{ t3_signal_evidence.get(metric_id, '') }}">{{ t3_metric_labels.get(metric_id, metric_id | replace("_", " ") | title) }}</span>
{% endfor %}
</td>
<td class="t3-reasoning">
@@ -2694,6 +2727,18 @@
</td>
</tr>
{% endfor %}
+ {% for dim in t3_na_dimensions %}
+ <tr>
+ <td>
+ <strong>{{ "Skill Efficiency" if dim.id == "efficiency" else (dim.id | title) }}</strong>
+ {% if t3_dim_hints.get(dim.id) %}<div class="t3-dim-hint">{{ t3_dim_hints.get(dim.id) }}</div>{% endif %}
+ </td>
+ <td>N/A</td><td>N/A</td><td>N/A</td>
+ <td><span class="t3-pill warning">Not applicable</span></td>
+ <td></td>
+ <td class="t3-reasoning">{{ dim.reason or "Not applicable: its signals had nothing to judge in any eval case." }} It has no score, so the agent's verdict stays NEUTRAL and cannot PASS.</td>
+ </tr>
+ {% endfor %}
</tbody>
</table>
</div>
@@ -2810,8 +2855,8 @@
</div>
{% endfor %}
</div>
- {% set dimension_reference = ('tier3-multi-agent-dimension-table' if t3_agents | length > 1 else 'tier3-dimension-table') if tier3.dimensions else 'tier3-json-downloads' %}
- {% set dimension_reference_label = 'Read the dimension scores table' if tier3.dimensions else 'Read JSON Downloads (dimension scores unavailable)' %}
+ {% set dimension_reference = ('tier3-multi-agent-dimension-table' if t3_agents | length > 1 else 'tier3-dimension-table') if tier3_display_dimensions else 'tier3-json-downloads' %}
+ {% set dimension_reference_label = 'Read the dimension scores table' if tier3_display_dimensions else 'Read JSON Downloads (dimension scores unavailable)' %}
<div class="chart-card">{{ tier3_chart('tier3-radar', 'Dimension Comparison (Radar)', 'Dimension scores for each agent, on a scale from 0 to 1; missing measurements are gaps.', dimension_reference, dimension_reference_label, 260) }}</div>
<div class="chart-card">{{ tier3_chart('tier3-metric-bar', 'Dimension Scores by Agent', 'Dimension scores grouped by dimension and agent, on a scale from 0 to 1; missing measurements are gaps.', dimension_reference, dimension_reference_label, 260) }}</div>
</div>
diff --git a/tests/golden/benchmark_fail/BENCHMARK.md b/tests/golden/benchmark_fail/BENCHMARK.md
index d0003870..26d99753 100644
--- a/tests/golden/benchmark_fail/BENCHMARK.md
+++ b/tests/golden/benchmark_fail/BENCHMARK.md
@@ -49,6 +49,8 @@ The three-tier evaluation checks whether the skill:
Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline.
+**Not scored** means a signal the dimension needs produced no score, and **Not applicable** means no eval case gave its signals anything to judge; neither is a score of zero. Scoring Methodology below explains each dimension and where its evidence comes from.
+
## Tier Status
| Tier | Purpose | Status | Evidence |
@@ -76,13 +78,13 @@ Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92
<details>
<summary>Show dimension definitions, source signals, and thresholds</summary>
-| Dimension | Question | Scored signals |
-|---|---|---|
-| Security | Is it safe to use? | `security` (100%) |
-| Correctness | Is the answer correct? | `accuracy` (100%) |
-| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) |
-| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) |
-| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) |
+| Dimension | What it measures | Scored signals | Evidence source | Baseline in this run |
+|---|---|---|---|---|
+| Security | Is it safe to use? Avoids unsafe operations, secret leakage, and unauthorized access. | `security` (100%) | Deterministic checks of the agent's commands and outputs | Yes, a no-skill baseline was scored |
+| Correctness | Is the answer correct? Produces factually accurate output. | `accuracy` (100%) | An LLM judge comparing the final answer with the reference answer | Yes, a no-skill baseline was scored |
+| Discoverability | Was the right skill loaded when needed? Activates the right skill when relevant; ignores unrelated skills. | `skill_execution` (100%) | Deterministic checks of the agent's trajectory | Yes, a no-skill baseline was scored |
+| Effectiveness | Did the skill help complete the task? Reaches the user's goal and follows the expected behavior. | `goal_accuracy` (50%) + `behavior_check` (50%) | An LLM judge of whether the user's goal was met; an LLM judge of the expected workflow | Yes, a no-skill baseline was scored |
+| Efficiency | Did it avoid wasted tool or skill usage? Uses the skill and tools productively without wasted work. | `skill_efficiency` (100%) | Deterministic checks of the agent's trajectory | Yes, a no-skill baseline was scored |
- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
@@ -90,6 +92,16 @@ Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92
- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`).
- Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict.
+- The baseline is the same eval cases run by the same agent without the skill. Every dimension is compared with it when the baseline arm ran and scored that dimension.
+
+Which number answers which question:
+
+- **Overall verdict** (top of this card): whether the evidence supports publishing the skill. It needs every required tier, not only Tier 3 scores.
+- **Dimension score**: how well the agent did on one question above when it had the skill, from 0% to 100%.
+- **Overall** row: the equal-weight mean of the five dimension scores with the skill.
+- **Uplift**: the displayed skill score minus the displayed no-skill baseline score. Published Skill Lift compares Overall only on the metrics and cases both arms scored, so it can differ. Uplift does not set the verdict.
+- **pass@k** (HTML report and `pass_at_k_lift.json`): the share of eval cases with at least one passing attempt out of k. It measures how repeatable success is, separately from the dimension scores.
+- **Quality Score** (Tier 1, out of 100): a static review of the skill's files. It is not an input to any Tier 3 score or uplift.
Signals present in this run:
diff --git a/tests/golden/benchmark_pass/BENCHMARK.md b/tests/golden/benchmark_pass/BENCHMARK.md
index e73863d2..fdbd124a 100644
--- a/tests/golden/benchmark_pass/BENCHMARK.md
+++ b/tests/golden/benchmark_pass/BENCHMARK.md
@@ -51,6 +51,8 @@ The three-tier evaluation checks whether the skill:
Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline.
+**Not scored** means a signal the dimension needs produced no score, and **Not applicable** means no eval case gave its signals anything to judge; neither is a score of zero. Scoring Methodology below explains each dimension and where its evidence comes from.
+
## Tier Status
| Tier | Purpose | Status | Evidence |
@@ -73,13 +75,13 @@ Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92
<details>
<summary>Show dimension definitions, source signals, and thresholds</summary>
-| Dimension | Question | Scored signals |
-|---|---|---|
-| Security | Is it safe to use? | `security` (100%) |
-| Correctness | Is the answer correct? | `accuracy` (100%) |
-| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) |
-| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) |
-| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) |
+| Dimension | What it measures | Scored signals | Evidence source | Baseline in this run |
+|---|---|---|---|---|
+| Security | Is it safe to use? Avoids unsafe operations, secret leakage, and unauthorized access. | `security` (100%) | Deterministic checks of the agent's commands and outputs | Yes, a no-skill baseline was scored |
+| Correctness | Is the answer correct? Produces factually accurate output. | `accuracy` (100%) | An LLM judge comparing the final answer with the reference answer | Yes, a no-skill baseline was scored |
+| Discoverability | Was the right skill loaded when needed? Activates the right skill when relevant; ignores unrelated skills. | `skill_execution` (100%) | Deterministic checks of the agent's trajectory | Yes, a no-skill baseline was scored |
+| Effectiveness | Did the skill help complete the task? Reaches the user's goal and follows the expected behavior. | `goal_accuracy` (50%) + `behavior_check` (50%) | An LLM judge of whether the user's goal was met; an LLM judge of the expected workflow | Yes, a no-skill baseline was scored |
+| Efficiency | Did it avoid wasted tool or skill usage? Uses the skill and tools productively without wasted work. | `skill_efficiency` (100%) | Deterministic checks of the agent's trajectory | Yes, a no-skill baseline was scored |
- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
@@ -87,6 +89,16 @@ Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92
- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`).
- Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict.
+- The baseline is the same eval cases run by the same agent without the skill. Every dimension is compared with it when the baseline arm ran and scored that dimension.
+
+Which number answers which question:
+
+- **Overall verdict** (top of this card): whether the evidence supports publishing the skill. It needs every required tier, not only Tier 3 scores.
+- **Dimension score**: how well the agent did on one question above when it had the skill, from 0% to 100%.
+- **Overall** row: the equal-weight mean of the five dimension scores with the skill.
+- **Uplift**: the displayed skill score minus the displayed no-skill baseline score. Published Skill Lift compares Overall only on the metrics and cases both arms scored, so it can differ. Uplift does not set the verdict.
+- **pass@k** (HTML report and `pass_at_k_lift.json`): the share of eval cases with at least one passing attempt out of k. It measures how repeatable success is, separately from the dimension scores.
+- **Quality Score** (Tier 1, out of 100): a static review of the skill's files. It is not an input to any Tier 3 score or uplift.
Signals present in this run:
diff --git a/tests/golden/test_benchmark_card.py b/tests/golden/test_benchmark_card.py
index 4f24d804..170e1f2a 100644
--- a/tests/golden/test_benchmark_card.py
+++ b/tests/golden/test_benchmark_card.py
@@ -1137,7 +1137,7 @@ def test_benchmark_zero_and_negative_lift_use_percentage_points() -> None:
assert _score_transition_values(0.50004, 0.500049) == "50.0% → 50.0% (±0.0 points)"
assert _score_transition_values(0.500049, 0.50004) == "50.0% → 50.0% (±0.0 points)"
assert _score_transition_values(None, 0.92) == "92.0% — baseline not run; uplift unavailable"
- assert _score_transition_values(0.5, None) == "Not available"
+ assert _score_transition_values(0.5, None) == "Not scored with the skill — baseline 50.0%; uplift unavailable"
assert _score_transition({"baseline": float("nan"), "with_skill": 0.92}) == "Not available"
assert _score_transition({"baseline": 0.5, "with_skill": float("inf")}) == "Not available"
diff --git a/tests/reporting/test_dimension_explanations.py b/tests/reporting/test_dimension_explanations.py
new file mode 100644
index 00000000..5eceec15
--- /dev/null
+++ b/tests/reporting/test_dimension_explanations.py
@@ -0,0 +1,230 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Reports explain each dimension, its evidence and baseline, and why a score is missing."""
+
+from __future__ import annotations
+
+import re
+from copy import deepcopy
+from pathlib import Path
+
+from scripts.ci import check_public_benchmarks as benchmark_gate
+
+from skillevaluator.constants import DIMENSION_HINTS, SIGNAL_EVIDENCE_SOURCES
+from skillevaluator.evaluation.tier3_report import _validation_result_from_payload, build_agent_eval_payload
+from skillevaluator.reporting import BenchmarkReporter, HTMLReporter
+from skillevaluator.reporting.benchmark import _baseline_coverage, _dimension_cell
+from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS
+
+_SUCCEEDED = {"execution_status": "succeeded", "expected_attempts": 1, "scored_attempts": 1}
+
+
+def _agent(*, baseline: bool = True, with_scores: dict[str, float | None] | None = None) -> dict:
+ scores = {**dict.fromkeys(DEFAULT_METRICS, 0.9), **(with_scores or {})}
+ reward = {**scores, "entry_id": "case-1", "trial_id": "case-1__with"}
+ agent = {
+ "execution_status": "succeeded",
+ "expected_attempts": 2 if baseline else 1,
+ "scored_attempts": 2 if baseline else 1,
+ "num_trials": 1,
+ "num_trials_baseline": int(baseline),
+ "with_skill": scores,
+ "rewards": [reward],
+ "rewards_complete": True,
+ "conditions": {"with_skill": _SUCCEEDED, "without_skill": {"execution_status": "skipped"}},
+ }
+ if baseline:
+ baseline_scores = dict.fromkeys(DEFAULT_METRICS, 0.5)
+ agent.update(
+ without_skill=baseline_scores,
+ rewards_baseline=[{**baseline_scores, "entry_id": "case-1", "trial_id": "case-1__without"}],
+ rewards_baseline_complete=True,
+ )
+ agent["conditions"]["without_skill"] = _SUCCEEDED
+ return agent
+
+
+def _payload(agents: dict[str, dict]) -> dict:
+ payload = build_agent_eval_payload("demo", agents, use_llm_judge=False)
+ assert payload is not None
+ return payload
+
+
+def _render(payload: dict) -> tuple[str, str]:
+ result = _validation_result_from_payload(payload)
+ assert result is not None
+ markdown = BenchmarkReporter(include_timestamp=False).render_all([result])
+ html = HTMLReporter(include_timestamp=False).render_all([result])
+ return markdown, html
+
+
+def _row(markdown: str, label: str) -> str:
+ glance = markdown.split("## Results at a Glance", 1)[1].split("## Tier Status", 1)[0]
+ return next(line for line in glance.splitlines() if line.startswith(f"| {label} |"))
+
+
+def _methodology_row(markdown: str, label: str) -> list[str]:
+ methodology = markdown.split("## Scoring Methodology", 1)[1]
+ line = next(line for line in methodology.splitlines() if line.startswith(f"| {label} |"))
+ return [cell.strip() for cell in line.strip("|").split("|")]
+
+
+def _dimension_table(html: str) -> str:
+ match = re.search(r'<table class="summary-table" id="tier3-dimension-table">.*?</table>', html, re.DOTALL)
+ assert match is not None
+ return re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", match.group(0)))
+
+
+def _assert_gate_accepts(markdown: str, tmp_path: Path) -> None:
+ card = tmp_path / "BENCHMARK.md"
+ card.write_text(markdown, encoding="utf-8")
+ _files, offenders = benchmark_gate.find_offenders([card])
+ assert offenders == []
+
+
+def test_benchmark_methodology_explains_meaning_evidence_and_baseline(tmp_path: Path) -> None:
+ markdown, _html = _render(_payload({"paired": _agent(), "solo": _agent(baseline=False)}))
+
+ assert "| Dimension | What it measures | Scored signals | Evidence source | Baseline in this run |" in markdown
+ security = _methodology_row(markdown, "Security")
+ assert security[1] == f"Is it safe to use? {DIMENSION_HINTS['security']}"
+ assert security[3] == "Deterministic checks of the agent's commands and outputs"
+ assert security[4] == "Baseline for 1 of 2 agents"
+ effectiveness = _methodology_row(markdown, "Effectiveness")
+ assert effectiveness[3] == "An LLM judge of whether the user's goal was met; an LLM judge of the expected workflow"
+ for term in ("**Overall verdict**", "**Uplift**", "**pass@k**", "**Quality Score** (Tier 1"):
+ assert term in markdown
+ assert (
+ "**Uplift**: the displayed skill score minus the displayed no-skill baseline score. Published Skill Lift "
+ "compares Overall only on the metrics and cases both arms scored, so it can differ. Uplift does not set the "
+ "verdict."
+ ) in markdown
+ _assert_gate_accepts(markdown, tmp_path)
+
+
+def test_benchmark_methodology_states_when_no_baseline_ran() -> None:
+ markdown, _html = _render(_payload({"solo": _agent(baseline=False)}))
+
+ assert _methodology_row(markdown, "Correctness")[4] == "No baseline score; uplift unavailable"
+ assert _row(markdown, "Correctness").endswith("| 90.0% — baseline not run; uplift unavailable |")
+
+
+def test_unscored_dimension_is_explained_not_zero(tmp_path: Path) -> None:
+ markdown, html = _render(_payload({"agent": _agent(with_scores={"accuracy": None})}))
+
+ assert _row(markdown, "Correctness") == (
+ "| Correctness | Not scored with the skill — baseline 50.0%; uplift unavailable |"
+ )
+ assert "**Not scored** means a signal the dimension needs produced no score" in markdown
+ _assert_gate_accepts(markdown, tmp_path)
+
+ table = _dimension_table(html)
+ assert "Correctness" in table
+ assert "N/A not scored with the skill 0.50 N/A with-skill unavailable No verdict" in table
+ assert "With-skill score unavailable; no verdict was computed." in table
+
+
+def test_missing_and_not_applicable_dimensions_have_reasons(tmp_path: Path) -> None:
+ payload = _payload({"agent": _agent()})
+ agent = payload["agents"]["agent"]
+ agent["dimensions"] = [d for d in agent["dimensions"] if d["id"] not in {"discoverability", "efficiency"}]
+ reason = "Not applicable: Skill Execution had nothing to judge against in any eval case"
+ agent["not_applicable_dimensions"] = [{"id": "discoverability", "reason": reason}]
+ payload["dimensions"] = deepcopy(agent["dimensions"])
+ payload["not_applicable_dimensions"] = deepcopy(agent["not_applicable_dimensions"])
+
+ markdown, html = _render(payload)
+
+ assert _row(markdown, "Discoverability") == (
+ "| Discoverability | Not applicable — no eval case gave its signals anything to judge |"
+ )
+ assert (
+ _row(markdown, "Efficiency") == "| Efficiency | Not scored — a signal this dimension needs produced no score |"
+ )
+ _assert_gate_accepts(markdown, tmp_path)
+
+ table = _dimension_table(html)
+ assert "Discoverability" in table
+ assert "N/A N/A N/A Not applicable" in table
+ assert "Skill Efficiency" in table
+ assert "N/A not scored with the skill N/A baseline not scored N/A with-skill unavailable No verdict" in table
+ assert "A signal this dimension needs produced no score." in table
+ assert f"{reason} It has no score, so the agent's verdict stays NEUTRAL and cannot PASS." in table
+
+
+def test_html_dimension_table_names_evidence_and_baseline_status() -> None:
+ _markdown, html = _render(_payload({"solo": _agent(baseline=False)}))
+
+ table = _dimension_table(html)
+ assert f"Evidence: {SIGNAL_EVIDENCE_SOURCES['security']}." in table.replace("&#39;", "'")
+ assert "0.90 N/A baseline not run N/A no baseline PASS" in table
+ assert "N/A means no measurement, never a score of zero; the note under it says why." in html
+
+
+def test_html_number_key_separates_verdict_scores_lift_and_quality() -> None:
+ _markdown, html = _render(_payload({"agent": _agent()}))
+
+ key = html.split('id="tier3-number-key"', 1)[1].split("</details>", 1)[0]
+ for term in ("Verdict", "Overall Score", "Skill Lift", "pass@k", "Quality Score", "Publication eligibility"):
+ assert f"<strong>{term}</strong>" in key
+ assert (
+ "Skill Lift</strong>: compares Overall on the metrics and cases both arms scored, so it can differ "
+ "from subtracting the displayed Overall scores."
+ ) in key
+ assert "not an input to any live-evaluation score" in key
+
+
+def test_baseline_only_cells_show_the_baseline_without_a_skill_score(tmp_path: Path) -> None:
+ payload = _payload({"agent": _agent()})
+ payload["mode"] = "baseline_only"
+ agent = payload["agents"]["agent"]
+ agent["with_skill"] = None
+ for dimension in agent["dimensions"]:
+ dimension.update(with_skill=None, score=None, lift=None)
+
+ markdown, _html = _render(payload)
+
+ assert (
+ _row(markdown, "Overall") == "| Overall | Baseline 50.0% — baseline-only run; no with-skill score or uplift |"
+ )
+ assert _row(markdown, "Security") == (
+ "| Security | Baseline 50.0% — baseline-only run; no with-skill score or uplift |"
+ )
+ assert _methodology_row(markdown, "Security")[4] == "Baseline only; no with-skill arm ran"
+ _assert_gate_accepts(markdown, tmp_path)
+
+
+def test_html_does_not_add_unscored_rows_for_custom_only_metrics() -> None:
+ payload = _payload({"agent": _agent()})
+ payload["agents"]["agent"]["metrics_with_skill"] = ["custom_metric"]
+ payload["agents"]["agent"]["dimensions"] = []
+ payload["dimensions"] = []
+ payload["not_applicable_dimensions"] = []
+
+ _markdown, html = _render(payload)
+
+ assert 'id="tier3-dimension-table"' not in html
+
+
+def test_benchmark_missing_dimension_requires_a_successful_active_signal() -> None:
+ active = {
+ "execution_status": "succeeded",
+ "dimensions": [],
+ "metrics_with_skill": ["accuracy"],
+ }
+ assert _dimension_cell(active, "correctness") == "Not scored — a signal this dimension needs produced no score"
+ assert _baseline_coverage({"agent": active}, "correctness", baseline_only=False) == (
+ "No baseline score; uplift unavailable"
+ )
+
+
+def test_benchmark_does_not_blame_missing_signals_for_custom_failed_or_legacy_agents() -> None:
+ agents = [
+ {"execution_status": "succeeded", "dimensions": [], "metrics_with_skill": ["custom_metric"]},
+ {"execution_status": "failed", "dimensions": [], "metrics_with_skill": ["accuracy"]},
+ {"execution_status": "succeeded", "dimensions": []},
+ ]
+ for agent in agents:
+ assert _dimension_cell(agent, "correctness") == "Not available"
+ assert _baseline_coverage({"agent": agent}, "correctness", baseline_only=False) == "Not available"
diff --git a/tests/reporting/test_unified_tier3_report.py b/tests/reporting/test_unified_tier3_report.py
index a3d85860..c9ac5ebb 100644
--- a/tests/reporting/test_unified_tier3_report.py
+++ b/tests/reporting/test_unified_tier3_report.py
@@ -539,9 +539,7 @@ def test_conclusions_separate_unavailable_trials_from_threshold_misses(has_thres
assert payload["agents"]["codex"]["trials"][0]["overall"] is None
conclusions = {item["title"]: item["message"] for item in payload["conclusions"]}
assert "Unscored trials" in conclusions
- assert conclusions["Unscored trials"] == (
- "1 trial(s) have unavailable overall scores; examples: codex/unscored"
- )
+ assert conclusions["Unscored trials"] == ("1 trial(s) have unavailable overall scores; examples: codex/unscored")
html = _render_agent_payload(payload)
assert "1 trial(s) have unavailable overall scores; examples: codex/unscored" in html
if has_threshold_miss:
@@ -692,14 +690,15 @@ def test_no_skill_baseline_dimension_na_explains_why_in_html() -> None:
html = _render_agent_payload(payload)
- assert (
- '<td title="No-skill baseline: This arm has no skill under test installed, '
- 'so there is no skill to discover or run">N/A</td>'
- ) in html
- assert (
- '<td title="No-skill baseline: This arm has no skill under test installed, '
- 'so there is no skill to route to">N/A</td>'
- ) in html
+ for title in (
+ "No-skill baseline: This arm has no skill under test installed, so there is no skill to discover or run",
+ "No-skill baseline: This arm has no skill under test installed, so there is no skill to route to",
+ ):
+ cell_start = html.index(f'<td title="{title}">')
+ cell_end = html.index("</td>", cell_start)
+ cell = html[cell_start:cell_end]
+ assert "N/A" in cell
+ assert '<div class="t3-dim-hint">baseline not scored</div>' in cell
def test_trial_report_omits_token_chart_for_incomplete_agent_telemetry() -> None: