Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier3): show N/A for N/A metrics in the terminal Skill Lift and the HTML trial drill-down (#411)
Narendran Raghavan5b2cd251e6e4
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 6688c6c9..893be9c1 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -409,6 +409,21 @@ All notable changes to SkillEvaluator are documented in this file.`Bedrock request failed`, like a Bedrock error that ends the judge and likethe default `legacy` policy.+- The Tier 3 terminal summary now shows `N/A`, not `NO SCORE`, as the Skill+ Lift of an evaluator or dimension that is N/A in either arm, as the Markdown+ and HTML reports already do. Most runs with a baseline hit this: Skill+ Execution, Efficiency and Discoverability are N/A in a baseline with no+ skill it could activate, `behavior_check` is N/A when the cases have no+ `assertions` (legacy `expected_behavior`), and `accuracy` and+ `goal_accuracy` when they have no `expected_output` (legacy+ `ground_truth`). In a baseline-only run (`--skip-with-skill`) the lift of+ every other row now reads `skipped`, as the docs and the Markdown report+ say. A lift that is missing for another reason still reads `NO SCORE`. The+ HTML report's Per-Evaluator Drill-Down now shows `N/A`, with the reason on+ hover, instead of `—` for a metric a trial recorded as N/A. The reports'+ reasons for an N/A Accuracy, Goal Accuracy or Behavior Check now name the+ dataset fields `expected_output` and `assertions` before their legacy names.+- `skillevaluator tier3 feedback` now writes the evaluated skill's directory(known from `--skill` or a run in the skill's `evals/results`) as its name andthe home directory as `~` in the evidence it sends to the LLM, so an errordiff --git a/docs/reports.mdx b/docs/reports.mdxindex c84af0d4..e9da4d19 100644--- a/docs/reports.mdx+++ b/docs/reports.mdx@@ -565,9 +565,15 @@ A finished `tier3 evaluate` run prints two tables and closes with a compactwith its per-arm scores and Skill Lift; **Results by Dimension** rolls thosesignals up into the five dimensions below. A metric the run couldn't scorerenders as `NO SCORE` — it is never coerced to 0.0 — and unscored metrics are-omitted from the averages, so `overall_score` can be `null` in the JSON. In a-baseline-only run (`--skip-with-skill`) the with-skill and Skill Lift cells read-`skipped` instead, because that arm never ran. The+omitted from the averages, so `overall_score` can be `null` in the JSON. A+metric that does not apply to an arm renders as `N/A`, and so does its Skill+Lift: for example Behavior Check when the cases have no `assertions` (legacy+`expected_behavior`), Accuracy when they have no `expected_output` (legacy+`ground_truth`), or Skill Execution in a baseline with no skill it could+activate. N/A metrics are left out of the scores and the lift. In a+baseline-only run (`--skip-with-skill`) the with-skill cells read `skipped`,+because that arm never ran, and so does the Skill Lift of every row that is+not N/A. TheArtifacts panel shows the HTML report path, the output directory, and — onlywhen a retained Harbor jobs directory exists — a ready-to-paste `harbor-view`command.@@ -945,7 +951,9 @@ list. A case that scored below 0.80 also quotes a short excerpt of theoffending command or artifact when the grader recorded one. When the attemptsof one case disagree, the lowest-scoring attempt's reason is shown. In the**Per-Evaluator Drill-Down** table, hover a dotted-underlined score below 0.80-to read the reason for that trial.+to read the reason for that trial. A metric the trial recorded as N/A reads+`N/A`, and hovering it says why; a score missing for any other reason reads+`—`.Reasons are built from each trial's `reward.json` `details`, joining up tothree distinct parts from the grader's `reason` or `message`, failing findingsdiff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdxindex e9dc7a11..76792cc6 100644--- a/docs/tier3-live-evaluation.mdx+++ b/docs/tier3-live-evaluation.mdx@@ -1111,6 +1111,39 @@ generated standard grader rejects non-finite values and clamps numeric scoreswith `max(0.0, min(1.0, score))`. If a required judge fails, the result isunscored rather than converted to `0.0`.+A metric can also be N/A (not applicable) for an eval case. It never counts as+`1.0` or `0.0`:++- `behavior_check` is N/A when the case has no `assertions` (legacy+ `expected_behavior`), and `accuracy` and `goal_accuracy` when it has no+ `expected_output` (legacy `ground_truth`). An empty or whitespace-only+ reference, or an empty list, counts as missing. The verifier does not call+ those judges. `skillevaluator tier3 ./my-skill` and `tier3 init-harbor-task`+ reject an empty `expected_output`, so with them only legacy cases without+ `ground_truth` make `accuracy` and `goal_accuracy` N/A.+- `skill_execution` and `skill_efficiency` are N/A in a baseline arm that has+ no skill it could activate.+- Security is never N/A.++In `skill_evaluator_reward.json` an N/A metric's value is `null`, and+`details.<metric>` records `status: "not_applicable"` and a `reason`.+Harbor's numeric `reward.json` leaves the metric out. The trial's overall+score, the dimension scores, pass@k and Skill Lift leave N/A metrics out, and+a dimension whose metrics are all N/A drops out of the overall score. A judge+that answers N/A for a case that defines its reference counts as a failed+judge, so the trial is unscored. Each arm's `summary.json` lists the metrics+that were N/A in every trial (`not_applicable_metrics`) and how many trials+marked each one (`not_applicable_counts`). The terminal summary and the+Markdown and HTML reports show `N/A` for those scores and for their lift.++An agent's verdict reads only the with-skill arm, and the agent can pass only+with a with-skill score in every dimension. When a dimension's metrics are N/A+in every with-skill trial, for example Correctness when no case defines+`expected_output` (legacy `ground_truth`), that dimension has no with-skill+score, so the agent cannot PASS (see the verdict table below). A dimension that+is N/A only in the baseline, such as Discoverability and Efficiency in a+baseline with no skill it could activate, does not affect the verdict.+| Dimension | Key question | Maps to | Weights | Source || --- | --- | --- | --- | --- || Security | Is it safe to use? | `security` | 1.0 | Deterministic checks of the trace and agent actions |diff --git a/src/skillevaluator/reporting/html.py b/src/skillevaluator/reporting/html.pyindex f442f7f6..71d3a4e4 100644--- a/src/skillevaluator/reporting/html.py+++ b/src/skillevaluator/reporting/html.py@@ -501,6 +501,13 @@ def _tier3_score_reasons(value: object) -> dict[str, str]:return reasons+def _tier3_not_applicable_reason(metric: object) -> str:+ """Return why a Tier 3 metric can be N/A, for the tooltip of an N/A drill-down cell."""+ from skillevaluator.tier3.harbor.metrics import NOT_APPLICABLE_REASONS++ return NOT_APPLICABLE_REASONS.get(str(metric), "Not applicable to this eval case")++def _tier3_text_mapping(value: object) -> dict[str, str]:"""Normalize public label maps without silently overwriting aliases."""if not isinstance(value, dict):@@ -1008,6 +1015,11 @@ def _sanitize_tier3_trial(value: dict[str, Any], *, fill_defaults: bool = True)score_reasons = _tier3_score_reasons(trial.pop("score_reasons", None))if score_reasons:trial["score_reasons"] = score_reasons+ if "not_applicable_metrics" in trial:+ from skillevaluator.tier3.harbor.metrics import not_applicable_list++ # Only metrics that can be N/A count, so a malformed payload cannot mark security N/A.+ trial["not_applicable_metrics"] = not_applicable_list(trial["not_applicable_metrics"])# Unavailable telemetry stays absent rather than becoming zero, so the# template can omit charts and show "—" instead of reporting no usage.raw_tokens = trial.pop("tokens", None)@@ -1927,6 +1939,7 @@ class HTMLReporter(ReporterBase):environment.filters["adaptive_percent"] = _adaptive_percentenvironment.filters["adaptive_interval_percent"] = _adaptive_interval_percentenvironment.filters["skip_reason"] = get_skip_reason+ environment.filters["not_applicable_reason"] = _tier3_not_applicable_reasonreturn environment@staticmethoddiff --git a/src/skillevaluator/reporting/templates/report.html.j2 b/src/skillevaluator/reporting/templates/report.html.j2index 8e37b04e..78175add 100644--- a/src/skillevaluator/reporting/templates/report.html.j2+++ b/src/skillevaluator/reporting/templates/report.html.j2@@ -3385,9 +3385,10 @@</td>{% for eval_id in t3_eval_ids %}{% set s = scores.get(eval_id) %}- {% set cell_reason = reasons.get(eval_id) if s is not none else none %}- {% set cell_text = "%.2f" | format(s) if s is not none else "—" %}- <td style="{% if s is not none %}background:{{ 'rgba(34,197,94,0.14)' if s >= 0.7 else ('rgba(234,179,8,0.14)' if s >= 0.4 else 'rgba(239,68,68,0.14)') }}; color:{{ 'var(--success-color)' if s >= 0.7 else ('var(--warning-color)' if s >= 0.4 else 'var(--danger-color)') }}; font-weight:600;{% endif %}"{% if cell_reason %} title="{{ eval_id | replace('_', ' ') | title }}: {{ cell_reason }}"{% endif %}>{% if cell_reason %}<span class="t3-score-reason">{{ cell_text }}</span>{% else %}{{ cell_text }}{% endif %}</td>+ {% set cell_na = s is none and eval_id in (trial.not_applicable_metrics or []) %}+ {% set cell_reason = reasons.get(eval_id) if s is not none else (("Not applicable: " ~ (eval_id | not_applicable_reason)) if cell_na else none) %}+ {% set cell_text = "%.2f" | format(s) if s is not none else ("N/A" if cell_na else "—") %}+ <td style="{% if s is not none %}background:{{ 'rgba(34,197,94,0.14)' if s >= 0.7 else ('rgba(234,179,8,0.14)' if s >= 0.4 else 'rgba(239,68,68,0.14)') }}; color:{{ 'var(--success-color)' if s >= 0.7 else ('var(--warning-color)' if s >= 0.4 else 'var(--danger-color)') }}; font-weight:600;{% elif cell_na %}color:var(--text-secondary);{% endif %}"{% if cell_na %} data-not-applicable="true"{% endif %}{% if cell_reason %} title="{{ eval_id | replace('_', ' ') | title }}: {{ cell_reason }}"{% endif %}>{% if cell_reason %}<span class="t3-score-reason">{{ cell_text }}</span>{% else %}{{ cell_text }}{% endif %}</td>{% endfor %}<td>{{ trial.steps if trial.steps is number else "—" }}</td><td>{% if tokens.prompt is number and tokens.completion is number %}{{ tokens.prompt + tokens.completion }}{% else %}—{% endif %}</td>diff --git a/src/skillevaluator/tier3/harbor/metrics.py b/src/skillevaluator/tier3/harbor/metrics.pyindex a7bcf804..a88fc3ee 100644--- a/src/skillevaluator/tier3/harbor/metrics.py+++ b/src/skillevaluator/tier3/harbor/metrics.py@@ -93,9 +93,10 @@ DIMENSION_QUESTIONS = {}# A judged metric is "not applicable" when its dataset case has nothing to-# judge against (no ground_truth / expected_behavior). The skill metrics are-# not applicable in an arm that has no skill under test to discover or route to-# (the no-skill or no-plugin baseline). The verifier records the metric as+# judge against (no expected_output / assertions, which the legacy format calls+# ground_truth / expected_behavior). The skill metrics are not applicable in an+# arm that has no skill under test to discover or route to (the no-skill or+# no-plugin baseline). The verifier records the metric as# ``null`` with ``details[metric].status == "not_applicable"``; N/A metrics are# excluded from the trial overall, the arm averages, and lift. Security never# becomes N/A, and an absent or non-finite metric without the explicit marker@@ -105,9 +106,11 @@ NOT_APPLICABLE_ELIGIBLE_METRICS = ("skill_execution", "skill_efficiency", "accurNOT_APPLICABLE_REASONS = {"skill_execution": "This arm has no skill under test installed, so there is no skill to discover or run","skill_efficiency": "This arm has no skill under test installed, so there is no skill to route to",- "accuracy": "No ground_truth defined, so there is no reference answer to judge against",- "goal_accuracy": "No ground_truth defined, so there is no expected outcome to judge against",- "behavior_check": "No expected_behavior defined, so there is no workflow to check",+ "accuracy": "No expected_output (legacy ground_truth) defined, so there is no reference answer to judge against",+ "goal_accuracy": (+ "No expected_output (legacy ground_truth) defined, so there is no expected outcome to judge against"+ ),+ "behavior_check": "No assertions (legacy expected_behavior) defined, so there is no workflow to check",}_RESERVED_METADATA_KEYS = {diff --git a/src/skillevaluator/tier3/result_display.py b/src/skillevaluator/tier3/result_display.pyindex 9abae801..38fbacf6 100644--- a/src/skillevaluator/tier3/result_display.py+++ b/src/skillevaluator/tier3/result_display.py@@ -79,8 +79,11 @@ def _score_cell(value: object, *, unavailable: str = "NO SCORE") -> tuple[Text,return Text(f"{numeric:.2f}", style=f"bold {style}"), Text(_score_bar(numeric), style=style)+_NOT_APPLICABLE_LABEL = "N/A"++def _not_applicable_cell() -> tuple[Text, Text]:- return Text("N/A", style="dim italic"), Text("")+ return Text(_NOT_APPLICABLE_LABEL, style="dim italic"), Text("")def _not_applicable_metrics(data: Mapping[str, Any], variant: str) -> frozenset[str]:@@ -91,9 +94,15 @@ def _not_applicable_metrics(data: Mapping[str, Any], variant: str) -> frozenset[return frozenset(not_applicable_list(raw))-def _delta_cell(value: object) -> Text:+def _delta_cell(value: object, *, not_applicable: bool = False, skipped: bool = False) -> Text:+ """Lift cell. With no delta it reads N/A for a row that is N/A in either arm, then+ ``skipped`` when the with-skill arm never ran, then NO SCORE, like the other reporters."""numeric = _finite_number(value)if numeric is None:+ if not_applicable:+ return Text(_NOT_APPLICABLE_LABEL, style="dim italic")+ if skipped:+ return Text("skipped", style="dim")return Text("NO SCORE", style="dim")style = lift_style(numeric)return Text(f"{numeric:+.2f}", style=f"bold {style}")@@ -551,7 +560,8 @@ def _render_agent_scores() -> None:with_usable = _condition_usable(data, "with_skill")# A baseline-only run (--skip-with-skill) deliberately leaves the with-skill arm unscored.- with_unavailable = "skipped" if _condition_status(data, "with_skill") == "skipped" else "NO SCORE"+ with_skipped = _condition_status(data, "with_skill") == "skipped"+ with_unavailable = "skipped" if with_skipped else "NO SCORE"baseline_status = _condition_status(data, "without_skill")baseline_skipped = baseline_status == "skipped"baseline_usable = _condition_usable(data, "without_skill") and not baseline_skipped@@ -603,14 +613,18 @@ def _render_agent_scores(else None)baseline_score, baseline_bar = _not_applicable_cell() if baseline_na else _score_cell(baseline_value)- row.extend([baseline_score, baseline_bar, _delta_cell(delta)])+ lift_cell = _delta_cell(delta, not_applicable=with_na or baseline_na, skipped=with_skipped)+ row.extend([baseline_score, baseline_bar, lift_cell])table.add_row(*row)basis = lift.get("basis")if isinstance(basis, Mapping) and basis.get("pairing") == "same_cases_per_metric":table.caption = "Lift compares the cases scored by both arms for each evaluator."if _not_applicable_metrics(data, "with_skill") or _not_applicable_metrics(data, "without_skill"):- table.caption = (str(table.caption or "") + " N/A: no applicable reference or skill in this arm.").strip()+ table.caption = (+ str(table.caption or "")+ + " N/A: no applicable reference or skill in this arm, so it is left out of scores and lift."+ ).strip()if custom_with or custom_without:if metrics:table.add_section()@@ -628,7 +642,7 @@ def _render_agent_scores(else None)baseline_score, baseline_bar = _score_cell(baseline_value)- row.extend([baseline_score, baseline_bar, _delta_cell(delta)])+ row.extend([baseline_score, baseline_bar, _delta_cell(delta, skipped=with_skipped)])table.add_row(*row)if not metrics and not custom_with and not custom_without and not show_baseline:@@ -820,7 +834,7 @@ def _render_dimensions(with_bar,baseline_cell,baseline_bar,- _delta_cell(delta),+ _delta_cell(delta, not_applicable=with_na or baseline_na, skipped=with_skipped),]table.add_row(*row)diff --git a/tests/reporting/test_applicability_saved_runs.py b/tests/reporting/test_applicability_saved_runs.pyindex e2e56b86..885ca200 100644--- a/tests/reporting/test_applicability_saved_runs.py+++ b/tests/reporting/test_applicability_saved_runs.py@@ -5,16 +5,18 @@from __future__ import annotationsimport json+from pathlib import Pathimport pytest-from skillevaluator.evaluation.tier3_report import build_agent_eval_payload+from skillevaluator.evaluation.tier3_report import VERDICT_NEUTRAL, VERDICT_PASS, build_agent_eval_payloadfrom skillevaluator.tier3.harbor.metrics import (DEFAULT_METRIC_SET,DEFAULT_METRICS,DEFAULT_SCORE_POLICY,average_metrics,dimension_scores,+ mark_not_applicable,not_applicable_metrics,overall_score,)@@ -311,3 +313,128 @@ def test_saved_run_does_not_renormalize_a_missing_metric_without_validated_na(tmassert agent["evaluators"]["behavior_check"]["with_skill_status"] == "missing"assert agent["with_skill"] is Noneassert not any(row["id"] == "effectiveness" and row["with_skill"] is not None for row in agent["dimensions"])+++def _uniform_run(root, *, with_na=(), baseline_na=(), with_scores=None):+ """Write a two-case saved run scoring 0.95 with the skill and 0.50 without it on every metric not N/A.++ ``with_scores`` overrides individual with-skill metric scores.+ """+ for condition, value, na_metrics, overrides in (+ ("with-skill", 0.95, with_na, with_scores or {}),+ ("without-skill", 0.5, baseline_na, {}),+ ):+ rows = []+ for case in ("a", "b"):+ row = dict.fromkeys(DEFAULT_METRICS, value)+ row.update(overrides)+ row.update(+ entry_id=case,+ trial_id=f"{case}-trial",+ metric_set=DEFAULT_METRIC_SET,+ score_policy=DEFAULT_SCORE_POLICY,+ details={},+ )+ for metric in na_metrics:+ row[metric] = None+ row["details"][metric] = {"score": None, "status": "not_applicable", "reason": "No reference"}+ rows.append(row)+ scores = average_metrics(rows)[0]+ na = not_applicable_metrics(rows)+ directory = root / "codex" / condition+ directory.mkdir(parents=True)+ summary = {+ "scores": scores,+ "metrics": list(DEFAULT_METRICS),+ "metric_set": DEFAULT_METRIC_SET,+ "score_policy": DEFAULT_SCORE_POLICY,+ "not_applicable_metrics": na,+ "dimensions": dimension_scores(scores, na),+ "num_trials": 2,+ "num_reward_rows": 2,+ "execution_status": "succeeded",+ "overall_score": sum(overall_score(row) for row in rows) / 2,+ }+ (directory / "summary.json").write_text(json.dumps(summary), encoding="utf-8")+ for row in rows:+ trial = directory / "trials" / row["entry_id"]+ trial.mkdir(parents=True)+ (trial / "reward.json").write_text(json.dumps(row), encoding="utf-8")+ return root+++ ("with_na", "verdict", "with_skill_na_dimension"),+ [+ ((), VERDICT_PASS, None),+ (("accuracy", "goal_accuracy"), VERDICT_NEUTRAL, "correctness"),+ ],+ ids=["baseline-only-na-dimensions", "with-skill-na-correctness"],+)+def test_verdict_reads_only_with_skill_na_dimensions(tmp_path, with_na, verdict, with_skill_na_dimension):+ # Discoverability and Efficiency are N/A in a baseline with no skill it could activate.+ root = _uniform_run(tmp_path, with_na=with_na, baseline_na=("skill_execution", "skill_efficiency"))++ payload = _report(root)++ dimensions = {row["id"]: row for row in payload["agents"]["codex"]["dimensions"]}+ for dimension in ("discoverability", "efficiency"):+ assert dimensions[dimension]["baseline"] is None+ assert dimensions[dimension]["with_skill"] == pytest.approx(0.95)+ for dimension_id, row in dimensions.items():+ assert (row["with_skill"] is None) is (dimension_id == with_skill_na_dimension)+ assert payload["verdict"] == verdict+++ ("with_scores", "activation"),+ [({}, None), ({"security": 0.1}, None), ({}, "never_activated")],+ ids=["other-dimensions-pass", "security-below-0.40", "skill-never-activated"],+)+def test_with_skill_na_dimension_never_lets_the_agent_pass(tmp_path, with_scores, activation):+ # The docs promise only that the agent cannot PASS. Which other verdict it gets depends on the+ # rest of the run, such as a dimension below 0.40 or a skill that never activated.+ root = _uniform_run(tmp_path, with_na=("accuracy", "goal_accuracy"), with_scores=with_scores)+ loaded = load_agent_data(root)+ if activation:+ loaded["codex"]["skill_activation"] = {"status": activation}++ payload = build_agent_eval_payload("demo", loaded, use_llm_judge=False)++ assert payload is not None+ agent = payload["agents"]["codex"]+ dimensions = {row["id"]: row for row in agent["dimensions"]}+ assert dimensions["correctness"]["with_skill"] is None+ assert dimensions["security"]["with_skill"] == pytest.approx(with_scores.get("security", 0.95))+ assert (agent["skill_activation"].get("status") == "never_activated") is (activation == "never_activated")+ assert payload["verdict"] != VERDICT_PASS+++def test_docs_name_only_the_shared_outcome_of_a_with_skill_na_dimension():+ # The other dimensions and the skill's activation decide which verdict other than PASS applies,+ # so the N/A paragraph must not name one; the verdict table covers that.+ docs = (Path(__file__).resolve().parents[2] / "docs" / "tier3-live-evaluation.mdx").read_text(encoding="utf-8")+ normalized = " ".join(docs.split())+ start = normalized.index("An agent's verdict reads only the with-skill arm")+ paragraph = normalized[start : normalized.index("does not affect the verdict.", start)]++ assert "that dimension has no with-skill score, so the agent cannot PASS" in paragraph+ for verdict in ("NEUTRAL", "FAIL", "INCONCLUSIVE"):+ assert verdict not in paragraph+++def test_na_evaluator_reasons_name_the_dataset_field_and_its_legacy_name(tmp_path):+ root = _uniform_run(tmp_path, with_na=("accuracy", "goal_accuracy", "behavior_check"))++ reasons = {item["id"]: item["reason"] for item in _report(root)["not_applicable_evaluators"]}++ assert set(reasons) == {"accuracy", "goal_accuracy", "behavior_check"}+ for metric, field, legacy in (+ ("accuracy", "expected_output", "ground_truth"),+ ("goal_accuracy", "expected_output", "ground_truth"),+ ("behavior_check", "assertions", "expected_behavior"),+ ):+ assert reasons[metric].startswith(f"No {field} (legacy {legacy}) defined, so there is no ")+ reward: dict = {}+ mark_not_applicable(reward, metric)+ assert reward["details"][metric]["reason"] == reasons[metric]diff --git a/tests/reporting/test_tier3_baseline_trial_views.py b/tests/reporting/test_tier3_baseline_trial_views.pyindex fc09f67f..8d118f49 100644--- a/tests/reporting/test_tier3_baseline_trial_views.py+++ b/tests/reporting/test_tier3_baseline_trial_views.py@@ -291,6 +291,92 @@ def test_drilldown_adds_a_column_for_a_metric_only_baseline_rows_scored() -> Nonassert _cell(table, "With skill", "Security") == "—"+def _na(reward: dict[str, Any], *metrics: str) -> dict[str, Any]:+ """Record *metrics* as N/A the way the Harbor verifier does: no score plus the sidecar marker."""+ details = reward.setdefault("details", {})+ for metric in metrics:+ reward[metric] = None+ details[metric] = {"score": None, "status": "not_applicable", "reason": "No reference"}+ return reward+++def _raw_cell(table: str, condition: str, column: str) -> str:+ headers = re.findall(r"<th>(.*?)</th>", table)+ cells = re.findall(r"<td[^>]*>.*?</td>", _row(table, condition), re.DOTALL)+ return cells[headers.index(column)]+++def test_drilldown_reads_na_for_not_applicable_trial_cells() -> None:+ # Each arm records a metric as N/A that the other arm scores, so both keep its column.+ with_reward = _na(_reward("case-1__attempt1", "case-1", score=0.9, steps=5, prompt=10), "accuracy")+ # A metric that is missing without the N/A marker is unscored, not N/A.+ with_reward["goal_accuracy"] = None+ baseline_reward = _na(+ _reward("case-1__baseline1", "case-1", score=0.5, steps=4, prompt=10), "skill_execution", "skill_efficiency"+ )+ payload = build_agent_eval_payload(+ "demo",+ {+ "codex": {+ "execution_status": "succeeded",+ "expected_attempts": 2,+ "scored_attempts": 2,+ "with_skill": {**dict.fromkeys(METRICS, 0.9), "accuracy": None},+ "without_skill": {**dict.fromkeys(METRICS, 0.5), "skill_execution": None, "skill_efficiency": None},+ "rewards": [with_reward],+ "rewards_baseline": [baseline_reward],+ "num_trials": 1,+ "num_trials_baseline": 1,+ }+ },+ use_llm_judge=False,+ )+ assert payload is not None++ table = _drilldown_table(_render(payload))++ assert _cell(table, "With skill", "Accuracy") == "N/A"+ assert 'data-not-applicable="true"' in _raw_cell(table, "With skill", "Accuracy")+ assert 'title="Accuracy: Not applicable: No expected_output (legacy ground_truth) defined' in _raw_cell(+ table, "With skill", "Accuracy"+ )+ assert _cell(table, "With skill", "Goal Accuracy") == "—"+ assert "data-not-applicable" not in _raw_cell(table, "With skill", "Goal Accuracy")+ assert _cell(table, "With skill", "Skill Execution") == "0.90"+ for column in ("Skill Execution", "Skill Efficiency"):+ assert _cell(table, "Without skill", column) == "N/A"+ assert 'data-not-applicable="true"' in _raw_cell(table, "Without skill", column)+ assert 'title="Skill Execution: Not applicable: This arm has no skill under test installed' in _raw_cell(+ table, "Without skill", "Skill Execution"+ )+ assert _cell(table, "Without skill", "Accuracy") == "0.50"+++ ("recorded", "kept"),+ [+ (["security", "accuracy", "unknown"], ["accuracy"]),+ ("accuracy", []),+ ({"accuracy": True}, []),+ ],+ ids=["ineligible-names", "string", "mapping"],+)+def test_display_keeps_only_eligible_na_metric_names(recorded: object, kept: list[str]) -> None:+ payload = _payload()+ payload["agents"]["codex"]["trials_baseline"][0]["not_applicable_metrics"] = recorded+ payload["agents"]["codex"]["trials_baseline"][0]["scores"] = {}+ result = ValidationResult(validator_name="AGENT_EVAL", validator_description="Live evaluation")+ result.metadata["agent_eval"] = payload++ data = HTMLReporter(include_timestamp=False)._tier3_report_data([result])++ assert data is not None+ assert data["agents"]["codex"]["trials_baseline"][0]["not_applicable_metrics"] == kept+ table = _drilldown_table(_render(payload))+ assert _cell(table, "Without skill", "Security") == "—"+ assert _cell(table, "Without skill", "Accuracy") == ("N/A" if kept else "—")++def test_drilldown_columns_include_metrics_scored_only_on_baseline_rows() -> None:payload = _payload()with_trial = payload["trials"][0]diff --git a/tests/tier3/test_metric_applicability_display.py b/tests/tier3/test_metric_applicability_display.pyindex dc1e192c..1c2e8aaa 100644--- a/tests/tier3/test_metric_applicability_display.py+++ b/tests/tier3/test_metric_applicability_display.py@@ -2,9 +2,12 @@# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.from __future__ import annotations+import re+from rich.console import Console-from skillevaluator.tier3.result_display import _default_with_skill_overall, _render_agent_scores+from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS, dimension_scores+from skillevaluator.tier3.result_display import _default_with_skill_overall, _render_agent_scores, render_resultdef test_terminal_keeps_not_applicable_distinct_from_missing_evidence():@@ -32,3 +35,86 @@ def test_terminal_overall_renormalizes_only_explicit_na_metrics():== 1.0)assert _default_with_skill_overall({"with_skill": values, "execution_status": "succeeded"}) is None+++def _rows(output: str) -> dict[str, list[str]]:+ """Map each table row label to its cells, dropping borders and score bars."""+ rows: dict[str, list[str]] = {}+ for line in output.splitlines():+ cells = re.sub(r"[─-╿█░]", " ", line).split(" ")+ cells = [cell.strip() for cell in cells if cell.strip()]+ if cells:+ rows.setdefault(cells[0], cells[1:])+ return rows+++def test_terminal_lift_reads_na_when_either_arm_is_na():+ with_scores = {"security": 1.0, "skill_execution": 0.8, "skill_efficiency": 0.6, "accuracy": 0.9}+ baseline_scores = {"security": 1.0, "accuracy": 0.5}+ with_na = ["behavior_check"]+ baseline_na = ["skill_execution", "skill_efficiency", "behavior_check"]+ result = {+ "execution_status": "succeeded",+ "metrics": list(DEFAULT_METRICS),+ "agents": {+ "codex": {+ "execution_status": "succeeded",+ "conditions": {+ "with_skill": {"execution_status": "succeeded"},+ "without_skill": {"execution_status": "succeeded"},+ },+ "with_skill": with_scores,+ "without_skill": baseline_scores,+ "not_applicable_metrics": {"with_skill": with_na, "without_skill": baseline_na},+ "lift": {"security": {"delta": 0.0}, "accuracy": {"delta": 0.4}},+ "dimensions_with_skill": dimension_scores(with_scores, with_na),+ "dimensions_without_skill": dimension_scores(baseline_scores, baseline_na),+ }+ },+ }++ rows = _rows(render_result(result))++ # N/A in one arm or both: no lift, and N/A says why instead of NO SCORE.+ assert rows["Skill Execution"] == ["0.80", "N/A", "N/A"]+ assert rows["Behavior Check"] == ["N/A", "N/A", "N/A"]+ assert rows["Discoverability"] == ["0.80", "N/A", "N/A"]+ # goal_accuracy is missing, not N/A: its lift still reads NO SCORE.+ assert rows["Goal Accuracy"] == ["NO SCORE", "NO SCORE", "NO SCORE"]+ assert rows["Effectiveness"] == ["NO SCORE", "NO SCORE", "NO SCORE"]+ # Scored rows keep their numeric lift.+ assert rows["Accuracy"] == ["0.90", "0.50", "+0.40"]+ assert rows["Correctness"] == ["0.90", "0.50", "+0.40"]+++def test_terminal_lift_reads_skipped_in_a_baseline_only_run_unless_the_row_is_na():+ baseline_scores = {"security": 1.0, "accuracy": 0.5, "goal_accuracy": 0.5, "behavior_check": 0.5}+ baseline_na = ["skill_execution", "skill_efficiency"]+ result = {+ "execution_status": "succeeded",+ "metrics": list(DEFAULT_METRICS),+ "agents": {+ "codex": {+ "execution_status": "succeeded",+ "conditions": {+ "with_skill": {"execution_status": "skipped"},+ "without_skill": {"execution_status": "succeeded"},+ },+ "with_skill": {},+ "without_skill": baseline_scores,+ "not_applicable_metrics": {"with_skill": [], "without_skill": baseline_na},+ "lift": {},+ "dimensions_with_skill": {},+ "dimensions_without_skill": dimension_scores(baseline_scores, baseline_na),+ }+ },+ }++ rows = _rows(render_result(result))++ # --skip-with-skill: the arm never ran, so its cells and the lift read skipped, as in the other reports.+ assert rows["Accuracy"] == ["skipped", "0.50", "skipped"]+ assert rows["Correctness"] == ["skipped", "0.50", "skipped"]+ # N/A wins over skipped: the baseline has no skill to activate.+ assert rows["Skill Execution"] == ["skipped", "N/A", "N/A"]+ assert rows["Discoverability"] == ["skipped", "N/A", "N/A"]diff --git a/tests/tier3/test_metric_applicability_producer.py b/tests/tier3/test_metric_applicability_producer.pyindex 1647897d..b55f0db7 100644--- a/tests/tier3/test_metric_applicability_producer.py+++ b/tests/tier3/test_metric_applicability_producer.py@@ -44,6 +44,29 @@ def test_false_and_zero_are_real_reference_answers(value):assert _has_judge_reference(value)+ ("value", "expected"),+ [+ (0, True),+ (0.0, True),+ (False, True),+ ([False], True),+ ("0", True),+ (None, False),+ ("", False),+ (" ", False),+ ([], False),+ (["", None], False),+ ],+ ids=["0", "0.0", "False", "list-False", "string-0", "None", "empty", "blank", "empty-list", "blank-items"],+)+def test_template_reference_check_counts_false_and_zero_but_not_empty_values(value, expected):+ from skillevaluator.tier3.harbor.templates import eval as verifier++ # The Harbor verifier keeps its own copy of the check, and that copy decides N/A in a trial.+ assert verifier._has_judge_reference(value) is expected++def test_only_explicit_markers_skip_eligible_metrics():payload = reward(accuracy=None, goal_accuracy=None, behavior_check=None)assert metrics.overall_score(payload) == 1.0@@ -284,6 +307,104 @@ def test_template_no_reference_main_writes_numeric_reward_and_rich_na(tmp_path,assert all(rich[metric] is None for metric in metrics.NOT_APPLICABLE_ELIGIBLE_METRICS)+def _template_verifier(tmp_path, monkeypatch, entry, *, trajectory=True, judges=None):+ """Point the Harbor verifier at tmp_path, with stubbed trajectory, security and judges."""+ from skillevaluator.tier3.harbor.templates import eval as verifier++ entry_path = tmp_path / "entry.json"+ entry_path.write_text(json.dumps(entry), encoding="utf-8")+ loaded = (+ ({"steps": [{"source": "agent", "message": "done"}]}, {"source": "trajectory.json"})+ if trajectory+ else (None, {"source": None})+ )+ monkeypatch.setattr(verifier, "ENTRY_PATH", entry_path)+ monkeypatch.setattr(verifier, "load_trajectory_with_fallback", lambda *_args: loaded)+ monkeypatch.setattr(verifier, "check_security", lambda *_a: {"score": 1.0})+ monkeypatch.setattr(verifier, "REWARD_JSON", tmp_path / "reward.json")+ monkeypatch.setattr(verifier, "REWARD_TXT", tmp_path / "reward.txt")+ monkeypatch.setattr(verifier, "SKILL_EVALUATOR_REWARD_JSON", tmp_path / "skill_evaluator_reward.json")+ monkeypatch.setattr(verifier, "call_public_llm", lambda *_a, **_k: pytest.fail("Unexpected LLM call"))+ for name, judge in (judges or {}).items():+ monkeypatch.setattr(verifier, name, judge)+ return verifier+++def _template_outputs(tmp_path):+ numeric = json.loads((tmp_path / "reward.json").read_text(encoding="utf-8"))+ rich = json.loads((tmp_path / "skill_evaluator_reward.json").read_text(encoding="utf-8"))+ return numeric, rich+++def test_template_trial_without_trajectory_keeps_reference_less_judges_na(tmp_path, monkeypatch):+ entry = {"id": "a", "question": "q", "expected_behavior": ["Answers"], "expected_skill": "demo", "has_skill": True}+ verifier = _template_verifier(tmp_path, monkeypatch, entry, trajectory=False)++ verifier.main()++ numeric, rich = _template_outputs(tmp_path)+ # The crashed trial fails every metric the case defines ...+ assert numeric == {+ "security": 0.0,+ "skill_execution": 0.0,+ "skill_efficiency": 0.0,+ "behavior_check": 0.0,+ "overall": 0.0,+ }+ # ... but a judge without its reference stays N/A, so it cannot pull the arm's mean down to 0.0.+ for metric in ("accuracy", "goal_accuracy"):+ assert rich[metric] is None+ assert metrics.metric_is_not_applicable(rich, metric)+ scores, _, _ = metrics.average_metrics([reward("b", accuracy=0.9, goal_accuracy=None), rich])+ assert scores["accuracy"] == 0.9+ assert "goal_accuracy" not in scores+++def test_template_baseline_trial_without_trajectory_keeps_skill_metrics_na(tmp_path, monkeypatch):+ entry = {"id": "a", "question": "q", "ground_truth": "42", "expected_skill": "demo", "has_skill": False}+ verifier = _template_verifier(tmp_path, monkeypatch, entry, trajectory=False)++ verifier.main()++ numeric, rich = _template_outputs(tmp_path)+ # A crashed baseline trial has no skill to activate, so its skill metrics stay N/A like the+ # other baseline trials. A 0.0 here would hand the with-skill arm lift from activation alone.+ for metric in ("skill_execution", "skill_efficiency"):+ assert metric not in numeric+ assert rich[metric] is None+ assert metrics.metric_is_not_applicable(rich, metric)+ assert numeric["accuracy"] == 0.0++[email protected]("metric", ["accuracy", "goal_accuracy", "behavior_check"])+def test_template_trial_is_unscored_when_a_judge_claims_na_for_a_defined_reference(tmp_path, monkeypatch, metric):+ from skillevaluator.tier3.harbor.collector import _merge_reward_sidecars, _partition_scoreable_rewards++ def scored(*_args, **_kwargs):+ return {"score": 1.0, "reason": "ok", "results": []}++ def claims_na(*_args, **_kwargs):+ return {"score": None, "status": "not_applicable", "reason": "Nothing to judge"}++ judges = dict.fromkeys(("judge_accuracy", "judge_goal_accuracy", "judge_behavior_check"), scored)+ judges[f"judge_{metric}"] = claims_na+ entry = {"id": "a", "question": "q", "ground_truth": "42", "expected_behavior": ["Answers"], "has_skill": False}+ verifier = _template_verifier(tmp_path, monkeypatch, entry, judges=judges)++ with pytest.raises(SystemExit) as stopped:+ verifier.main()++ assert stopped.value.code == 1+ numeric, rich = _template_outputs(tmp_path)+ assert rich["evaluation_status"] == "failed"+ assert "although the eval case defines its reference" in rich["evaluation_errors"][metric]+ assert not metrics.metric_is_not_applicable(rich, metric)+ _merge_reward_sidecars(numeric, tmp_path)+ scoreable, failed = _partition_scoreable_rewards([numeric])+ assert scoreable == []+ assert len(failed) == 1++def test_authoritative_multistep_na_requires_every_constituent_sidecar(tmp_path):from skillevaluator.tier3.harbor.collector import _merge_constituent_default_reward_failure