Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

docs(tier3): document per-arm dimension applicability for the no-skill baseline (#398)

Christopher Kevinbe71254ea1da
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7858a472..be6b0d50 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -67,11 +67,13 @@ All notable changes to SkillEvaluator are documented in this file.
### Changed
+- Document per-arm Tier 3 dimension applicability and test how N/A baseline
+ metrics are handled in Overall scores and Skill Lift comparisons.
+
- Run GitLab Tier 2 without Docker-in-Docker while keeping TLS-protected Docker
for Tier 3. Both native tiers retain shared caller timeout and runner-tag
overrides, advisory outcomes, and aggregate artifacts; local mode still
disables the Tier 3 Docker service.
-
- Align terminal grade and lift colors across compact, full, and direct Tier 3
reports; displayed zero lift is neutral. Narrow terminals stack complete
condition score/bar groups. Full-report Details retain successful quality and
diff --git a/docs/reports.mdx b/docs/reports.mdx
index 9416db99..d45d9d20 100644
--- a/docs/reports.mdx
+++ b/docs/reports.mdx
@@ -640,14 +640,11 @@ The **Quality Score** displayed out of 100 in the HTML report is the Tier 1
static-analysis score. It is not an input to the Tier 3 dimensions, overall
score, or Skill Lift.
-The standard overall score gives each dimension one equal vote:
+For standard grading, each arm's Overall is the mean of the dimensions it
+scored, and each dimension is the mean of its scored metrics. N/A dimensions
+and metrics are excluded rather than counted as 0 or 1.
-```text
-overall = (Security + Correctness + Discoverability + Effectiveness + Efficiency) / 5
-Effectiveness = (goal_accuracy + behavior_check) / 2
-```
-
-This formula is shared by the report, per-attempt pass@k thresholding, Skill
+This per-arm rule is shared by the report, per-attempt pass@k thresholding, Skill
Lift, comparisons, and best-agent selection. Standard artifacts identify it as
`skill-evaluator-dimension-mean-v1` in `score_policy` and in the persisted
attempt policy. Older metric sets and partial historical artifacts keep their
@@ -667,8 +664,7 @@ agents scored under different policies or metric sets.
### Skill Lift
-Skill Lift is `with-skill overall score - without-skill overall score` for
-comparable cases. It uses the same score units as both arms and ranges from
+Skill Lift compares Overall on the metrics and cases both arms scored, so it can differ from the displayed with-skill Overall minus baseline Overall when the baseline has N/A dimensions. It uses the same score units as both arms and ranges from
-1.0 to +1.0. Because live agent runs are noisy, especially at low attempt
counts, small deltas are kept neutral. The verdict bands:
diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdx
index 01da045b..6d849a98 100644
--- a/docs/tier3-live-evaluation.mdx
+++ b/docs/tier3-live-evaluation.mdx
@@ -1004,12 +1004,14 @@ The **Quality Score** shown as a value out of 100 in the HTML report comes from
Tier 1 static analysis. It is separate from the Tier 3 evaluator scores,
dimensions, overall score, and Skill Lift.
-For standard grading, the overall score is the equal-weight mean of these five
-dimensions:
+For standard grading, each arm's overall score is the equal-weight mean of the
+dimensions that arm can be scored on. An N/A dimension is left out; it is not
+counted as 0 or 1. Within Effectiveness, N/A source metrics are also left out,
+and the dimension is N/A only when neither source metric can be scored:
```text
-overall = (Security + Correctness + Discoverability + Effectiveness + Efficiency) / 5
-Effectiveness = (goal_accuracy + behavior_check) / 2
+overall = mean(scored(Security, Correctness, Discoverability, Effectiveness, Efficiency))
+Effectiveness = mean(scored(goal_accuracy, behavior_check))
```
Effectiveness therefore enters the overall score once; its two source
@@ -1020,6 +1022,32 @@ Artifacts record this contract as
historical partial reports retain their recorded scoring semantics instead of
being silently reinterpreted.
+### Which dimensions apply to each arm
+
+Security is scored for every arm. Correctness uses `accuracy`, which is N/A
+when `ground_truth` is missing. Effectiveness uses `goal_accuracy`
+(`ground_truth`) and `behavior_check` (`expected_behavior`); each missing
+reference makes its metric N/A. The dimension is N/A only if both metrics are
+N/A.
+
+| Arm | Security | Correctness | Discoverability | Effectiveness | Efficiency |
+| --- | --- | --- | --- | --- | --- |
+| With skill, `should_trigger=true` | Scored | Scored with `ground_truth`; otherwise N/A | Scored: the skill is present to discover and run | Scored for each present reference; N/A if both are missing | Scored: routing and tool use are evaluated |
+| With skill, `should_trigger=false` (negative case) | Scored | Scored with `ground_truth`; otherwise N/A | Scored: `skill_execution` evaluates the negative case | Scored for each present reference; N/A if both are missing | Scored as 1.0 when no skill is expected to trigger |
+| Without skill and no applicable alternate staged | Scored | Scored with `ground_truth`; otherwise N/A | N/A: no skill under test to discover or run | Scored for each present reference; N/A if both are missing | N/A: no skill under test to route to |
+| Without skill, with an acceptable alternate staged | Scored | Scored with `ground_truth`; otherwise N/A | Scored when the staged skill matches a case candidate | Scored for each present reference; N/A if both are missing | Scored when the staged skill matches a case candidate |
+
+For a positive case, an alternate candidate is the expected skill or an
+acceptable alternate. For a negative case, the candidate is the evaluated
+skill, or the expected skill when no evaluated skill is recorded. In either
+case, the staged name must match a candidate for the no-skill arm's skill
+metrics to apply.
+
+Skill Lift per dimension and overall uses only cases and metrics both arms
+scored; the `basis` block in `lift.json` records the shared comparison ([Skill
+Lift reporting](reports.mdx#skill-lift)). A baseline is never credited or
+penalized for a skill it was not given.
+
The Security trace check excludes plain `rm -rf` commands whose operands are
recognized regenerable caches, such as `.pytest_cache`, `__pycache__`, or
`~/.cache/uv`. Absolute cache paths are recognized only under `/workspace/` and
@@ -1085,10 +1113,7 @@ agent passes only when every dimension passes. The top-level Tier 3 verdict is:
| Neutral | No agent passes, but at least one has no dimension below 0.40 |
| Fail | Every successful agent has a dimension below 0.40 |
-Skill Lift is `with-skill overall score - without-skill overall score` for
-comparable cases. Judge-model exclusions can make it differ from subtracting
-the displayed arm means; when no complete comparison with matching judges
-remains, lift is unavailable. It uses the same score units as the two arms;
+Skill Lift compares Overall on the metrics and cases both arms scored, so it can differ from the displayed with-skill Overall minus baseline Overall when the baseline has N/A dimensions. Judge-model exclusions can make it differ from subtracting the displayed arm means; when no complete comparison with matching judges remains, lift is unavailable. It uses the same score units as the two arms;
because it is a difference, it ranges
from -1.0 to +1.0. Its independent band is **PASS** at +0.05 or above,
**NEUTRAL** between -0.10 and +0.05, and **FAIL** at -0.10 or below. Lift is
diff --git a/src/skillevaluator/reporting/templates/report.html.j2 b/src/skillevaluator/reporting/templates/report.html.j2
index fa6b4565..346f9202 100644
--- a/src/skillevaluator/reporting/templates/report.html.j2
+++ b/src/skillevaluator/reporting/templates/report.html.j2
@@ -2639,6 +2639,8 @@
<th>Verdict</th><th>Signals</th><th>Reasoning</th>
</tr></thead>
<tbody>
+ {% set t3_best_agent = t3_agents.get(t3_best_name, {}) %}
+ {% set baseline_na_metrics = t3_best_agent.get('not_applicable_metrics', {}).get('without_skill', []) %}
{% for dim in tier3.dimensions %}
{% set dim_score = dim.score if dim.score is not none else dim.with_skill %}
{% set dim_color = 'var(--success-color)' if dim_score >= 0.7 else ('var(--warning-color)' if dim_score >= 0.4 else 'var(--danger-color)') %}
@@ -2653,7 +2655,7 @@
<div class="mini-bar"><span style="width:{{ (dim_score * 100) | round(0) }}%; background:{{ dim_color }};"></span></div>
</div>
</td>
- <td>{{ "%.2f" | format(dim.baseline) if dim.baseline is not none else "N/A" }}</td>
+ <td{% if dim.id == 'discoverability' and 'skill_execution' in baseline_na_metrics %} title="No-skill baseline: This arm has no skill under test installed, so there is no skill to discover or run"{% elif dim.id == 'efficiency' and 'skill_efficiency' in baseline_na_metrics %} title="No-skill baseline: This arm has no skill under test installed, so there is no skill to route to"{% endif %}>{{ "%.2f" | format(dim.baseline) if dim.baseline is not none else "N/A" }}</td>
<td style="color:{{ 'var(--success-color)' if dim.lift is not none and dim.lift >= tier3_lift_pass_threshold else ('var(--danger-color)' if dim.lift is not none and dim.lift <= tier3_lift_fail_threshold else 'var(--text-secondary)') }};font-weight:600;">
{{ "%+.2f" | format(dim.lift) if dim.lift is not none else "N/A" }}
</td>
diff --git a/tests/reporting/test_unified_tier3_report.py b/tests/reporting/test_unified_tier3_report.py
index bb8b7e83..b0add7a5 100644
--- a/tests/reporting/test_unified_tier3_report.py
+++ b/tests/reporting/test_unified_tier3_report.py
@@ -612,6 +612,41 @@ def test_trial_report_renders_unavailable_tokens_as_unavailable() -> None:
assert re.search(r"<td>4</td>\s*<td>—</td>", trials_html)
+def test_no_skill_baseline_dimension_na_explains_why_in_html() -> None:
+ payload = build_agent_eval_payload(
+ "demo",
+ {
+ "codex": {
+ "execution_status": "succeeded",
+ "expected_attempts": 1,
+ "scored_attempts": 1,
+ "with_skill": dict.fromkeys(DEFAULT_METRICS, 1.0),
+ "without_skill": {
+ "security": 0.8,
+ "accuracy": 0.7,
+ "goal_accuracy": 0.6,
+ "behavior_check": 0.8,
+ },
+ "not_applicable_validated_without_skill": True,
+ "not_applicable_without_skill": ["skill_execution", "skill_efficiency"],
+ }
+ },
+ use_llm_judge=False,
+ )
+ assert payload is not None
+
+ html = _render_agent_payload(payload)
+
+ assert (
+ '<td title="No-skill baseline: This arm has no skill under test installed, '
+ 'so there is no skill to discover or run">N/A</td>'
+ ) in html
+ assert (
+ '<td title="No-skill baseline: This arm has no skill under test installed, '
+ 'so there is no skill to route to">N/A</td>'
+ ) in html
+
+
def test_trial_report_omits_token_chart_for_incomplete_agent_telemetry() -> None:
payload = build_agent_eval_payload(
"demo",
diff --git a/tests/tier3/test_baseline_dimension_applicability.py b/tests/tier3/test_baseline_dimension_applicability.py
new file mode 100644
index 00000000..7f0ac8e5
--- /dev/null
+++ b/tests/tier3/test_baseline_dimension_applicability.py
@@ -0,0 +1,115 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+"""Per-arm applicability for the no-skill baseline."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from skillevaluator.constants import DEFAULT_SCORE_POLICY
+from skillevaluator.tier3.harbor import metrics
+from skillevaluator.tier3.harbor.collector import _compute_lift
+from skillevaluator.tier3.harbor.templates import eval as verifier
+
+
+def reward(case: str, **overrides: float | None) -> dict[str, Any]:
+ result = dict.fromkeys(metrics.DEFAULT_METRICS, 1.0)
+ result.update(
+ metric_set=metrics.DEFAULT_METRIC_SET,
+ score_policy=DEFAULT_SCORE_POLICY,
+ entry_id=case,
+ details={},
+ )
+ for metric, value in overrides.items():
+ result[metric] = value
+ if value is None:
+ result["details"][metric] = {
+ "score": None,
+ "status": "not_applicable",
+ "reason": "N/A: this metric has no applicable reference or skill",
+ }
+ return result
+
+
+def test_skill_metrics_apply_to_skill_and_matching_acceptable_alternate():
+ assert not verifier.skill_metrics_applicable(
+ False,
+ True,
+ "target-skill",
+ workspace_skill_names=[],
+ )
+ assert verifier.skill_metrics_applicable(True, True, "target-skill")
+ assert verifier.skill_metrics_applicable(
+ False,
+ True,
+ "target-skill",
+ acceptable_skills=["alternate-skill"],
+ workspace_skill_names=["alternate-skill"],
+ )
+
+
+def test_reward_overall_averages_three_scored_dimensions_when_skill_metrics_are_na():
+ result = {
+ "security": 0.8,
+ "accuracy": 0.6,
+ "skill_execution": None,
+ "goal_accuracy": 0.4,
+ "behavior_check": 0.8,
+ "skill_efficiency": None,
+ }
+ details = {
+ metric: {
+ "score": None,
+ "status": "not_applicable",
+ "reason": verifier.SKILL_METRICS_NOT_APPLICABLE_REASON,
+ }
+ for metric in ("skill_execution", "skill_efficiency")
+ }
+
+ assert verifier._reward_overall(result, details) == 0.6667
+
+
+def test_lift_basis_excludes_baseline_na_skill_metrics_but_pairs_other_metrics():
+ with_skill = reward(
+ "case-1",
+ security=0.9,
+ accuracy=0.8,
+ skill_execution=0.7,
+ goal_accuracy=0.6,
+ behavior_check=0.8,
+ skill_efficiency=0.9,
+ )
+ without_skill = reward(
+ "case-1",
+ security=0.7,
+ accuracy=0.6,
+ skill_execution=None,
+ goal_accuracy=0.4,
+ behavior_check=0.6,
+ skill_efficiency=None,
+ )
+ lift = _compute_lift(
+ metrics.average_metrics([with_skill])[0],
+ metrics.average_metrics([without_skill])[0],
+ [with_skill],
+ [without_skill],
+ ["case-1"],
+ score_policy=DEFAULT_SCORE_POLICY,
+ )
+
+ assert "skill_execution" not in lift
+ assert "skill_efficiency" not in lift
+ excluded = {item["metric"]: item for item in lift["basis"]["excluded_metrics"]}
+ assert excluded["skill_execution"] == {
+ "metric": "skill_execution",
+ "reason": "not_applicable",
+ "arms": ["without_skill"],
+ }
+ assert excluded["skill_efficiency"] == {
+ "metric": "skill_efficiency",
+ "reason": "not_applicable",
+ "arms": ["without_skill"],
+ }
+ for metric in ("security", "accuracy", "goal_accuracy", "behavior_check"):
+ assert lift[metric]["delta"] == 0.2
+ assert lift["basis"]["metrics"][metric]["cases"] == 1