Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
docs(tier3): document per-arm dimension applicability for the no-skill baseline (#398)
Christopher Kevinbe71254ea1da
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7858a472..be6b0d50 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -67,11 +67,13 @@ All notable changes to SkillEvaluator are documented in this file.### Changed+- Document per-arm Tier 3 dimension applicability and test how N/A baseline+ metrics are handled in Overall scores and Skill Lift comparisons.+- Run GitLab Tier 2 without Docker-in-Docker while keeping TLS-protected Dockerfor Tier 3. Both native tiers retain shared caller timeout and runner-tagoverrides, advisory outcomes, and aggregate artifacts; local mode stilldisables the Tier 3 Docker service.-- Align terminal grade and lift colors across compact, full, and direct Tier 3reports; displayed zero lift is neutral. Narrow terminals stack completecondition score/bar groups. Full-report Details retain successful quality anddiff --git a/docs/reports.mdx b/docs/reports.mdxindex 9416db99..d45d9d20 100644--- a/docs/reports.mdx+++ b/docs/reports.mdx@@ -640,14 +640,11 @@ The **Quality Score** displayed out of 100 in the HTML report is the Tier 1static-analysis score. It is not an input to the Tier 3 dimensions, overallscore, or Skill Lift.-The standard overall score gives each dimension one equal vote:+For standard grading, each arm's Overall is the mean of the dimensions it+scored, and each dimension is the mean of its scored metrics. N/A dimensions+and metrics are excluded rather than counted as 0 or 1.-```text-overall = (Security + Correctness + Discoverability + Effectiveness + Efficiency) / 5-Effectiveness = (goal_accuracy + behavior_check) / 2-```--This formula is shared by the report, per-attempt pass@k thresholding, Skill+This per-arm rule is shared by the report, per-attempt pass@k thresholding, SkillLift, comparisons, and best-agent selection. Standard artifacts identify it as`skill-evaluator-dimension-mean-v1` in `score_policy` and in the persistedattempt policy. Older metric sets and partial historical artifacts keep their@@ -667,8 +664,7 @@ agents scored under different policies or metric sets.### Skill Lift-Skill Lift is `with-skill overall score - without-skill overall score` for-comparable cases. It uses the same score units as both arms and ranges from+Skill Lift compares Overall on the metrics and cases both arms scored, so it can differ from the displayed with-skill Overall minus baseline Overall when the baseline has N/A dimensions. It uses the same score units as both arms and ranges from-1.0 to +1.0. Because live agent runs are noisy, especially at low attemptcounts, small deltas are kept neutral. The verdict bands:diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdxindex 01da045b..6d849a98 100644--- a/docs/tier3-live-evaluation.mdx+++ b/docs/tier3-live-evaluation.mdx@@ -1004,12 +1004,14 @@ The **Quality Score** shown as a value out of 100 in the HTML report comes fromTier 1 static analysis. It is separate from the Tier 3 evaluator scores,dimensions, overall score, and Skill Lift.-For standard grading, the overall score is the equal-weight mean of these five-dimensions:+For standard grading, each arm's overall score is the equal-weight mean of the+dimensions that arm can be scored on. An N/A dimension is left out; it is not+counted as 0 or 1. Within Effectiveness, N/A source metrics are also left out,+and the dimension is N/A only when neither source metric can be scored:```text-overall = (Security + Correctness + Discoverability + Effectiveness + Efficiency) / 5-Effectiveness = (goal_accuracy + behavior_check) / 2+overall = mean(scored(Security, Correctness, Discoverability, Effectiveness, Efficiency))+Effectiveness = mean(scored(goal_accuracy, behavior_check))```Effectiveness therefore enters the overall score once; its two source@@ -1020,6 +1022,32 @@ Artifacts record this contract ashistorical partial reports retain their recorded scoring semantics instead ofbeing silently reinterpreted.+### Which dimensions apply to each arm++Security is scored for every arm. Correctness uses `accuracy`, which is N/A+when `ground_truth` is missing. Effectiveness uses `goal_accuracy`+(`ground_truth`) and `behavior_check` (`expected_behavior`); each missing+reference makes its metric N/A. The dimension is N/A only if both metrics are+N/A.++| Arm | Security | Correctness | Discoverability | Effectiveness | Efficiency |+| --- | --- | --- | --- | --- | --- |+| With skill, `should_trigger=true` | Scored | Scored with `ground_truth`; otherwise N/A | Scored: the skill is present to discover and run | Scored for each present reference; N/A if both are missing | Scored: routing and tool use are evaluated |+| With skill, `should_trigger=false` (negative case) | Scored | Scored with `ground_truth`; otherwise N/A | Scored: `skill_execution` evaluates the negative case | Scored for each present reference; N/A if both are missing | Scored as 1.0 when no skill is expected to trigger |+| Without skill and no applicable alternate staged | Scored | Scored with `ground_truth`; otherwise N/A | N/A: no skill under test to discover or run | Scored for each present reference; N/A if both are missing | N/A: no skill under test to route to |+| Without skill, with an acceptable alternate staged | Scored | Scored with `ground_truth`; otherwise N/A | Scored when the staged skill matches a case candidate | Scored for each present reference; N/A if both are missing | Scored when the staged skill matches a case candidate |++For a positive case, an alternate candidate is the expected skill or an+acceptable alternate. For a negative case, the candidate is the evaluated+skill, or the expected skill when no evaluated skill is recorded. In either+case, the staged name must match a candidate for the no-skill arm's skill+metrics to apply.++Skill Lift per dimension and overall uses only cases and metrics both arms+scored; the `basis` block in `lift.json` records the shared comparison ([Skill+Lift reporting](reports.mdx#skill-lift)). A baseline is never credited or+penalized for a skill it was not given.+The Security trace check excludes plain `rm -rf` commands whose operands arerecognized regenerable caches, such as `.pytest_cache`, `__pycache__`, or`~/.cache/uv`. Absolute cache paths are recognized only under `/workspace/` and@@ -1085,10 +1113,7 @@ agent passes only when every dimension passes. The top-level Tier 3 verdict is:| Neutral | No agent passes, but at least one has no dimension below 0.40 || Fail | Every successful agent has a dimension below 0.40 |-Skill Lift is `with-skill overall score - without-skill overall score` for-comparable cases. Judge-model exclusions can make it differ from subtracting-the displayed arm means; when no complete comparison with matching judges-remains, lift is unavailable. It uses the same score units as the two arms;+Skill Lift compares Overall on the metrics and cases both arms scored, so it can differ from the displayed with-skill Overall minus baseline Overall when the baseline has N/A dimensions. Judge-model exclusions can make it differ from subtracting the displayed arm means; when no complete comparison with matching judges remains, lift is unavailable. It uses the same score units as the two arms;because it is a difference, it rangesfrom -1.0 to +1.0. Its independent band is **PASS** at +0.05 or above,**NEUTRAL** between -0.10 and +0.05, and **FAIL** at -0.10 or below. Lift isdiff --git a/src/skillevaluator/reporting/templates/report.html.j2 b/src/skillevaluator/reporting/templates/report.html.j2index fa6b4565..346f9202 100644--- a/src/skillevaluator/reporting/templates/report.html.j2+++ b/src/skillevaluator/reporting/templates/report.html.j2@@ -2639,6 +2639,8 @@<th>Verdict</th><th>Signals</th><th>Reasoning</th></tr></thead><tbody>+ {% set t3_best_agent = t3_agents.get(t3_best_name, {}) %}+ {% set baseline_na_metrics = t3_best_agent.get('not_applicable_metrics', {}).get('without_skill', []) %}{% for dim in tier3.dimensions %}{% set dim_score = dim.score if dim.score is not none else dim.with_skill %}{% set dim_color = 'var(--success-color)' if dim_score >= 0.7 else ('var(--warning-color)' if dim_score >= 0.4 else 'var(--danger-color)') %}@@ -2653,7 +2655,7 @@<div class="mini-bar"><span style="width:{{ (dim_score * 100) | round(0) }}%; background:{{ dim_color }};"></span></div></div></td>- <td>{{ "%.2f" | format(dim.baseline) if dim.baseline is not none else "N/A" }}</td>+ <td{% if dim.id == 'discoverability' and 'skill_execution' in baseline_na_metrics %} title="No-skill baseline: This arm has no skill under test installed, so there is no skill to discover or run"{% elif dim.id == 'efficiency' and 'skill_efficiency' in baseline_na_metrics %} title="No-skill baseline: This arm has no skill under test installed, so there is no skill to route to"{% endif %}>{{ "%.2f" | format(dim.baseline) if dim.baseline is not none else "N/A" }}</td><td style="color:{{ 'var(--success-color)' if dim.lift is not none and dim.lift >= tier3_lift_pass_threshold else ('var(--danger-color)' if dim.lift is not none and dim.lift <= tier3_lift_fail_threshold else 'var(--text-secondary)') }};font-weight:600;">{{ "%+.2f" | format(dim.lift) if dim.lift is not none else "N/A" }}</td>diff --git a/tests/reporting/test_unified_tier3_report.py b/tests/reporting/test_unified_tier3_report.pyindex bb8b7e83..b0add7a5 100644--- a/tests/reporting/test_unified_tier3_report.py+++ b/tests/reporting/test_unified_tier3_report.py@@ -612,6 +612,41 @@ def test_trial_report_renders_unavailable_tokens_as_unavailable() -> None:assert re.search(r"<td>4</td>\s*<td>—</td>", trials_html)+def test_no_skill_baseline_dimension_na_explains_why_in_html() -> None:+ payload = build_agent_eval_payload(+ "demo",+ {+ "codex": {+ "execution_status": "succeeded",+ "expected_attempts": 1,+ "scored_attempts": 1,+ "with_skill": dict.fromkeys(DEFAULT_METRICS, 1.0),+ "without_skill": {+ "security": 0.8,+ "accuracy": 0.7,+ "goal_accuracy": 0.6,+ "behavior_check": 0.8,+ },+ "not_applicable_validated_without_skill": True,+ "not_applicable_without_skill": ["skill_execution", "skill_efficiency"],+ }+ },+ use_llm_judge=False,+ )+ assert payload is not None++ html = _render_agent_payload(payload)++ assert (+ '<td title="No-skill baseline: This arm has no skill under test installed, '+ 'so there is no skill to discover or run">N/A</td>'+ ) in html+ assert (+ '<td title="No-skill baseline: This arm has no skill under test installed, '+ 'so there is no skill to route to">N/A</td>'+ ) in html++def test_trial_report_omits_token_chart_for_incomplete_agent_telemetry() -> None:payload = build_agent_eval_payload("demo",diff --git a/tests/tier3/test_baseline_dimension_applicability.py b/tests/tier3/test_baseline_dimension_applicability.pynew file mode 100644index 00000000..7f0ac8e5--- /dev/null+++ b/tests/tier3/test_baseline_dimension_applicability.py@@ -0,0 +1,115 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0+"""Per-arm applicability for the no-skill baseline."""++from __future__ import annotations++from typing import Any++from skillevaluator.constants import DEFAULT_SCORE_POLICY+from skillevaluator.tier3.harbor import metrics+from skillevaluator.tier3.harbor.collector import _compute_lift+from skillevaluator.tier3.harbor.templates import eval as verifier+++def reward(case: str, **overrides: float | None) -> dict[str, Any]:+ result = dict.fromkeys(metrics.DEFAULT_METRICS, 1.0)+ result.update(+ metric_set=metrics.DEFAULT_METRIC_SET,+ score_policy=DEFAULT_SCORE_POLICY,+ entry_id=case,+ details={},+ )+ for metric, value in overrides.items():+ result[metric] = value+ if value is None:+ result["details"][metric] = {+ "score": None,+ "status": "not_applicable",+ "reason": "N/A: this metric has no applicable reference or skill",+ }+ return result+++def test_skill_metrics_apply_to_skill_and_matching_acceptable_alternate():+ assert not verifier.skill_metrics_applicable(+ False,+ True,+ "target-skill",+ workspace_skill_names=[],+ )+ assert verifier.skill_metrics_applicable(True, True, "target-skill")+ assert verifier.skill_metrics_applicable(+ False,+ True,+ "target-skill",+ acceptable_skills=["alternate-skill"],+ workspace_skill_names=["alternate-skill"],+ )+++def test_reward_overall_averages_three_scored_dimensions_when_skill_metrics_are_na():+ result = {+ "security": 0.8,+ "accuracy": 0.6,+ "skill_execution": None,+ "goal_accuracy": 0.4,+ "behavior_check": 0.8,+ "skill_efficiency": None,+ }+ details = {+ metric: {+ "score": None,+ "status": "not_applicable",+ "reason": verifier.SKILL_METRICS_NOT_APPLICABLE_REASON,+ }+ for metric in ("skill_execution", "skill_efficiency")+ }++ assert verifier._reward_overall(result, details) == 0.6667+++def test_lift_basis_excludes_baseline_na_skill_metrics_but_pairs_other_metrics():+ with_skill = reward(+ "case-1",+ security=0.9,+ accuracy=0.8,+ skill_execution=0.7,+ goal_accuracy=0.6,+ behavior_check=0.8,+ skill_efficiency=0.9,+ )+ without_skill = reward(+ "case-1",+ security=0.7,+ accuracy=0.6,+ skill_execution=None,+ goal_accuracy=0.4,+ behavior_check=0.6,+ skill_efficiency=None,+ )+ lift = _compute_lift(+ metrics.average_metrics([with_skill])[0],+ metrics.average_metrics([without_skill])[0],+ [with_skill],+ [without_skill],+ ["case-1"],+ score_policy=DEFAULT_SCORE_POLICY,+ )++ assert "skill_execution" not in lift+ assert "skill_efficiency" not in lift+ excluded = {item["metric"]: item for item in lift["basis"]["excluded_metrics"]}+ assert excluded["skill_execution"] == {+ "metric": "skill_execution",+ "reason": "not_applicable",+ "arms": ["without_skill"],+ }+ assert excluded["skill_efficiency"] == {+ "metric": "skill_efficiency",+ "reason": "not_applicable",+ "arms": ["without_skill"],+ }+ for metric in ("security", "accuracy", "goal_accuracy", "behavior_check"):+ assert lift[metric]["delta"] == 0.2+ assert lift["basis"]["metrics"][metric]["cases"] == 1