Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier3): apply case-specific skill metric applicability across reports (#425)
Narendran Raghavan88506620b2c4
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 19f10569..f5142dbe 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -562,6 +562,34 @@ All notable changes to SkillEvaluator are documented in this file.reasons for an N/A Accuracy, Goal Accuracy or Behavior Check now name thedataset fields `expected_output` and `assertions` before their legacy names.+- Tier 3 no longer scores a with-skill skill metric `1.0` when the eval case+ gives it nothing to score. `skill_efficiency` (Efficiency) is now N/A for a+ negative case, such as the one `create-eval-dataset --full` writes, and both+ `skill_execution` (Discoverability) and `skill_efficiency` are N/A for a case+ that is not negative and names no `expected_skill`, such as a legacy case+ without `expected_skill` and `should_trigger`. The same holds in a baseline+ arm that stages a skill the case could activate. The trial overall, pass@k+ and the arm's averages leave N/A metrics out, so absolute with-skill scores+ of these cases change: a negative case scoring 1.0 on Security and+ Discoverability and 0.8 on Correctness and Effectiveness now has an overall+ of 0.90, not 0.92. If no case is a positive case that names an+ `expected_skill`, for example when every case is negative, Efficiency has no+ with-skill score and the agent cannot PASS; Discoverability has none either+ unless some case is negative. A trial without a trajectory keeps these+ metrics N/A too. A positive case with no tool calls still scores `0.0`, and+ Skill Lift is unchanged when the baseline has no skill it could activate,+ because both metrics are already N/A there. The HTML report now explains a+ with-skill N/A skill metric by the eval case, not by a missing skill, and a+ multi-step trial's collected reward keeps the N/A reason its steps' verifier+ recorded.++- The per-arm Tier 3 applicability table reflects negative-case Efficiency N/A+ and the expected-skill requirement for positive-case skill metrics.++- Tier 3 baseline N/A explanations cover case-level applicability when a staged+ skill is available, including negative cases, instead of claiming no skill is+ installed. Dimension tooltips use the same explanation as evaluator summaries.+- `skillevaluator tier3 feedback` now writes the evaluated skill's directory(known from `--skill` or a run in the skill's `evals/results`) as its name andthe home directory as `~` in the evidence it sends to the LLM, so an errordiff --git a/docs/eval-datasets.mdx b/docs/eval-datasets.mdxindex 21db8b51..3ab3bf02 100644--- a/docs/eval-datasets.mdx+++ b/docs/eval-datasets.mdx@@ -155,6 +155,12 @@ checked because that skill is not installed.| `expected_behavior` (recommended) | `assertions` || `expected_skill` (recommended) | `expected_skill` (unchanged) || `acceptable_alternates` | `acceptable_skills` |++ A legacy entry without `expected_skill` and `should_trigger` has no skill+ to discover or measure, so its Discoverability and Efficiency are N/A. If+ no entry is a positive case that names an `expected_skill`, the agent+ cannot PASS. See+ [Reading results](/skills/skillevaluator/tier3-live-evaluation#reading-results).</Accordion></AccordionGroup>diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdxindex 611a59b4..7015b8c2 100644--- a/docs/tier3-live-evaluation.mdx+++ b/docs/tier3-live-evaluation.mdx@@ -1176,7 +1176,23 @@ A metric can also be N/A (not applicable) for an eval case. It never counts asreject an empty `expected_output`, so with them only legacy cases without`ground_truth` make `accuracy` and `goal_accuracy` N/A.- `skill_execution` and `skill_efficiency` are N/A in a baseline arm that has- no skill it could activate.+ no skill it could activate. A baseline arm that stages a skill the case could+ activate is scored on them, and the case rules below apply to it too.+- `skill_efficiency` is N/A for a negative case (`should_trigger: false`, or a+ `null` or empty `expected_skill` without `should_trigger`, which is how+ `create-eval-dataset` writes one), because the case expects no skill use to+ measure. `skill_execution` still scores a negative case: it checks that the+ agent left the skill under test alone.+- Both are N/A for a case that is not negative and names no `expected_skill`: a+ legacy case without `expected_skill` and `should_trigger`, or one with+ `should_trigger: true` and no `expected_skill`. In the agentskills.io shape a+ case without `expected_skill` gets the dataset's `skill_name`.+- A positive case whose agent made no tool calls still scores `0.0` on both.+- Efficiency has a with-skill score only when some case is positive and names+ an `expected_skill`. Without such a case the agent cannot PASS (see below),+ for example when every case is negative, as in a run filtered to the negative+ cases, or when no case names an `expected_skill`. Discoverability then has no+ with-skill score either, unless some case is negative.- Security is never N/A.In `skill_evaluator_reward.json` an N/A metric's value is `null`, and@@ -1349,16 +1365,19 @@ N/A.| Arm | Security | Correctness | Discoverability | Effectiveness | Efficiency || --- | --- | --- | --- | --- | --- |-| With skill, `should_trigger=true` | Scored | Scored with `ground_truth`; otherwise N/A | Scored: the skill is present to discover and run | Scored for each present reference; N/A if both are missing | Scored: routing and tool use are evaluated |-| With skill, `should_trigger=false` (negative case) | Scored | Scored with `ground_truth`; otherwise N/A | Scored: `skill_execution` evaluates the negative case | Scored for each present reference; N/A if both are missing | Scored as 1.0 when no skill is expected to trigger |+| With skill, `should_trigger=true` | Scored | Scored with `ground_truth`; otherwise N/A | Scored when `expected_skill` names a skill; otherwise N/A | Scored for each present reference; N/A if both are missing | Scored when `expected_skill` names a skill; otherwise N/A |+| With skill, `should_trigger=false` (negative case) | Scored | Scored with `ground_truth`; otherwise N/A | Scored: `skill_execution` evaluates the negative case | Scored for each present reference; N/A if both are missing | N/A: a negative case has no skill use to measure || Without skill and no applicable alternate staged | Scored | Scored with `ground_truth`; otherwise N/A | N/A: no skill under test to discover or run | Scored for each present reference; N/A if both are missing | N/A: no skill under test to route to |-| Without skill, with an acceptable alternate staged | Scored | Scored with `ground_truth`; otherwise N/A | Scored when the staged skill matches a case candidate | Scored for each present reference; N/A if both are missing | Scored when the staged skill matches a case candidate |+| Without skill, with an acceptable alternate staged | Scored | Scored with `ground_truth`; otherwise N/A | Scored when the staged skill matches a case candidate and the case is negative or names an `expected_skill`; otherwise N/A | Scored for each present reference; N/A if both are missing | Scored for positive cases that name an `expected_skill` and whose staged skill matches a case candidate; otherwise N/A |For a positive case, an alternate candidate is the expected skill or anacceptable alternate. For a negative case, the candidate is the evaluatedskill, or the expected skill when no evaluated skill is recorded. In eithercase, the staged name must match a candidate for the no-skill arm's skill-metrics to apply.+metrics to apply. The eval-case rules still apply after matching: a negative+case always leaves Efficiency N/A, and a positive case without `expected_skill`+leaves both skill metrics N/A. A case with neither `expected_skill` nor a+resolved `should_trigger` label also leaves both skill metrics N/A.Skill Lift per dimension and overall uses only cases and metrics both armsscored; the `basis` block in `lift.json` records the shared comparison ([Skilldiff --git a/src/skillevaluator/evaluation/tier3_report.py b/src/skillevaluator/evaluation/tier3_report.pyindex 774cad8b..97cd0b55 100644--- a/src/skillevaluator/evaluation/tier3_report.py+++ b/src/skillevaluator/evaluation/tier3_report.py@@ -1835,15 +1835,19 @@ def _not_applicable_evaluators(metrics: list[str],with_scores: dict[str, Any],not_applicable: list[str],+ *,+ with_skill: bool = True,) -> list[dict[str, str]]:- """Evaluators with no with-skill score because no eval case gave them a reference."""- from skillevaluator.tier3.harbor.metrics import METRIC_DISPLAY, NOT_APPLICABLE_REASONS+ """Evaluators with no score in the reported arm because every trial recorded them as N/A."""+ from skillevaluator.tier3.harbor.metrics import METRIC_DISPLAY, not_applicable_reasonreturn [{"id": metric,"label": METRIC_DISPLAY.get(metric, metric.replace("_", " ").title()),- "reason": NOT_APPLICABLE_REASONS.get(metric, "Not applicable to these eval cases"),+ "reason": not_applicable_reason(+ metric, with_skill=with_skill, default="Not applicable to these eval cases"+ ),}for metric in metricsif metric in not_applicable and _finite_float(with_scores.get(metric)) is None@@ -2169,7 +2173,7 @@ def _build_agent(# N/A notes describe the scored arm: the baseline in a baseline-only run.if baseline_only:not_applicable_evaluators = _not_applicable_evaluators(- baseline_metrics, without_scores, baseline_not_applicable+ baseline_metrics, without_scores, baseline_not_applicable, with_skill=False)not_applicable_dimensions = _not_applicable_dimensions(dimensions, baseline_not_applicable)else:diff --git a/src/skillevaluator/reporting/html.py b/src/skillevaluator/reporting/html.pyindex b116bf12..7f23b21b 100644--- a/src/skillevaluator/reporting/html.py+++ b/src/skillevaluator/reporting/html.py@@ -37,7 +37,7 @@ from pathlib import Pathfrom typing import TYPE_CHECKING, Any, ClassVarfrom urllib.parse import urlparse-from jinja2 import BaseLoader, Environment+from jinja2 import BaseLoader, Environment, pass_contextfrom skillevaluator import __version__from skillevaluator.constants import (@@ -501,11 +501,12 @@ def _tier3_score_reasons(value: object) -> dict[str, str]:return reasons-def _tier3_not_applicable_reason(metric: object) -> str:- """Return why a Tier 3 metric can be N/A, for the tooltip of an N/A drill-down cell."""- from skillevaluator.tier3.harbor.metrics import NOT_APPLICABLE_REASONS+@pass_context+def _tier3_not_applicable_reason(_context: Any, metric: object, with_skill: bool = False) -> str:+ """Describe N/A metrics at render time, keeping Tier 3 imports out of base template compilation."""+ from skillevaluator.tier3.harbor.metrics import not_applicable_reason- return NOT_APPLICABLE_REASONS.get(str(metric), "Not applicable to this eval case")+ return not_applicable_reason(str(metric), with_skill=with_skill is True)def _tier3_text_mapping(value: object) -> dict[str, str]:diff --git a/src/skillevaluator/reporting/templates/report.html.j2 b/src/skillevaluator/reporting/templates/report.html.j2index 5fbc2ed2..8e408a8e 100644--- a/src/skillevaluator/reporting/templates/report.html.j2+++ b/src/skillevaluator/reporting/templates/report.html.j2@@ -2792,7 +2792,7 @@<div class="t3-dim-hint">{{ "with-skill arm not run" if t3_baseline_only else "not scored with the skill" }}</div>{% endif %}</td>- <td{% if dim.id == 'discoverability' and 'skill_execution' in baseline_na_metrics %} title="No-skill baseline: This arm has no skill under test installed, so there is no skill to discover or run"{% elif dim.id == 'efficiency' and 'skill_efficiency' in baseline_na_metrics %} title="No-skill baseline: This arm has no skill under test installed, so there is no skill to route to"{% endif %}>+ <td{% if dim.id == 'discoverability' and 'skill_execution' in baseline_na_metrics %} title="No-skill baseline: {{ 'skill_execution' | not_applicable_reason }}"{% elif dim.id == 'efficiency' and 'skill_efficiency' in baseline_na_metrics %} title="No-skill baseline: {{ 'skill_efficiency' | not_applicable_reason }}"{% endif %}>{% if dim.baseline is not none %}{{ "%.2f" | format(dim.baseline) }}{% else %}N/A<div class="t3-dim-hint">{{ "baseline not scored" if t3_best and t3_best.baseline is not none else "baseline not run" }}</div>{% endif %}</td>@@ -3432,7 +3432,7 @@{% for eval_id in t3_eval_ids %}{% set s = scores.get(eval_id) %}{% set cell_na = s is none and eval_id in (trial.not_applicable_metrics or []) %}- {% set cell_reason = reasons.get(eval_id) if s is not none else (("Not applicable: " ~ (eval_id | not_applicable_reason)) if cell_na else none) %}+ {% set cell_reason = reasons.get(eval_id) if s is not none else (("Not applicable: " ~ (eval_id | not_applicable_reason(row.condition == 'With skill'))) if cell_na else none) %}{% set cell_text = "%.2f" | format(s) if s is not none else ("N/A" if cell_na else "—") %}<td style="{% if s is not none %}background:{{ 'rgba(34,197,94,0.14)' if s >= 0.7 else ('rgba(234,179,8,0.14)' if s >= 0.4 else 'rgba(239,68,68,0.14)') }}; color:{{ 'var(--success-color)' if s >= 0.7 else ('var(--warning-color)' if s >= 0.4 else 'var(--danger-color)') }}; font-weight:600;{% elif cell_na %}color:var(--text-secondary);{% endif %}"{% if cell_na %} data-not-applicable="true"{% endif %}{% if cell_reason %} title="{{ eval_id | replace('_', ' ') | title }}: {{ cell_reason }}"{% endif %}>{% if cell_reason %}<span class="t3-score-reason">{{ cell_text }}</span>{% else %}{{ cell_text }}{% endif %}</td>{% endfor %}diff --git a/src/skillevaluator/tier3/eval_core/checks.py b/src/skillevaluator/tier3/eval_core/checks.pyindex de306fbc..b532f9c4 100644--- a/src/skillevaluator/tier3/eval_core/checks.py+++ b/src/skillevaluator/tier3/eval_core/checks.py@@ -5647,6 +5647,41 @@ def check_token_efficiency(total_tokens: int) -> dict[str, Any]:# Composite scorers (combine sub-checks into eval-level scores)# ---------------------------------------------------------------------------+# Why an eval case leaves a skill metric N/A. The verifier records the reason+# under ``details.<metric>`` with ``status: "not_applicable"`` and a null score.+SKILL_CASE_UNLABELED_REASON = "N/A: no expected_skill or should_trigger defined for this eval case"+SKILL_CASE_NO_EXPECTED_SKILL_REASON = "N/A: no expected_skill defined for this eval case"+SKILL_CASE_NEGATIVE_EFFICIENCY_REASON = "N/A: negative case, so there is no skill use to measure"+SKILL_CASE_NEGATIVE_NO_SKILL_REASON = "N/A: negative case with no skill under test to check"+++def skill_execution_not_applicable_reason(should_trigger: bool | None, expected_skill: str | None) -> str | None:+ """Return why an eval case gives ``skill_execution`` nothing to score, or ``None`` when it applies.++ A negative case still scores: it checks that the agent left the skill under test alone.+ """+ if should_trigger is None:+ return SKILL_CASE_UNLABELED_REASON+ if should_trigger and not expected_skill:+ return SKILL_CASE_NO_EXPECTED_SKILL_REASON+ return None+++def skill_efficiency_not_applicable_reason(should_trigger: bool | None, expected_skill: str | None) -> str | None:+ """Return why an eval case gives ``skill_efficiency`` nothing to measure, or ``None`` when it applies."""+ if should_trigger is None:+ return SKILL_CASE_UNLABELED_REASON+ if not should_trigger:+ return SKILL_CASE_NEGATIVE_EFFICIENCY_REASON+ if not expected_skill:+ return SKILL_CASE_NO_EXPECTED_SKILL_REASON+ return None+++def _case_not_applicable(reason: str) -> dict[str, Any]:+ """A scoreless composite result whose details carry the verifier's N/A marker."""+ return {"score": None, "details": {"score": None, "status": "not_applicable", "reason": reason}}+def score_skill_execution(tool_calls: list[dict[str, Any]],@@ -5670,7 +5705,7 @@ def score_skill_execution("""expected_script = normalize_optional_reference(expected_script)if should_trigger is None:- return {"score": 1.0, "details": {"message": "No expected_skill -- skipped"}}+ return _case_not_applicable(SKILL_CASE_UNLABELED_REASON)if not should_trigger:if require_evaluated_skill and not evaluated_skill:@@ -5683,7 +5718,7 @@ def score_skill_execution(}skill_under_test = evaluated_skill or expected_skillif not skill_under_test:- return {"score": 1.0, "details": {"message": "Negative case, no skill identified"}}+ return _case_not_applicable(SKILL_CASE_NEGATIVE_NO_SKILL_REASON)if not tool_calls:neg = {"passed": True, "score": 1.0, "reason": "No tool calls"}else:@@ -5691,7 +5726,7 @@ def score_skill_execution(return {"score": neg["score"], "details": {"negative_check": neg, "should_trigger": False}}if not expected_skill:- return {"score": 1.0, "details": {"message": "No expected_skill -- skipped"}}+ return _case_not_applicable(SKILL_CASE_NO_EXPECTED_SKILL_REASON)checks: dict[str, dict[str, Any]] = {}scores: list[float] = []@@ -5741,7 +5776,7 @@ def score_skill_efficiency(tool_calls: list[dict[str, Any]],expected_skill: str,expected_script: str | None = None,- should_trigger: bool = True,+ should_trigger: bool | None = True,*,skill_tool_names: list[str] | None = None,workspace_skill_names: list[str] | None = None,@@ -5751,10 +5786,12 @@ def score_skill_efficiency("""Compute the ``skill_efficiency`` eval score (average of sub-checks)."""expected_script = normalize_optional_reference(expected_script)if not should_trigger:- return {"score": 1.0, "details": {"message": "Negative case -- efficiency not applicable"}}+ return _case_not_applicable(+ SKILL_CASE_UNLABELED_REASON if should_trigger is None else SKILL_CASE_NEGATIVE_EFFICIENCY_REASON+ )if not expected_skill:- return {"score": 1.0, "details": {"message": "No expected_skill -- skipped"}}+ return _case_not_applicable(SKILL_CASE_NO_EXPECTED_SKILL_REASON)if not tool_calls:return {"score": 0.0, "details": {"message": "No tool calls in trajectory"}}diff --git a/src/skillevaluator/tier3/harbor/activation.py b/src/skillevaluator/tier3/harbor/activation.pyindex b23f5501..1223ffdb 100644--- a/src/skillevaluator/tier3/harbor/activation.py+++ b/src/skillevaluator/tier3/harbor/activation.py@@ -12,6 +12,7 @@ from typing import Anyfrom skillevaluator.tier3.harbor.metrics import (DEFAULT_METRIC_SET,LEGACY_METRIC_SET,+ NOT_APPLICABLE_STATUS,metric_set_for_reward,metric_value,)@@ -23,13 +24,19 @@ def _activation_state(reward: Mapping[str, Any]) -> str:if not isinstance(execution, Mapping):return "unknown"message = execution.get("message")- if execution.get("should_trigger") is False or (- isinstance(message, str)- and message- in {- "No expected_skill -- skipped",- "Negative case, no skill identified",- }+ # The verifier records an N/A skill_execution with ``status``; older rewards+ # used one of the legacy messages below for the same cases.+ if (+ execution.get("should_trigger") is False+ or execution.get("status") == NOT_APPLICABLE_STATUS+ or (+ isinstance(message, str)+ and message+ in {+ "No expected_skill -- skipped",+ "Negative case, no skill identified",+ }+ )):return "not_applicable"standard_sets = {DEFAULT_METRIC_SET, LEGACY_METRIC_SET}diff --git a/src/skillevaluator/tier3/harbor/collector.py b/src/skillevaluator/tier3/harbor/collector.pyindex a2951f63..3c203318 100644--- a/src/skillevaluator/tier3/harbor/collector.py+++ b/src/skillevaluator/tier3/harbor/collector.py@@ -61,10 +61,12 @@ from skillevaluator.tier3.harbor.metrics import (not_applicable_metrics,overall_score,overall_score_from_metrics,+ recorded_not_applicable_reason,rewards_have_mixed_metric_contracts,score_definition,score_policy_for_metrics,score_value,+ shared_not_applicable_reason,)from skillevaluator.tier3.output_provenance import write_output_file_atomicallyfrom skillevaluator.tier3.toml_utils import extract_toml_metadata_entry_id@@ -3409,7 +3411,7 @@ def _restore_constituent_not_applicable(return resultroot_rewards = dict(original)steps = []- for _name, step in zip(names, result["step_results"], strict=True):+ for name, step in zip(names, result["step_results"], strict=True):verifier = step.get("verifier_result")rewards = verifier.get("rewards") if isinstance(verifier, dict) else Noneif not isinstance(rewards, dict):@@ -3419,12 +3421,14 @@ def _restore_constituent_not_applicable(# A conflicting numeric value remains authoritative and prevents restoration.if metric in rewards and not metric_is_not_applicable(rewards, metric):return result- mark_not_applicable(rewards, metric)+ # Keep the reason the step's verifier recorded, such as a negative eval case.+ mark_not_applicable(rewards, metric, recorded_not_applicable_reason(by_name[name], metric))steps.append({**step, "verifier_result": {**verifier, "rewards": rewards}})for metric in restored:- mark_not_applicable(root_rewards, metric)+ reason = shared_not_applicable_reason([by_name[name] for name in names], metric)+ mark_not_applicable(root_rewards, metric, reason)if metric not in data:- mark_not_applicable(data, metric)+ mark_not_applicable(data, metric, reason)return {**result, "verifier_result": {**root_verifier, "rewards": root_rewards}, "step_results": steps}@@ -4651,7 +4655,7 @@ def _logical_attempt_rewards(rewards: list[dict[str, Any]]) -> list[dict[str, Anif metric in standard_scores:logical_reward[metric] = standard_scores[metric]for metric in not_applicable_metrics(rows, metrics):- mark_not_applicable(logical_reward, metric)+ mark_not_applicable(logical_reward, metric, shared_not_applicable_reason(rows, metric))logical_reward.update(custom_scores)if custom_scores:logical_reward["custom_metrics"] = custom_scoresdiff --git a/src/skillevaluator/tier3/harbor/metrics.py b/src/skillevaluator/tier3/harbor/metrics.pyindex a88fc3ee..ace6f0c3 100644--- a/src/skillevaluator/tier3/harbor/metrics.py+++ b/src/skillevaluator/tier3/harbor/metrics.py@@ -96,7 +96,8 @@ DIMENSION_QUESTIONS = {# judge against (no expected_output / assertions, which the legacy format calls# ground_truth / expected_behavior). The skill metrics are not applicable in an# arm that has no skill under test to discover or route to (the no-skill or-# no-plugin baseline). The verifier records the metric as+# no-plugin baseline), and for an eval case that names no expected_skill;+# skill_efficiency also for a negative case. The verifier records the metric as# ``null`` with ``details[metric].status == "not_applicable"``; N/A metrics are# excluded from the trial overall, the arm averages, and lift. Security never# becomes N/A, and an absent or non-finite metric without the explicit marker@@ -112,6 +113,35 @@ NOT_APPLICABLE_REASONS = {),"behavior_check": "No assertions (legacy expected_behavior) defined, so there is no workflow to check",}+# A reason carried over from a verifier sidecar is cut to this length.+NOT_APPLICABLE_REASON_MAX_CHARS = 512+# With the skill under test installed, a skill metric is N/A only because of the eval case.+WITH_SKILL_NOT_APPLICABLE_REASONS = {+ "skill_execution": "No expected_skill defined, so the eval case names no skill to discover or run",+ "skill_efficiency": "Negative case or no expected_skill defined, so there is no skill use to measure",+}+++# Reporting may have only an arm-level N/A marker, without the verifier's reason.+# A baseline that stages a matching candidate still follows the eval-case rules.+BASELINE_NOT_APPLICABLE_REASONS = {+ "skill_execution": ("No applicable skill is available in this arm, or the eval case names no expected_skill"),+ "skill_efficiency": (+ "No applicable skill is available in this arm, or the case is negative or names no expected_skill"+ ),+}+++def not_applicable_reason(+ metric: str, *, with_skill: bool = False, default: str = "Not applicable to this eval case"+) -> str:+ """Return why *metric* can be N/A in an arm with or without the skill under test."""+ if with_skill and metric in WITH_SKILL_NOT_APPLICABLE_REASONS:+ return WITH_SKILL_NOT_APPLICABLE_REASONS[metric]+ if not with_skill and metric in BASELINE_NOT_APPLICABLE_REASONS:+ return BASELINE_NOT_APPLICABLE_REASONS[metric]+ return NOT_APPLICABLE_REASONS.get(metric, default)+_RESERVED_METADATA_KEYS = {"custom_details",@@ -357,8 +387,36 @@ def not_applicable_counts(return counts-def mark_not_applicable(reward: dict[str, Any], metric: str) -> None:- """Record *metric* as N/A on *reward* using the verifier's marker shape."""+def _bounded_reason(value: object) -> str | None:+ """Return a non-empty reason string cut to ``NOT_APPLICABLE_REASON_MAX_CHARS``, or ``None``."""+ if not isinstance(value, str) or not value.strip():+ return None+ return value.strip()[:NOT_APPLICABLE_REASON_MAX_CHARS]+++def recorded_not_applicable_reason(reward: dict[str, Any], metric: str) -> str | None:+ """Return the reason *reward* records for its N/A *metric*, or ``None`` when it records none."""+ details = reward.get("details")+ detail = details.get(metric) if isinstance(details, dict) else None+ if not isinstance(detail, dict) or detail.get("status") != NOT_APPLICABLE_STATUS:+ return None+ return _bounded_reason(detail.get("reason"))+++def shared_not_applicable_reason(rewards: list[dict[str, Any]], metric: str) -> str | None:+ """Return the distinct reasons *rewards* record for their N/A *metric*, joined, or ``None``."""+ reasons = [reason for reward in rewards if (reason := recorded_not_applicable_reason(reward, metric))]+ return _bounded_reason("; ".join(dict.fromkeys(reasons)))+++def mark_not_applicable(reward: dict[str, Any], metric: str, reason: str | None = None) -> None:+ """Record *metric* as N/A on *reward* using the verifier's marker shape.++ A reason the reward already records for its N/A metric is kept. Otherwise+ *reason*, such as the one the verifier recorded for the eval case, is used,+ and without one the generic reason for the metric.+ """+ kept = recorded_not_applicable_reason(reward, metric) or _bounded_reason(reason)reward[metric] = Nonedetails = reward.get("details")if not isinstance(details, dict):@@ -370,7 +428,7 @@ def mark_not_applicable(reward: dict[str, Any], metric: str) -> None:{"score": None,"status": NOT_APPLICABLE_STATUS,- "reason": NOT_APPLICABLE_REASONS.get(metric, "Not applicable to this eval case"),+ "reason": kept or NOT_APPLICABLE_REASONS.get(metric, "Not applicable to this eval case"),})details[metric] = detaildiff --git a/src/skillevaluator/tier3/harbor/templates/eval.py b/src/skillevaluator/tier3/harbor/templates/eval.pyindex f9370156..7ad5e719 100644--- a/src/skillevaluator/tier3/harbor/templates/eval.py+++ b/src/skillevaluator/tier3/harbor/templates/eval.py@@ -308,6 +308,8 @@ normalize_optional_reference = _shared_checks.normalize_optional_referencenormalize_expected_behaviors = _shared_checks.normalize_expected_behaviorsresolve_should_trigger = _shared_checks.resolve_should_triggerscore_skill_execution = _shared_checks.score_skill_execution+skill_efficiency_not_applicable_reason = _shared_checks.skill_efficiency_not_applicable_reason+skill_execution_not_applicable_reason = _shared_checks.skill_execution_not_applicable_reason# All 148 constant initializers are AST-identical at the fixed base; shared# constants added since are aliased here too.@@ -3838,9 +3840,22 @@ def skill_metrics_applicable(return any(str(name).strip().lower() in staged for name in candidates if name and str(name).strip())-def _skill_metric_not_applicable():- """Scoreless ``skill_execution`` / ``skill_efficiency`` result for an arm without the skill."""- return {"score": None, "status": NOT_APPLICABLE_STATUS, "reason": SKILL_METRICS_NOT_APPLICABLE_REASON}+def _skill_metric_not_applicable(reason=SKILL_METRICS_NOT_APPLICABLE_REASON):+ """Scoreless ``skill_execution`` / ``skill_efficiency`` result.++ The default reason covers an arm without the skill. An eval case that gives+ the metric nothing to score passes its own reason.+ """+ return {"score": None, "status": NOT_APPLICABLE_STATUS, "reason": reason}+++def _case_skill_metric_reasons(should_trigger, expected_skill):+ """Return the skill metrics this eval case leaves N/A, each with its reason."""+ reasons = {+ "skill_execution": skill_execution_not_applicable_reason(should_trigger, expected_skill),+ "skill_efficiency": skill_efficiency_not_applicable_reason(should_trigger, expected_skill),+ }+ return {metric: reason for metric, reason in reasons.items() if reason}def _reward_overall(result, details):@@ -4003,6 +4018,11 @@ def main():skill_execution=_skill_metric_not_applicable(),skill_efficiency=_skill_metric_not_applicable(),)+ else:+ # A skill metric the case gives nothing to score stays N/A, as on the judged path.+ for metric, na_reason in _case_skill_metric_reasons(should_trigger, expected_skill).items():+ skill_scores[metric] = None+ judge_details[metric] = _skill_metric_not_applicable(na_reason)result = {"security": 0,**skill_scores,@@ -4113,8 +4133,10 @@ def main():sef_score = Nonedetails["skill_efficiency"] = _skill_metric_not_applicable()elif not should_trigger or not expected_skill:- sef_score = 1.0- details["skill_efficiency"] = {"message": "Skipped (negative or no expected_skill)"}+ sef_score = None+ details["skill_efficiency"] = _skill_metric_not_applicable(+ skill_efficiency_not_applicable_reason(should_trigger, expected_skill)+ )elif not tool_calls:sef_score = 0.0details["skill_efficiency"] = {"message": "No tool calls in trajectory"}diff --git a/tests/reporting/test_applicability_saved_runs.py b/tests/reporting/test_applicability_saved_runs.pyindex 885ca200..5ced1e34 100644--- a/tests/reporting/test_applicability_saved_runs.py+++ b/tests/reporting/test_applicability_saved_runs.py@@ -438,3 +438,196 @@ def test_na_evaluator_reasons_name_the_dataset_field_and_its_legacy_name(tmp_patreward: dict = {}mark_not_applicable(reward, metric)assert reward["details"][metric]["reason"] == reasons[metric]+++def test_with_skill_na_skill_metrics_name_the_eval_case_not_a_missing_skill(tmp_path):+ # Cases that name no expected_skill leave both skill metrics N/A even with the skill installed.+ skill_metrics = ("skill_execution", "skill_efficiency")+ root = _uniform_run(tmp_path, with_na=skill_metrics, baseline_na=skill_metrics)++ payload = _report(root)++ reasons = {item["id"]: item["reason"] for item in payload["not_applicable_evaluators"]}+ assert reasons == {+ "skill_execution": "No expected_skill defined, so the eval case names no skill to discover or run",+ "skill_efficiency": "Negative case or no expected_skill defined, so there is no skill use to measure",+ }+ dimensions = {row["id"] for row in payload["agents"]["codex"]["dimensions"]}+ assert {"discoverability", "efficiency"}.isdisjoint(dimensions)+ assert {item["id"] for item in payload["not_applicable_dimensions"]} == {"discoverability", "efficiency"}+ # No with-skill Discoverability or Efficiency score, so the agent cannot PASS.+ assert payload["verdict"] == VERDICT_NEUTRAL+++SKILL_METRICS = ("skill_execution", "skill_efficiency")+++def _trial_row(case, *, has_skill, na=(), reason="N/A: no skill under test installed", **scores):+ """A verifier reward scoring 0.5 on every metric but *scores*, with *na* recorded as N/A."""+ row = dict.fromkeys(DEFAULT_METRICS, 0.5)+ row.update(scores)+ row.update(entry_id=case, metric_set=DEFAULT_METRIC_SET, score_policy=DEFAULT_SCORE_POLICY, has_skill=has_skill)+ row["details"] = {}+ for metric in na:+ row[metric] = None+ row["details"][metric] = {"score": None, "status": "not_applicable", "reason": reason}+ return row+++def _write_job(job, rows):+ """Write a Harbor job with one trial per row: numeric ``reward.json`` plus the verifier's sidecar."""+ trial_names = []+ for row in rows:+ trial = f"{row['entry_id']}__attempt1"+ trial_names.append(trial)+ verifier = job / trial / "verifier"+ verifier.mkdir(parents=True)+ numeric = {key: value for key, value in row.items() if isinstance(value, float)}+ numeric["overall"] = overall_score(row)+ (verifier / "reward.json").write_text(json.dumps(numeric), encoding="utf-8")+ (verifier / "skill_evaluator_reward.json").write_text(json.dumps(row), encoding="utf-8")+ stats = {+ "n_completed_trials": len(trial_names),+ "n_errored_trials": 0,+ "n_running_trials": 0,+ "n_pending_trials": 0,+ "n_cancelled_trials": 0,+ "n_retries": 0,+ "evals": {+ "agent__model___harbor-tasks": {+ "n_trials": len(trial_names),+ "n_errors": 0,+ "reward_stats": {"reward": {"0.1": trial_names}},+ }+ },+ }+ (job / "result.json").write_text(json.dumps({"n_total_trials": len(trial_names), "stats": stats}), encoding="utf-8")+++def _collect(tmp_path, cases, **options):+ from skillevaluator.tier3.harbor.collector import collect_harbor_results++ collect_harbor_results(+ skill_name="demo",+ agents=["codex"],+ output_dir=tmp_path / "results",+ jobs_dir=tmp_path / "jobs",+ expected_cases=len(cases),+ expected_case_ids=list(cases),+ expected_trials=len(cases),+ **options,+ )+ return _report(tmp_path / "results")++[email protected]("baseline_only", [True, False], ids=["baseline-only", "paired-arms"])[email protected]("view", ["saved-report", "html"])+def test_staged_negative_baseline_na_explains_case_context_in_saved_report_and_html(tmp_path, baseline_only, view):+ from skillevaluator.models import ValidationResult+ from skillevaluator.reporting import HTMLReporter+ from skillevaluator.tier3.eval_core import checks+ from skillevaluator.tier3.harbor.templates.eval import skill_metrics_applicable++ entry = {"has_skill": False, "workspace_skill_names": ["demo"], "evaluated_skill": "demo", "expected_skill": None}+ should_trigger = checks.resolve_should_trigger(entry)+ assert skill_metrics_applicable(+ entry["has_skill"],+ should_trigger,+ entry["expected_skill"],+ evaluated_skill=entry["evaluated_skill"],+ workspace_skill_names=entry["workspace_skill_names"],+ )+ execution = checks.score_skill_execution(+ [], entry["expected_skill"], should_trigger=should_trigger, evaluated_skill=entry["evaluated_skill"]+ )+ efficiency = checks.score_skill_efficiency([], entry["expected_skill"], should_trigger=should_trigger)+ assert execution["score"] == 1.0+ assert efficiency["score"] is None and efficiency["details"]["status"] == "not_applicable"+ row = _trial_row("negative", has_skill=False, na=("skill_efficiency",), skill_execution=execution["score"])+ row.update(entry)+ row["details"]["skill_execution"] = execution["details"]+ row["details"]["skill_efficiency"] = {"score": None, **efficiency["details"]}+ _write_job(tmp_path / "jobs" / "demo-codex-without", [row])+ if not baseline_only:+ _write_job(tmp_path / "jobs" / "demo-codex-with", [{**row, "has_skill": True}])++ payload = _collect(tmp_path, ("negative",), skip_with_skill=baseline_only)++ agent = payload["agents"]["codex"]+ assert agent["evaluators"]["skill_execution"]["baseline"] == 1.0+ assert agent["evaluators"]["skill_execution"]["baseline_status"] == "scored"+ assert agent["evaluators"]["skill_efficiency"]["baseline"] is None+ assert agent["evaluators"]["skill_efficiency"]["baseline_status"] == "not_applicable"+ dimensions = {item["id"]: item for item in agent["dimensions"]}+ assert dimensions["discoverability"]["baseline"] == 1.0+ assert {item["id"] for item in payload["not_applicable_dimensions"]} == {"efficiency"}+ reason = next(item["reason"] for item in payload["not_applicable_evaluators"] if item["id"] == "skill_efficiency")+ if view == "saved-report":+ assert "case" in reason.lower()+ assert "no skill under test installed" not in reason.lower()+ return++ result = ValidationResult(validator_name="AGENT_EVAL", validator_description="Live evaluation")+ result.metadata["agent_eval"] = payload+ html = HTMLReporter(include_timestamp=False).render_all([result])+ assert "skill_efficiency" in html+ assert reason in html+ assert "no skill under test installed" not in html.lower()+++def test_baseline_only_run_explains_both_arm_and_case_skill_na(tmp_path):+ # --skip-with-skill scores only the baseline, which has no skill under test installed.+ cases = ("a", "b")+ _write_job(+ tmp_path / "jobs" / "demo-codex-without",+ [_trial_row(case, has_skill=False, na=SKILL_METRICS) for case in cases],+ )++ payload = _collect(tmp_path, cases, skip_with_skill=True)++ assert payload["mode"] == "baseline_only"+ reasons = {item["id"]: item["reason"] for item in payload["not_applicable_evaluators"]}+ assert reasons == {+ "skill_execution": "No applicable skill is available in this arm, or the eval case names no expected_skill",+ "skill_efficiency": "No applicable skill is available in this arm, or the case is negative or names no expected_skill",+ }+ assert {item["id"] for item in payload["not_applicable_dimensions"]} == {"discoverability", "efficiency"}+++def test_all_negative_dataset_scores_discoverability_but_cannot_pass_without_efficiency(tmp_path):+ # Every case is negative, as in a run filtered to the negative cases: the negative check still+ # scores Discoverability, but no case gives Efficiency a skill use to measure.+ cases = ("n1", "n2")+ with_skill = []+ for case in cases:+ row = _trial_row(+ case,+ has_skill=True,+ na=("skill_efficiency",),+ reason="N/A: negative case, so there is no skill use to measure",+ # Goals met and no security finding, so only the N/A dimension can stop a PASS.+ security=1.0,+ accuracy=0.9,+ skill_execution=1.0,+ goal_accuracy=1.0,+ behavior_check=0.9,+ )+ row["details"]["skill_execution"] = {"negative_check": {"passed": True, "score": 1.0}, "should_trigger": False}+ with_skill.append(row)+ _write_job(tmp_path / "jobs" / "demo-codex-with", with_skill)+ _write_job(+ tmp_path / "jobs" / "demo-codex-without",+ [_trial_row(case, has_skill=False, na=SKILL_METRICS) for case in cases],+ )++ payload = _collect(tmp_path, cases)++ assert payload["mode"] == "both"+ dimensions = {row["id"]: row for row in payload["agents"]["codex"]["dimensions"]}+ assert dimensions["discoverability"]["with_skill"] == pytest.approx(1.0)+ assert "efficiency" not in dimensions+ assert {item["id"] for item in payload["not_applicable_dimensions"]} == {"efficiency"}+ assert {item["id"]: item["reason"] for item in payload["not_applicable_evaluators"]} == {+ "skill_efficiency": "Negative case or no expected_skill defined, so there is no skill use to measure",+ }+ assert payload["verdict"] == VERDICT_NEUTRALdiff --git a/tests/reporting/test_tier3_baseline_trial_views.py b/tests/reporting/test_tier3_baseline_trial_views.pyindex 8d118f49..a2832108 100644--- a/tests/reporting/test_tier3_baseline_trial_views.py+++ b/tests/reporting/test_tier3_baseline_trial_views.py@@ -346,12 +346,56 @@ def test_drilldown_reads_na_for_not_applicable_trial_cells() -> None:for column in ("Skill Execution", "Skill Efficiency"):assert _cell(table, "Without skill", column) == "N/A"assert 'data-not-applicable="true"' in _raw_cell(table, "Without skill", column)- assert 'title="Skill Execution: Not applicable: This arm has no skill under test installed' in _raw_cell(+ assert 'title="Skill Execution: Not applicable: No applicable skill is available in this arm' in _raw_cell(table, "Without skill", "Skill Execution")assert _cell(table, "Without skill", "Accuracy") == "0.50"+def test_drilldown_explains_with_skill_skill_metric_na_by_the_eval_case() -> None:+ # With the skill installed, a negative case leaves Efficiency N/A and a positive case scores it.+ # The baseline has no skill at all.+ with_rewards = [+ _na(_reward("negative__attempt1", "negative", score=0.9, steps=5, prompt=10), "skill_efficiency"),+ _reward("positive__attempt1", "positive", score=0.9, steps=5, prompt=10),+ ]+ baseline_rewards = [+ _na(_reward(f"{case}__baseline1", case, score=0.5, steps=4, prompt=10), "skill_execution", "skill_efficiency")+ for case in ("negative", "positive")+ ]+ payload = build_agent_eval_payload(+ "demo",+ {+ "codex": {+ "execution_status": "succeeded",+ "expected_attempts": 4,+ "scored_attempts": 4,+ "with_skill": dict.fromkeys(METRICS, 0.9),+ "without_skill": {**dict.fromkeys(METRICS, 0.5), "skill_execution": None, "skill_efficiency": None},+ "rewards": with_rewards,+ "rewards_baseline": baseline_rewards,+ "num_trials": 2,+ "num_trials_baseline": 2,+ }+ },+ use_llm_judge=False,+ )+ assert payload is not None++ table = _drilldown_table(_render(payload))++ assert _cell(table, "With skill", "Skill Efficiency") == "N/A"+ assert (+ 'title="Skill Efficiency: Not applicable: Negative case or no expected_skill defined, '+ 'so there is no skill use to measure"'+ ) in _raw_cell(table, "With skill", "Skill Efficiency")+ assert _cell(table, "Without skill", "Skill Efficiency") == "N/A"+ assert 'title="Skill Efficiency: Not applicable: No applicable skill is available in this arm' in _raw_cell(+ table, "Without skill", "Skill Efficiency"+ )+ assert _cell(table, "With skill", "Skill Execution") == "0.90"++@pytest.mark.parametrize(("recorded", "kept"),[diff --git a/tests/reporting/test_unified_tier3_report.py b/tests/reporting/test_unified_tier3_report.pyindex 11b0a138..3016cd40 100644--- a/tests/reporting/test_unified_tier3_report.py+++ b/tests/reporting/test_unified_tier3_report.py@@ -843,8 +843,8 @@ def test_no_skill_baseline_dimension_na_explains_why_in_html() -> None:html = _render_agent_payload(payload)for title in (- "No-skill baseline: This arm has no skill under test installed, so there is no skill to discover or run",- "No-skill baseline: This arm has no skill under test installed, so there is no skill to route to",+ "No-skill baseline: No applicable skill is available in this arm, or the eval case names no expected_skill",+ "No-skill baseline: No applicable skill is available in this arm, or the case is negative or names no expected_skill",):cell_start = html.index(f'<td title="{title}">')cell_end = html.index("</td>", cell_start)diff --git a/tests/test_harbor_skill_activation.py b/tests/test_harbor_skill_activation.pyindex c4de8655..aaa57bf2 100644--- a/tests/test_harbor_skill_activation.py+++ b/tests/test_harbor_skill_activation.py@@ -72,7 +72,18 @@ def test_unknown_or_malformed_evidence_does_not_assert_never_activated(unknown:@pytest.mark.parametrize("detail",- [{"should_trigger": False}, {"message": "No expected_skill -- skipped"}],+ [+ {"should_trigger": False},+ # Rewards from earlier releases recorded an unlabelled case with a message and a 1.0 score.+ {"message": "No expected_skill -- skipped"},+ {"message": "Negative case, no skill identified"},+ {+ "score": None,+ "status": "not_applicable",+ "reason": "N/A: no expected_skill or should_trigger defined for this eval case",+ },+ ],+ ids=("negative", "legacy-unlabelled", "legacy-negative-without-skill", "not-applicable"),)def test_negative_and_unlabelled_cases_do_not_count_as_activation_opportunities(detail: dict) -> None:negative = {"details": {"skill_execution": detail}}diff --git a/tests/tier3/test_judge_failure_artifacts.py b/tests/tier3/test_judge_failure_artifacts.pyindex cb71479f..978326ed 100644--- a/tests/tier3/test_judge_failure_artifacts.py+++ b/tests/tier3/test_judge_failure_artifacts.py@@ -273,10 +273,10 @@ def test_verifier_main_fails_closed_after_collecting_every_required_judge(assert set(rich["evaluation_errors"]) == {"accuracy", "goal_accuracy", "behavior_check"}assert all(0 < len(reason) <= 512 for reason in rich["evaluation_errors"].values())+ # The case is a negative case, so skill_efficiency is N/A and Harbor's numeric reward leaves it out.assert numeric == {"security": 1.0,"skill_execution": 1.0,- "skill_efficiency": 1.0,"overall": 0.0,}assert verifier.REWARD_TXT.read_text(encoding="utf-8") == "0.0"@@ -861,16 +861,20 @@ def test_verifier_main_keeps_genuine_zero_judge_verdicts_scoreable(assert "evaluation_status" not in richassert "evaluation_errors" not in richassert rich["score_policy"] == DEFAULT_SCORE_POLICY- assert {metric: numeric[metric] for metric in verifier.DISPLAY_METRICS} == {+ # The case is a negative case, so skill_efficiency is N/A: Harbor's numeric reward leaves it out,+ # and the overall averages the other four dimensions.+ assert {metric: numeric.get(metric) for metric in verifier.DISPLAY_METRICS} == {"security": 1.0,"skill_execution": 1.0,- "skill_efficiency": 1.0,+ "skill_efficiency": None,"accuracy": 0.0,"goal_accuracy": 0.0,"behavior_check": 0.0,}- assert numeric["overall"] == 0.6- assert overall_score(numeric) == 0.6+ assert numeric["overall"] == 0.5+ collected = dict(numeric)+ collector._merge_reward_sidecars(collected, verifier.VERIFIER_DIR)+ assert overall_score(collected) == 0.5def test_verifier_main_recovers_malformed_accuracy_and_goal_judges(diff --git a/tests/tier3/test_negative_control_invocation_evidence.py b/tests/tier3/test_negative_control_invocation_evidence.pyindex 4ffbb68f..10e496bd 100644--- a/tests/tier3/test_negative_control_invocation_evidence.py+++ b/tests/tier3/test_negative_control_invocation_evidence.py@@ -413,7 +413,7 @@ def test_legacy_direct_negative_uses_expected_skill_compatibly() -> None:assert shared["details"]["negative_check"]["score"] == 1.0-def test_legacy_unlabeled_case_retains_skip_pass_behavior() -> None:+def test_legacy_unlabeled_case_is_not_applicable() -> None:kwargs = {"tool_calls": TARGET_READ,"expected_skill": "",@@ -425,8 +425,8 @@ def test_legacy_unlabeled_case_retains_skip_pass_behavior() -> None:template = TEMPLATE.score_skill_execution(**kwargs)assert shared == template- assert shared["score"] == 1.0- assert "skipped" in shared["details"]["message"].lower()+ assert shared["score"] is None+ assert shared["details"]["status"] == "not_applicable"def test_positive_scoring_ignores_evaluated_skill_and_remains_unchanged() -> None:@@ -498,9 +498,12 @@ def test_template_main_scores_standard_negative_fail_closed(tmp_path, monkeypatcresult = json.loads(rich_reward_json.read_text(encoding="utf-8"))harbor_reward = json.loads(reward_json.read_text(encoding="utf-8"))- expected_reward_keys = {*TEMPLATE.DISPLAY_METRICS, "overall"}+ # A negative case gives skill_efficiency nothing to measure, so Harbor's numeric reward leaves it out.+ expected_reward_keys = {*TEMPLATE.DISPLAY_METRICS, "overall"} - {"skill_efficiency"}expected_overall = overall_score(result)+ assert result["skill_efficiency"] is None+ assert result["details"]["skill_efficiency"]["status"] == "not_applicable"assert result["skill_execution"] == 0.0if include_trusted_identity:assert result["details"]["skill_execution"]["negative_check"]["score"] == 0.0diff --git a/tests/tier3/test_skill_activation_scoring.py b/tests/tier3/test_skill_activation_scoring.pyindex 189322b4..1c442712 100644--- a/tests/tier3/test_skill_activation_scoring.py+++ b/tests/tier3/test_skill_activation_scoring.py@@ -111,17 +111,36 @@ def test_negative_control_does_not_require_positive_activation(scorer: ModuleTyp@pytest.mark.parametrize("expected_skill", (None, "", "demo"))-def test_unlabeled_case_remains_skipped(scorer: ModuleType, expected_skill: str | None) -> None:+def test_unlabeled_case_is_not_applicable(scorer: ModuleType, expected_skill: str | None) -> None:result = scorer.score_skill_execution([], expected_skill, should_trigger=None)- assert result["score"] == 1.0- assert "skipped" in result["details"]["message"]- assert "activation" not in result["details"]+ assert result == {+ "score": None,+ "details": {+ "score": None,+ "status": "not_applicable",+ "reason": "N/A: no expected_skill or should_trigger defined for this eval case",+ },+ }-def test_missing_expected_skill_remains_skipped(scorer: ModuleType) -> None:- result = scorer.score_skill_execution([], "", should_trigger=True)[email protected]("expected_skill", (None, ""))+def test_positive_case_without_expected_skill_is_not_applicable(scorer: ModuleType, expected_skill: str | None) -> None:+ calls = [{"action": "Read", "action_input": {"file_path": "/skills/demo/SKILL.md"}}]+ result = scorer.score_skill_execution(calls, expected_skill, should_trigger=True)- assert result["score"] == 1.0- assert "skipped" in result["details"]["message"]- assert "activation" not in result["details"]+ assert result["score"] is None+ assert result["details"] == {+ "score": None,+ "status": "not_applicable",+ "reason": "N/A: no expected_skill defined for this eval case",+ }+++def test_negative_case_without_any_skill_identity_is_not_applicable(scorer: ModuleType) -> None:+ # Only a caller that does not require the trusted evaluated_skill reaches this branch.+ result = scorer.score_skill_execution([], "", should_trigger=False)++ assert result["score"] is None+ assert result["details"]["status"] == "not_applicable"+ assert result["details"]["reason"] == "N/A: negative case with no skill under test to check"diff --git a/tests/tier3/test_skill_metric_applicability.py b/tests/tier3/test_skill_metric_applicability.pynew file mode 100644index 00000000..bef41bdb--- /dev/null+++ b/tests/tier3/test_skill_metric_applicability.py@@ -0,0 +1,435 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0+"""Skill metrics an eval case gives nothing to score are N/A in the with-skill arm, not 1.0."""++from __future__ import annotations++import json+from typing import Any++import pytest++from skillevaluator.constants import DEFAULT_SCORE_POLICY+from skillevaluator.tier3.eval_core import checks+from skillevaluator.tier3.harbor import metrics+from skillevaluator.tier3.harbor.activation import summarize_skill_activation++UNLABELED = "N/A: no expected_skill or should_trigger defined for this eval case"+NO_EXPECTED_SKILL = "N/A: no expected_skill defined for this eval case"+NEGATIVE = "N/A: negative case, so there is no skill use to measure"++READ_SKILL = {"action": "Read", "action_input": {"file_path": "/skills/demo/SKILL.md"}, "observation": "ok"}+READ_OTHER = {"action": "Read", "action_input": {"file_path": "/workspace/README.md"}, "observation": "ok"}+READ_ALT = {"action": "Read", "action_input": {"file_path": "/skills/alt/SKILL.md"}, "observation": "ok"}+++def _judge(*_args: Any, **_kwargs: Any) -> dict[str, Any]:+ return {"score": 0.8, "reason": "ok", "results": []}+++def _verifier(tmp_path, monkeypatch, entry, *, tool_calls=None, trajectory=True):+ """Point the Harbor verifier at tmp_path with stubbed trajectory, tool calls, security and judges."""+ from skillevaluator.tier3.harbor.templates import eval as verifier++ entry_path = tmp_path / "entry.json"+ entry_path.write_text(json.dumps(entry), encoding="utf-8")+ loaded = (+ ({"steps": [{"source": "agent", "message": "done"}]}, {"source": "trajectory.json"})+ if trajectory+ else (None, {"source": None})+ )+ monkeypatch.setattr(verifier, "ENTRY_PATH", entry_path)+ monkeypatch.setattr(verifier, "load_trajectory_with_fallback", lambda *_args: loaded)+ monkeypatch.setattr(verifier, "extract_tool_calls_as_dicts", lambda _traj: list(tool_calls or []))+ monkeypatch.setattr(verifier, "get_skill_tool_calls", lambda _traj: [])+ monkeypatch.setattr(verifier, "check_security", lambda *_a: {"score": 1.0})+ monkeypatch.setattr(verifier, "REWARD_JSON", tmp_path / "reward.json")+ monkeypatch.setattr(verifier, "REWARD_TXT", tmp_path / "reward.txt")+ monkeypatch.setattr(verifier, "SKILL_EVALUATOR_REWARD_JSON", tmp_path / "skill_evaluator_reward.json")+ monkeypatch.setattr(verifier, "call_public_llm", lambda *_a, **_k: pytest.fail("Unexpected LLM call"))+ for name in ("judge_accuracy", "judge_goal_accuracy", "judge_behavior_check"):+ monkeypatch.setattr(verifier, name, _judge)+ return verifier+++def _outputs(tmp_path):+ numeric = json.loads((tmp_path / "reward.json").read_text(encoding="utf-8"))+ rich = json.loads((tmp_path / "skill_evaluator_reward.json").read_text(encoding="utf-8"))+ return numeric, rich+++def _case(**fields: Any) -> dict[str, Any]:+ return {+ "id": "case-1",+ "question": "q",+ "ground_truth": "42",+ "expected_behavior": ["Answers"],+ "evaluated_skill": "demo",+ "has_skill": True,+ **fields,+ }+++# ── Verifier: judged trials ─────────────────────────────────────────────+++ ("tool_calls", "skill_execution"),+ [([READ_OTHER], 1.0), ([READ_SKILL], 0.0)],+ ids=["skill-left-alone", "skill-wrongly-read"],+)+ "labels",+ # create-eval-dataset writes a negative case with expected_skill null and no should_trigger.+ [{"expected_skill": None}, {"expected_skill": ""}, {"expected_skill": "demo", "should_trigger": False}],+ ids=["generated", "empty-expected-skill", "explicit"],+)+def test_negative_case_leaves_efficiency_na_and_keeps_its_negative_check(+ tmp_path, monkeypatch, labels, tool_calls, skill_execution+):+ verifier = _verifier(tmp_path, monkeypatch, _case(**labels), tool_calls=tool_calls)++ verifier.main()++ numeric, rich = _outputs(tmp_path)+ assert "skill_efficiency" not in numeric+ assert rich["skill_efficiency"] is None+ assert rich["details"]["skill_efficiency"] == {"score": None, "status": "not_applicable", "reason": NEGATIVE}+ assert metrics.metric_is_not_applicable(rich, "skill_efficiency")+ # Discoverability still checks that the agent left the skill under test alone.+ assert numeric["skill_execution"] == skill_execution+ assert rich["details"]["skill_execution"]["should_trigger"] is False+ assert "negative_check" in rich["details"]["skill_execution"]+ # Security, Correctness, Discoverability and Effectiveness; Efficiency drops out.+ expected = round((1.0 + 0.8 + skill_execution + 0.8) / 4, 4)+ assert numeric["overall"] == expected+ assert (tmp_path / "reward.txt").read_text(encoding="utf-8") == str(expected)+ assert metrics.overall_score(rich) == expected+++ ("labels", "reason"),+ [+ ({}, UNLABELED),+ ({"should_trigger": True}, NO_EXPECTED_SKILL),+ ({"should_trigger": True, "expected_skill": ""}, NO_EXPECTED_SKILL),+ ],+ ids=["legacy-unlabeled", "positive-without-expected-skill", "positive-with-empty-expected-skill"],+)+def test_case_without_expected_skill_leaves_both_skill_metrics_na(tmp_path, monkeypatch, labels, reason):+ verifier = _verifier(tmp_path, monkeypatch, _case(**labels), tool_calls=[READ_SKILL])++ verifier.main()++ numeric, rich = _outputs(tmp_path)+ for metric in ("skill_execution", "skill_efficiency"):+ assert metric not in numeric+ assert rich[metric] is None+ assert rich["details"][metric] == {"score": None, "status": "not_applicable", "reason": reason}+ # Security, Correctness and Effectiveness only.+ assert numeric["overall"] == round((1.0 + 0.8 + 0.8) / 3, 4)+ # An N/A Discoverability is not an activation opportunity.+ assert summarize_skill_activation([rich])["status"] == "not_applicable"+++def test_positive_case_without_tool_calls_keeps_the_zero_efficiency_penalty(tmp_path, monkeypatch):+ verifier = _verifier(tmp_path, monkeypatch, _case(expected_skill="demo"), tool_calls=[])++ verifier.main()++ numeric, rich = _outputs(tmp_path)+ assert numeric["skill_execution"] == 0.0+ assert numeric["skill_efficiency"] == 0.0+ assert rich["details"]["skill_efficiency"] == {"message": "No tool calls in trajectory"}+ assert numeric["overall"] == round((1.0 + 0.8 + 0.0 + 0.8 + 0.0) / 5, 4)+++def test_positive_case_scores_both_skill_metrics(tmp_path, monkeypatch):+ verifier = _verifier(tmp_path, monkeypatch, _case(expected_skill="demo"), tool_calls=[READ_SKILL])++ verifier.main()++ numeric, rich = _outputs(tmp_path)+ assert numeric["skill_execution"] == 1.0+ assert numeric["skill_efficiency"] == 1.0+ assert set(rich["details"]["skill_efficiency"]) == {"routing", "tool_efficiency"}+ assert not metrics.not_applicable_metrics([rich])+++ ("labels", "tool_calls", "na_reasons"),+ [+ ({"expected_skill": None}, [READ_OTHER], {"skill_efficiency": NEGATIVE}),+ (+ {"should_trigger": True, "expected_skill": "", "acceptable_skills": ["alt"]},+ [READ_ALT],+ {"skill_execution": NO_EXPECTED_SKILL, "skill_efficiency": NO_EXPECTED_SKILL},+ ),+ ({"expected_skill": "demo", "acceptable_skills": ["alt"]}, [READ_ALT], {}),+ ],+ ids=["negative", "positive-without-expected-skill", "positive-with-alternate"],+)+def test_baseline_staging_a_case_candidate_follows_the_case_rules(+ tmp_path, monkeypatch, labels, tool_calls, na_reasons+):+ # A no-skill arm that stages a skill the case could activate is scored on the+ # skill metrics, but a metric the case gives nothing to score stays N/A there too.+ entry = _case(has_skill=False, workspace_skill_names=["demo", "alt"], skill_workspace_mode="group", **labels)+ verifier = _verifier(tmp_path, monkeypatch, entry, tool_calls=tool_calls)++ verifier.main()++ numeric, rich = _outputs(tmp_path)+ for metric in ("skill_execution", "skill_efficiency"):+ if metric in na_reasons:+ assert metric not in numeric+ assert rich["details"][metric] == {"score": None, "status": "not_applicable", "reason": na_reasons[metric]}+ else:+ # Scored, not the baseline's N/A: the staged skill matches a case candidate.+ assert rich["details"][metric].get("status") != "not_applicable"+ assert numeric[metric] > 0+ assert set(metrics.not_applicable_metrics([rich])) == set(na_reasons)+ assert numeric["overall"] == metrics.overall_score(rich)+++# ── Verifier: trials without a trajectory ───────────────────────────────+++ ("labels", "na_reasons"),+ [+ ({"expected_skill": None}, {"skill_efficiency": NEGATIVE}),+ ({}, {"skill_execution": UNLABELED, "skill_efficiency": UNLABELED}),+ ({"expected_skill": "demo"}, {}),+ ],+ ids=["negative", "legacy-unlabeled", "positive"],+)+def test_trial_without_trajectory_keeps_case_level_skill_na(tmp_path, monkeypatch, labels, na_reasons):+ verifier = _verifier(tmp_path, monkeypatch, _case(**labels), trajectory=False)++ verifier.main()++ numeric, rich = _outputs(tmp_path)+ for metric in ("skill_execution", "skill_efficiency"):+ if metric in na_reasons:+ # A crash cannot turn a metric the case gives nothing to score into 0.0.+ assert metric not in numeric+ assert rich["details"][metric] == {+ "score": None,+ "status": "not_applicable",+ "reason": na_reasons[metric],+ }+ else:+ assert numeric[metric] == 0.0+ assert numeric["overall"] == 0.0+++# ── Library scorers ─────────────────────────────────────────────────────+++ ("should_trigger", "expected_skill", "reason"),+ [(False, "demo", NEGATIVE), (None, "demo", UNLABELED), (None, None, UNLABELED), (True, "", NO_EXPECTED_SKILL)],+ ids=["negative", "unlabeled-with-skill-name", "unlabeled", "positive-without-expected-skill"],+)+def test_library_skill_efficiency_is_na_without_a_skill_use_to_measure(should_trigger, expected_skill, reason):+ result = checks.score_skill_efficiency([READ_SKILL], expected_skill, should_trigger=should_trigger)++ assert result == {"score": None, "details": {"score": None, "status": "not_applicable", "reason": reason}}+++def test_library_skill_efficiency_keeps_scores_for_positive_cases():+ assert checks.score_skill_efficiency([], "demo")["score"] == 0.0+ assert checks.score_skill_efficiency([READ_SKILL], "demo")["score"] == 1.0++[email protected]("should_trigger", [None, False, True])[email protected]("expected_skill", [None, "", "demo"])+def test_case_reasons_match_the_library_scorers(should_trigger, expected_skill):+ # The verifier decides N/A for trials without a trajectory with the same helpers.+ execution = checks.score_skill_execution(+ [READ_OTHER], expected_skill, should_trigger=should_trigger, evaluated_skill="demo"+ )+ efficiency = checks.score_skill_efficiency([READ_OTHER], expected_skill, should_trigger=should_trigger)++ for result, reason in (+ (execution, checks.skill_execution_not_applicable_reason(should_trigger, expected_skill)),+ (efficiency, checks.skill_efficiency_not_applicable_reason(should_trigger, expected_skill)),+ ):+ if reason is None:+ assert isinstance(result["score"], float)+ else:+ assert result["score"] is None+ assert result["details"]["reason"] == reason+++# ── Collection: some trials N/A ─────────────────────────────────────────+++def _reward(case: str, *, has_skill: bool = True, **overrides: Any) -> dict[str, Any]:+ row: dict[str, Any] = dict.fromkeys(metrics.DEFAULT_METRICS, 0.8)+ row.update(+ metric_set=metrics.DEFAULT_METRIC_SET,+ score_policy=DEFAULT_SCORE_POLICY,+ entry_id=case,+ has_skill=has_skill,+ details={},+ )+ for metric, value in overrides.items():+ row[metric] = value+ if value is None:+ row["details"][metric] = {"score": None, "status": "not_applicable", "reason": "No skill use"}+ return row+++def _write_arm(job, rows):+ trial_names = []+ for row in rows:+ trial = f"{row['entry_id']}__attempt1"+ trial_names.append(trial)+ verifier = job / trial / "verifier"+ verifier.mkdir(parents=True)+ numeric = {key: value for key, value in row.items() if isinstance(value, float)}+ numeric["overall"] = metrics.overall_score(row)+ (verifier / "reward.json").write_text(json.dumps(numeric), encoding="utf-8")+ (verifier / "skill_evaluator_reward.json").write_text(json.dumps(row), encoding="utf-8")+ stats = {+ "n_completed_trials": len(rows),+ "n_errored_trials": 0,+ "n_running_trials": 0,+ "n_pending_trials": 0,+ "n_cancelled_trials": 0,+ "n_retries": 0,+ "evals": {+ "agent__model___harbor-tasks": {+ "n_trials": len(rows),+ "n_errors": 0,+ "reward_stats": {"reward": {"0.1": trial_names}},+ }+ },+ }+ (job / "result.json").write_text(json.dumps({"n_total_trials": len(rows), "stats": stats}), encoding="utf-8")+++def test_arm_with_some_na_efficiency_trials_stays_scored(tmp_path):+ from skillevaluator.tier3.harbor.collector import collect_harbor_results++ baseline_na = {"skill_execution": None, "skill_efficiency": None}+ arms = {+ # A positive case scores Efficiency; the negative case leaves it N/A. Its agent read the+ # skill under test, so its negative check scores 0.0. Counting the N/A Efficiency as 0.0+ # would fail the negative case (0.44) instead of passing it (0.55).+ "with": [+ _reward("pos", skill_efficiency=0.6),+ _reward(+ "neg",+ security=1.0,+ accuracy=0.6,+ skill_execution=0.0,+ goal_accuracy=1.0,+ behavior_check=0.2,+ skill_efficiency=None,+ ),+ ],+ "without": [+ _reward("pos", has_skill=False, accuracy=0.5, **baseline_na),+ _reward("neg", has_skill=False, accuracy=0.5, **baseline_na),+ ],+ }+ for arm, rows in arms.items():+ _write_arm(tmp_path / "jobs" / f"demo-opencode-{arm}", rows)++ result = collect_harbor_results(+ skill_name="demo",+ agents=["opencode"],+ output_dir=tmp_path / "results",+ jobs_dir=tmp_path / "jobs",+ expected_cases=2,+ expected_case_ids=["pos", "neg"],+ expected_trials=2,+ )++ agent = result["agents"]["opencode"]+ assert agent["execution_status"] == "succeeded"+ assert "skill_efficiency" not in agent["not_applicable_metrics"]["with_skill"]+ summary = json.loads((tmp_path / "results/opencode/with-skill/summary.json").read_text(encoding="utf-8"))+ assert summary["not_applicable_counts"] == {"skill_efficiency": 1}+ # The arm mean is the positive case's score alone.+ assert summary["scores"]["skill_efficiency"] == 0.6+ assert summary["dimensions"]["efficiency"]["score"] == 0.6+ # pass@k scores each attempt without its N/A metric: (1.0 + 0.6 + 0.0 + 0.6) / 4.+ neg = summary["pass_at_k"]["cases"]["neg"]+ assert neg["best_score"] == pytest.approx(0.55)+ assert neg["passed"] is True+ # Lift pairs Efficiency on no case: the baseline never has a skill to use.+ assert "skill_efficiency" not in agent["lift"]+ assert agent["lift"]["accuracy"]["delta"] == pytest.approx(0.2)+ assert agent["lift"]["basis"]["metrics"]["accuracy"]["cases"] == 2+++# ── Collection: the N/A reason the verifier recorded survives ──────────+++def _negative_step_row(name: str, reason: str = NEGATIVE) -> dict[str, Any]:+ row = _reward("neg", skill_efficiency=None)+ row["details"]["skill_efficiency"]["reason"] = reason+ return dict(row, _trial_root_name="trial", _trial_name=f"trial/{name}", _step_name=name)+++def test_mark_not_applicable_keeps_the_reason_the_verifier_recorded():+ recorded = _reward("neg", skill_efficiency=None)+ recorded["details"]["skill_efficiency"]["reason"] = NEGATIVE+ metrics.mark_not_applicable(recorded, "skill_efficiency", "Some other reason")+ assert recorded["details"]["skill_efficiency"]["reason"] == NEGATIVE++ given: dict[str, Any] = {"skill_efficiency": 0.5}+ metrics.mark_not_applicable(given, "skill_efficiency", NEGATIVE)+ assert given["skill_efficiency"] is None+ assert given["details"]["skill_efficiency"] == {"score": None, "status": "not_applicable", "reason": NEGATIVE}++ # A scored detail's text is no N/A reason; without a given one the metric's own reason applies.+ scored: dict[str, Any] = {"details": {"skill_efficiency": {"score": 0.5, "reason": "Two redundant reads"}}}+ metrics.mark_not_applicable(scored, "skill_efficiency")+ assert scored["details"]["skill_efficiency"]["reason"] == metrics.NOT_APPLICABLE_REASONS["skill_efficiency"]++ long: dict[str, Any] = {}+ metrics.mark_not_applicable(long, "skill_efficiency", "x" * 5000)+ assert len(long["details"]["skill_efficiency"]["reason"]) == metrics.NOT_APPLICABLE_REASON_MAX_CHARS+++def test_logical_multistep_reward_keeps_the_negative_case_reason():+ from skillevaluator.tier3.harbor.collector import _logical_attempt_rewards++ [logical] = _logical_attempt_rewards([_negative_step_row("first"), _negative_step_row("second")])++ assert metrics.metric_is_not_applicable(logical, "skill_efficiency")+ assert logical["details"]["skill_efficiency"]["reason"] == NEGATIVE++ # Steps that record different reasons keep each of them.+ [mixed] = _logical_attempt_rewards([_negative_step_row("first"), _negative_step_row("second", UNLABELED)])+ assert mixed["details"]["skill_efficiency"]["reason"] == f"{NEGATIVE}; {UNLABELED}"+++def test_restored_multistep_na_keeps_the_reason_each_step_recorded(tmp_path):+ from skillevaluator.tier3.harbor.collector import _restore_constituent_not_applicable++ rich = _negative_step_row("step")+ numeric = {key: value for key, value in rich.items() if isinstance(value, float)}+ result: dict[str, Any] = {"verifier_result": {"rewards": dict(numeric)}, "step_results": []}+ for name in ("first", "second"):+ directory = tmp_path / "steps" / name / "verifier"+ directory.mkdir(parents=True)+ (directory / "skill_evaluator_reward.json").write_text(json.dumps(rich), encoding="utf-8")+ result["step_results"].append({"step_name": name, "verifier_result": {"rewards": dict(numeric)}})+ data = dict(numeric)++ restored = _restore_constituent_not_applicable(data, result, tmp_path)++ rewards = [+ data,+ restored["verifier_result"]["rewards"],+ *(step["verifier_result"]["rewards"] for step in restored["step_results"]),+ ]+ for reward in rewards:+ assert metrics.metric_is_not_applicable(reward, "skill_efficiency")+ assert reward["details"]["skill_efficiency"]["reason"] == NEGATIVE