Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
feat(tier3): report dataset coverage and label one-case starters (#381)
Christopher Kevin770c599fae90
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex fa9ec850..0c17504a 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -6,6 +6,10 @@ All notable changes to SkillEvaluator are documented in this file.### Added+- `tier3 validate` reports informational dataset coverage counts and concrete+ gaps in text and JSON. Generated one-case starters now explain their limits+ and point authors to full dataset generation and coverage validation.+- Document an additive qualitative assessment review worksheet with boundedevidence from real successful, failed-setup and invalid-judge runs. Preservenumeric grading and identify the separate decisions needed for scoreless support.@@ -39,7 +43,6 @@ All notable changes to SkillEvaluator are documented in this file.listener reaches the default branch.Document the shared chat/judge/agent model input and model-availabilitytroubleshooting for provider-backed CI.-- `skills/skillevaluator/` is an agent skill for operating SkillEvaluator: itroutes requests to the right tier, readiness check, dataset, report, or CIcommand, and ships Tier 3 evaluation cases for itself. A test keeps everydiff --git a/docs/eval-datasets.mdx b/docs/eval-datasets.mdxindex 8bdc163d..c8d8614d 100644--- a/docs/eval-datasets.mdx+++ b/docs/eval-datasets.mdx@@ -353,6 +353,28 @@ skillevaluator tier3 validate ./my-skillRun it in CI or before a long evaluation: it catches an empty dataset, missingrequired fields, duplicate case IDs, and invalid config in seconds.+The informational **Dataset coverage** block also shows case count, positive+(should-trigger), negative/non-trigger and unlabeled-trigger counts, plus cases+with an expected output and assertions/expected behaviors. These use the same+field normalization and trigger resolution as evaluation. Gaps name missing+negative or positive examples and missing judge references: without an expected+output, accuracy and goal_accuracy are N/A; without assertions or expected+behaviors, behavior_check is N/A.++A single-case dataset is labelled **a starting point, not a representative+evaluation**. Larger datasets show their counts and concrete gaps without any+claim of sufficiency. Counts alone do not establish representative coverage;+review task diversity, failure modes and outcomes independently.++Coverage is informational and does not add warnings or change `--strict` exit+codes. `--json` remains a list of check objects. It adds an entry with path+`evals/evals.json#coverage` (or the actual dataset filename), status `ok`, a+human-readable `message` and a structured `coverage` object containing+`case_count`, `starter`, `positive_cases`, `negative_cases`,+`unlabeled_trigger_cases`, `cases_with_expected_outcome`,+`cases_with_expected_behaviors` and `gaps`. Native Harbor task sources do not+receive this eval-dataset summary.+`validate PATH` also checks the selected Tier 3 source (dataset entries andcase IDs, `config.yml`, or the native Harbor task layout) before any live run.When that check fails, each error names the entry or file and what is wrong,diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdxindex 875f2df3..25f1ae62 100644--- a/docs/tier3-live-evaluation.mdx+++ b/docs/tier3-live-evaluation.mdx@@ -309,8 +309,15 @@ for NVIDIA Build, Codex for OpenAI, or Claude Code for Anthropic. If no accepteddataset or task source exists, it generates one starter case and proceeds to thelive with-skill and without-skill comparison. It reuses valid existing sourcesunchanged; invalid sources produce an error rather than being replaced.-A generated starter case lets you try the pipeline, but does not establish-broad positive and negative coverage.+A generated one-case starter is a starting point, not a representative+evaluation. Generation prints its concrete coverage gaps to stderr, such as+the absence of a negative/non-trigger case or missing expected outcomes and+assertions. Replace the starter with a full dataset using+`skillevaluator create-eval-dataset ./my-skill --full --force`. `--force`+overwrites the starter and discards any edits to it. Then inspect the counts+and gaps with `skillevaluator tier3 validate ./my-skill`. Larger case counts+do not by themselves establish representative coverage. Native Harbor task+sources do not receive an eval-dataset coverage summary.The same agent, model, environment, grading, and reporting options described for`tier3 evaluate` apply to this direct workflow. For example:diff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.pyindex 8d54c47d..97fbf99e 100644--- a/src/skillevaluator/cli.py+++ b/src/skillevaluator/cli.py@@ -1086,6 +1086,7 @@ def _ensure_autopilot_dataset(from skillevaluator.evaluation.results import DatasetGenerationErrorfrom skillevaluator.inference.diagnostics import llm_failure_diagnostic, safe_llm_labelsfrom skillevaluator.provider_config import ProviderConfigurationError, resolve_llm_provider+ from skillevaluator.tier3.dataset_coverage import starter_guidancefrom skillevaluator.tier3.dataset_progress import dataset_generation_progressfrom skillevaluator.tier3.harbor.adapter import find_evals_file@@ -1128,11 +1129,13 @@ def _ensure_autopilot_dataset(echo(f"Autopilot: generating one case with {provider_name} / {model_name}.")dataset_note = "auto-generated by autopilot — review evals/"+ generated = Falsewith dataset_generation_progress(enabled=not quiet and progress != "off", stream=click.get_text_stream("stderr")):try:service.create_autopilot_dataset(skill_path, use_llm=not no_llm)+ generated = Trueexcept FileExistsError:echo("Autopilot: an evaluation source appeared concurrently; reusing it unchanged.")except Exception as exc:@@ -1145,8 +1148,13 @@ def _ensure_autopilot_dataset(dataset_note = f"deterministic starter — LLM generation failed: {diagnostic} Review evals/."if not eval_source_exists():service.create_autopilot_dataset(skill_path, use_llm=False)+ generated = Trueif not eval_source_exists():raise click.ClickException("Autopilot dataset generation did not produce an evaluation source.")+ if generated:+ guidance = starter_guidance(skill_path)+ if guidance is not None:+ echo(strip_terminal_controls(guidance))return dataset_noteexcept click.ClickException:raisediff --git a/src/skillevaluator/tier3/commands.py b/src/skillevaluator/tier3/commands.pyindex 11154ce8..a2ce8e82 100644--- a/src/skillevaluator/tier3/commands.py+++ b/src/skillevaluator/tier3/commands.py@@ -29,6 +29,7 @@ from skillevaluator.evaluation.results import DatasetGenerationError, DatasetGenfrom skillevaluator.evaluation.tier3_report import render_agent_eval_html_reportfrom skillevaluator.provider_config import ProviderConfigurationError, resolve_llm_providerfrom skillevaluator.tier3.case_ids import safe_child, validate_case_id, validate_case_ids+from skillevaluator.tier3.dataset_coverage import dataset_coverage_resultfrom skillevaluator.tier3.dataset_utils import DATASET_EXTENSIONS, load_dataset_entries_with_formatfrom skillevaluator.tier3.evals_config import CONFIG_FILENAMES, _validate_config, load_evals_configfrom skillevaluator.tier3.evals_spec import validate_harbor_contract, validate_skillevaluators@@ -966,15 +967,19 @@ def validate_evals(skill_path: Path, *, as_json: bool, strict: bool, harbor_cont)]+ coverage = dataset_coverage_result(skill_path)if as_json:console.print_json(json.dumps(- [{"path": r.path, "status": r.status, "message": r.message} for r in results],+ [{"path": r.path, "status": r.status, "message": r.message} for r in results]+ + ([coverage] if coverage is not None else []),indent=2,))else:_print_validate_results(skill_path, results)+ if coverage is not None:+ console.print(Text(coverage["message"]))has_errors = any(r.status == "error" for r in results)has_warnings = any(r.status == "warning" for r in results)diff --git a/src/skillevaluator/tier3/dataset_coverage.py b/src/skillevaluator/tier3/dataset_coverage.pynew file mode 100644index 00000000..91d8e7c6--- /dev/null+++ b/src/skillevaluator/tier3/dataset_coverage.py@@ -0,0 +1,99 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Descriptive dataset coverage, independent of validity and run gates."""++from __future__ import annotations++import shlex+from pathlib import Path+from typing import Any++from skillevaluator.tier3.dataset_utils import find_eval_file, load_dataset_entries+from skillevaluator.tier3.eval_core.checks import resolve_should_trigger+from skillevaluator.tier3.eval_core.llm_judge import _has_judge_reference+from skillevaluator.tier3.evals_config import load_evals_config+++def summarize_dataset(entries: list[dict[str, Any]]) -> dict[str, Any]:+ """Count normalized cases without interpreting any count as sufficiency."""+ count = len(entries)+ positive = sum(resolve_should_trigger(entry) is True for entry in entries)+ negative = sum(resolve_should_trigger(entry) is False for entry in entries)+ outcomes = sum(_has_judge_reference(entry.get("ground_truth")) for entry in entries)+ behaviors = sum(_has_judge_reference(entry.get("expected_behavior")) for entry in entries)+ gaps = []+ if not count:+ gaps.append("no evaluation cases")+ if not positive:+ gaps.append("no positive case that should trigger the skill")+ if not negative:+ gaps.append("no negative case that should not trigger the skill")+ unknown = count - positive - negative+ if unknown:+ gaps.append(f"{unknown} cases have no resolved should_trigger label")+ if count - outcomes:+ gaps.append(f"{count - outcomes} cases have no expected output, so accuracy and goal_accuracy are N/A for them")+ if count - behaviors:+ prefix = "no assertions; " if not behaviors else ""+ gaps.append(+ f"{prefix}{count - behaviors} cases have no assertions or expected behaviors, so behavior_check is N/A for them"+ )+ return {+ "case_count": count,+ "starter": count == 1,+ "positive_cases": positive,+ "negative_cases": negative,+ "unlabeled_trigger_cases": unknown,+ "cases_with_expected_outcome": outcomes,+ "cases_with_expected_behaviors": behaviors,+ "gaps": gaps,+ }+++def coverage_text(coverage: dict[str, Any]) -> str:+ count = coverage["case_count"]+ lines = [+ f"Dataset coverage: {count} cases; {coverage['positive_cases']} positive, "+ f"{coverage['negative_cases']} negative/non-trigger, "+ f"{coverage['unlabeled_trigger_cases']} unlabeled trigger.",+ f"Expected outcome: {coverage['cases_with_expected_outcome']}/{count}; "+ f"assertions/expected behaviors: {coverage['cases_with_expected_behaviors']}/{count}.",+ ]+ if coverage["starter"]:+ lines.append("One-case starter: a starting point, not a representative evaluation.")+ lines.extend(f"Gap: {gap}." for gap in coverage["gaps"])+ return "\n".join(lines)+++def dataset_coverage_result(skill_path: Path) -> dict[str, Any] | None:+ """Load authored entries; leave invalid and native sources to their validators."""+ try:+ config, _ = load_evals_config(skill_path)+ if (config.get("harbor") or {}).get("task_source") == "native_harbor":+ return None+ dataset = find_eval_file(skill_path)+ if dataset is None or dataset.parent.name != "evals":+ return None+ coverage = summarize_dataset(load_dataset_entries(dataset))+ except Exception:+ return None+ return {+ "path": f"{dataset.relative_to(skill_path)}#coverage",+ "status": "ok",+ "message": coverage_text(coverage),+ "coverage": coverage,+ }+++def starter_guidance(skill_path: Path) -> str | None:+ result = dataset_coverage_result(skill_path)+ if result is None:+ return None+ path = shlex.quote(str(skill_path))+ return (+ result["message"] + "\n"+ f"Replace the starter with a full dataset using: skillevaluator create-eval-dataset {path} --full --force\n"+ "`--force` overwrites the starter and discards any edits to it.\n"+ f"Inspect coverage with: skillevaluator tier3 validate {path}"+ )diff --git a/tests/tier3/test_dataset_coverage.py b/tests/tier3/test_dataset_coverage.pynew file mode 100644index 00000000..b88647b8--- /dev/null+++ b/tests/tier3/test_dataset_coverage.py@@ -0,0 +1,151 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++from __future__ import annotations++import json+from pathlib import Path++import pytest+from click.testing import CliRunner++from skillevaluator.cli import cli+from skillevaluator.tier3.dataset_coverage import (+ coverage_text,+ dataset_coverage_result,+ summarize_dataset,+)+++def skill(tmp_path):+ root = tmp_path / "demo"+ (root / "evals").mkdir(parents=True)+ (root / "SKILL.md").write_text("---\nname: demo\ndescription: Compute values.\n---\n# Demo\n")+ return root+++def write_dataset(skill, cases):+ (skill / "evals/evals.json").write_text(json.dumps({"skill_name": "demo", "evals": cases}))+++def test_summary_counts_and_routing_resolution():+ result = summarize_dataset(+ [+ {"expected_skill": "demo", "ground_truth": "answer", "expected_behavior": ["act"]},+ {"expected_skill": "demo", "should_trigger": False, "ground_truth": 0, "expected_behavior": [""]},+ {"expected_skill": "", "ground_truth": False},+ {},+ ]+ )+ assert result["case_count"] == 4+ assert result["positive_cases"] == 1+ assert result["negative_cases"] == 2+ assert result["unlabeled_trigger_cases"] == 1+ assert result["cases_with_expected_outcome"] == 3+ assert result["cases_with_expected_behaviors"] == 1+ assert "1 cases have no resolved should_trigger label" in result["gaps"]+ assert any("1 cases have no expected output" in gap for gap in result["gaps"])+ assert any("3 cases have no assertions" in gap for gap in result["gaps"])+ assert result["starter"] is False+++def test_starter_and_empty_summary():+ starter = summarize_dataset([{"expected_skill": "demo", "ground_truth": " ", "expected_behavior": []}])+ assert starter["starter"] is True+ assert "a starting point, not a representative evaluation" in coverage_text(starter)+ assert "no negative case that should not trigger the skill" in starter["gaps"]+ assert any("no assertions" in gap for gap in starter["gaps"])+ empty = summarize_dataset([])+ assert empty["case_count"] == 0+ assert empty["starter"] is False+ assert "no evaluation cases" in empty["gaps"]+++def test_more_cases_make_no_sufficiency_claim():+ summary = summarize_dataset(+ [+ {"should_trigger": value, "ground_truth": "result", "expected_behavior": ["act"]}+ for value in [True, False] * 20+ ]+ )+ assert summary["gaps"] == []+ text = coverage_text(summary)+ assert "40 cases" in text+ assert "representative" not in text+ assert "enough" not in text+++def test_loader_normalizes_author_fields(skill):+ write_dataset(skill, [{"id": "one", "prompt": "Compute.", "expected_output": "0", "assertions": ["Return 0."]}])+ result = dataset_coverage_result(skill)+ assert result["coverage"]["positive_cases"] == 1+ assert result["coverage"]["cases_with_expected_outcome"] == 1+ assert result["coverage"]["cases_with_expected_behaviors"] == 1+++def test_validate_text_json_and_strict_info(skill):+ write_dataset(skill, [{"id": "one", "prompt": "Compute.", "expected_output": ""}])+ runner = CliRunner()+ text = runner.invoke(cli, ["tier3", "validate", str(skill), "--strict"])+ assert text.exit_code == 0, text.output+ assert "Dataset coverage: 1 cases" in text.output+ assert "a starting point, not a representative evaluation" in text.output+ assert "no negative case" in text.output+ assert "accuracy and goal_accuracy are N/A" in text.output+ assert "no assertions" in text.output+ result = runner.invoke(cli, ["tier3", "validate", str(skill), "--json", "--strict"])+ assert result.exit_code == 0, result.output+ payload = json.loads(result.output)+ assert isinstance(payload, list)+ coverage = next(row for row in payload if row["path"] == "evals/evals.json#coverage")+ assert coverage["status"] == "ok"+ assert coverage["coverage"]["starter"] is True+ assert all({"path", "status", "message"} <= row.keys() for row in payload)+ assert not any(row["status"] == "warning" for row in payload)++[email protected]("content", ["{invalid", '{"evals": []}', '["bad"]'])+def test_invalid_source_keeps_failure_exit(skill, content):+ (skill / "evals/evals.json").write_text(content)+ result = CliRunner().invoke(cli, ["tier3", "validate", str(skill), "--json"])+ assert result.exit_code == 1, result.output+ assert any(row["status"] == "error" for row in json.loads(result.output))+++def test_missing_and_malformed_yaml_have_no_coverage(skill):+ assert dataset_coverage_result(skill) is None+ (skill / "evals/evals.yaml").write_text("cases: [")+ assert dataset_coverage_result(skill) is None+++def test_recursive_yaml_alias_does_not_abort_coverage(skill):+ (skill / "evals/evals.yaml").write_text(+ "skill_name: demo\nevals:\n - id: one\n prompt: Q\n expected_output: &a [*a]\n"+ )+ assert dataset_coverage_result(skill) is None+++def test_legacy_dataset_is_not_summarized_as_active_coverage(skill):+ legacy = skill.parent / "eval"+ legacy.mkdir()+ (legacy / "dataset.json").write_text(+ json.dumps({"skill_name": "demo", "evals": [{"id": "one", "prompt": "Q", "expected_output": "A"}]})+ )+ assert dataset_coverage_result(skill) is None+++def test_native_source_skips_even_if_dataset_present(skill):+ write_dataset(skill, [{"id": "one", "prompt": "Compute."}])+ (skill / "evals/config.yml").write_text("harbor:\n task_source: native_harbor\n")+ assert dataset_coverage_result(skill) is None+ result = CliRunner().invoke(cli, ["tier3", "validate", str(skill), "--json"])+ assert not any("#coverage" in row["path"] for row in json.loads(result.output))+++def test_reference_starter_and_calculator():+ references = Path(__file__).parents[2] / "src/skillevaluator/tier3/reference_skills"+ assert dataset_coverage_result(references / "text-analyzer")["coverage"]["starter"] is True+ calculator = dataset_coverage_result(references / "calculator")["coverage"]+ assert calculator["case_count"] == 2+ assert calculator["starter"] is Falsediff --git a/tests/tier3/test_dataset_generation_progress.py b/tests/tier3/test_dataset_generation_progress.pyindex 4ab9ada9..f8cd030e 100644--- a/tests/tier3/test_dataset_generation_progress.py+++ b/tests/tier3/test_dataset_generation_progress.py@@ -49,6 +49,11 @@ def test_generation_announces_provider_and_handoff_on_stderr(generation_skill):assert result.stdout == ""assert "nv_build / nvidia/" in result.stderrassert "evaluation source ready" in result.stderr+ assert "One-case starter: a starting point, not a representative evaluation" in result.stderr+ assert "no negative case that should not trigger the skill" in result.stderr+ assert f"skillevaluator create-eval-dataset {generation_skill} --full --force" in result.stderr+ assert "`--force` overwrites the starter and discards any edits to it." in result.stderr+ assert f"skillevaluator tier3 validate {generation_skill}" in result.stderrassert (generation_skill / "evals" / "evals.json").exists()diff --git a/tests/tier3/test_workflow.py b/tests/tier3/test_workflow.pyindex e807a7c0..df1e3e59 100644--- a/tests/tier3/test_workflow.py+++ b/tests/tier3/test_workflow.py@@ -11,7 +11,7 @@ from pathlib import Pathimport pytestfrom click.testing import CliRunner-from skillevaluator.cli import evaluate+from skillevaluator.cli import create_dataset, evaluatefrom skillevaluator.evaluation import EvaluationServicefrom skillevaluator.tier3.workflow import build_tier3_workflow@@ -111,11 +111,24 @@ def test_direct_workflow_generates_missing_dataset_then_evaluates(skill, configuresult = CliRunner().invoke(_command(), [str(skill), "--progress", "off"])assert result.exit_code == 0, result.outputassert calls == [True]+ assert "One-case starter: a starting point, not a representative evaluation" in result.stderr+ assert "no negative case that should not trigger the skill" in result.stderr+ assert f"skillevaluator create-eval-dataset {skill} --full --force" in result.stderr+ assert "`--force` overwrites the starter and discards any edits to it." in result.stderr+ assert f"skillevaluator tier3 validate {skill}" in result.stderrassert len(configured) == 1assert configured[0].agents == "opencode"assert configured[0].skip_baseline is False+def test_starter_guidance_command_replaces_the_existing_dataset(skill):+ dataset = _dataset(skill)+ original = dataset.read_bytes()+ result = CliRunner().invoke(create_dataset, [str(skill), "--full", "--force", "--no-llm"])+ assert result.exit_code == 0, result.output+ assert dataset.read_bytes() != original++@pytest.mark.parametrize(("provider", "credential", "expected_agent"),[