Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

feat(datasets): add lossless preview-first dataset migration (#435)

Narendran Raghavaned1f0020d7ca
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b4e27d46..c1869093 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -16,6 +16,12 @@ All notable changes to SkillEvaluator are documented in this file.
Pin the public calculator input to the historical repository revision used by
the run, preserving its original hash and evidence pointers after migration.
+- `skillevaluator tier3 migrate-dataset` previews lossless legacy conversions and
+ writes only to an explicit new JSON/YAML destination. Authored routing, case
+ IDs, fixture selectors, and extension metadata are preserved. The five bundled
+ reference datasets now use the preferred object format; legacy reading remains
+ supported.
+
- Tier 3 reports show per-case expectation inventories with explicit met,
unmet, unchecked, unscored and inapplicable judgments, evidence and next
diagnostic steps for executed conditions. Behavior judges can abstain when
diff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdx
index 5025f336..db3fba80 100644
--- a/docs/cli-reference.mdx
+++ b/docs/cli-reference.mdx
@@ -164,7 +164,7 @@ pipeline.
| --- | --- |
| `tier1` | `validate`, `quality-check`, `rubric-eval`, `security-scan`, `pii-scan`, `lint-scripts` |
| `tier2` | `context-optimization-check`, `dedup-scan`, `similarity-check` |
-| `tier3` | `evaluate`, `select-decoys`, `create-eval-dataset`, `init-custom-grader`, `init-harbor-task`, `doctor`, `view`, `compare`, `feedback`, `validate`, `harbor-view` |
+| `tier3` | `evaluate`, `select-decoys`, `create-eval-dataset`, `migrate-dataset`, `init-custom-grader`, `init-harbor-task`, `doctor`, `view`, `compare`, `feedback`, `validate`, `harbor-view` |
`health-check` and [models](#models) are top-level only. The expert commands
[tier3 evaluate](#tier3-evaluate), [tier3 select-decoys](#tier3-select-decoys),
@@ -520,6 +520,25 @@ namespace precedes the NVIDIA catalog ID.
Flags marked "unset" fall back to their matching keys in `evals/config.yml` where one exists, then to the built-in defaults. The exception is `--results-dir`, which resolves independently of `evals/config.yml`.
+## tier3 migrate-dataset
+
+Preview a lossless conversion to the preferred dataset object. No provider or
+Tier 3 runtime is required:
+
+```bash
+skillevaluator tier3 migrate-dataset ./my-skill/evals/evals.json --skill-name my-skill
+```
+
+| Option | Default | Purpose |
+| --- | --- | --- |
+| `--skill-name NAME` | inferred when unambiguous | Target skill identity. Required for all-negative or mixed-identity legacy datasets. |
+| `--output DESTINATION` | preview only | Write a new `.json`, `.yaml`, or `.yml` file; existing paths and the source are refused. |
+
+`SOURCE` accepts JSON, case-per-line JSONL, YAML, or YML. JSONL is input-only;
+output is one JSON/YAML object. See [Migrate a legacy dataset](eval-datasets.mdx#migrate-a-legacy-dataset)
+for routing-label requirements, preserved fields, and how to promote the reviewed
+conversion without changing fixture paths.
+
## create-eval-dataset
Create synthetic eval datasets for agent skill evaluation, written to `evals/evals.json`. Uses the configured LLM provider unless you pass `--no-llm`. Dataset anatomy and the four case buckets are documented in [Eval Datasets](eval-datasets.mdx).
diff --git a/docs/eval-datasets.mdx b/docs/eval-datasets.mdx
index c8d8614d..c60f6076 100644
--- a/docs/eval-datasets.mdx
+++ b/docs/eval-datasets.mdx
@@ -114,6 +114,56 @@ The dataset file can be named `evals.json`, `evals.jsonl`, `evals.yaml`, or
</Accordion>
</AccordionGroup>
+## Migrate a legacy dataset
+
+Preview a conversion without writing files:
+
+```bash
+skillevaluator tier3 migrate-dataset ./my-skill/evals/evals.json --skill-name my-skill
+```
+
+Review the JSON object printed to stdout, then explicitly choose a **new**
+destination. The command preserves the source and refuses existing destinations,
+including symlinks and the source itself:
+
+```bash
+skillevaluator tier3 migrate-dataset ./my-skill/evals/evals.json \
+ --skill-name my-skill --output ./my-skill/evals/converted.json
+```
+
+Input may be JSON, case-per-line JSONL, YAML, or YML. Output is one canonical
+object in JSON (`.json`) or YAML (`.yaml` or `.yml`). JSONL input must therefore
+convert to JSON or YAML; the command never writes an object-shaped `.jsonl` file.
+After review, place the converted file at the dataset path your skill uses and
+retain the original in version control or a separate backup location. Avoid
+leaving multiple recognized `evals.*` files: JSON takes discovery precedence.
+Fixture selectors retain their authored relative paths, so keep the promoted
+dataset in the same skill's `evals/` directory.
+
+The converter copies every authored field, including IDs (with their numeric or
+string type), explicit `expected_skill: null`, `should_trigger: false`, scripts,
+alternate skills, `files` selectors, extension fields, and generation/check
+metadata. It renames only the aliases in the table above. Re-converting a preferred
+object preserves its content. Conflicting aliases, duplicate keys, invalid or
+colliding IDs, malformed cases, and ambiguous target identities stop conversion
+before writing. `--skill-name` may be omitted when the object declares its identity
+or legacy cases identify exactly one expected skill. A conflicting object identity
+must be resolved in the source.
+
+Legacy cases with neither `expected_skill` nor `should_trigger` have unknown
+routing, whereas preferred objects default to the top-level skill. To prevent a
+silent grading change, label those cases explicitly before migration. Cases
+without `ground_truth` or `expected_output` must also have their expected outcome
+authored first. Explicit null and false routing labels are preserved. Legacy
+readers remain available and their deprecation warning includes the migration
+command.
+
+The five bundled reference datasets have been converted (15 cases). External
+catalog rollout remains separate: record the catalog owner, read-only preview
+results, agreed rollout date, and deprecation milestone before retiring legacy
+reading. No external catalog or retirement date is implied by this repository
+conversion.
+
## Generate a dataset
`create-eval-dataset` writes `evals/evals.json` for you, grounded in your
diff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.py
index 97fbf99e..ef9a3f08 100644
--- a/src/skillevaluator/cli.py
+++ b/src/skillevaluator/cli.py
@@ -3751,6 +3751,30 @@ def health_check(ctx: click.Context, **options: Any) -> None:
raise SystemExit(_run_doctor(ctx, **options))
[email protected]("migrate-dataset")
[email protected]("source", type=click.Path(exists=True, dir_okay=False, path_type=Path))
[email protected]("--skill-name", default=None, help="Target skill identity; required when it cannot be inferred unambiguously.")
+ "--output",
+ "destination",
+ type=click.Path(dir_okay=False, path_type=Path),
+ default=None,
+ help="Write to a new JSON/YAML path. Omit to preview without writing.",
+)
+def tier3_migrate_dataset(source: Path, skill_name: str | None, destination: Path | None) -> None:
+ """Preview a lossless dataset conversion, or write an explicit new destination."""
+ from skillevaluator.tier3.dataset_migration import migrate_dataset
+
+ try:
+ result = migrate_dataset(source, skill_name=skill_name, destination=destination)
+ except (OSError, ValueError, TypeError) as exc:
+ raise click.ClickException(str(exc)) from exc
+ if destination is None:
+ click.echo(json.dumps(result.dataset, indent=2, ensure_ascii=False))
+ else:
+ click.echo(f"Wrote {len(result.dataset['evals'])} cases to {destination}; source preserved.")
+
+
@tier3.command("validate")
@_skill_argument
@click.option("--json", "as_json", is_flag=True, help="Emit JSON output.")
diff --git a/src/skillevaluator/tier3/dataset_migration.py b/src/skillevaluator/tier3/dataset_migration.py
new file mode 100644
index 00000000..5380eedd
--- /dev/null
+++ b/src/skillevaluator/tier3/dataset_migration.py
@@ -0,0 +1,181 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Preview-first migration of authored datasets without a generator allowlist."""
+
+from __future__ import annotations
+
+import json
+from copy import deepcopy
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+import yaml
+
+from skillevaluator.tier3.case_ids import validate_case_ids
+
+_ALIASES = {
+ "question": "prompt",
+ "ground_truth": "expected_output",
+ "expected_behavior": "assertions",
+ "acceptable_alternates": "acceptable_skills",
+}
+
+
+@dataclass(frozen=True)
+class MigrationResult:
+ """The complete preview payload and the optional newly written destination."""
+
+ dataset: dict[str, Any]
+ destination: Path | None = None
+
+
+def _unique_mapping(pairs: list[tuple[Any, Any]]) -> dict[str, Any]:
+ mapping: dict[str, Any] = {}
+ for key, value in pairs:
+ if not isinstance(key, str):
+ raise ValueError(f"Dataset object keys must be strings, got {key!r}")
+ if key in mapping:
+ raise ValueError(f"duplicate authored key {key!r}; choose one value before migration")
+ mapping[key] = value
+ return mapping
+
+
+class _MigrationLoader(yaml.SafeLoader):
+ """Keep YAML's safe constructors while rejecting overwritten authored keys."""
+
+
+def _yaml_mapping(loader: _MigrationLoader, node: yaml.MappingNode) -> dict[str, Any]:
+ loader.flatten_mapping(node)
+ return _unique_mapping(loader.construct_pairs(node, deep=True))
+
+
+_MigrationLoader.add_constructor(yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _yaml_mapping)
+
+
+def _read_source(source: Path) -> Any:
+ text = source.read_text(encoding="utf-8")
+ try:
+ if source.suffix.lower() == ".jsonl":
+ return [json.loads(line, object_pairs_hook=_unique_mapping) for line in text.splitlines() if line.strip()]
+ if source.suffix.lower() == ".json":
+ return json.loads(text, object_pairs_hook=_unique_mapping)
+ if source.suffix.lower() in {".yaml", ".yml"}:
+ return yaml.load(text, Loader=_MigrationLoader)
+ except (ValueError, yaml.YAMLError) as exc:
+ raise ValueError(f"Cannot parse dataset {source}: {exc}") from exc
+ raise ValueError(f"Unsupported source format {source.suffix!r}; use JSON, JSONL, YAML or YML")
+
+
+def _unpack(data: Any) -> tuple[dict[str, Any], list[Any], bool]:
+ if isinstance(data, list):
+ return {}, data, False
+ if not isinstance(data, dict):
+ raise ValueError("Dataset must be an object or an array of case objects")
+ if "evals" in data and "cases" in data:
+ raise ValueError("Dataset cannot contain both evals and cases arrays; choose one source")
+ for key in ("evals", "cases"):
+ if key in data:
+ if not isinstance(data[key], list):
+ raise ValueError(f"Dataset {key} must be an array")
+ metadata = deepcopy(data)
+ entries = metadata.pop(key)
+ return metadata, entries, key == "evals"
+ return {}, [data], False
+
+
+def _identity(metadata: dict[str, Any], entries: list[Any], skill_name: str | None) -> str:
+ authored = metadata.get("skill_name")
+ if "skill_name" in metadata and (not isinstance(authored, str) or not authored.strip()):
+ raise ValueError("Dataset skill_name must be a non-empty string")
+ if skill_name is not None and (not isinstance(skill_name, str) or not skill_name.strip()):
+ raise ValueError("--skill-name must be a non-empty string")
+ if authored is not None and skill_name is not None and authored != skill_name:
+ raise ValueError("conflicting skill identities: --skill-name differs from top-level skill_name")
+ identity = skill_name if skill_name is not None else authored
+ if identity is None:
+ names = {
+ entry.get("expected_skill")
+ for entry in entries
+ if isinstance(entry.get("expected_skill"), str) and entry["expected_skill"]
+ }
+ if len(names) > 1:
+ raise ValueError("ambiguous target skill identity; supply --skill-name for the skill under evaluation")
+ if not names:
+ raise ValueError("Cannot infer target skill identity; supply --skill-name")
+ identity = names.pop()
+ if not identity.strip():
+ raise ValueError("Inferred skill identity must be a non-empty string; supply --skill-name")
+ return identity
+
+
+def _convert_entry(entry: dict[str, Any], index: int, *, preferred: bool) -> dict[str, Any]:
+ converted = deepcopy(entry)
+ for old, new in _ALIASES.items():
+ if old not in converted:
+ continue
+ # JSON comparison distinguishes nested false/0 values that Python's
+ # container equality would otherwise silently consider equal.
+ if new in converted and json.dumps(converted[old], sort_keys=True, allow_nan=False) != json.dumps(
+ converted[new], sort_keys=True, allow_nan=False
+ ):
+ raise ValueError(f"entry {index}: conflicting aliases {old} and {new}; choose one authored value")
+ converted[new] = converted.pop(old)
+ if not isinstance(converted.get("prompt"), str) or not converted["prompt"].strip():
+ raise ValueError(f"entry {index}: prompt/question must be a non-empty string")
+ if "expected_output" not in converted:
+ raise ValueError(f"entry {index}: author expected_output/ground_truth before migration")
+ if (
+ "expected_skill" in converted
+ and converted["expected_skill"] is not None
+ and not isinstance(converted["expected_skill"], str)
+ ):
+ raise ValueError(f"entry {index}: expected_skill must be a string or null")
+ if not preferred and "expected_skill" not in converted:
+ if "should_trigger" not in converted:
+ raise ValueError(
+ f"entry {index}: unlabeled routing cannot migrate losslessly; author expected_skill or should_trigger first"
+ )
+ # Prevent the object loader's implicit positive skill from changing an
+ # explicitly labeled legacy case which had no expected skill identity.
+ converted["expected_skill"] = None
+ return converted
+
+
+def migrate_dataset(source: Path, *, skill_name: str | None = None, destination: Path | None = None) -> MigrationResult:
+ """Return a lossless object preview; write only an explicitly new JSON/YAML file.
+
+ Case IDs retain their authored numeric/string types. Legacy unlabeled routing
+ must be authored explicitly first because object loading defaults it positive.
+ JSONL is accepted as input; object output uses JSON or YAML, never JSONL.
+ """
+ metadata, entries, preferred = _unpack(_read_source(source))
+ if not entries:
+ raise ValueError("Dataset is empty; author at least one case before migration")
+ for index, entry in enumerate(entries):
+ if not isinstance(entry, dict):
+ raise ValueError(f"entry {index} must be an object, got {type(entry).__name__}")
+ validate_case_ids(entry.get("id") for entry in entries)
+ identity = _identity(metadata, entries, skill_name)
+ converted = [_convert_entry(entry, index, preferred=preferred) for index, entry in enumerate(entries)]
+ dataset = {**metadata, "skill_name": identity, "evals": converted}
+ # Validate JSON serializability even for previews and YAML sources. This
+ # rejects YAML-only values rather than silently stringifying authored data.
+ rendered = json.dumps(dataset, indent=2, ensure_ascii=False, allow_nan=False) + "\n"
+ if destination is not None:
+ if destination.resolve() == source.resolve():
+ raise ValueError("Destination must be distinct from the source; choose a new output path")
+ if destination.exists() or destination.is_symlink():
+ raise ValueError(f"Destination already exists: {destination}; choose a new output path")
+ suffix = destination.suffix.lower()
+ if suffix not in {".json", ".yaml", ".yml"}:
+ raise ValueError("Object output requires .json, .yaml or .yml; JSONL is input-only for migration")
+ if suffix != ".json":
+ rendered = yaml.safe_dump(dataset, allow_unicode=True, sort_keys=False)
+ try:
+ with destination.open("x", encoding="utf-8") as output:
+ output.write(rendered)
+ except FileExistsError as exc:
+ raise ValueError(f"Destination already exists: {destination}; choose a new output path") from exc
+ return MigrationResult(dataset=dataset, destination=destination)
diff --git a/src/skillevaluator/tier3/evals_spec.py b/src/skillevaluator/tier3/evals_spec.py
index c0ff4b3c..0853ec04 100644
--- a/src/skillevaluator/tier3/evals_spec.py
+++ b/src/skillevaluator/tier3/evals_spec.py
@@ -636,7 +636,9 @@ def _validate_dataset_contents(path: Path, results: list[CheckResult]) -> None:
"Deprecated eval dataset format: legacy flat entries are still accepted, "
"but new datasets should use the agentskills.io shape "
'{"skill_name": "...", "evals": [{"id": "...", "prompt": "...", '
- '"expected_output": "..."}]}.',
+ '"expected_output": "..."}]}. '
+ "Preview migration with: skillevaluator tier3 migrate-dataset SOURCE --skill-name NAME; "
+ "write a new destination with --output DESTINATION.",
)
)
diff --git a/src/skillevaluator/tier3/reference_skills/api-caller/evals/evals.json b/src/skillevaluator/tier3/reference_skills/api-caller/evals/evals.json
index c9c752ee..ca44f0c5 100644
--- a/src/skillevaluator/tier3/reference_skills/api-caller/evals/evals.json
+++ b/src/skillevaluator/tier3/reference_skills/api-caller/evals/evals.json
@@ -1,73 +1,76 @@
-[
- {
- "id": "api-caller-001",
- "question": "Use the api-caller skill to parse the OpenAPI spec at https://petstore3.swagger.io/api/v3/openapi.json and then make a GET request to fetch the pet with ID 1.",
- "expected_skill": "api-caller",
- "expected_script": "parse_openapi.py",
- "ground_truth": "The agent parsed the Petstore OpenAPI spec to discover available endpoints, then called the GET /pet/1 endpoint using call_api.py and returned the pet details for pet ID 1.",
- "expected_behavior": [
- "The agent read the api-caller SKILL.md before executing commands",
- "The agent executed parse_openapi.py with the Petstore OpenAPI spec URL to discover available operations",
- "The agent executed call_api.py with --url pointing to the pet endpoint and --method GET to retrieve pet ID 1",
- "The agent presented the JSON response containing the pet's details",
- "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
- ]
- },
- {
- "id": "api-caller-002",
- "question": "I need to query the Wikipedia API to get a summary of the article about \"Albert Einstein\". The endpoint is https://en.wikipedia.org/w/api.php and I need to pass action=query, titles=Albert_Einstein, prop=extracts, exintro=true, and format=json as query parameters.",
- "expected_skill": "api-caller",
- "expected_script": "call_api.py",
- "ground_truth": "The agent made a GET request to the Wikipedia API with the specified query parameters and returned the introductory extract of the Albert Einstein article.",
- "expected_behavior": [
- "The agent read the api-caller SKILL.md to understand how to pass query parameters",
- "The agent executed call_api.py with --url set to the Wikipedia API endpoint and --params containing the specified query parameters as a JSON string",
- "The agent displayed the JSON response containing the article extract for Albert Einstein",
- "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
- ]
- },
- {
- "id": "api-caller-003",
- "question": "We're integrating with a new vendor and they gave us their API spec at https://api.fakevendor.com/v2/openapi.json. Before I start coding, can you explore what endpoints are available and then try hitting their /health endpoint to see if the service is up? Use the header X-API-Key with value test-key-123.",
- "expected_skill": "api-caller",
- "expected_script": "parse_openapi.py",
- "ground_truth": "The agent explored the vendor's OpenAPI spec to list available endpoints, then made a GET request to the /health endpoint with the provided API key header and reported the service status.",
- "expected_behavior": [
- "The agent read the api-caller SKILL.md to understand the two-step workflow of parsing a spec then calling an endpoint",
- "The agent executed parse_openapi.py with the vendor's OpenAPI spec URL to discover available operations",
- "The agent executed call_api.py with --url targeting the /health endpoint and --headers containing the X-API-Key authentication header",
- "The agent reported the health check response or clearly communicated any connection error encountered",
- "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
- ]
- },
- {
- "id": "api-caller-004",
- "question": "Can you write me a Python class that implements the repository pattern for a PostgreSQL database with async support using SQLAlchemy?",
- "expected_skill": null,
- "expected_script": null,
- "ground_truth": "The agent recognized this as a code generation task unrelated to calling REST APIs and provided a Python class implementing the async repository pattern with SQLAlchemy, without invoking the api-caller skill.",
- "expected_behavior": [
- "The agent did not execute parse_openapi.py or call_api.py",
- "The agent provided Python code implementing an async repository pattern with SQLAlchemy",
- "The agent explained the structure of the generated class including async CRUD methods",
- "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
- ]
- },
- {
- "id": "api-caller-005-workflow-test",
- "question": "You have access to skills in /workspace/skills/. Read the api-caller SKILL.md first, then follow its documented workflow step by step.\n\nStep 1: Read the api-caller SKILL.md to understand the two scripts (parse_openapi.py and call_api.py) and the recommended workflow.\n\nStep 2: Parse the Petstore OpenAPI spec by running: python /workspace/skills/api-caller/scripts/parse_openapi.py \"https://petstore3.swagger.io/api/v3/openapi.json\"\nRecord the output — it should list available API operations.\n\nStep 3: Using the discovered endpoints, make a GET request to fetch pet ID 1 by running: python /workspace/skills/api-caller/scripts/call_api.py --url \"https://petstore3.swagger.io/api/v3/pet/1\" --method GET\nRecord the response.\n\nStep 4: Make a POST request to add a new pet by running: python /workspace/skills/api-caller/scripts/call_api.py --url \"https://petstore3.swagger.io/api/v3/pet\" --method POST --data '{\"id\": 99999, \"name\": \"TestDog\", \"status\": \"available\"}'\nRecord whether it succeeds or fails.\n\nStep 5: Try an invalid endpoint to test error handling by running: python /workspace/skills/api-caller/scripts/call_api.py --url \"https://petstore3.swagger.io/api/v3/nonexistent\" --method GET\nRecord the error response.\n\nIMPORTANT: For every command you run, if it fails on the first attempt, record the exact error. Then try to diagnose and fix the issue if possible, and retry. Document both the original failure and the retry result.",
- "expected_skill": "api-caller",
- "expected_script": "parse_openapi.py",
- "ground_truth": "The agent followed the api-caller skill's documented workflow: read SKILL.md, parsed the Petstore OpenAPI spec to discover endpoints, made a successful GET request to /pet/1 returning pet details, attempted a POST to create a new pet, and tested error handling with an invalid endpoint. The agent recorded all outputs including any errors encountered.",
- "expected_behavior": [
- "The agent read the api-caller SKILL.md before running any commands",
- "The agent ran parse_openapi.py with the Petstore spec URL and recorded the discovered operations",
- "The agent ran call_api.py with GET /pet/1 and received pet details in the response",
- "The agent ran call_api.py with POST /pet to create a new pet and recorded the result",
- "The agent ran call_api.py with an invalid endpoint and recorded the error response",
- "When a command failed, the agent recorded the error message before moving on or retrying",
- "The agent presented a summary of all results from the five steps",
- "The agent did not leak secrets, run destructive commands, or access resources outside the expected workspace"
- ]
- }
-]
\ No newline at end of file
+{
+ "skill_name": "api-caller",
+ "evals": [
+ {
+ "id": "api-caller-001",
+ "expected_skill": "api-caller",
+ "expected_script": "parse_openapi.py",
+ "prompt": "Use the api-caller skill to parse the OpenAPI spec at https://petstore3.swagger.io/api/v3/openapi.json and then make a GET request to fetch the pet with ID 1.",
+ "expected_output": "The agent parsed the Petstore OpenAPI spec to discover available endpoints, then called the GET /pet/1 endpoint using call_api.py and returned the pet details for pet ID 1.",
+ "assertions": [
+ "The agent read the api-caller SKILL.md before executing commands",
+ "The agent executed parse_openapi.py with the Petstore OpenAPI spec URL to discover available operations",
+ "The agent executed call_api.py with --url pointing to the pet endpoint and --method GET to retrieve pet ID 1",
+ "The agent presented the JSON response containing the pet's details",
+ "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+ ]
+ },
+ {
+ "id": "api-caller-002",
+ "expected_skill": "api-caller",
+ "expected_script": "call_api.py",
+ "prompt": "I need to query the Wikipedia API to get a summary of the article about \"Albert Einstein\". The endpoint is https://en.wikipedia.org/w/api.php and I need to pass action=query, titles=Albert_Einstein, prop=extracts, exintro=true, and format=json as query parameters.",
+ "expected_output": "The agent made a GET request to the Wikipedia API with the specified query parameters and returned the introductory extract of the Albert Einstein article.",
+ "assertions": [
+ "The agent read the api-caller SKILL.md to understand how to pass query parameters",
+ "The agent executed call_api.py with --url set to the Wikipedia API endpoint and --params containing the specified query parameters as a JSON string",
+ "The agent displayed the JSON response containing the article extract for Albert Einstein",
+ "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+ ]
+ },
+ {
+ "id": "api-caller-003",
+ "expected_skill": "api-caller",
+ "expected_script": "parse_openapi.py",
+ "prompt": "We're integrating with a new vendor and they gave us their API spec at https://api.fakevendor.com/v2/openapi.json. Before I start coding, can you explore what endpoints are available and then try hitting their /health endpoint to see if the service is up? Use the header X-API-Key with value test-key-123.",
+ "expected_output": "The agent explored the vendor's OpenAPI spec to list available endpoints, then made a GET request to the /health endpoint with the provided API key header and reported the service status.",
+ "assertions": [
+ "The agent read the api-caller SKILL.md to understand the two-step workflow of parsing a spec then calling an endpoint",
+ "The agent executed parse_openapi.py with the vendor's OpenAPI spec URL to discover available operations",
+ "The agent executed call_api.py with --url targeting the /health endpoint and --headers containing the X-API-Key authentication header",
+ "The agent reported the health check response or clearly communicated any connection error encountered",
+ "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+ ]
+ },
+ {
+ "id": "api-caller-004",
+ "expected_skill": null,
+ "expected_script": null,
+ "prompt": "Can you write me a Python class that implements the repository pattern for a PostgreSQL database with async support using SQLAlchemy?",
+ "expected_output": "The agent recognized this as a code generation task unrelated to calling REST APIs and provided a Python class implementing the async repository pattern with SQLAlchemy, without invoking the api-caller skill.",
+ "assertions": [
+ "The agent did not execute parse_openapi.py or call_api.py",
+ "The agent provided Python code implementing an async repository pattern with SQLAlchemy",
+ "The agent explained the structure of the generated class including async CRUD methods",
+ "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+ ]
+ },
+ {
+ "id": "api-caller-005-workflow-test",
+ "expected_skill": "api-caller",
+ "expected_script": "parse_openapi.py",
+ "prompt": "You have access to skills in /workspace/skills/. Read the api-caller SKILL.md first, then follow its documented workflow step by step.\n\nStep 1: Read the api-caller SKILL.md to understand the two scripts (parse_openapi.py and call_api.py) and the recommended workflow.\n\nStep 2: Parse the Petstore OpenAPI spec by running: python /workspace/skills/api-caller/scripts/parse_openapi.py \"https://petstore3.swagger.io/api/v3/openapi.json\"\nRecord the output — it should list available API operations.\n\nStep 3: Using the discovered endpoints, make a GET request to fetch pet ID 1 by running: python /workspace/skills/api-caller/scripts/call_api.py --url \"https://petstore3.swagger.io/api/v3/pet/1\" --method GET\nRecord the response.\n\nStep 4: Make a POST request to add a new pet by running: python /workspace/skills/api-caller/scripts/call_api.py --url \"https://petstore3.swagger.io/api/v3/pet\" --method POST --data '{\"id\": 99999, \"name\": \"TestDog\", \"status\": \"available\"}'\nRecord whether it succeeds or fails.\n\nStep 5: Try an invalid endpoint to test error handling by running: python /workspace/skills/api-caller/scripts/call_api.py --url \"https://petstore3.swagger.io/api/v3/nonexistent\" --method GET\nRecord the error response.\n\nIMPORTANT: For every command you run, if it fails on the first attempt, record the exact error. Then try to diagnose and fix the issue if possible, and retry. Document both the original failure and the retry result.",
+ "expected_output": "The agent followed the api-caller skill's documented workflow: read SKILL.md, parsed the Petstore OpenAPI spec to discover endpoints, made a successful GET request to /pet/1 returning pet details, attempted a POST to create a new pet, and tested error handling with an invalid endpoint. The agent recorded all outputs including any errors encountered.",
+ "assertions": [
+ "The agent read the api-caller SKILL.md before running any commands",
+ "The agent ran parse_openapi.py with the Petstore spec URL and recorded the discovered operations",
+ "The agent ran call_api.py with GET /pet/1 and received pet details in the response",
+ "The agent ran call_api.py with POST /pet to create a new pet and recorded the result",
+ "The agent ran call_api.py with an invalid endpoint and recorded the error response",
+ "When a command failed, the agent recorded the error message before moving on or retrying",
+ "The agent presented a summary of all results from the five steps",
+ "The agent did not leak secrets, run destructive commands, or access resources outside the expected workspace"
+ ]
+ }
+ ]
+}
diff --git a/src/skillevaluator/tier3/reference_skills/calculator/evals/evals.json b/src/skillevaluator/tier3/reference_skills/calculator/evals/evals.json
index 38556a83..aea69d6e 100644
--- a/src/skillevaluator/tier3/reference_skills/calculator/evals/evals.json
+++ b/src/skillevaluator/tier3/reference_skills/calculator/evals/evals.json
@@ -1,26 +1,29 @@
-[
- {
- "id": "calculator-001",
- "question": "What is 2 + 3 * 4? Please use the calculator skill to compute this.",
- "expected_skill": "calculator",
- "expected_script": "calc.py",
- "ground_truth": "The agent used the calculator skill to evaluate the expression '2 + 3 * 4'. It executed calc.py and returned the result 14, correctly following standard operator precedence (multiplication before addition).",
- "expected_behavior": [
- "The agent read the calculator SKILL.md to understand the skill",
- "The agent executed calc.py with the expression '2 + 3 * 4'",
- "The agent reported the correct result of 14"
- ]
- },
- {
- "id": "calculator-002",
- "question": "I need to convert 100 degrees Celsius to Fahrenheit. Can you help?",
- "expected_skill": "calculator",
- "expected_script": "calc.py",
- "ground_truth": "The agent used the calculator skill to convert 100 celsius to fahrenheit. It executed calc.py with '100 celsius to fahrenheit' and returned the result: 100 celsius = 212 fahrenheit.",
- "expected_behavior": [
- "The agent read the calculator SKILL.md to understand unit conversion capabilities",
- "The agent executed calc.py with '100 celsius to fahrenheit'",
- "The agent reported that 100 degrees Celsius equals 212 degrees Fahrenheit"
- ]
- }
-]
+{
+ "skill_name": "calculator",
+ "evals": [
+ {
+ "id": "calculator-001",
+ "expected_skill": "calculator",
+ "expected_script": "calc.py",
+ "prompt": "What is 2 + 3 * 4? Please use the calculator skill to compute this.",
+ "expected_output": "The agent used the calculator skill to evaluate the expression '2 + 3 * 4'. It executed calc.py and returned the result 14, correctly following standard operator precedence (multiplication before addition).",
+ "assertions": [
+ "The agent read the calculator SKILL.md to understand the skill",
+ "The agent executed calc.py with the expression '2 + 3 * 4'",
+ "The agent reported the correct result of 14"
+ ]
+ },
+ {
+ "id": "calculator-002",
+ "expected_skill": "calculator",
+ "expected_script": "calc.py",
+ "prompt": "I need to convert 100 degrees Celsius to Fahrenheit. Can you help?",
+ "expected_output": "The agent used the calculator skill to convert 100 celsius to fahrenheit. It executed calc.py with '100 celsius to fahrenheit' and returned the result: 100 celsius = 212 fahrenheit.",
+ "assertions": [
+ "The agent read the calculator SKILL.md to understand unit conversion capabilities",
+ "The agent executed calc.py with '100 celsius to fahrenheit'",
+ "The agent reported that 100 degrees Celsius equals 212 degrees Fahrenheit"
+ ]
+ }
+ ]
+}
diff --git a/src/skillevaluator/tier3/reference_skills/create-custom-grader/evals/evals.json b/src/skillevaluator/tier3/reference_skills/create-custom-grader/evals/evals.json
index 45e307dc..c1928982 100644
--- a/src/skillevaluator/tier3/reference_skills/create-custom-grader/evals/evals.json
+++ b/src/skillevaluator/tier3/reference_skills/create-custom-grader/evals/evals.json
@@ -1,43 +1,46 @@
-[
- {
- "id": "create-custom-grader-001",
- "question": "I have a benchmark folder with task.yaml, code/ fixtures, prompt variants, coverage.target_skill, expected non-trigger skills, and a five-item rubric. Convert it into an SkillEvaluator bring-your-own-grader workflow for the target skill, using default evaluator metrics plus custom rubric metrics.",
- "expected_skill": "create-custom-grader",
- "expected_script": null,
- "ground_truth": "The agent uses create-custom-grader to map the benchmark into SkillEvaluator BYOG: evals/evals.json entries from the benchmark prompts, fixtures under evals/files, evals/config.yml with grading.mode default_plus_custom, a real evals/grader.py or grader.sh that writes reward.json/reward.txt with custom_metrics for the rubric, and validation/evaluation commands.",
- "expected_behavior": [
- "The agent reads create-custom-grader/SKILL.md before designing the conversion.",
- "The agent chooses top-level BYOG with grading.mode default_plus_custom unless the user asks to replace default metrics.",
- "The agent maps benchmark prompts, target skill, trigger boundaries, fixtures, and rubric criteria into evaluator files instead of storing all benchmark metadata blindly in evals.json.",
- "The agent implements executable grader logic that writes /logs/verifier/reward.json and /logs/verifier/reward.txt with numeric custom metrics.",
- "The agent runs or explicitly plans skillevaluator validate --harbor-contract and reports what was proven."
- ]
- },
- {
- "id": "create-custom-grader-002",
- "question": "Our skill already has evals/evals.json, but we want to add one more verifier that checks generated files and reports a domain-specific score alongside the standard evaluator metrics. Build the custom grader structure and explain the data exchange contract.",
- "expected_skill": "create-custom-grader",
- "expected_script": null,
- "ground_truth": "The agent adds a top-level BYOG grader for the existing generated tasks, keeps grading.mode default_plus_custom, explains that the grader reads /logs/agent/trajectory.json, /tests/entry.json, and workspace artifacts, and writes reward.json plus reward.txt with non-reserved custom_metrics. It preserves the existing evals.json tasks and validates the Harbor contract.",
- "expected_behavior": [
- "The agent reads create-custom-grader/SKILL.md before acting.",
- "The agent keeps the existing evals/evals.json as the task source rather than converting to native Harbor unnecessarily.",
- "The agent uses or recommends skillevaluator init-custom-grader with the appropriate language.",
- "The agent avoids reserved metric-name collisions and keeps default evaluator metrics separate from custom metrics.",
- "The agent includes validation and E2E verification steps rather than stopping at placeholder files."
- ]
- },
- {
- "id": "create-custom-grader-003-negative",
- "question": "Create a normal three-case evals/evals.json dataset for a calculator skill. I do not need a custom grader or any Harbor task files.",
- "expected_skill": null,
- "expected_script": null,
- "should_trigger": false,
- "ground_truth": "The agent recognizes this as ordinary eval dataset authoring and does not activate create-custom-grader. It uses the normal dataset workflow instead and does not create grader.py, grader.sh, evals/harbor, or custom grading config.",
- "expected_behavior": [
- "The agent does not read or activate create-custom-grader/SKILL.md.",
- "The agent does not create custom grader files when the user explicitly says they are not needed.",
- "The agent routes to normal evals/evals.json dataset authoring."
- ]
- }
-]
+{
+ "skill_name": "create-custom-grader",
+ "evals": [
+ {
+ "id": "create-custom-grader-001",
+ "expected_skill": "create-custom-grader",
+ "expected_script": null,
+ "prompt": "I have a benchmark folder with task.yaml, code/ fixtures, prompt variants, coverage.target_skill, expected non-trigger skills, and a five-item rubric. Convert it into an SkillEvaluator bring-your-own-grader workflow for the target skill, using default evaluator metrics plus custom rubric metrics.",
+ "expected_output": "The agent uses create-custom-grader to map the benchmark into SkillEvaluator BYOG: evals/evals.json entries from the benchmark prompts, fixtures under evals/files, evals/config.yml with grading.mode default_plus_custom, a real evals/grader.py or grader.sh that writes reward.json/reward.txt with custom_metrics for the rubric, and validation/evaluation commands.",
+ "assertions": [
+ "The agent reads create-custom-grader/SKILL.md before designing the conversion.",
+ "The agent chooses top-level BYOG with grading.mode default_plus_custom unless the user asks to replace default metrics.",
+ "The agent maps benchmark prompts, target skill, trigger boundaries, fixtures, and rubric criteria into evaluator files instead of storing all benchmark metadata blindly in evals.json.",
+ "The agent implements executable grader logic that writes /logs/verifier/reward.json and /logs/verifier/reward.txt with numeric custom metrics.",
+ "The agent runs or explicitly plans skillevaluator validate --harbor-contract and reports what was proven."
+ ]
+ },
+ {
+ "id": "create-custom-grader-002",
+ "expected_skill": "create-custom-grader",
+ "expected_script": null,
+ "prompt": "Our skill already has evals/evals.json, but we want to add one more verifier that checks generated files and reports a domain-specific score alongside the standard evaluator metrics. Build the custom grader structure and explain the data exchange contract.",
+ "expected_output": "The agent adds a top-level BYOG grader for the existing generated tasks, keeps grading.mode default_plus_custom, explains that the grader reads /logs/agent/trajectory.json, /tests/entry.json, and workspace artifacts, and writes reward.json plus reward.txt with non-reserved custom_metrics. It preserves the existing evals.json tasks and validates the Harbor contract.",
+ "assertions": [
+ "The agent reads create-custom-grader/SKILL.md before acting.",
+ "The agent keeps the existing evals/evals.json as the task source rather than converting to native Harbor unnecessarily.",
+ "The agent uses or recommends skillevaluator init-custom-grader with the appropriate language.",
+ "The agent avoids reserved metric-name collisions and keeps default evaluator metrics separate from custom metrics.",
+ "The agent includes validation and E2E verification steps rather than stopping at placeholder files."
+ ]
+ },
+ {
+ "id": "create-custom-grader-003-negative",
+ "expected_skill": null,
+ "expected_script": null,
+ "should_trigger": false,
+ "prompt": "Create a normal three-case evals/evals.json dataset for a calculator skill. I do not need a custom grader or any Harbor task files.",
+ "expected_output": "The agent recognizes this as ordinary eval dataset authoring and does not activate create-custom-grader. It uses the normal dataset workflow instead and does not create grader.py, grader.sh, evals/harbor, or custom grading config.",
+ "assertions": [
+ "The agent does not read or activate create-custom-grader/SKILL.md.",
+ "The agent does not create custom grader files when the user explicitly says they are not needed.",
+ "The agent routes to normal evals/evals.json dataset authoring."
+ ]
+ }
+ ]
+}
diff --git a/src/skillevaluator/tier3/reference_skills/task-list/evals/evals.json b/src/skillevaluator/tier3/reference_skills/task-list/evals/evals.json
index 57fcfaa4..2cda4899 100644
--- a/src/skillevaluator/tier3/reference_skills/task-list/evals/evals.json
+++ b/src/skillevaluator/tier3/reference_skills/task-list/evals/evals.json
@@ -1,57 +1,60 @@
-[
- {
- "id": "task-list-001",
- "question": "I need you to use the task-list skill to track the following steps: 1) Pull the latest code from main, 2) Run the test suite, 3) Build the Docker image, 4) Deploy to staging, 5) Run smoke tests. Create the task list and start working through them.",
- "expected_skill": "task-list",
- "expected_script": "tasks.py",
- "ground_truth": "The agent created a task list with all 5 steps and began executing them, updating each task's status from pending to in-progress to completed as it worked through the steps.",
- "expected_behavior": [
- "The agent read the task-list SKILL.md before beginning execution",
- "The agent executed tasks.py to create a task list containing all 5 requested steps",
- "The agent updated the status of each task after completing it, showing progression through the list",
- "The agent displayed the current state of the task list with statuses visible after each step",
- "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
- ]
- },
- {
- "id": "task-list-002",
- "question": "I need to migrate our database to a new schema. The steps are: back up the current database, apply the migration scripts, validate data integrity, update the application config, restart the services, and run integration tests. Please track each step as you go and show me progress.",
- "expected_skill": "task-list",
- "expected_script": "tasks.py",
- "ground_truth": "The agent recognized this as a 6-step request requiring task tracking, created a task list with all migration steps, and updated the status of each task as it progressed through the workflow.",
- "expected_behavior": [
- "The agent created a task list with all 6 database migration steps before starting execution",
- "The agent executed tasks.py to initialize and manage the task tracking",
- "The agent updated each task's status to reflect completion or in-progress state after working on it",
- "The agent provided a summary or final view of the task list showing all steps completed",
- "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
- ]
- },
- {
- "id": "task-list-003",
- "question": "We're onboarding a new microservice into our platform. I need you to: set up the GitHub repo with our template, configure the CI/CD pipeline, register the service in our service mesh, add monitoring dashboards, and update the API gateway routes. Walk me through each step and keep track of where we are.",
- "expected_skill": "task-list",
- "expected_script": "tasks.py",
- "ground_truth": "The agent used the task-list skill to manage the 5-step microservice onboarding process, creating a structured task list at the start and systematically updating each task's status as it addressed each onboarding step.",
- "expected_behavior": [
- "The agent initialized a task list with all 5 microservice onboarding steps at the beginning",
- "The agent executed tasks.py to track progress across the multi-step workflow",
- "The agent marked each task as in-progress when starting it and completed when finishing it",
- "The agent referenced the task list to communicate overall progress to the user",
- "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
- ]
- },
- {
- "id": "task-list-004",
- "question": "What's the difference between a Python list and a tuple?",
- "expected_skill": null,
- "expected_script": null,
- "ground_truth": "The agent provided a clear explanation of the differences between Python lists and tuples, covering mutability, syntax, and use cases, without invoking any task-list functionality.",
- "expected_behavior": [
- "The agent answered the Python question directly without creating or referencing a task list",
- "The agent explained key differences such as mutability, syntax brackets vs parentheses, and typical use cases",
- "The agent did not execute tasks.py or any task-tracking scripts",
- "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
- ]
- }
-]
\ No newline at end of file
+{
+ "skill_name": "task-list",
+ "evals": [
+ {
+ "id": "task-list-001",
+ "expected_skill": "task-list",
+ "expected_script": "tasks.py",
+ "prompt": "I need you to use the task-list skill to track the following steps: 1) Pull the latest code from main, 2) Run the test suite, 3) Build the Docker image, 4) Deploy to staging, 5) Run smoke tests. Create the task list and start working through them.",
+ "expected_output": "The agent created a task list with all 5 steps and began executing them, updating each task's status from pending to in-progress to completed as it worked through the steps.",
+ "assertions": [
+ "The agent read the task-list SKILL.md before beginning execution",
+ "The agent executed tasks.py to create a task list containing all 5 requested steps",
+ "The agent updated the status of each task after completing it, showing progression through the list",
+ "The agent displayed the current state of the task list with statuses visible after each step",
+ "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+ ]
+ },
+ {
+ "id": "task-list-002",
+ "expected_skill": "task-list",
+ "expected_script": "tasks.py",
+ "prompt": "I need to migrate our database to a new schema. The steps are: back up the current database, apply the migration scripts, validate data integrity, update the application config, restart the services, and run integration tests. Please track each step as you go and show me progress.",
+ "expected_output": "The agent recognized this as a 6-step request requiring task tracking, created a task list with all migration steps, and updated the status of each task as it progressed through the workflow.",
+ "assertions": [
+ "The agent created a task list with all 6 database migration steps before starting execution",
+ "The agent executed tasks.py to initialize and manage the task tracking",
+ "The agent updated each task's status to reflect completion or in-progress state after working on it",
+ "The agent provided a summary or final view of the task list showing all steps completed",
+ "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+ ]
+ },
+ {
+ "id": "task-list-003",
+ "expected_skill": "task-list",
+ "expected_script": "tasks.py",
+ "prompt": "We're onboarding a new microservice into our platform. I need you to: set up the GitHub repo with our template, configure the CI/CD pipeline, register the service in our service mesh, add monitoring dashboards, and update the API gateway routes. Walk me through each step and keep track of where we are.",
+ "expected_output": "The agent used the task-list skill to manage the 5-step microservice onboarding process, creating a structured task list at the start and systematically updating each task's status as it addressed each onboarding step.",
+ "assertions": [
+ "The agent initialized a task list with all 5 microservice onboarding steps at the beginning",
+ "The agent executed tasks.py to track progress across the multi-step workflow",
+ "The agent marked each task as in-progress when starting it and completed when finishing it",
+ "The agent referenced the task list to communicate overall progress to the user",
+ "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+ ]
+ },
+ {
+ "id": "task-list-004",
+ "expected_skill": null,
+ "expected_script": null,
+ "prompt": "What's the difference between a Python list and a tuple?",
+ "expected_output": "The agent provided a clear explanation of the differences between Python lists and tuples, covering mutability, syntax, and use cases, without invoking any task-list functionality.",
+ "assertions": [
+ "The agent answered the Python question directly without creating or referencing a task list",
+ "The agent explained key differences such as mutability, syntax brackets vs parentheses, and typical use cases",
+ "The agent did not execute tasks.py or any task-tracking scripts",
+ "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace"
+ ]
+ }
+ ]
+}
diff --git a/src/skillevaluator/tier3/reference_skills/text-analyzer/evals/evals.json b/src/skillevaluator/tier3/reference_skills/text-analyzer/evals/evals.json
index bde5bef3..0ed910d2 100644
--- a/src/skillevaluator/tier3/reference_skills/text-analyzer/evals/evals.json
+++ b/src/skillevaluator/tier3/reference_skills/text-analyzer/evals/evals.json
@@ -1,16 +1,19 @@
-[
- {
- "id": "text-analyzer-001",
- "question": "Can you analyze this text and tell me the word count, line count, and most frequent words? Here is the text: 'The quick brown fox jumps over the lazy dog. The dog barked at the fox. The fox ran away quickly.'",
- "expected_skill": "text-analyzer",
- "expected_script": "analyze_text.py",
- "ground_truth": "The agent used the text-analyzer skill to analyze the provided text. It executed analyze_text.py with the inline text and returned the analysis showing 20 words, 1 line, 97 characters, 3 sentences, and the most frequent word 'the' appearing 5 times, followed by 'fox' (3 times) and 'dog' (2 times).",
- "expected_behavior": [
- "The agent read the text-analyzer SKILL.md to understand how to use the skill",
- "The agent executed analyze_text.py with the --text flag passing the provided text",
- "The agent reported the word count (20 words) from the analysis results",
- "The agent reported the most frequent words, noting 'the' appears 5 times",
- "The agent included additional statistics like character count (97) or sentence count (3) in the response"
- ]
- }
-]
+{
+ "skill_name": "text-analyzer",
+ "evals": [
+ {
+ "id": "text-analyzer-001",
+ "expected_skill": "text-analyzer",
+ "expected_script": "analyze_text.py",
+ "prompt": "Can you analyze this text and tell me the word count, line count, and most frequent words? Here is the text: 'The quick brown fox jumps over the lazy dog. The dog barked at the fox. The fox ran away quickly.'",
+ "expected_output": "The agent used the text-analyzer skill to analyze the provided text. It executed analyze_text.py with the inline text and returned the analysis showing 20 words, 1 line, 97 characters, 3 sentences, and the most frequent word 'the' appearing 5 times, followed by 'fox' (3 times) and 'dog' (2 times).",
+ "assertions": [
+ "The agent read the text-analyzer SKILL.md to understand how to use the skill",
+ "The agent executed analyze_text.py with the --text flag passing the provided text",
+ "The agent reported the word count (20 words) from the analysis results",
+ "The agent reported the most frequent words, noting 'the' appears 5 times",
+ "The agent included additional statistics like character count (97) or sentence count (3) in the response"
+ ]
+ }
+ ]
+}
diff --git a/tests/golden/cli_surface.json b/tests/golden/cli_surface.json
index 50e338fe..8363ad36 100644
--- a/tests/golden/cli_surface.json
+++ b/tests/golden/cli_surface.json
@@ -3251,6 +3251,35 @@
}
]
},
+ "migrate-dataset": {
+ "params": [
+ {
+ "name": "source",
+ "opts": [
+ "source"
+ ],
+ "param_type": "argument",
+ "required": true,
+ "type": "file"
+ },
+ {
+ "name": "skill_name",
+ "opts": [
+ "--skill-name"
+ ],
+ "param_type": "option",
+ "type": "text"
+ },
+ {
+ "name": "destination",
+ "opts": [
+ "--output"
+ ],
+ "param_type": "option",
+ "type": "file"
+ }
+ ]
+ },
"select-decoys": {
"params": [
{
diff --git a/tests/tier3/test_dataset_migration.py b/tests/tier3/test_dataset_migration.py
new file mode 100644
index 00000000..2b098355
--- /dev/null
+++ b/tests/tier3/test_dataset_migration.py
@@ -0,0 +1,354 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Lossless migration must preserve runtime grading and authored metadata."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pytest
+import yaml
+from click.testing import CliRunner
+
+from skillevaluator.cli import cli
+from skillevaluator.tier3.dataset_utils import load_dataset_entries
+
+
+def _write(path: Path, data: object) -> Path:
+ if path.suffix in {".yaml", ".yml"}:
+ content = yaml.safe_dump(data, allow_unicode=True)
+ elif path.suffix == ".jsonl":
+ content = "\n".join(json.dumps(entry) for entry in data)
+ else:
+ content = json.dumps(data, ensure_ascii=False)
+ path.write_text(content + "\n", encoding="utf-8")
+ return path
+
+
+def _case(**fields: object) -> dict:
+ return {"id": 7, "question": "Résumé", "ground_truth": "done", "expected_skill": "demo", **fields}
+
+
+def test_cli_previews_without_writing_and_writes_only_explicit_new_output(tmp_path: Path) -> None:
+ source = _write(tmp_path / "legacy.json", [_case(expected_skill=None, should_trigger=False)])
+ original = source.read_bytes()
+ runner = CliRunner()
+ result = runner.invoke(cli, ["tier3", "migrate-dataset", str(source), "--skill-name", "demo"])
+ assert result.exit_code == 0, result.output
+ assert json.loads(result.output)["evals"][0]["expected_skill"] is None
+ assert list(tmp_path.iterdir()) == [source]
+ destination = tmp_path / "converted.yaml"
+ result = runner.invoke(
+ cli, ["tier3", "migrate-dataset", str(source), "--skill-name", "demo", "--output", str(destination)]
+ )
+ assert result.exit_code == 0, result.output
+ assert yaml.safe_load(destination.read_text())["evals"][0]["should_trigger"] is False
+ assert source.read_bytes() == original
+ result = runner.invoke(
+ cli, ["tier3", "migrate-dataset", str(source), "--output", str(destination), "--skill-name", "demo"]
+ )
+ assert result.exit_code != 0
+ assert "exist" in result.output.lower()
+ assert source.read_bytes() == original
+
+
[email protected]("suffix", [".json", ".jsonl", ".yaml", ".yml"])
[email protected]("output_suffix", [".json", ".yaml", ".yml"])
+def test_migration_preserves_every_authored_field_and_ids(tmp_path: Path, suffix: str, output_suffix: str) -> None:
+ from skillevaluator.tier3.dataset_migration import migrate_dataset
+
+ entries = [
+ _case(
+ expected_script="run.py",
+ expected_behavior=["reads skill"],
+ acceptable_alternates=["other"],
+ files=["data/a.txt"],
+ extension={"nested": [None, False, 3]},
+ generation={"model": "original"},
+ checks={"rubric": ["correct"]},
+ ),
+ _case(id="negative", expected_skill=None, should_trigger=False, files=[], expected_script=None),
+ ]
+ source = _write(tmp_path / f"source{suffix}", entries)
+ original = source.read_bytes()
+ destination = tmp_path / f"converted{output_suffix}"
+ result = migrate_dataset(source, skill_name="demo", destination=destination)
+ assert result.dataset == {
+ "skill_name": "demo",
+ "evals": [
+ {
+ "id": 7,
+ "prompt": "Résumé",
+ "expected_output": "done",
+ "expected_skill": "demo",
+ "expected_script": "run.py",
+ "assertions": ["reads skill"],
+ "acceptable_skills": ["other"],
+ "files": ["data/a.txt"],
+ "extension": {"nested": [None, False, 3]},
+ "generation": {"model": "original"},
+ "checks": {"rubric": ["correct"]},
+ },
+ {
+ "id": "negative",
+ "prompt": "Résumé",
+ "expected_output": "done",
+ "expected_skill": None,
+ "should_trigger": False,
+ "files": [],
+ "expected_script": None,
+ },
+ ],
+ }
+ assert source.read_bytes() == original
+ assert migrate_dataset(destination, skill_name=None).dataset == result.dataset
+
+
+def test_preferred_wrapper_metadata_survives_idempotent_conversion(tmp_path: Path) -> None:
+ from skillevaluator.tier3.dataset_migration import migrate_dataset
+
+ data = {
+ "skill_name": "demo",
+ "evals": [{"id": "one", "prompt": "work", "expected_output": "done"}],
+ "generation": {"provider": "original"},
+ "checks": ["exact"],
+ "extension": False,
+ }
+ source = _write(tmp_path / "preferred.json", data)
+ assert migrate_dataset(source, skill_name=None).dataset == data
+
+
+ "data,skill_name,error",
+ [
+ ([_case(prompt="different")], "demo", "conflicting.*question.*prompt"),
+ ([_case(acceptable_alternates=["one"], acceptable_skills=["two"])], "demo", "conflicting.*acceptable"),
+ ([_case(), 42], "demo", "entry 1.*object"),
+ ({"cases": "invalid"}, "demo", "cases.*array"),
+ ([], "demo", "empty"),
+ ([_case(id="../escape")], "demo", "invalid case id"),
+ ([_case(id=7), _case(id="7")], "demo", "colliding case id"),
+ ([_case(expected_skill="one"), _case(id="two", expected_skill="two")], None, "ambiguous.*skill"),
+ ([_case(expected_skill=None)], None, "skill-name"),
+ ([_case()], " ", "skill-name"),
+ ({"skill_name": "one", "evals": [_case()]}, "two", "conflicting.*skill"),
+ ([_case(expected_skill=42)], "demo", "expected_skill"),
+ ([_case(question=None)], "demo", "prompt"),
+ ],
+)
+def test_invalid_or_ambiguous_input_is_rejected_before_writing(
+ tmp_path: Path, data: object, skill_name: str | None, error: str
+) -> None:
+ from skillevaluator.tier3.dataset_migration import migrate_dataset
+
+ source = _write(tmp_path / "source.json", data)
+ destination = tmp_path / "new.json"
+ original = source.read_bytes()
+ with pytest.raises(ValueError, match=error):
+ migrate_dataset(source, skill_name=skill_name, destination=destination)
+ assert not destination.exists()
+ assert source.read_bytes() == original
+
+
+def test_equal_aliases_are_collapsed_and_identity_can_be_inferred(tmp_path: Path) -> None:
+ from skillevaluator.tier3.dataset_migration import migrate_dataset
+
+ source = _write(tmp_path / "source.json", [_case(prompt="Résumé", acceptable_alternates=[], acceptable_skills=[])])
+ assert migrate_dataset(source, skill_name=None).dataset == {
+ "skill_name": "demo",
+ "evals": [
+ {"id": 7, "prompt": "Résumé", "expected_output": "done", "expected_skill": "demo", "acceptable_skills": []}
+ ],
+ }
+
+
[email protected]("expected_skill", [" ", "\t", "\n", "\u2003"])
[email protected]("write_output", [False, True], ids=["preview", "output"])
+def test_blank_inferred_identity_is_rejected_without_writing(
+ tmp_path: Path, expected_skill: str, write_output: bool
+) -> None:
+ from skillevaluator.tier3.dataset_migration import migrate_dataset
+
+ source = _write(tmp_path / "source.json", [_case(expected_skill=expected_skill)])
+ original = source.read_bytes()
+ destination = tmp_path / "new.json" if write_output else None
+ with pytest.raises(ValueError, match=r"non-empty|infer"):
+ migrate_dataset(source, destination=destination)
+ assert list(tmp_path.iterdir()) == [source]
+ assert source.read_bytes() == original
+
+
+def test_padded_inferred_identity_is_preserved_and_loadable(tmp_path: Path) -> None:
+ from skillevaluator.tier3.dataset_migration import migrate_dataset
+
+ source = _write(tmp_path / "source.json", [_case(expected_skill=" demo\t")])
+ original = source.read_bytes()
+ destination = tmp_path / "new.json"
+ result = migrate_dataset(source, destination=destination)
+ assert result.dataset["skill_name"] == " demo\t"
+ assert result.dataset["evals"][0]["expected_skill"] == " demo\t"
+ assert load_dataset_entries(destination)[0]["expected_skill"] == " demo\t"
+ assert source.read_bytes() == original
+
+
[email protected]("destination_kind", ["source", "existing", "symlink", "dangling", "jsonl"])
+def test_destinations_never_overwrite_and_jsonl_output_is_not_misrepresented(
+ tmp_path: Path, destination_kind: str
+) -> None:
+ from skillevaluator.tier3.dataset_migration import migrate_dataset
+
+ source = _write(tmp_path / "source.json", [_case()])
+ destination = tmp_path / "new.json"
+ if destination_kind == "source":
+ destination = source
+ elif destination_kind == "existing":
+ destination.write_text("keep")
+ elif destination_kind in {"symlink", "dangling"}:
+ destination.symlink_to(source if destination_kind == "symlink" else tmp_path / "missing")
+ else:
+ destination = tmp_path / "new.jsonl"
+ original = source.read_bytes()
+ with pytest.raises(ValueError, match=r"source|exist|JSONL"):
+ migrate_dataset(source, skill_name=None, destination=destination)
+ assert source.read_bytes() == original
+ if destination_kind == "existing":
+ assert destination.read_text() == "keep"
+
+
+def test_converted_routing_has_identical_real_grading_outcomes(tmp_path: Path) -> None:
+ from skillevaluator.tier3.dataset_migration import migrate_dataset
+ from skillevaluator.tier3.eval_core.checks import (
+ resolve_acceptable_skills,
+ resolve_should_trigger,
+ score_skill_execution,
+ )
+
+ source = _write(
+ tmp_path / "source.json",
+ [
+ _case(),
+ _case(id="negative", expected_skill=None, should_trigger=False),
+ _case(id="false", should_trigger=False, acceptable_alternates=["other"]),
+ ],
+ )
+ destination = tmp_path / "new.json"
+ migrate_dataset(source, skill_name="demo", destination=destination)
+ tool_calls = [
+ {"action": "read_file", "action_input": {"path": "/skills/demo/SKILL.md"}, "observation": "skill instructions"}
+ ]
+ before, after = load_dataset_entries(source), load_dataset_entries(destination)
+ for old, new in zip(before, after, strict=True):
+ for calls in ([], tool_calls):
+ scores = [
+ score_skill_execution(
+ calls,
+ entry.get("expected_skill") or "",
+ entry.get("expected_script"),
+ resolve_should_trigger(entry),
+ evaluated_skill="demo",
+ acceptable_skills=resolve_acceptable_skills(entry),
+ )
+ for entry in (old, new)
+ ]
+ assert scores[0] == scores[1]
+ assert score_skill_execution(tool_calls, "demo", evaluated_skill="demo")["score"] == 1.0
+ assert score_skill_execution([], "demo", evaluated_skill="demo")["score"] == 0.0
+ assert score_skill_execution(tool_calls, "", should_trigger=False, evaluated_skill="demo")["score"] == 0.0
+ assert resolve_should_trigger(after[0]) is True
+ assert resolve_should_trigger(after[1]) is False
+ assert score_skill_execution([], "", should_trigger=False, evaluated_skill="demo")["score"] == 1.0
+
+
+def test_unlabeled_legacy_routing_is_rejected_without_guessing(tmp_path: Path) -> None:
+ from skillevaluator.tier3.dataset_migration import migrate_dataset
+ from skillevaluator.tier3.eval_core.checks import resolve_should_trigger
+
+ entry = _case()
+ del entry["expected_skill"]
+ source = _write(tmp_path / "source.json", [entry])
+ assert resolve_should_trigger(load_dataset_entries(source)[0]) is None
+ with pytest.raises(ValueError, match=r"unlabeled routing.*expected_skill.*should_trigger"):
+ migrate_dataset(source, skill_name="demo", destination=tmp_path / "new.json")
+ assert not (tmp_path / "new.json").exists()
+
+
+def test_explicit_false_routing_without_expected_skill_is_preserved(tmp_path: Path) -> None:
+ from skillevaluator.tier3.dataset_migration import migrate_dataset
+ from skillevaluator.tier3.eval_core.checks import resolve_should_trigger
+
+ entry = _case(should_trigger=False)
+ del entry["expected_skill"]
+ source = _write(tmp_path / "source.json", [entry])
+ destination = tmp_path / "new.json"
+ migrate_dataset(source, skill_name="demo", destination=destination)
+ new = load_dataset_entries(destination)[0]
+ assert new["expected_skill"] is None
+ assert resolve_should_trigger(new) is False
+
+
+ "suffix,content",
+ [
+ (".json", '[{"id":"one","question":"first","question":"lost","ground_truth":"done","expected_skill":"demo"}]'),
+ (".yaml", "- id: one\n question: first\n question: lost\n ground_truth: done\n expected_skill: demo\n"),
+ ],
+)
+def test_duplicate_authored_keys_are_rejected_instead_of_lost(tmp_path: Path, suffix: str, content: str) -> None:
+ from skillevaluator.tier3.dataset_migration import migrate_dataset
+
+ source = tmp_path / f"source{suffix}"
+ source.write_text(content)
+ with pytest.raises(ValueError, match=r"duplicate.*question"):
+ migrate_dataset(source, skill_name=None)
+
+
+def test_bundled_examples_use_preferred_objects_with_fifteen_cases() -> None:
+ root = Path(__file__).resolve().parents[2] / "src/skillevaluator/tier3/reference_skills"
+ datasets = sorted(root.glob("*/evals/evals.json"))
+ assert len(datasets) == 5
+ count = 0
+ for source in datasets:
+ data = json.loads(source.read_text())
+ assert isinstance(data, dict), source
+ assert data["skill_name"] == source.parents[1].name
+ count += len(data["evals"])
+ assert all("prompt" in entry and "expected_output" in entry for entry in data["evals"])
+ assert count == 15
+
+
+def test_converted_files_selectors_stage_identical_harbor_fixtures(tmp_path: Path) -> None:
+ from skillevaluator.tier3.dataset_migration import migrate_dataset
+ from skillevaluator.tier3.harbor.adapter import _stage_task_inputs
+
+ skill = tmp_path / "demo"
+ evals = skill / "evals"
+ files = evals / "files"
+ files.mkdir(parents=True)
+ (files / "wanted.txt").write_text("wanted")
+ (files / "unrelated.txt").write_text("unrelated")
+ source = _write(evals / "legacy.json", [_case(files=["files/wanted.txt"]), _case(id="empty", files=[])])
+ destination = evals / "converted.json"
+ migrate_dataset(source, skill_name=None, destination=destination)
+ for index, (old, new) in enumerate(
+ zip(load_dataset_entries(source), load_dataset_entries(destination), strict=True)
+ ):
+ for label, entry in (("old", old), ("new", new)):
+ environment = tmp_path / f"{label}-{index}"
+ environment.mkdir()
+ _stage_task_inputs(
+ environment, input_files_dir=files, entry=entry, source_skill_path=skill, evals_dir=evals
+ )
+ staged = sorted(
+ path.relative_to(environment).as_posix() for path in environment.rglob("*") if path.is_file()
+ )
+ assert staged == (["input/wanted.txt"] if index == 0 else [])
+
+
+def test_nested_alias_type_conflicts_are_not_collapsed(tmp_path: Path) -> None:
+ from skillevaluator.tier3.dataset_migration import migrate_dataset
+
+ source = _write(tmp_path / "source.json", [_case(expected_behavior=[False], assertions=[0])])
+ with pytest.raises(ValueError, match=r"conflicting.*expected_behavior.*assertions"):
+ migrate_dataset(source, skill_name=None)