Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

feat(tier3): import precomputed custom metrics (#375)

Christopher Kevin2adb700d198d
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4669f695..2be969c2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,12 @@ All notable changes to SkillEvaluator are documented in this file.
### Added
+- `tier3 import-metrics` accepts versioned, normalized per-case custom scores
+ from external evaluations and writes clearly labeled JSON/Markdown reports.
+ Imports validate identities, metric metadata and values, retain declared
+ provenance and source hashes, and compute custom lift only on matched
+ case/attempt pairs, without assigning standard scores or claiming execution.
+
- Show per-agent evidence volume next to Tier 3 scores: distinct cases, planned
and scored attempts, execution and judge failures, and per-arm counts when a
baseline ran. Flag limited case coverage or incomplete attempt coverage;
@@ -71,7 +77,7 @@ All notable changes to SkillEvaluator are documented in this file.
onboarding and status recognize this filename collision without replacing
reusable workflow definitions. Automatic comments activate after the
listener reaches the default branch.
- Document the shared chat/judge/agent model input and model-availability
+ Document the shared chat/judge/agent model input and model-availability
troubleshooting for provider-backed CI.
- `skills/skillevaluator/` is an agent skill for operating SkillEvaluator: it
routes requests to the right tier, readiness check, dataset, report, or CI
diff --git a/docs/assets/simulator-metrics.json b/docs/assets/simulator-metrics.json
new file mode 100644
index 00000000..b437c0cd
--- /dev/null
+++ b/docs/assets/simulator-metrics.json
@@ -0,0 +1,58 @@
+{
+ "schema_version": "1",
+ "evaluation": {
+ "dataset_id": "circuit-simulation",
+ "dataset_version": "1",
+ "skill_id": "timing-analysis",
+ "skill_version": "1",
+ "agent": "simulation-driver",
+ "agent_version": "1",
+ "model": "example-model-v1",
+ "environment": "example-simulator-v3"
+ },
+ "provenance": {
+ "producer": "example-simulator-exporter-v1",
+ "run_id": "synthetic-example-1",
+ "created_at": "2026-10-01T12:00:00Z"
+ },
+ "metric_definitions": {
+ "timing_error_fraction": {
+ "version": "1",
+ "unit": "fraction_of_budget",
+ "direction": "lower_is_better",
+ "aggregation": "mean"
+ },
+ "convergence_rate": {
+ "version": "1",
+ "unit": "fraction",
+ "direction": "higher_is_better",
+ "aggregation": "mean"
+ }
+ },
+ "observations": [
+ {
+ "case_id": "adder",
+ "attempt": 1,
+ "arm": "with_skill",
+ "custom_metrics": {"timing_error_fraction": 0.1, "convergence_rate": 0.9}
+ },
+ {
+ "case_id": "adder",
+ "attempt": 1,
+ "arm": "without_skill",
+ "custom_metrics": {"timing_error_fraction": 0.3, "convergence_rate": 0.7}
+ },
+ {
+ "case_id": "multiplier",
+ "attempt": 1,
+ "arm": "with_skill",
+ "custom_metrics": {"timing_error_fraction": 0.2, "convergence_rate": 0.8}
+ },
+ {
+ "case_id": "multiplier",
+ "attempt": 1,
+ "arm": "without_skill",
+ "custom_metrics": {"timing_error_fraction": 0.3, "convergence_rate": 0.6}
+ }
+ ]
+}
diff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdx
index 0e2b261f..4de3dc48 100644
--- a/docs/cli-reference.mdx
+++ b/docs/cli-reference.mdx
@@ -44,6 +44,19 @@ commands referenced below.
| [health-check](#health-check) | Quick CLI + backend readiness check, or another tier with `--tier` | Nothing |
| [models](#models) | List models visible through the selected provider's catalog | Provider configuration/key |
| [tier3 validate](#tier3-validate) | Validate `evals/` and the optional Harbor contract | Nothing |
+| [tier3 import-metrics](#tier3-import-metrics) | Report precomputed custom scores as external observations | A versioned metrics JSON file; no runtime or provider key |
+
+## tier3 import-metrics
+
+```bash
+skillevaluator tier3 import-metrics INPUT.json --output-dir NEW_DIRECTORY
+```
+
+`INPUT.json` must exist. Required `-o, --output-dir` names a directory that must
+not exist. The command validates the complete input before writing
+`external_metrics.json` and `external_metrics.md`. It never executes tests or
+assigns canonical scores; exit 0 means a valid import, not passing tests.
+See the [input contract and simulator example](custom-graders.mdx#import-precomputed-custom-metrics).
## Global conventions
@@ -164,7 +177,7 @@ pipeline.
| --- | --- |
| `tier1` | `validate`, `quality-check`, `rubric-eval`, `security-scan`, `pii-scan`, `lint-scripts` |
| `tier2` | `context-optimization-check`, `dedup-scan`, `similarity-check` |
-| `tier3` | `evaluate`, `select-decoys`, `create-eval-dataset`, `migrate-dataset`, `init-custom-grader`, `init-harbor-task`, `doctor`, `view`, `compare`, `feedback`, `validate`, `harbor-view` |
+| `tier3` | `evaluate`, `select-decoys`, `create-eval-dataset`, `migrate-dataset`, `init-custom-grader`, `init-harbor-task`, `doctor`, `view`, `compare`, `feedback`, `validate`, `harbor-view`, `import-metrics` |
`health-check` and [models](#models) are top-level only. The expert commands
[tier3 evaluate](#tier3-evaluate), [tier3 select-decoys](#tier3-select-decoys),
diff --git a/docs/custom-graders.mdx b/docs/custom-graders.mdx
index 9f586f55..8ef4e83b 100644
--- a/docs/custom-graders.mdx
+++ b/docs/custom-graders.mdx
@@ -327,6 +327,114 @@ Custom lift needs both arms — a `--skip-baseline` run reports your custom
scores but no lift. [Reports & Results](reports.mdx) covers the full on-disk
layout and the machine-readable contract.
+## Import precomputed custom metrics
+
+Use `tier3 import-metrics` when evaluation already happened outside
+SkillEvaluator, such as a hardware simulator that cannot run in the standard
+sandbox. The initial import contract accepts **per-case, per-attempt precomputed
+scores**, not executable graders, predictions, raw trajectories, arbitrary
+physical quantities, or aggregate-only benchmark results. Continue using the
+grading modes above to execute custom grading logic against live trajectories.
+
+Imports reuse the existing `custom_metrics` reward envelope, numeric bounds,
+reserved-name protection and custom-score averaging. They do **not** select a
+grading mode, supply `overall`, or replace canonical dimensions. In particular,
+an import does not imply that `custom_only` grading or a standard verifier ran.
+
+### Run the simulator example
+
+From a repository checkout, import the bundled
+[synthetic simulator results](assets/simulator-metrics.json):
+
+```bash
+uv run skillevaluator tier3 import-metrics docs/assets/simulator-metrics.json \
+ --output-dir reports/imported-simulator
+```
+
+No provider key, agent, Docker or Tier 3 extra is needed. The output directory
+must not exist; imports never overwrite live reports or a previous import.
+The command creates `external_metrics.json` and `external_metrics.md`. These are
+external-only artifacts, not a live run's `result.json`; `view`, `compare`, the
+dashboard and publication gates do not consume them. Exit 0 means the import
+was valid and reports were written, **not** that the external evaluation passed.
+
+The example is fabricated data, not a simulator run. Its Markdown report includes:
+
+| Custom metric | Version | Unit | Direction | With skill (n/total) | Without skill (n/total) | Paired delta | Improvement | Paired cases/attempts |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| convergence\_rate | 1 | fraction | higher\_is\_better | 0.8500 (2/2) | 0.6500 (2/2) | 0.2000 | 0.2000 | 2/2 |
+| timing\_error\_fraction | 1 | fraction\_of\_budget | lower\_is\_better | 0.1500 (2/2) | 0.3000 (2/2) | -0.1500 | 0.1500 | 2/2 |
+
+Both reports label the observations as external. SkillEvaluator performs only
+input validation, aggregation and paired comparison, not test execution or
+independent verification of the producer's claims. JSON records
+`executed_by_skillevaluator: false`, the declared provenance, import time,
+SHA-256 of the exact input bytes, definitions, observations and coverage counts.
+It assigns no Overall, pass/fail result or publication evidence.
+
+### Input contract (version 1)
+
+The input is a UTF-8 JSON object with exactly these fields:
+
+| Field | Contract |
+| --- | --- |
+| `schema_version` | String `"1"`. Unknown versions are rejected. |
+| `evaluation` | Required nonempty strings: `dataset_id`, `dataset_version`, `skill_id`, `skill_version`, `agent`, `agent_version`, `model`, `environment`. Use stable versions or content digests for datasets, skills, models and environment configurations. A non-model producer may explicitly use `model: "not_applicable"`. |
+| `provenance` | Required nonempty strings: `producer` (exporter identity/version), `run_id` (external source run), `created_at` (ISO 8601 timestamp including timezone). |
+| `metric_definitions` | Map of custom names to `{ "version": "1", "unit": "fraction", "direction": "higher_is_better", "aggregation": "mean" }`, for example. Version and unit are required nonempty strings; direction is `higher_is_better` or `lower_is_better`; only mean aggregation is supported. |
+| `observations` | Nonempty list of `{ "case_id": "adder", "attempt": 1, "arm": "with_skill", "custom_metrics": { "convergence_rate": 0.9 } }`, for example. Each score references a declared definition. |
+
+Scores must be finite JSON numbers in **[0, 1]**, the same scale as executable
+custom graders. Booleans, numeric strings and `{ "score": ... }` wrappers are
+not accepted by this initial import format. `unit` describes the normalized
+quantity; the importer does not convert units or normalize raw measurements.
+For example, export timing error divided by a fixed budget, and version that
+normalization formula. Values above 1 must be transformed by an explicitly
+documented, versioned scoring formula before export; never silently clip them.
+Raw milliseconds, unrestricted counts and arbitrary ranges are not supported.
+Change the metric version whenever its meaning, normalization or unit changes.
+
+An omitted metric or `null` means **unscored**, never zero. Every observation
+must have a nonempty `custom_metrics` object, but its declared metrics may all
+be null. Arm means exclude missing values and report a scored count per metric
+against total observations. All-missing metrics render `NO SCORE`.
+
+Case IDs are strings following the [dataset identity rules](eval-datasets.mdx). Attempts are
+integers from 1 through 1000, not booleans. Only `with_skill` and `without_skill`
+arms are accepted; duplicate case/attempt/arm identities and case IDs differing
+only in capitalization are rejected. Standard metric names, metadata names
+reserved by custom graders, and canonical dimension names cannot be custom
+names. Names retain the existing bounded, publishable-name protections.
+
+Unknown fields (including per-row context/definition overrides), duplicate JSON
+keys and invalid values fail the whole import before creating output. Text
+identities are printable, trimmed strings of at most 256 characters. Inputs are
+bounded to 4 MiB, 1000 observations, 128 metric definitions, 128 metrics per
+observation, 12 nesting levels and 500,000 parser tokens. Do not put credentials,
+private data or sensitive infrastructure identifiers in exported metadata.
+
+### Compatible paired custom lift
+
+One document declares **one** dataset/version, skill/version, agent/version,
+model, environment and metric-definition set, shared by both arms. Export only
+comparable observations into that document: in particular, a case/attempt pair
+must use the same workload and seed. These are producer assertions, not facts
+SkillEvaluator can independently verify. Export different contexts or metric
+versions separately; this initial path does not merge files or compare imports
+with retained live runs.
+
+For each metric, pair only matching case IDs **and attempt numbers** with finite
+scores in both arms. Compute with-skill minus without-skill for each pair, mean
+the matched attempts within each case, then equally weight cases. Unmatched
+observations still contribute to arm means, not paired lift, so paired lift can
+differ from subtraction of arm means. Reports show matched case and attempt
+counts and make no statistical significance claim.
+
+`custom_lift[NAME].delta` preserves the existing custom lift sign convention
+(with minus without). `improvement` reverses it for lower-is-better metrics, so
+positive always means better. A missing baseline, disjoint attempts or missing
+scores produce null delta/improvement and `NO SCORE`, not a fabricated zero.
+
## Troubleshooting
<AccordionGroup>
diff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.py
index c79b00ed..768bf22b 100644
--- a/src/skillevaluator/cli.py
+++ b/src/skillevaluator/cli.py
@@ -3761,9 +3761,32 @@ def health_check(ctx: click.Context, **options: Any) -> None:
raise SystemExit(_run_doctor(ctx, **options))
[email protected]("import-metrics")
[email protected]("source", type=click.Path(exists=True, dir_okay=False, path_type=Path))
+ "-o",
+ "--output-dir",
+ type=click.Path(file_okay=False, path_type=Path),
+ required=True,
+ help="New directory for external-only JSON and Markdown reports; must not exist.",
+)
+def tier3_import_metrics(source: Path, output_dir: Path) -> None:
+ """Report precomputed custom scores without executing tests or assigning standard scores."""
+ from skillevaluator.reporting.external_metrics import import_external_metrics
+
+ try:
+ import_external_metrics(source, output_dir)
+ except (OSError, ValueError) as exc:
+ raise click.ClickException(str(exc)) from exc
+ click.echo("Imported external observations; SkillEvaluator did not execute these tests.")
+ click.echo(f"Reports: {output_dir / 'external_metrics.json'}, {output_dir / 'external_metrics.md'}")
+
+
@tier3.command("migrate-dataset")
@click.argument("source", type=click.Path(exists=True, dir_okay=False, path_type=Path))
[email protected]("--skill-name", default=None, help="Target skill identity; required when it cannot be inferred unambiguously.")
+ "--skill-name", default=None, help="Target skill identity; required when it cannot be inferred unambiguously."
+)
@click.option(
"--output",
"destination",
diff --git a/src/skillevaluator/reporting/external_metrics.py b/src/skillevaluator/reporting/external_metrics.py
new file mode 100644
index 00000000..c7f72503
--- /dev/null
+++ b/src/skillevaluator/reporting/external_metrics.py
@@ -0,0 +1,95 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Plain reports for imported observations, deliberately separate from live runs."""
+
+from __future__ import annotations
+
+import html
+import json
+import re
+from pathlib import Path
+from typing import Any
+
+from skillevaluator.tier3.external_metrics import ARMS, load_external_metrics, summarize_external_metrics
+
+
+def _text(value: object) -> str:
+ return re.sub(r"([\\`*{}\[\]()#+.!_|>~-])", r"\\\1", html.escape(str(value), quote=False))
+
+
+def _score(value: float | None) -> str:
+ return "NO SCORE" if value is None else f"{value:.4f}"
+
+
+def render_external_metrics_markdown(report: dict[str, Any]) -> str:
+ lines = [
+ "# Imported custom metrics",
+ "",
+ "External observations — SkillEvaluator did not execute these tests or verify their source claims.",
+ "Only JSON validation, custom-metric aggregation and paired comparison ran locally.",
+ "No canonical dimensions, Overall, pass/fail or publication evidence are assigned.",
+ "",
+ f"Source SHA-256: `{report['source_sha256']}`",
+ f"Imported at: {_text(report['imported_at'])}",
+ "",
+ "## Declared provenance and evaluation context",
+ "",
+ ]
+ for key, value in {**report["provenance"], **report["evaluation"]}.items():
+ lines.append(f"- {_text(key)}: {_text(value)}")
+ lines.extend(
+ [
+ "",
+ "## Custom scores (0-1)",
+ "",
+ "Arm scores are means over available observations; n/total shows coverage, not test success.",
+ "Missing scores are excluded, never replaced with zero.",
+ "",
+ "| Custom metric | Version | Unit | Direction | With skill (n/total) | Without skill (n/total) | Paired delta | Improvement | Paired cases/attempts |",
+ "| --- | --- | --- | --- | --- | --- | --- | --- | --- |",
+ ]
+ )
+ for name, definition in sorted(report["metric_definitions"].items()):
+ values = []
+ for arm in ARMS:
+ summary = report["arms"][arm]
+ score = _score(summary["custom_metrics"].get(name))
+ values.append(f"{score} ({summary['counts'][name]}/{summary['observations']})")
+ lift = report["custom_lift"][name]
+ cells = [
+ _text(name),
+ _text(definition["version"]),
+ _text(definition["unit"]),
+ _text(definition["direction"]),
+ *values,
+ _score(lift["delta"]),
+ _score(lift["improvement"]),
+ f"{lift['paired_cases']}/{lift['paired_attempts']}",
+ ]
+ lines.append("| " + " | ".join(cells) + " |")
+ lines.extend(
+ [
+ "",
+ "Paired delta is with-skill minus without-skill for matching case/attempt IDs in this declared context.",
+ "Matched attempt deltas are averaged within each case, then equally across cases.",
+ "It may differ from subtraction of arm means when coverage differs.",
+ "Improvement reverses the delta for lower-is-better metrics; positive means better.",
+ "NO SCORE means no compatible scored pairs, not zero lift. No statistical significance is claimed.",
+ "",
+ ]
+ )
+ return "\n".join(lines)
+
+
+def import_external_metrics(source: Path, output_dir: Path) -> dict[str, Any]:
+ """Validate before writing to a new directory; never overwrite a saved live run."""
+ contract, digest = load_external_metrics(source)
+ report = summarize_external_metrics(contract, digest)
+ markdown = render_external_metrics_markdown(report)
+ output_dir.mkdir(parents=True, exist_ok=False)
+ (output_dir / "external_metrics.json").write_text(
+ json.dumps(report, indent=2, ensure_ascii=True, allow_nan=False) + "\n", encoding="utf-8"
+ )
+ (output_dir / "external_metrics.md").write_text(markdown, encoding="utf-8")
+ return report
diff --git a/src/skillevaluator/tier3/external_metrics.py b/src/skillevaluator/tier3/external_metrics.py
new file mode 100644
index 00000000..fe0e11e7
--- /dev/null
+++ b/src/skillevaluator/tier3/external_metrics.py
@@ -0,0 +1,260 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Import precomputed custom rewards without claiming a local evaluation ran."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import math
+from collections import defaultdict
+from datetime import UTC, datetime
+from pathlib import Path
+from typing import Annotated, Any, Literal
+
+from pydantic import AfterValidator, BaseModel, ConfigDict, Field, ValidationError, field_validator, model_validator
+
+from skillevaluator.tier3.case_ids import validate_case_id
+from skillevaluator.tier3.harbor.metrics import (
+ DIMENSION_DEFINITIONS,
+ MAX_CUSTOM_METRICS,
+ RESERVED_METRIC_NAMES,
+ average_custom_metrics,
+ custom_metric_name_is_publishable,
+ score_value,
+)
+from skillevaluator.utils.redaction import contains_credential_value
+from skillevaluator.utils.structured_data import preflight_json_structure
+
+MAX_INPUT_BYTES = 4 * 1024 * 1024
+MAX_OBSERVATIONS = 1000
+ARMS = ("with_skill", "without_skill")
+_NO_MAP_VALUE = object()
+
+
+def _identity(value: str) -> str:
+ if not value.strip() or value != value.strip() or not value.isprintable() or contains_credential_value(value):
+ raise ValueError("identities must be nonempty printable text without credentials or surrounding whitespace")
+ return value
+
+
+Identity = Annotated[str, Field(min_length=1, max_length=256), AfterValidator(_identity)]
+
+
+class _Contract(BaseModel):
+ model_config = ConfigDict(extra="forbid", strict=True)
+
+
+class MetricDefinition(_Contract):
+ """All imported scores use the existing custom grader's normalized 0..1 scale."""
+
+ version: Identity
+ unit: Identity
+ direction: Literal["higher_is_better", "lower_is_better"]
+ aggregation: Literal["mean"]
+
+
+class EvaluationIdentity(_Contract):
+ """One compatible dataset, skill, agent/model and environment per import."""
+
+ dataset_id: Identity
+ dataset_version: Identity
+ skill_id: Identity
+ skill_version: Identity
+ agent: Identity
+ agent_version: Identity
+ model: Identity
+ environment: Identity
+
+
+class ExternalProvenance(_Contract):
+ producer: Identity
+ run_id: Identity
+ created_at: Identity
+
+ @field_validator("created_at")
+ @classmethod
+ def _timestamp(cls, value: str) -> str:
+ try:
+ parsed = datetime.fromisoformat(value)
+ except ValueError as exc:
+ raise ValueError("created_at must be an ISO 8601 timestamp with a timezone") from exc
+ if parsed.tzinfo is None:
+ raise ValueError("created_at must include a timezone")
+ return value
+
+
+class ExternalObservation(_Contract):
+ case_id: Identity
+ attempt: Annotated[int, Field(ge=1, le=MAX_OBSERVATIONS)]
+ arm: Literal["with_skill", "without_skill"]
+ custom_metrics: dict[str, float | None] = Field(min_length=1, max_length=MAX_CUSTOM_METRICS)
+
+ @field_validator("case_id")
+ @classmethod
+ def _case_id(cls, value: str) -> str:
+ return validate_case_id(value)
+
+ @field_validator("custom_metrics", mode="before")
+ @classmethod
+ def _scores(cls, values: Any) -> Any:
+ if not isinstance(values, dict) or any(v is not None and score_value(v) is None for v in values.values()):
+ raise ValueError("custom metrics must be finite numbers in [0, 1] or null, never booleans or strings")
+ return values
+
+
+class ExternalMetrics(_Contract):
+ schema_version: Literal["1"]
+ evaluation: EvaluationIdentity
+ provenance: ExternalProvenance
+ metric_definitions: dict[str, MetricDefinition] = Field(min_length=1, max_length=MAX_CUSTOM_METRICS)
+ observations: list[ExternalObservation] = Field(min_length=1, max_length=MAX_OBSERVATIONS)
+
+ @field_validator("metric_definitions")
+ @classmethod
+ def _names(cls, definitions: dict[str, MetricDefinition]) -> dict[str, MetricDefinition]:
+ reserved = RESERVED_METRIC_NAMES | DIMENSION_DEFINITIONS.keys()
+ if any(name in reserved or not custom_metric_name_is_publishable(name) for name in definitions):
+ raise ValueError("custom metric names must be publishable and cannot use reserved metrics or dimensions")
+ return definitions
+
+ @model_validator(mode="after")
+ def _observations(self) -> ExternalMetrics:
+ seen = set()
+ case_names: dict[str, str] = {}
+ for row in self.observations:
+ identity = (row.case_id, row.attempt, row.arm)
+ if identity in seen:
+ raise ValueError(
+ f"duplicate case/attempt/arm observation for case {row.case_id!r}, attempt {row.attempt}"
+ )
+ seen.add(identity)
+ if case_names.setdefault(row.case_id.casefold(), row.case_id) != row.case_id:
+ raise ValueError("case IDs must not differ only in letter case")
+ if set(row.custom_metrics) - self.metric_definitions.keys():
+ raise ValueError("observations reference undefined custom metrics")
+ return self
+
+
+def _unique_object(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
+ result: dict[str, Any] = {}
+ for key, value in pairs:
+ if key in result:
+ raise ValueError("duplicate JSON object key")
+ result[key] = value
+ return result
+
+
+def _reject_constant(value: str) -> None:
+ raise ValueError("non-finite JSON number")
+
+
+def _safe_validation_location(location: tuple[Any, ...]) -> str:
+ current_model: type[BaseModel] | None = ExternalMetrics
+ map_value_model: type[BaseModel] | None | object = _NO_MAP_VALUE
+ path = ""
+ for segment in location:
+ if isinstance(segment, int):
+ path += f"[{segment}]"
+ continue
+ if map_value_model is not _NO_MAP_VALUE:
+ safe_segment = "<metric>"
+ current_model = map_value_model
+ map_value_model = _NO_MAP_VALUE
+ elif isinstance(segment, str) and current_model is not None and segment in current_model.model_fields:
+ safe_segment = segment
+ if current_model is ExternalMetrics:
+ if segment == "evaluation":
+ current_model = EvaluationIdentity
+ elif segment == "provenance":
+ current_model = ExternalProvenance
+ elif segment == "metric_definitions":
+ map_value_model = MetricDefinition
+ current_model = None
+ elif segment == "observations":
+ current_model = ExternalObservation
+ else:
+ current_model = None
+ elif current_model is ExternalObservation and segment == "custom_metrics":
+ map_value_model = None
+ current_model = None
+ else:
+ current_model = None
+ else:
+ safe_segment = "<field>"
+ current_model = None
+ path += ("" if not path else ".") + safe_segment
+ return path or "<input>"
+
+
+def load_external_metrics(path: Path) -> tuple[ExternalMetrics, str]:
+ """Validate a bounded, duplicate-free JSON document and hash its exact bytes."""
+ with path.open("rb") as stream:
+ raw = stream.read(MAX_INPUT_BYTES + 1)
+ if len(raw) > MAX_INPUT_BYTES:
+ raise ValueError("external metric input exceeds 4 MiB")
+ try:
+ text = raw.decode("utf-8")
+ preflight_json_structure(text, max_depth=12, max_tokens=500_000, max_collection_items=MAX_OBSERVATIONS)
+ data = json.loads(text, object_pairs_hook=_unique_object, parse_constant=_reject_constant)
+ except (ValueError, RecursionError) as exc:
+ raise ValueError("invalid external metrics JSON (check syntax, duplicate keys and size limits)") from exc
+ try:
+ contract = ExternalMetrics.model_validate(data)
+ except ValidationError as exc:
+ error = exc.errors(include_input=False, include_context=False)[0]
+ location = _safe_validation_location(error.get("loc", ()))
+ raise ValueError(f"invalid external metrics contract at {location}: {error['msg']}") from exc
+ return contract, hashlib.sha256(raw).hexdigest()
+
+
+def _paired_summary(contract: ExternalMetrics, name: str) -> dict[str, Any]:
+ scores = {
+ (row.case_id, row.attempt, row.arm): score
+ for row in contract.observations
+ if (score := row.custom_metrics.get(name)) is not None
+ }
+ by_case: dict[str, list[float]] = defaultdict(list)
+ for (case_id, attempt, arm), value in scores.items():
+ baseline = scores.get((case_id, attempt, "without_skill"))
+ if arm == "with_skill" and baseline is not None:
+ by_case[case_id].append(value - baseline)
+ delta = None
+ if by_case:
+ delta = round(math.fsum(math.fsum(values) / len(values) for values in by_case.values()) / len(by_case), 4)
+ if delta == 0:
+ delta = 0.0
+ direction = contract.metric_definitions[name].direction
+ return {
+ "delta": delta,
+ "improvement": delta if delta is None or delta == 0 or direction == "higher_is_better" else -delta,
+ "paired_cases": len(by_case),
+ "paired_attempts": sum(len(values) for values in by_case.values()),
+ }
+
+
+def summarize_external_metrics(contract: ExternalMetrics, source_sha256: str) -> dict[str, Any]:
+ """Reuse custom reward aggregation, keeping imported evidence out of canonical scores."""
+ arms: dict[str, Any] = {}
+ for arm in ARMS:
+ rows = [row for row in contract.observations if row.arm == arm]
+ arms[arm] = {
+ "observations": len(rows),
+ "custom_metrics": average_custom_metrics([{"custom_metrics": row.custom_metrics} for row in rows]),
+ "counts": {
+ name: sum(row.custom_metrics.get(name) is not None for row in rows)
+ for name in contract.metric_definitions
+ },
+ }
+ return {
+ "report_kind": "external_custom_metrics",
+ "schema_version": "1",
+ "observation_source": "external",
+ "executed_by_skillevaluator": False,
+ "imported_at": datetime.now(UTC).isoformat(),
+ "source_sha256": source_sha256,
+ **contract.model_dump(),
+ "arms": arms,
+ "custom_lift": {name: _paired_summary(contract, name) for name in sorted(contract.metric_definitions)},
+ }
diff --git a/tests/golden/cli_surface.json b/tests/golden/cli_surface.json
index 0f3b3d99..cda043c8 100644
--- a/tests/golden/cli_surface.json
+++ b/tests/golden/cli_surface.json
@@ -3153,6 +3153,29 @@
}
]
},
+ "import-metrics": {
+ "params": [
+ {
+ "name": "source",
+ "opts": [
+ "source"
+ ],
+ "param_type": "argument",
+ "required": true,
+ "type": "file"
+ },
+ {
+ "name": "output_dir",
+ "opts": [
+ "--output-dir",
+ "-o"
+ ],
+ "param_type": "option",
+ "required": true,
+ "type": "directory"
+ }
+ ]
+ },
"init-custom-grader": {
"params": [
{
diff --git a/tests/tier3/test_external_metrics.py b/tests/tier3/test_external_metrics.py
new file mode 100644
index 00000000..ed0759ea
--- /dev/null
+++ b/tests/tier3/test_external_metrics.py
@@ -0,0 +1,409 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Precomputed scores share custom grader bounds, never canonical score semantics."""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from pathlib import Path
+
+import pytest
+from click.testing import CliRunner
+from pydantic import ValidationError
+
+from skillevaluator.cli import cli
+from skillevaluator.reporting.dashboard_data import parse_dashboard_report
+from skillevaluator.reporting.external_metrics import import_external_metrics, render_external_metrics_markdown
+from skillevaluator.tier3.external_metrics import (
+ MAX_INPUT_BYTES,
+ ExternalMetrics,
+ load_external_metrics,
+ summarize_external_metrics,
+)
+from skillevaluator.tier3.harbor.metrics import RESERVED_METRIC_NAMES, average_custom_metrics
+
+
+def payload() -> dict:
+ return {
+ "schema_version": "1",
+ "evaluation": {
+ "dataset_id": "circuit-simulation",
+ "dataset_version": "sha256:example",
+ "skill_id": "timing-analysis",
+ "skill_version": "v1",
+ "agent": "sim-driver",
+ "agent_version": "v2",
+ "model": "example-model-v1",
+ "environment": "simulator-v3",
+ },
+ "provenance": {"producer": "sim-exporter-v1", "run_id": "sim-123", "created_at": "2026-10-01T12:00:00Z"},
+ "metric_definitions": {
+ "timing_error_fraction": {
+ "version": "1",
+ "unit": "fraction_of_budget",
+ "direction": "lower_is_better",
+ "aggregation": "mean",
+ }
+ },
+ "observations": [
+ {"case_id": "adder", "attempt": 1, "arm": "with_skill", "custom_metrics": {"timing_error_fraction": 0.1}},
+ {
+ "case_id": "adder",
+ "attempt": 1,
+ "arm": "without_skill",
+ "custom_metrics": {"timing_error_fraction": 0.4},
+ },
+ ],
+ }
+
+
+def summarize(payload: dict) -> dict:
+ return summarize_external_metrics(ExternalMetrics.model_validate(payload), "0" * 64)
+
+
+def test_reuses_custom_reward_averages_without_canonical_scores(payload):
+ report = summarize(payload)
+ expected = average_custom_metrics([{"custom_metrics": {"timing_error_fraction": 0.1}}])
+ assert report["arms"]["with_skill"]["custom_metrics"] == expected
+ assert report["custom_lift"]["timing_error_fraction"] == {
+ "delta": -0.3,
+ "improvement": 0.3,
+ "paired_cases": 1,
+ "paired_attempts": 1,
+ }
+ assert report["observation_source"] == "external"
+ assert report["executed_by_skillevaluator"] is False
+ assert not {"overall", "overall_score", "dimensions", "passed", "score_policy"} & report.keys()
+ assert report["provenance"] == payload["provenance"]
+ assert report["observations"] == payload["observations"]
+
+
+ "value", [True, False, "0.5", -0.01, 1.01, float("nan"), float("inf"), -float("inf"), 10**400, {}]
+)
+def test_rejects_invalid_scores(payload, value):
+ payload["observations"][0]["custom_metrics"]["timing_error_fraction"] = value
+ with pytest.raises(ValidationError):
+ ExternalMetrics.model_validate(payload)
+
+
+ "name",
+ [
+ *sorted(RESERVED_METRIC_NAMES),
+ "correctness",
+ "efficiency",
+ "discoverability",
+ "effectiveness",
+ "bad\nname",
+ " ",
+ "a" * 257,
+ ],
+)
+def test_rejects_reserved_and_unpublishable_names(payload, name):
+ definition = payload["metric_definitions"].pop("timing_error_fraction")
+ payload["metric_definitions"][name] = definition
+ with pytest.raises(ValidationError):
+ ExternalMetrics.model_validate(payload)
+
+
+ ("field", "value"),
+ [("direction", "up"), ("aggregation", "sum"), ("version", ""), ("unit", ""), ("unit", "line\nbreak")],
+)
+def test_rejects_invalid_metric_metadata(payload, field, value):
+ payload["metric_definitions"]["timing_error_fraction"][field] = value
+ with pytest.raises(ValidationError):
+ ExternalMetrics.model_validate(payload)
+
+
+ "field",
+ ["dataset_id", "dataset_version", "skill_id", "skill_version", "agent", "agent_version", "model", "environment"],
+)
+def test_requires_complete_evaluation_identity(payload, field):
+ del payload["evaluation"][field]
+ with pytest.raises(ValidationError):
+ ExternalMetrics.model_validate(payload)
+
+
[email protected]("timestamp", ["yesterday", "2026-10-01", "2026-10-01T12:00:00"])
+def test_requires_timezone_provenance(payload, timestamp):
+ payload["provenance"]["created_at"] = timestamp
+ with pytest.raises(ValidationError):
+ ExternalMetrics.model_validate(payload)
+
+
+ ("field", "value"),
+ [("case_id", "../bad"), ("case_id", ""), ("attempt", True), ("attempt", 0), ("attempt", "1"), ("arm", "baseline")],
+)
+def test_requires_valid_observation_identity(payload, field, value):
+ payload["observations"][0][field] = value
+ with pytest.raises(ValidationError):
+ ExternalMetrics.model_validate(payload)
+
+
+def test_rejects_duplicate_or_case_colliding_observations(payload):
+ payload["observations"].append(copy.deepcopy(payload["observations"][0]))
+ with pytest.raises(ValidationError, match="duplicate"):
+ ExternalMetrics.model_validate(payload)
+ payload["observations"][-1]["case_id"] = "Adder"
+ with pytest.raises(ValidationError, match="letter case"):
+ ExternalMetrics.model_validate(payload)
+
+
+def test_rejects_undefined_metrics_and_per_row_context_overrides(payload):
+ payload["observations"][0]["custom_metrics"]["undefined"] = 0.2
+ with pytest.raises(ValidationError, match="undefined"):
+ ExternalMetrics.model_validate(payload)
+ del payload["observations"][0]["custom_metrics"]["undefined"]
+ payload["observations"][0]["evaluation"] = {"dataset_version": "different"}
+ with pytest.raises(ValidationError, match="Extra inputs"):
+ ExternalMetrics.model_validate(payload)
+
+
[email protected]("missing", [None, "absent"])
+def test_missing_scores_are_counted_not_zeroed_or_paired(payload, missing):
+ if missing is None:
+ payload["observations"][1]["custom_metrics"]["timing_error_fraction"] = None
+ else:
+ payload["observations"] = payload["observations"][:1]
+ report = summarize(payload)
+ assert report["arms"]["without_skill"]["custom_metrics"] == {}
+ assert report["arms"]["without_skill"]["counts"] == {"timing_error_fraction": 0}
+ assert report["custom_lift"]["timing_error_fraction"] == {
+ "delta": None,
+ "improvement": None,
+ "paired_cases": 0,
+ "paired_attempts": 0,
+ }
+
+
+def test_absent_metric_is_allowed_and_reported_unscored(payload):
+ payload["metric_definitions"]["other_metric"] = copy.deepcopy(
+ payload["metric_definitions"]["timing_error_fraction"]
+ )
+ report = summarize(payload)
+ assert report["arms"]["with_skill"]["counts"]["other_metric"] == 0
+ assert report["custom_lift"]["other_metric"]["delta"] is None
+
+
+def test_pairing_requires_same_attempt_not_just_same_case(payload):
+ payload["observations"][1]["attempt"] = 2
+ report = summarize(payload)
+ assert report["custom_lift"]["timing_error_fraction"]["delta"] is None
+
+
+def test_lift_is_case_weighted_on_matched_attempts_only(payload):
+ rows = payload["observations"]
+ for case_id, attempt, with_score, without_score in [("adder", 2, 0.3, 0.4), ("multiplier", 1, 0.6, 0.5)]:
+ for arm, score in [("with_skill", with_score), ("without_skill", without_score)]:
+ rows.append(
+ {"case_id": case_id, "attempt": attempt, "arm": arm, "custom_metrics": {"timing_error_fraction": score}}
+ )
+ rows.append({"case_id": "adder", "attempt": 3, "arm": "with_skill", "custom_metrics": {"timing_error_fraction": 1}})
+ lift = summarize(payload)["custom_lift"]["timing_error_fraction"]
+ assert lift == {"delta": -0.05, "improvement": 0.05, "paired_cases": 2, "paired_attempts": 3}
+ payload["metric_definitions"]["timing_error_fraction"]["direction"] = "higher_is_better"
+ assert summarize(payload)["custom_lift"]["timing_error_fraction"]["improvement"] == -0.05
+
+
+def test_loader_preserves_hash_and_rejects_duplicate_keys(payload, tmp_path):
+ path = tmp_path / "metrics.json"
+ raw = json.dumps(payload).encode()
+ path.write_bytes(raw)
+ _, digest = load_external_metrics(path)
+ assert digest == hashlib.sha256(raw).hexdigest()
+ path.write_text('{"schema_version":"1","schema_version":"1"}')
+ with pytest.raises(ValueError, match="duplicate keys"):
+ load_external_metrics(path)
+
+
[email protected]("raw", ['{"x":NaN}', '{"x":Infinity}', "[" * 14 + "0" + "]" * 14, "\ud800", "not JSON"])
+def test_loader_rejects_malformed_or_unbounded_input(tmp_path, raw):
+ path = tmp_path / "metrics.json"
+ path.write_bytes(raw.encode("utf-8", errors="surrogatepass"))
+ with pytest.raises(ValueError):
+ load_external_metrics(path)
+
+
+def test_input_size_limit(tmp_path):
+ path = tmp_path / "metrics.json"
+ path.write_bytes(b" " * (MAX_INPUT_BYTES + 1))
+ with pytest.raises(ValueError, match="4 MiB"):
+ load_external_metrics(path)
+
+
[email protected]("container", ["metric_definitions", "observations"])
+def test_empty_containers_rejected(payload, container):
+ payload[container] = {} if container == "metric_definitions" else []
+ with pytest.raises(ValidationError):
+ ExternalMetrics.model_validate(payload)
+
+
+def test_documented_simulator_import_end_to_end(tmp_path):
+ example = Path(__file__).resolve().parents[2] / "docs" / "assets" / "simulator-metrics.json"
+ output = tmp_path / "external"
+ result = CliRunner().invoke(cli, ["tier3", "import-metrics", str(example), "-o", str(output)])
+ assert result.exit_code == 0, result.output
+ assert "did not execute these tests" in result.output
+ assert {path.name for path in output.iterdir()} == {"external_metrics.json", "external_metrics.md"}
+ report = json.loads((output / "external_metrics.json").read_text())
+ assert parse_dashboard_report(report, "external_metrics.json").rows == []
+ assert report["source_sha256"] == hashlib.sha256(example.read_bytes()).hexdigest()
+ assert report["provenance"]["run_id"] == "synthetic-example-1"
+ assert report["arms"]["with_skill"]["custom_metrics"] == {"convergence_rate": 0.85, "timing_error_fraction": 0.15}
+ assert report["arms"]["without_skill"]["custom_metrics"] == {"convergence_rate": 0.65, "timing_error_fraction": 0.3}
+ assert report["custom_lift"]["timing_error_fraction"] == {
+ "delta": -0.15,
+ "improvement": 0.15,
+ "paired_cases": 2,
+ "paired_attempts": 2,
+ }
+ markdown = (output / "external_metrics.md").read_text()
+ assert "did not execute these tests" in markdown
+ assert "No canonical dimensions, Overall" in markdown
+ assert (
+ "| Custom metric | Version | Unit | Direction | With skill (n/total) | Without skill (n/total) | "
+ "Paired delta | Improvement | Paired cases/attempts |"
+ ) in markdown
+ assert (
+ "| convergence\\_rate | 1 | fraction | higher\\_is\\_better | 0.8500 (2/2) | 0.6500 (2/2) | "
+ "0.2000 | 0.2000 | 2/2 |"
+ ) in markdown
+ assert (
+ "| timing\\_error\\_fraction | 1 | fraction\\_of\\_budget | lower\\_is\\_better | "
+ "0.1500 (2/2) | 0.3000 (2/2) | -0.1500 | 0.1500 | 2/2 |"
+ ) in markdown
+
+
+def test_invalid_import_writes_nothing(payload, tmp_path):
+ payload["observations"][0]["custom_metrics"]["timing_error_fraction"] = "not a score"
+ source = tmp_path / "source.json"
+ source.write_text(json.dumps(payload))
+ output = tmp_path / "not-created"
+ result = CliRunner().invoke(cli, ["tier3", "import-metrics", str(source), "-o", str(output)])
+ assert result.exit_code == 1
+ assert "invalid external metrics contract" in result.output
+ assert "observations[0].custom_metrics" in result.output
+ assert not output.exists()
+
+
+def test_existing_output_and_source_are_not_overwritten(payload, tmp_path):
+ source = tmp_path / "source.json"
+ source.write_text(json.dumps(payload))
+ output = tmp_path / "live-run"
+ output.mkdir()
+ saved = output / "result.json"
+ saved.write_text("untouched")
+ with pytest.raises(FileExistsError):
+ import_external_metrics(source, output)
+ assert saved.read_text() == "untouched"
+ assert list(output.iterdir()) == [saved]
+ assert json.loads(source.read_text()) == payload
+
+
+def test_markdown_escapes_untrusted_metadata(payload):
+ payload["provenance"]["producer"] = (
+ '<img src=x onerror="alert(1)">[link](https://example.org)|*text* Acme\'s sim exporter v1'
+ )
+ payload["metric_definitions"]["driver's_rate"] = payload["metric_definitions"].pop("timing_error_fraction")
+ for row in payload["observations"]:
+ row["custom_metrics"]["driver's_rate"] = row["custom_metrics"].pop("timing_error_fraction")
+ markdown = render_external_metrics_markdown(summarize(payload))
+ assert "<img" not in markdown
+ assert "[link](https://example.org)" not in markdown
+ assert "\\|\\*text\\*" in markdown
+ assert "Acme's sim exporter v1" in markdown
+ assert "driver's\\_rate" in markdown
+ assert "&#x27;" not in markdown
+
+
+def test_unscored_report_shows_no_score_not_zero(payload):
+ payload["observations"][1]["custom_metrics"]["timing_error_fraction"] = None
+ markdown = render_external_metrics_markdown(summarize(payload))
+ assert "NO SCORE (0/1) | NO SCORE | NO SCORE | 0/0" in markdown
+
+
+def test_unknown_schema_version_rejected(payload):
+ payload["schema_version"] = "2"
+ with pytest.raises(ValidationError):
+ ExternalMetrics.model_validate(payload)
+
+
+def test_row_and_metric_limits(payload):
+ payload["observations"] *= 501
+ with pytest.raises(ValidationError):
+ ExternalMetrics.model_validate(payload)
+ payload["observations"] = payload["observations"][:1]
+ template = payload["metric_definitions"]["timing_error_fraction"]
+ payload["metric_definitions"] = {f"metric_{index}": template for index in range(129)}
+ with pytest.raises(ValidationError):
+ ExternalMetrics.model_validate(payload)
+
+
+def test_loader_diagnostics_do_not_echo_secret_values_or_keys(payload, tmp_path):
+ secret_text = "sensitive-unexpected-field"
+ payload[secret_text] = "sensitive-value"
+ source = tmp_path / "source.json"
+ source.write_text(json.dumps(payload))
+ with pytest.raises(ValueError) as caught:
+ load_external_metrics(source)
+ assert secret_text not in str(caught.value)
+ assert "sensitive-value" not in str(caught.value)
+ assert "<field>" in str(caught.value)
+
+
+def test_loader_diagnostics_include_observation_field_path(payload, tmp_path):
+ payload["observations"][1]["attempt"] = 0
+ source = tmp_path / "source.json"
+ source.write_text(json.dumps(payload))
+ with pytest.raises(ValueError, match=r"observations\[1\]\.attempt"):
+ load_external_metrics(source)
+
+
+def test_loader_diagnostics_hide_metric_definition_keys(payload, tmp_path):
+ secret_metric = "sensitive_metric_name"
+ payload["metric_definitions"][secret_metric] = {
+ "version": 1,
+ "unit": "fraction",
+ "direction": "higher_is_better",
+ "aggregation": "mean",
+ }
+ source = tmp_path / "source.json"
+ source.write_text(json.dumps(payload))
+ with pytest.raises(ValueError) as caught:
+ load_external_metrics(source)
+ assert "metric_definitions.<metric>.version" in str(caught.value)
+ assert secret_metric not in str(caught.value)
+
+
+def test_duplicate_observation_diagnostic_names_case_and_attempt(payload, tmp_path):
+ payload["observations"].append(copy.deepcopy(payload["observations"][0]))
+ source = tmp_path / "source.json"
+ source.write_text(json.dumps(payload))
+ with pytest.raises(ValueError) as caught:
+ load_external_metrics(source)
+ assert "case 'adder', attempt 1" in str(caught.value)
+
+
[email protected]("value", [0, 1])
+def test_boundary_scores_and_zero_lift_are_not_missing(payload, value):
+ for row in payload["observations"]:
+ row["custom_metrics"]["timing_error_fraction"] = value
+ report = summarize(payload)
+ assert report["arms"]["with_skill"]["counts"] == {"timing_error_fraction": 1}
+ assert report["custom_lift"]["timing_error_fraction"] == {
+ "delta": 0,
+ "improvement": 0,
+ "paired_cases": 1,
+ "paired_attempts": 1,
+ }
+ markdown = render_external_metrics_markdown(report)
+ assert "NO SCORE (" not in markdown
+ assert "-0.0000" not in markdown