Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
feat(reporting): record the evaluated source identity on benchmark cards (#131)
Max Freedom Pollarda8d1614437fb
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex f1c585ff..c8f162ad 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -68,7 +68,7 @@ jobs:run: python scripts/check_oss_boundary.py --root . --allowlist config/oss_boundary_allowlist.json- name: Lint committed BENCHMARK.md fixtures for leak patternsif: ${{ needs.classify-changes.outputs.docs_only != 'true' }}- run: python scripts/ci/check_public_benchmarks.py --require-files tests/golden+ run: python scripts/ci/check_public_benchmarks.py --require-files tests/golden --require-source-provenance- name: Lintif: ${{ needs.classify-changes.outputs.docs_only != 'true' }}run: uv run ruff check .diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 6be4ceed..a2367a81 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -18,6 +18,41 @@ All notable changes to SkillEvaluator are documented in this file.HTTP proxy so its bearer token is not offered to an intermediary. Thetransport rechecks authorization before dispatch and rejects hosts thatare no longer allowed.+- Published benchmark cards record the evaluated source identity. `BENCHMARK.md`+ now carries `Evaluated source`, `Evaluated source revision` and+ `Evaluator container revision` as separate fields, so a reader can tell which+ source tree was evaluated apart from the evaluator build that evaluated it.+ Previously two skills evaluated from different repositories by the same+ evaluator container produced cards whose only recorded revision was the shared+ container tag. `validate` and `tier3 evaluate` take the identity as+ `--evaluated-source-repository`, `--evaluated-source-revision` and+ `--evaluator-container-revision`. `validate` records it on the card and in+ the top-level `evaluated_source` object of its JSON report and forwards it to+ every child of a parallel catalog run; both commands persist it into a Tier 3+ run's `run_config.json`. It can also arrive as the `evaluated_source`+ argument to `build_agent_eval_payload`, as an `evaluated_source` object in the+ run's `run_config.json`, or as `metadata["evaluated_source"]` on any+ validation result. It is never inferred from repository state while rendering,+ because the tree that renders a card is the evaluator checkout rather than the+ evaluated skill's source. Every populated carrier, including the payload of+ every Tier 3 result, is folded into one identity before any report is+ written, so carriers that disagree fail closed with nothing published instead+ of letting result ordering decide which source tree a card claims to describe.+ A revision is accepted only in an unambiguous shape: a full Git object id+ (40 or 64 hex characters), or a digest whose width matches the algorithm it+ names. A container revision is an image reference validated by component (a+ repository path of up to 255 characters, an optional tag of up to 128, and a+ digest at its algorithm's width), so a long repository name is no longer+ discarded. The 255 bound measures the path once the registry host is split+ off it. Path components are lower case, as the OCI grammar requires, while a+ registry host may use any case and is read as a host only when it is+ `localhost`, carries a dot, or carries a port.+ `check_public_benchmarks.py --require-source-provenance` requires the+ fields and fails any card publishing a `PASS` without them, including a+ `PASS` whose evaluator container is named by a mutable tag rather than+ pinned by digest. SkillEvaluator's own CI now runs the scan with that flag;+ it stays opt-in for trees whose cards predate the contract+ ([#72](https://github.com/NVIDIA/SkillEvaluator/issues/72)).- SARIF 2.1.0 reporter (`-r sarif`) for GitHub Code Scanning and other SARIFconsumers. Findings map to rule IDs, severity levels, and file locations fromTier 1 validation results.diff --git a/docs/benchmark-rollout.mdx b/docs/benchmark-rollout.mdxindex bba00384..193a9628 100644--- a/docs/benchmark-rollout.mdx+++ b/docs/benchmark-rollout.mdx@@ -110,6 +110,26 @@ python scripts/ci/check_public_benchmarks.py \./benchmark-backfill```+Add `--require-source-provenance` for any tree whose cards were generated with+the evaluated-source identity supplied. SkillEvaluator's own CI runs the scan+with the flag. It requires every scanned card to carry `Evaluated source`,+`Evaluated source revision` and `Evaluator container revision`, and fails any+card publishing a `PASS` that does not record the repository and revision it+evaluated, or whose evaluator container is named by a mutable tag rather than+pinned by digest. That includes a card whose Tier 3 row is skipped under an+`optional by policy` contract. The flag stays opt-in because cards generated+before the identity contract existed cannot satisfy it; without it the scan+behaves exactly as it did before.++Supply the identity when regenerating a card:++```bash+skillevaluator validate ./my-skill \+ --evaluated-source-repository NVIDIA/NVFlare \+ --evaluated-source-revision 2263a2ebdab903e87f7e7c0a001d22c3a926a9cf \+ --evaluator-container-revision ghcr.io/nvidia/skillevaluator@sha256:0117bc2e...+```+The linter rejects configured known leak patterns such as retired productidentities, any validation-profile metadata line, common absolute home paths,ambiguous legacy uplift cells, the old `Num` column, missing metadata/decisiondiff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdxindex d3cc8a46..9760cb24 100644--- a/docs/cli-reference.mdx+++ b/docs/cli-reference.mdx@@ -182,6 +182,16 @@ The following flags are forwarded to the live-eval engine **only when Tier 3 isFlags marked "unset" fall back to their matching keys in `evals/config.yml` where one exists, then to the built-in defaults ([Eval Datasets](eval-datasets.mdx) documents the config keys and precedence). The exception is `--results-dir`, which resolves independently of `evals/config.yml` (see [Reports & Results](reports.mdx)).+### Evaluated source provenance++| Flag | Default | Effect |+| --- | --- | --- |+| `--evaluated-source-repository TEXT` | unset | Repository (`owner/name`) of the source tree being evaluated. |+| `--evaluated-source-revision TEXT` | unset | Immutable revision of the evaluated source: a full Git object id, or a `sha256`/`sha384`/`sha512` digest. A short prefix is refused. |+| `--evaluator-container-revision TEXT` | unset | Digest-pinned evaluator image reference recorded beside the evaluated source. |++The identity is recorded on `BENCHMARK.md`, forwarded to every child of a parallel catalog run so each card names the same source tree, and persisted into a Tier 3 run's `run_config.json` (see [Reports & Results](reports.mdx)).+## quality-checkScore skill quality across correctness, discoverability, reliability, and efficiency. Runs entirely offline — no API key.@@ -327,6 +337,9 @@ The old top-level spelling `skillevaluator evaluate` still works for existing sc| `--override-cpus INTEGER` | unset | Override eval-environment CPU allocation (passed through to Harbor). || `--override-memory-mb INTEGER` | unset | Override eval-environment memory in MB (passed through to Harbor). || `--override-storage-mb INTEGER` | unset | Override eval-environment storage in MB (passed through to Harbor). |+| `--evaluated-source-repository TEXT` | unset | Repository (`owner/name`) of the source tree being evaluated, persisted into the run's `run_config.json`. |+| `--evaluated-source-revision TEXT` | unset | Immutable revision of the evaluated source: a full Git object id, or a `sha256`/`sha384`/`sha512` digest. A short prefix is refused. |+| `--evaluator-container-revision TEXT` | unset | Digest-pinned evaluator image reference recorded beside the evaluated source. || `--progress [auto\|rich\|plain\|off]` | `auto` | Progress display for the run: `rich` live view, `plain` line output, or `off`. |Without a `--model`/`--agent-model` override, each agent uses the selected provider's default model: `gpt-5.6-sol` on OpenAI, and Nemotron on the NVIDIA provider — `nvidia/nemotron-3-super-120b-a12b` for bridged agents, `nvidia/nemotron-3-nano-30b-a3b` for direct OpenCode.diff --git a/docs/reports.mdx b/docs/reports.mdxindex c1d3993a..a8ec8530 100644--- a/docs/reports.mdx+++ b/docs/reports.mdx@@ -88,6 +88,60 @@ by default, so a missing, skipped, or incomplete live evaluation producesexplicitly persist `benchmark_policy.tier3_required = false`; the carddiscloses that exception instead of implying that Tier 3 ran.+### Evaluated source provenance++The Evaluation Metadata section distinguishes the source tree that was evaluated+from the evaluator build that evaluated it:++```text+- Evaluator version: `0.8.2`+- Evaluated source: `NVIDIA/NVFlare`+- Evaluated source revision: `2263a2ebdab903e87f7e7c0a001d22c3a926a9cf`+- Evaluator container revision: `ghcr.io/nvidia/skillevaluator@sha256:0117bc2e...`+```++Without that separation, two skills evaluated from different repositories by the+same evaluator container publish cards whose only recorded revision is the shared+container tag, and a reader cannot tell which version of a skill was evaluated or+whether the card has gone stale.++The identity is supplied by the orchestration input and carried through the+evaluation-result contract unchanged. It is never inferred from repository state+while rendering, because the tree that renders a card is the evaluator checkout+rather than the evaluated skill's source. The supported seams are:++- `--evaluated-source-repository`, `--evaluated-source-revision` and+ `--evaluator-container-revision`, accepted by both `validate` and+ `tier3 evaluate`, which record the identity on the card and, for a Tier 3 run,+ persist it into that run's `run_config.json`;+- the `evaluated_source` argument to `build_agent_eval_payload` or+ `agent_eval_result_from_directory`;+- an `evaluated_source` object in the run's `run_config.json`. Note that+ SkillEvaluator writes that file itself, so a publication pipeline using this+ route has to add the key to the generated run artifact before the card is+ rendered;+- `metadata["evaluated_source"]` on any validation result, which is how a card+ that publishes without a completed Tier 3 run records its identity.++Where any two carriers supply a field and disagree, rendering fails rather than+silently choosing one, so the order results happen to arrive in can never decide+which source tree a card names. `Evaluated source revision` records a full Git+object id of 40 or 64 hex characters, or a canonical `sha256`/`sha384`/`sha512`+digest of the evaluated skill content at that algorithm's own width when a run+has no upstream commit. A short revision prefix is refused, because it can grow+a second match as a tree grows. `Evaluator container revision` records an image+reference validated component by component: a repository path of at most 255+characters, an optional tag of at most 128 characters, and an optional digest at+its own algorithm's width. The 255 bound measures the path once the registry+host has been split off it. Path components are lower case, as the OCI grammar+requires, while a registry host may use any case and is read as a host only when+it is `localhost`, carries a dot, or carries a port. A bare name is refused+because it identifies a repository rather than the build that ran, and a+published PASS additionally requires the digest, or a full implementation+revision, because a tag can be repointed after the card is published. Fields+the orchestration did not supply render as+`not recorded (not supplied by the orchestration input)`.+Keep the file with the skill and refresh it whenever the skill, eval dataset,agent/model, evaluator version, scoring policy, attempt policy, or executionenvironment materially changes. Follow the@@ -320,6 +374,13 @@ For CI and tooling, two JSON entry points matter:tiers. Each result carries finalized gating metadata. Tier 3 is advisory bydefault and becomes blocking with `--block-on-agent-eval`; Tier 2 is blockingby default and becomes advisory with `--no-block-on-dedup`.+- **The evaluated source identity**: every JSON report always carries a+ top-level `evaluated_source` object, resolved from every carrier the run+ recorded, so the JSON and `BENCHMARK.md` name the same source tree. It is+ `null` when the orchestration input supplied none. Carriers that name+ different sources abort the run before any report file is written, so a+ contradictory identity is never published in one format and reported in+ another.Both paths speak the same dialect: standalone `tier3 evaluate` and`validate --tier3` share one HTML renderer for `report.html`, and both embeddiff --git a/scripts/ci/check_public_benchmarks.py b/scripts/ci/check_public_benchmarks.pyindex eda94073..12e4e8ad 100644--- a/scripts/ci/check_public_benchmarks.py+++ b/scripts/ci/check_public_benchmarks.py@@ -80,53 +80,175 @@ _AGENT_MODEL_STATE = re.compile(flags=re.IGNORECASE,)_RECORDED_AGENT_MODEL_STATE = re.compile(r"^[^,]+ \(`[^`,]+`\)$")+# A verdict line states a verdict and nothing else. Every card's policy section+# restates the rule as a bullet ("- Overall verdict: PASS only when every+# configured dimension passes ..."), so a list item is never a verdict, and a+# stated verdict either ends its line or introduces the wording that explains+# it. Both detectors below are built from this one tail, so they cannot+# disagree about what a verdict line is; only the blockquote marker differs.+_VERDICT_LINE_START = r"^(?!\s*(?:[-*+]|\d+[.)])\s)\s*"+_VERDICT_PASS_TAIL = r"\s*.*Overall verdict:\s*PASS\b(?:\*\*)?\s*(?:$|[\u2014:\-(])"_OVERALL_PASS = re.compile(- r"^\s*>\s*.*Overall verdict:\s*PASS\b",+ rf"{_VERDICT_LINE_START}>{_VERDICT_PASS_TAIL}",flags=re.IGNORECASE | re.MULTILINE,)+# Each field carries the predicate that accepts its recorded value, because the+# container reference needs a length rule no single pattern can express._METADATA_FIELD_RULES = (("Evaluation date",- re.compile(r"(?:\d{4}-\d{2}-\d{2}|not recorded\b.*)", flags=re.IGNORECASE),+ re.compile(r"(?:\d{4}-\d{2}-\d{2}|not recorded\b.*)", flags=re.IGNORECASE).fullmatch,),("Evaluator version",- re.compile(r"(?:`[^`\s][^`]*`|not recorded\b.*)", flags=re.IGNORECASE),+ re.compile(r"(?:`[^`\s][^`]*`|not recorded\b.*)", flags=re.IGNORECASE).fullmatch,),("Tasks",- re.compile(r"(?:[1-9]\d*\s+evaluation tasks?(?:\s+\(.*\))?|not recorded\b.*)", flags=re.IGNORECASE),+ re.compile(r"(?:[1-9]\d*\s+evaluation tasks?(?:\s+\(.*\))?|not recorded\b.*)", flags=re.IGNORECASE).fullmatch,),("Dataset digest",- re.compile(r"(?:`[^`\s][^`]*`(?:\s+\([^)]*\))?|not recorded\b.*)", flags=re.IGNORECASE),+ re.compile(r"(?:`[^`\s][^`]*`(?:\s+\([^)]*\))?|not recorded\b.*)", flags=re.IGNORECASE).fullmatch,),("Attempts per task",- re.compile(r"(?:[1-9]\d*|not recorded\b.*)", flags=re.IGNORECASE),+ re.compile(r"(?:[1-9]\d*|not recorded\b.*)", flags=re.IGNORECASE).fullmatch,),("Environment",- re.compile(r"(?:`[^`\s][^`]*`|not recorded\b.*)", flags=re.IGNORECASE),+ re.compile(r"(?:`[^`\s][^`]*`|not recorded\b.*)", flags=re.IGNORECASE).fullmatch,),("Tier 3 evidence",- re.compile(r"(?:required for publication|optional by policy)", flags=re.IGNORECASE),+ re.compile(r"(?:required for publication|optional by policy)", flags=re.IGNORECASE).fullmatch,),)_PASS_METADATA_FIELD_RULES = (- ("Evaluation date", re.compile(r"\d{4}-\d{2}-\d{2}")),- ("Evaluator version", re.compile(r"`[^`\s][^`]*`")),- ("Tasks", re.compile(r"[1-9]\d*\s+evaluation tasks?(?:\s+\(.*\))?", flags=re.IGNORECASE)),+ ("Evaluation date", re.compile(r"\d{4}-\d{2}-\d{2}").fullmatch),+ ("Evaluator version", re.compile(r"`[^`\s][^`]*`").fullmatch),+ ("Tasks", re.compile(r"[1-9]\d*\s+evaluation tasks?(?:\s+\(.*\))?", flags=re.IGNORECASE).fullmatch),("Dataset digest",re.compile(r"`sha256:[0-9a-f]{64}`\s+\(skill-evaluator-dataset-snapshot/1\)",flags=re.IGNORECASE,- ),+ ).fullmatch,),- ("Attempts per task", re.compile(r"[1-9]\d*")),- ("Environment", re.compile(r"`[^`\s][^`]*`")),+ ("Attempts per task", re.compile(r"[1-9]\d*").fullmatch),+ ("Environment", re.compile(r"`[^`\s][^`]*`").fullmatch),+)+++# Evaluated-source provenance is opt-in. A card published before the identity+# contract existed cannot carry these fields, and the orchestration side has to+# supply them first, so the default scan stays byte-compatible with the previous+# behaviour and CI enables --require-source-provenance once the pipeline does.+_SOURCE_PROVENANCE_MARKERS = (+ "- Evaluated source:",+ "- Evaluated source revision:",+ "- Evaluator container revision:",+)+_SOURCE_REPOSITORY_VALUE = r"`[A-Za-z0-9][A-Za-z0-9._-]{0,38}/[A-Za-z0-9][A-Za-z0-9._-]{0,99}`"+# Mirrors ``skillevaluator.source_identity``: a full Git object id (SHA-1 or+# SHA-256), or a digest whose width matches the algorithm it names. A short+# prefix and an under-length digest are both rejected as ambiguous.+_SOURCE_REVISION_VALUE = r"`(?:[0-9a-f]{40}|[0-9a-f]{64}|sha256:[0-9a-f]{64}|sha384:[0-9a-f]{96}|sha512:[0-9a-f]{128})`"+_NOT_RECORDED_VALUE = re.compile(r"not recorded\b.*")+# Mirrors ``skillevaluator.source_identity`` character for character, including+# the path length, so the two sides cannot drift: an OCI reference is bounded+# component by component rather than as a whole string, because one cap over the+# whole reference counts the ``@sha256:`` suffix against the repository path and+# silently discards an ordinary name pinned by digest. The bound measures the+# path once the registry host has been split off it, which is where the+# reference grammar's RepositoryNameTotalLengthMax applies.+_CONTAINER_PATH_MAX = 255+# A first component is a registry host only where the reference grammar says so:+# it is ``localhost``, it carries a dot, or it carries a port. Anything else+# begins the path, whose components are lower case, while a host label is+# matched in either case because DNS is case-insensitive.+_CONTAINER_HOST_LABEL = r"[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?"+_CONTAINER_REFERENCE_PATTERN = (+ r"(?P<name>"+ rf"(?:(?:localhost|{_CONTAINER_HOST_LABEL}(?:\.{_CONTAINER_HOST_LABEL})+)(?::[0-9]+)?/"+ rf"|{_CONTAINER_HOST_LABEL}:[0-9]+/)?"+ r"(?P<path>"+ r"[a-z0-9]+(?:(?:[._]|__|-+)[a-z0-9]+)*"+ r"(?:/[a-z0-9]+(?:(?:[._]|__|-+)[a-z0-9]+)*)*"+ r")"+ r")"+ r"(?::(?P<tag>[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}))?"+ r"(?:@(?P<digest>sha256:[0-9a-f]{64}|sha384:[0-9a-f]{96}|sha512:[0-9a-f]{128}))?"+)+_CONTAINER_REFERENCE = re.compile(_CONTAINER_REFERENCE_PATTERN)+# Anchors included, so this is the same pattern text as ``_SOURCE_COMMIT`` and+# the drift test can compare the two strings rather than trusting the eye.+_GIT_OBJECT_ID = re.compile(r"^(?:[0-9a-f]{40}|[0-9a-f]{64})$")+++def _code_span_value(value: str) -> str | None:+ """Return what a metadata field renders inside its code span, or ``None``."""+ if len(value) > 2 and value.startswith("`") and value.endswith("`"):+ return value[1:-1]+ return None+++def _valid_container_reference(reference: str, *, require_digest: bool) -> bool:+ if _GIT_OBJECT_ID.fullmatch(reference):+ # An evaluator run from a source checkout names its implementation+ # revision instead of an image, and that revision is already immutable.+ return True+ match = _CONTAINER_REFERENCE.fullmatch(reference)+ if match is None or len(match["path"]) > _CONTAINER_PATH_MAX:+ return False+ # A bare name identifies a repository rather than the build that ran, so it+ # never records a revision on its own.+ return bool(match["digest"]) if require_digest else bool(match["tag"] or match["digest"])+++def _valid_container_revision(value: str) -> bool:+ """Accept a reference that names a revision, or the recorded absence."""+ if _NOT_RECORDED_VALUE.fullmatch(value):+ return True+ reference = _code_span_value(value)+ return reference is not None and _valid_container_reference(reference, require_digest=False)+++_SOURCE_METADATA_FIELD_RULES = (+ ("Evaluated source", re.compile(rf"(?:{_SOURCE_REPOSITORY_VALUE}|not recorded\b.*)").fullmatch),+ ("Evaluated source revision", re.compile(rf"(?:{_SOURCE_REVISION_VALUE}|not recorded\b.*)").fullmatch),+ ("Evaluator container revision", _valid_container_revision),+)++# A published PASS must name the source it evaluated, whatever the Tier 3 policy+# says, so an "optional by policy" card cannot publish without the identity.+# A hand-authored backfill card may state its verdict without a blockquote, and+# the rollout runbook points this flag at exactly those trees, so the source+# check matches the verdict line either way. Making the marker optional is what+# makes the list-item exclusion load-bearing here: the policy section restates+# the rule as a bullet on every card, so without it an INCOMPLETE or FAIL card+# reads as a published PASS and is charged for provenance it never claimed.+_PUBLISHED_PASS = re.compile(+ rf"{_VERDICT_LINE_START}>?{_VERDICT_PASS_TAIL}",+ flags=re.IGNORECASE | re.MULTILINE,+)+++def _valid_pass_container_revision(value: str) -> bool:+ """Require the digest of the build that ran, on the same grammar as above.++ A mutable tag can be repointed to a different build after the card is+ published, which leaves the reader unable to recover what actually ran, so a+ PASS has to pin the evaluator by digest or by a full implementation revision.+ """+ reference = _code_span_value(value)+ return reference is not None and _valid_container_reference(reference, require_digest=True)+++_PASS_SOURCE_PROVENANCE_RULES = (+ ("Evaluated source", re.compile(_SOURCE_REPOSITORY_VALUE).fullmatch),+ ("Evaluated source revision", re.compile(_SOURCE_REVISION_VALUE).fullmatch),+ ("Evaluator container revision", _valid_pass_container_revision),)@@ -154,7 +276,7 @@ def benchmark_files(roots: list[Path]) -> list[Path]:return sorted(found)-def scan_file(path: Path) -> list[Offender]:+def scan_file(path: Path, *, require_source_provenance: bool = False) -> list[Offender]:try:text = path.read_text(encoding="utf-8")except (OSError, UnicodeError) as error:@@ -162,7 +284,8 @@ def scan_file(path: Path) -> list[Offender]:offenders: list[Offender] = []- for marker in REQUIRED_MARKERS:+ markers = REQUIRED_MARKERS + (_SOURCE_PROVENANCE_MARKERS if require_source_provenance else ())+ for marker in markers:if marker not in text:offenders.append(Offender(path, 1, f"missing required section: {marker}"))@@ -171,24 +294,37 @@ def scan_file(path: Path) -> list[Offender]:if pattern.search(line):offenders.append(Offender(path, line_number, reason))- _check_metadata_semantics(path, text, offenders)+ _check_metadata_semantics(path, text, offenders, require_source_provenance=require_source_provenance)_check_verdict_tier_consistency(path, text, offenders)+ if require_source_provenance and _PUBLISHED_PASS.search(text):+ # Independent of the Tier 3 row: a card publishing a PASS must name the+ # source it evaluated, including one skipped under an optional policy.+ _check_source_provenance(path, text, offenders)return offenders-def _check_metadata_semantics(path: Path, text: str, offenders: list[Offender]) -> None:+def _check_metadata_semantics(+ path: Path,+ text: str,+ offenders: list[Offender],+ *,+ require_source_provenance: bool = False,+) -> None:metadata_lines = _metadata_section_lines(text)if not metadata_lines:return- for field, pattern in _METADATA_FIELD_RULES:+ rules = _METADATA_FIELD_RULES+ if require_source_provenance:+ rules += _SOURCE_METADATA_FIELD_RULES+ for field, is_valid in rules:matches = _metadata_field_matches(metadata_lines, field)marker = f"- {field}:"if not matches:offenders.append(Offender(path, metadata_lines[0][0], f"missing metadata field: {marker}"))continueline_number, value = matches[0]- if len(matches) > 1 or not pattern.fullmatch(value):+ if len(matches) > 1 or not is_valid(value):offenders.append(Offender(path, line_number, f"invalid metadata field: {marker}"))agent_matches = _metadata_field_matches(metadata_lines, "Agents")@@ -283,14 +419,25 @@ def _check_verdict_tier_consistency(path: Path, text: str, offenders: list[Offenoffenders.append(Offender(path, line_number, "publication PASS without completed Tier 3 evidence"))+def _check_source_provenance(path: Path, text: str, offenders: list[Offender]) -> None:+ """Reject a published PASS that does not record the source it evaluated."""+ metadata_lines = _metadata_section_lines(text)+ fallback_line = metadata_lines[0][0] if metadata_lines else 1+ for field, is_valid in _PASS_SOURCE_PROVENANCE_RULES:+ matches = _metadata_field_matches(metadata_lines, field)+ line_number = matches[0][0] if matches else fallback_line+ if len(matches) != 1 or not is_valid(matches[0][1]):+ offenders.append(Offender(path, line_number, f"publication PASS without recorded {field.lower()}"))++def _check_pass_provenance(path: Path, text: str, offenders: list[Offender]) -> None:"""Reject PASS cards that replace required provenance with legacy placeholders."""metadata_lines = _metadata_section_lines(text)fallback_line = metadata_lines[0][0] if metadata_lines else 1- for field, pattern in _PASS_METADATA_FIELD_RULES:+ for field, is_valid in _PASS_METADATA_FIELD_RULES:matches = _metadata_field_matches(metadata_lines, field)line_number = matches[0][0] if matches else fallback_line- if len(matches) != 1 or not pattern.fullmatch(matches[0][1]):+ if len(matches) != 1 or not is_valid(matches[0][1]):offenders.append(Offender(path, line_number, f"publication PASS without recorded {field.lower()}"))agent_matches = _metadata_field_matches(metadata_lines, "Agents")@@ -309,7 +456,11 @@ def _valid_recorded_agent_models(value: str) -> bool:return bool(agents) and all(_RECORDED_AGENT_MODEL_STATE.fullmatch(agent) for agent in agents)-def find_offenders(roots: list[Path]) -> tuple[list[Path], list[Offender]]:+def find_offenders(+ roots: list[Path],+ *,+ require_source_provenance: bool = False,+) -> tuple[list[Path], list[Offender]]:files: set[Path] = set()offenders: list[Offender] = []for root in roots:@@ -319,7 +470,11 @@ def find_offenders(roots: list[Path]) -> tuple[list[Path], list[Offender]]:offenders.append(Offender(root.expanduser().resolve(), 1, "input path does not exist"))sorted_files = sorted(files)- offenders.extend(offender for path in sorted_files for offender in scan_file(path))+ offenders.extend(+ offender+ for path in sorted_files+ for offender in scan_file(path, require_source_provenance=require_source_provenance)+ )return sorted_files, offenders@@ -337,12 +492,20 @@ def _parser() -> argparse.ArgumentParser:action="store_true",help="Fail when no BENCHMARK.md files are found",)+ parser.add_argument(+ "--require-source-provenance",+ action="store_true",+ help="Fail PASS cards that do not record an evaluated source repository and revision",+ )return parserdef main(argv: list[str] | None = None) -> int:args = _parser().parse_args(argv)- files, offenders = find_offenders(args.paths)+ files, offenders = find_offenders(+ args.paths,+ require_source_provenance=args.require_source_provenance,+ )if offenders:print("Public benchmark scan FAILED:")diff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.pyindex 67f24dbf..4e2cd673 100644--- a/src/skillevaluator/cli.py+++ b/src/skillevaluator/cli.py@@ -461,6 +461,101 @@ def _partial_agent_eval_result(return result+def _finalize_evaluated_source(+ results: list[ValidationResult],+ evaluated_source: dict[str, str] | None,+) -> None:+ """Resolve one source identity onto every result before a report is written.++ Provenance is part of every report, not just the card, so it has to be+ finalized ahead of ``emit_reports``. Resolving it afterwards published a JSON+ report that recorded neither the source repository nor its revision while+ BENCHMARK.md recorded both, and let a contradictory identity reach disk as+ JSON and HTML before benchmark generation failed the run.++ The supplied identity is one carrier among the ones the run recorded, not a+ default the results get to override. The fold either raises, when two+ carriers disagree, so a run that cannot say what it evaluated writes nothing+ at all, or yields the union of every carrier, and that union is written back+ so each report reads one complete identity rather than whichever fragment a+ producer happened to record. Leaving a recorded identity in place instead+ let a partial carrier shadow the rest: a result naming only the repository+ published a card calling the revision and the container not recorded, though+ the operator had supplied both.++ The identity rides on the results themselves, for every content type, because+ a PASS can be published without a completed Tier 3 run and so cannot rely on+ the Tier 3 payload as its carrier. A run that recorded no identity anywhere+ has none written, so an absent identity stays absent rather than becoming an+ empty one.+ """+ from skillevaluator.source_identity import (+ EvaluatedSourceConflict,+ merge_evaluated_sources,+ recorded_evaluated_source,+ )++ try:+ identity = merge_evaluated_sources(+ (evaluated_source, recorded_evaluated_source(result.metadata for result in results))+ )+ except EvaluatedSourceConflict as exc:+ # Name the values that disagreed rather than letting a report guess.+ raise click.ClickException(+ f"No report was written because the run records more than one evaluated source ({exc})."+ ) from exc+ if not identity:+ return+ for result in results:+ if isinstance(result.metadata, dict):+ # A fresh dict per result, so mutating one result's identity later+ # cannot rewrite what another result or the caller is holding.+ result.metadata["evaluated_source"] = dict(identity)+++def _evaluated_source_from_options(+ repository: str | None,+ revision: str | None,+ container_revision: str | None,+) -> dict[str, str] | None:+ """Build the evaluated-source identity from the orchestration input.++ The identity is supplied rather than inferred, because the tree running the+ evaluator is not the tree being evaluated. A value that is not in its+ canonical shape is rejected here rather than dropped, so a card never says+ ``not recorded`` for a field the operator believed they had supplied.+ """+ from skillevaluator.source_identity import normalized_evaluated_source++ supplied = {+ "repository": repository,+ # One option covers both immutable revision shapes the card accepts,+ # matching its single "Evaluated source revision" line.+ "commit": revision,+ "content_digest": revision,+ "evaluator_container_revision": container_revision,+ }+ identity = normalized_evaluated_source({key: value for key, value in supplied.items() if value}) or {}+ for option, value, accepted, expected in (+ ("--evaluated-source-repository", repository, ("repository",), "a forge name such as owner/repository"),+ (+ "--evaluated-source-revision",+ revision,+ ("commit", "content_digest"),+ "a full Git object id (40 or 64 hex characters) or a digest such as sha256:<64 hex characters>",+ ),+ (+ "--evaluator-container-revision",+ container_revision,+ ("evaluator_container_revision",),+ "an image reference such as ghcr.io/org/image@sha256:<64 hex characters>",+ ),+ ):+ if value and not any(field in identity for field in accepted):+ raise click.BadParameter(f"expected {expected}.", param_hint=option)+ return identity or None++def _run_agent_eval_or_skip(target_path: Path,*,@@ -482,6 +577,7 @@ def _run_agent_eval_or_skip(harbor_keep_jobs: bool = False,block_on_agent_eval: bool = False,validate_source: bool = True,+ evaluated_source: dict[str, str] | None = None,progress_reporter=None,) -> ValidationResult:"""Run Tier 3 live agent evaluation and fold the result into the combined report.@@ -530,6 +626,7 @@ def _run_agent_eval_or_skip(copy_repo=copy_repo,timeout_multiplier=timeout_multiplier,harbor_keep_jobs=harbor_keep_jobs,+ evaluated_source=evaluated_source,)try:service = EvaluationService()@@ -723,34 +820,6 @@ def _print_catalog_summary(total: int, failures: list[tuple[str, str]], reports_CATALOG_SUMMARY_FILENAME = "catalog-summary.json"-def _catalog_child_argv_from_sys(skill_dir: Path, output_dir: Path, parent_argv: list[str]) -> list[str]:- """Rebuild ``validate`` argv for one catalog skill from ``sys.argv``."""- argv = list(parent_argv)- try:- validate_idx = next(i for i, arg in enumerate(argv) if arg == "validate")- except StopIteration:- return ["validate", str(skill_dir), "-o", str(output_dir)]-- tail = argv[validate_idx + 1 :]-- child_tail: list[str] = []- skip_next = False- for arg in tail:- if skip_next:- skip_next = False- continue- if arg in {"--workers", "-o", "--output-dir"}:- skip_next = True- continue- if arg.startswith("--workers=") or arg.startswith("--output-dir=") or arg.startswith("-o="):- continue- if not arg.startswith("-"):- continue- child_tail.append(arg)-- return ["validate", str(skill_dir), *child_tail, "-o", str(output_dir)]--def _catalog_child_argv_from_ctx(ctx: click.Context, skill_dir: Path, output_dir: Path) -> list[str]:"""Rebuild ``validate`` argv from the active Click context (pytest-safe)."""params = ctx.params@@ -838,6 +907,15 @@ def _catalog_child_argv_from_ctx(ctx: click.Context, skill_dir: Path, output_dirargv.extend(["--timeout-multiplier", str(params["timeout_multiplier"])])if params.get("harbor_keep_jobs"):argv.append("--harbor-keep-jobs")+ # Every child renders its own BENCHMARK.md, so the identity has to reach+ # each one: dropping it here would publish a catalog of cards that all say+ # the evaluated source was never recorded.+ if params.get("evaluated_source_repository"):+ argv.extend(["--evaluated-source-repository", str(params["evaluated_source_repository"])])+ if params.get("evaluated_source_revision"):+ argv.extend(["--evaluated-source-revision", str(params["evaluated_source_revision"])])+ if params.get("evaluator_container_revision"):+ argv.extend(["--evaluator-container-revision", str(params["evaluator_container_revision"])])if _report_formats_explicit():for fmt in params.get("report_formats") or ():argv.extend(["-r", fmt])@@ -845,17 +923,6 @@ def _catalog_child_argv_from_ctx(ctx: click.Context, skill_dir: Path, output_dirreturn argv-def _catalog_child_argv(- ctx: click.Context,- skill_dir: Path,- output_dir: Path,- parent_argv: list[str],-) -> list[str]:- if "validate" in parent_argv:- return _catalog_child_argv_from_sys(skill_dir, output_dir, parent_argv)- return _catalog_child_argv_from_ctx(ctx, skill_dir, output_dir)--def _run_catalog_skill_worker(job: dict[str, Any]) -> tuple[str, bool, str, str | None]:"""Run one catalog skill validation in a child process."""import os@@ -1473,6 +1540,27 @@ def _print_run_banner(target_path: Path, content_type: str, profile: str | None)help_group=_TIER3_GROUP,help="Retain Harbor job dirs/artifacts after the run for inspection.",)+ "--evaluated-source-repository",+ default=None,+ cls=GroupedOption,+ help_group=_RUN_GROUP,+ help="Repository (owner/name) of the source tree being evaluated, recorded on BENCHMARK.md.",+)+ "--evaluated-source-revision",+ default=None,+ cls=GroupedOption,+ help_group=_RUN_GROUP,+ help="Immutable revision of the evaluated source: a full Git object id, or a sha256/sha384/sha512 digest.",+)+ "--evaluator-container-revision",+ default=None,+ cls=GroupedOption,+ help_group=_RUN_GROUP,+ help="Digest-pinned evaluator image reference recorded beside the evaluated source.",+)@_report_optionsdef validate(target_path: Path,@@ -1512,6 +1600,9 @@ def validate(timeout_multiplier: float | None,harbor_keep_jobs: bool,workers: int,+ evaluated_source_repository: str | None,+ evaluated_source_revision: str | None,+ evaluator_container_revision: str | None,report_formats: tuple[str, ...],output_dir: Path,) -> None:@@ -1531,6 +1622,11 @@ def validate(if dedup:_reject_linked_tier2_root(target_path)target_path = target_path.resolve()+ evaluated_source = _evaluated_source_from_options(+ evaluated_source_repository,+ evaluated_source_revision,+ evaluator_container_revision,+ )from skillevaluator.cli_core import detect_content_typefrom skillevaluator.constants import (@@ -1756,6 +1852,7 @@ def validate(harbor_keep_jobs=harbor_keep_jobs,block_on_agent_eval=block_on_agent_eval_effective,validate_source=preflight_tier3_source,+ evaluated_source=evaluated_source,progress_reporter=reporter,)results.append(tier3_result)@@ -1800,6 +1897,7 @@ def validate(# Reporters and the exit gate consume the same finalized result objects.apply_policy(results, policy)+ _finalize_evaluated_source(results, evaluated_source)content_label = {CONTENT_TYPE_SKILL: "Skill",@@ -1840,9 +1938,21 @@ def validate(if resolved_type == CONTENT_TYPE_SKILL:from skillevaluator.reporting import BenchmarkReporterfrom skillevaluator.reporting.naming import BENCHMARK_FILENAME+ from skillevaluator.source_identity import EvaluatedSourceConflictoutput_dir.mkdir(parents=True, exist_ok=True)- BenchmarkReporter(skill_name=target_path.name).save(results, output_dir / BENCHMARK_FILENAME)+ # ``_finalize_evaluated_source`` already attached and resolved the+ # identity, so a conflict aborts before any report file exists. This+ # stays as the last line of defence: the renderer re-validates the+ # carriers it is handed, and a producer can record one after the fact.+ try:+ BenchmarkReporter(skill_name=target_path.name).save(results, output_dir / BENCHMARK_FILENAME)+ except EvaluatedSourceConflict as exc:+ # Publication fails closed on a contradictory identity, so report which+ # values disagreed rather than letting the card write a guess.+ raise click.ClickException(+ f"{BENCHMARK_FILENAME} was not written because the run records more than one evaluated source ({exc})."+ ) from exceffective_gate_results = list(tier1_gate_results)if block_on_dedup_effective:@@ -2143,6 +2253,23 @@ def dedup_scan(@click.option("--override-cpus", type=int, default=None)@click.option("--override-memory-mb", type=int, default=None)@click.option("--override-storage-mb", type=int, default=None)+ "--evaluated-source-repository",+ default=None,+ # This command renders no card, so it names where it does record the value:+ # the run directory, which a later card is rendered from.+ help="Repository (owner/name) of the source tree being evaluated, persisted into the run's run_config.json.",+)+ "--evaluated-source-revision",+ default=None,+ help="Immutable revision of the evaluated source: a full Git object id, or a sha256/sha384/sha512 digest.",+)+ "--evaluator-container-revision",+ default=None,+ help="Digest-pinned evaluator image reference recorded beside the evaluated source.",+)@click.option("--progress",type=click.Choice(["auto", "rich", "plain", "off"]),@@ -2175,6 +2302,9 @@ def evaluate(override_cpus: int | None,override_memory_mb: int | None,override_storage_mb: int | None,+ evaluated_source_repository: str | None,+ evaluated_source_revision: str | None,+ evaluator_container_revision: str | None,progress: str,) -> None:"""Run Tier 3 live agent evaluation."""@@ -2182,6 +2312,13 @@ def evaluate(from skillevaluator.tier3.harbor.progress import create_progress_reporterservice = EvaluationService()+ # Resolved before the service is built so a non-canonical value is refused+ # at the boundary rather than after a long live run has already started.+ evaluated_source = _evaluated_source_from_options(+ evaluated_source_repository,+ evaluated_source_revision,+ evaluator_container_revision,+ )if autopilot:_ensure_autopilot_dataset(skill_path)@@ -2209,6 +2346,7 @@ def evaluate(override_cpus=override_cpus,override_memory_mb=override_memory_mb,override_storage_mb=override_storage_mb,+ evaluated_source=evaluated_source,)try:if env_mode == "local":diff --git a/src/skillevaluator/evaluation/options.py b/src/skillevaluator/evaluation/options.pyindex a131944c..819535f1 100644--- a/src/skillevaluator/evaluation/options.py+++ b/src/skillevaluator/evaluation/options.py@@ -46,6 +46,9 @@ class EvaluationOptions:override_cpus: int | None = Noneoverride_memory_mb: int | None = Noneoverride_storage_mb: int | None = None+ # Supplied by the orchestration input, never inferred from repository state:+ # the tree that runs the evaluator is not the tree being evaluated.+ evaluated_source: dict[str, str] | None = Nonedef engine_kwargs(self) -> dict[str, Any]:"""Return keyword arguments (excluding ``skill_path``) for the engine."""diff --git a/src/skillevaluator/evaluation/tier3_report.py b/src/skillevaluator/evaluation/tier3_report.pyindex 2bff758f..bcef8673 100644--- a/src/skillevaluator/evaluation/tier3_report.py+++ b/src/skillevaluator/evaluation/tier3_report.py@@ -39,6 +39,7 @@ from skillevaluator.constants import ()from skillevaluator.evidence import evidence_ref_identityfrom skillevaluator.models.result import Finding, Severity, ValidationResult+from skillevaluator.source_identity import resolve_evaluated_source# Verdict labels mirror SkillEvaluator's AGENT_EVAL_VERDICT_* values so the ported# reporters classify the overall outcome identically.@@ -219,6 +220,7 @@ def _advisory_agent_eval_payload("dataset_summary": dataset_summary,"dataset_digest": None,"dataset_digest_algorithm": None,+ "evaluated_source": None,"verdict_policy": verdict_policy,"execution_status": "skipped","execution_errors": [message],@@ -258,6 +260,7 @@ def _advisory_agent_eval_payload("dataset_summary": dataset_summary,"dataset_digest": None,"dataset_digest_algorithm": None,+ "evaluated_source": None,"verdict_policy": verdict_policy,"provenance": {"source": "advisory",@@ -420,6 +423,7 @@ def agent_eval_result_from_directory(engine_result: dict[str, Any] | None = None,evaluated_at: str | None = None,evaluator_version: str | None = None,+ evaluated_source: dict[str, Any] | None = None,use_llm_judge: bool = True,) -> ValidationResult | None:"""Build the canonical ``AGENT_EVAL`` result for one explicit Harbor run."""@@ -459,6 +463,7 @@ def agent_eval_result_from_directory(persisted_dataset_summary=run_truth.get("dataset_summary"),dataset_digest=run_truth.get("dataset_digest"),dataset_digest_algorithm=run_truth.get("dataset_digest_algorithm"),+ evaluated_source=evaluated_source,use_llm_judge=use_llm_judge,)return _validation_result_from_payload(payload)@@ -550,6 +555,7 @@ def build_agent_eval_payload(persisted_dataset_summary: dict[str, Any] | None = None,dataset_digest: str | None = None,dataset_digest_algorithm: str | None = None,+ evaluated_source: dict[str, Any] | None = None,use_llm_judge: bool = True,) -> dict[str, Any] | None:"""Assemble the canonical Tier 3 ``agent_eval`` payload from loaded agent data.@@ -632,6 +638,7 @@ def build_agent_eval_payload(effective_dataset_digest_algorithm = dataset_digest_algorithm or (str(computed_dataset_truth["dataset_digest_algorithm"]) if computed_dataset_truth else None)+ effective_evaluated_source = resolve_evaluated_source(evaluated_source, (run_config or {}).get("evaluated_source"))verdict_policy = _verdict_policy(policy)harbor_summary = _merge_harbor_viewer_summaries(_harbor_viewer_summary(canonical_trials),@@ -655,6 +662,7 @@ def build_agent_eval_payload("dataset_summary": dataset_summary,"dataset_digest": effective_dataset_digest,"dataset_digest_algorithm": effective_dataset_digest_algorithm,+ "evaluated_source": effective_evaluated_source,"verdict_policy": verdict_policy,"execution_status": execution_status,"execution_errors": execution_errors,@@ -714,6 +722,7 @@ def build_agent_eval_payload("dataset_summary": dataset_summary,"dataset_digest": effective_dataset_digest,"dataset_digest_algorithm": effective_dataset_digest_algorithm,+ "evaluated_source": effective_evaluated_source,"verdict_policy": verdict_policy,"agents": agent_payloads,"dimensions": best_dimensions,diff --git a/src/skillevaluator/reporting/benchmark.py b/src/skillevaluator/reporting/benchmark.pyindex 9980f73d..57e14112 100644--- a/src/skillevaluator/reporting/benchmark.py+++ b/src/skillevaluator/reporting/benchmark.py@@ -22,12 +22,17 @@ from skillevaluator.constants import (TIER3_LIFT_PASS_THRESHOLD,)from skillevaluator.reporting.base import ReporterBase, is_advisory_agent_eval_skip, passes_required_gate+from skillevaluator.source_identity import evaluated_source_revision, recorded_evaluated_sourcefrom skillevaluator.tier3_environments import HARBOR_ENV_MODESif TYPE_CHECKING:from skillevaluator.models import Finding, ValidationResult+# These fields are empty because the orchestration input did not supply them,+# which is a different cause from the legacy/non-live wording used elsewhere.+_SOURCE_UNRECORDED = "not recorded (not supplied by the orchestration input)"+_SIGNAL_DESCRIPTIONS = {"security": "unsafe operations, secret leakage, and unauthorized access","skill_execution": "whether the expected skill was found and executed",@@ -197,6 +202,29 @@ class BenchmarkReporter(ReporterBase):else "- Evaluator version: not recorded (legacy or non-live result)")+ # Validated by ``normalized_evaluated_source``, so the identity is+ # published verbatim: escaping would rewrite `_` and `@` and corrupt the+ # very value the card exists to record.+ source = _evaluated_source(results)+ repository = source.get("repository", "")+ lines.append(+ f"- Evaluated source: `{repository}`" if repository else "- Evaluated source: " + _SOURCE_UNRECORDED+ )++ revision = evaluated_source_revision(source)+ lines.append(+ f"- Evaluated source revision: `{revision}`"+ if revision+ else "- Evaluated source revision: " + _SOURCE_UNRECORDED+ )++ container_revision = source.get("evaluator_container_revision", "")+ lines.append(+ f"- Evaluator container revision: `{container_revision}`"+ if container_revision+ else "- Evaluator container revision: " + _SOURCE_UNRECORDED+ )+agents = _agents(ae)if agents:lines.append(@@ -555,6 +583,13 @@ class BenchmarkReporter(ReporterBase):def _agent_eval_payload(results: list[ValidationResult]) -> dict[str, Any] | None:+ """Return the payload the score and run-detail lines describe.++ A card reports one live evaluation, so the first payload found is the one+ rendered. That choice is safe for scores, which belong to a single run, but+ not for the evaluated-source identity: see ``_evaluated_source``, which folds+ every carrier instead so a second payload cannot be silently dropped.+ """for result in results:payload = result.metadata.get("agent_eval") if isinstance(result.metadata, dict) else Noneif isinstance(payload, dict):@@ -803,6 +838,30 @@ def _evaluation_date(value: str) -> str:return value[:10] if re.fullmatch(r"\d{4}-\d{2}-\d{2}.*", value) else value+def _evaluated_source(results: list[ValidationResult]) -> dict[str, str]:+ """Return the validated evaluated-source identity recorded for this run.++ Every card shape needs a carrier for the identity, not just a completed+ Tier 3 run: a Tier 1-only card and an advisory Tier 3 skip can both publish+ a PASS. The identity is therefore read from every result's metadata and from+ every nested agent-eval payload alike, mirroring how the persisted+ publication policy is resolved.++ The results are handed to ``recorded_evaluated_source`` whole rather than+ alongside the single payload the score lines use, because that payload is+ the first one found: a run carrying two agent-eval results would otherwise+ publish the identity of whichever came first and would name the other after+ the list was reversed. Folding every carrier makes that disagreement raise+ ``EvaluatedSourceConflict`` in either order.++ The value is re-validated by the fold rather than trusted, because a card can+ be rendered from a hand-built or legacy metadata dict that never passed the+ producer. Kept separate from the evaluator/container provenance so a reader+ can tell which source tree was evaluated from the build that evaluated it.+ """+ return recorded_evaluated_source(result.metadata for result in results) or {}++def _environment(ae: dict[str, Any] | None) -> str | None:summary = _mapping((ae or {}).get("summary"))value = summary.get("environment") or (ae or {}).get("environment")diff --git a/src/skillevaluator/reporting/json_reporter.py b/src/skillevaluator/reporting/json_reporter.pyindex 2fd32749..ef6bfb83 100644--- a/src/skillevaluator/reporting/json_reporter.py+++ b/src/skillevaluator/reporting/json_reporter.py@@ -19,6 +19,7 @@ from datetime import UTC, datetimefrom typing import TYPE_CHECKING, Anyfrom skillevaluator.reporting.base import ReporterBase, is_advisory_agent_eval_skip, passes_required_gate+from skillevaluator.source_identity import recorded_evaluated_sourceif TYPE_CHECKING:from skillevaluator.models import ValidationResult@@ -115,6 +116,16 @@ class JSONReporter(ReporterBase):if policy is not None:data["policy"] = policy+ # BENCHMARK.md records which source tree was evaluated, so the report a+ # CI job actually parses has to record it too: a consumer cannot verify+ # a card it never reads. The key is always present, so ``null`` says the+ # orchestration input supplied no identity rather than leaving a caller+ # to guess whether an older reporter simply omitted it. Every carrier is+ # folded, so a run recording two source trees raises rather than+ # publishing one of them. The CLI resolves the identity before any+ # report is written, so this fails closed only on programmatic misuse.+ data["evaluated_source"] = recorded_evaluated_source(r.metadata for r in results)+gating_by_tier: dict[str, dict[str, Any]] = {}for result in results:gating = result.metadata.get("gating")diff --git a/src/skillevaluator/source_identity.py b/src/skillevaluator/source_identity.pynew file mode 100644index 00000000..f53ac1e2--- /dev/null+++ b/src/skillevaluator/source_identity.py@@ -0,0 +1,247 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Evaluated-source identity carried from the orchestration input.++A published benchmark card has to say which source tree was evaluated, separately+from the evaluator build that evaluated it. That identity is supplied by the+orchestration input and carried through the evaluation-result contract unchanged;+it is never inferred from repository state while rendering, because the tree that+renders a card is the evaluator checkout rather than the evaluated skill's source.++The field patterns live here, outside both ``evaluation`` and ``reporting``, so+the producer, the renderer and the publication gate all validate against exactly+one definition of the identity.++Validation is what makes the identity safe to publish, so the card renders these+values verbatim rather than escaping them. Escaping would corrupt them: Markdown+inline escaping rewrites ``_`` to ``\\_`` and ``@`` to ``@``, which turns+``org/nv_examples`` and ``ghcr.io/x@sha256:...`` into strings that no longer name+the thing they identify. Every character below is inert inside a Markdown code+span -- backtick, backslash, angle brackets and whitespace are all excluded -- so+a validated value cannot break out of the span or inject markup.+"""++from __future__ import annotations++import re+from collections.abc import Iterable+from typing import Final++# Forge limits: GitHub owners are <=39 characters and repository names <=100.+_SOURCE_REPOSITORY: Final = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,38}/[A-Za-z0-9][A-Za-z0-9._-]{0,99}$")+# A Git object id is a full SHA-1 (40 hex) or SHA-256 (64 hex) name. A short+# prefix is ambiguous -- it can grow a second match as the tree grows -- so it+# must not render as the immutable revision a reader is asked to trust.+_SOURCE_COMMIT: Final = re.compile(r"^(?:[0-9a-f]{40}|[0-9a-f]{64})$")+# A digest names its algorithm, so the algorithm half is an allowlist rather+# than a charset ("totally-fake:0000..." must not read as a canonical digest),+# and each algorithm admits only its own exact width, so "sha512:" followed by+# 32 hex characters cannot pass as a canonical sha512 digest.+_SOURCE_CONTENT_DIGEST: Final = re.compile(r"^(?:sha256:[0-9a-f]{64}|sha384:[0-9a-f]{96}|sha512:[0-9a-f]{128})$")+# An OCI reference is bounded component by component rather than as a whole.+# One cap over the whole string counts the ``@sha256:`` suffix against the+# repository path, which silently discards an ordinary name pinned by digest,+# and the OCI grammar already bounds each component on its own terms. The bound+# is that grammar's RepositoryNameTotalLengthMax, which measures the path once+# the registry host has been split off it, so a long host name cannot spend the+# budget a path is entitled to.+_CONTAINER_PATH_MAX: Final = 255+# A registry host label. Host names are matched in either case because DNS is+# case-insensitive, unlike the path, which the grammar admits in lower case only.+_CONTAINER_HOST_LABEL: Final = r"[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?"+# name: an optional registry host followed by the slash-separated path; path:+# that path on its own, which is what the length bound above measures; tag: an+# optional mutable label of at most 128 characters; digest: the allowlist and+# exact widths used for the content digest above, so an under-length or+# invented algorithm cannot pass.+#+# A first component is a registry host only where the reference grammar says so:+# it is ``localhost``, it carries a dot, or it carries a port. Anything else+# begins the path and is held to the path's rules. Reading any single first+# label as a host instead made the grammar asymmetric, accepting+# ``NVIDIA/skillevaluator@sha256:...`` while refusing the same repository under+# a registry, ``ghcr.io/NVIDIA/skillevaluator@sha256:...``; both are uppercase+# path components, so both are refused.+_CONTAINER_REFERENCE_PATTERN: Final = (+ r"(?P<name>"+ rf"(?:(?:localhost|{_CONTAINER_HOST_LABEL}(?:\.{_CONTAINER_HOST_LABEL})+)(?::[0-9]+)?/"+ rf"|{_CONTAINER_HOST_LABEL}:[0-9]+/)?"+ r"(?P<path>"+ r"[a-z0-9]+(?:(?:[._]|__|-+)[a-z0-9]+)*"+ r"(?:/[a-z0-9]+(?:(?:[._]|__|-+)[a-z0-9]+)*)*"+ r")"+ r")"+ r"(?::(?P<tag>[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}))?"+ r"(?:@(?P<digest>sha256:[0-9a-f]{64}|sha384:[0-9a-f]{96}|sha512:[0-9a-f]{128}))?"+)+_CONTAINER_REFERENCE: Final = re.compile(_CONTAINER_REFERENCE_PATTERN)+++def is_container_revision(value: str) -> bool:+ """Return whether the value names the evaluator build that produced a card.++ The path length is checked in Python rather than in the pattern because a+ regex cannot bound one alternation-heavy group without either duplicating+ the grammar or capping the reference as a whole, which is the bug this+ replaces.+ """+ if _SOURCE_COMMIT.fullmatch(value):+ # An evaluator run from a source checkout has no image to name, so its+ # own implementation revision stands in, as the publication gate has+ # always allowed for a published PASS.+ return True+ reference = _CONTAINER_REFERENCE.fullmatch(value)+ if reference is None or len(reference["path"]) > _CONTAINER_PATH_MAX:+ return False+ # A bare name identifies a repository, not a revision: it cannot tell a+ # reader which build ran, so it is refused here rather than published as an+ # identity that does not identify anything.+ return bool(reference["tag"] or reference["digest"])+++# Each field carries the predicate that accepts its canonical shape, because+# the container reference needs a length rule the pattern cannot express.+EVALUATED_SOURCE_FIELDS: Final = (+ ("repository", _SOURCE_REPOSITORY.fullmatch),+ ("commit", _SOURCE_COMMIT.fullmatch),+ ("content_digest", _SOURCE_CONTENT_DIGEST.fullmatch),+ ("evaluator_container_revision", is_container_revision),+)++_CASE_FOLDED: Final = frozenset({"commit", "content_digest"})+++def normalized_evaluated_source(value: object) -> dict[str, str] | None:+ """Return the validated evaluated-source identity, or ``None``.++ Each field is accepted only in its canonical shape, so a published card can+ never carry an unverified or Markdown-injecting provenance value. A field+ that does not validate is dropped rather than guessed at, which keeps the+ card honest about what the orchestration input actually recorded.+ """+ if not isinstance(value, dict):+ return None++ normalized: dict[str, str] = {}+ for field_name, is_canonical in EVALUATED_SOURCE_FIELDS:+ raw = value.get(field_name)+ if not isinstance(raw, str):+ continue+ candidate = raw.strip()+ if field_name in _CASE_FOLDED:+ # Hex revisions are case-insensitive, so fold them to one spelling+ # rather than dropping an otherwise valid uppercase digest.+ candidate = candidate.lower()+ if candidate and is_canonical(candidate):+ normalized[field_name] = candidate+ return normalized or None+++def evaluated_source_revision(source: dict[str, str] | None) -> str:+ """Return the immutable revision of the evaluated source.++ Issue #72 accepts either an evaluated source commit SHA *or* a canonical+ digest of the evaluated skill content, so a run with no upstream commit+ still records an immutable identity. The card labels this ``Evaluated+ source revision`` rather than ``commit`` because it may legitimately be+ either.+ """+ if not source:+ return ""+ return str(source.get("commit") or source.get("content_digest") or "")+++class EvaluatedSourceConflict(ValueError):+ """Two orchestration inputs disagree about what was evaluated."""+++def merge_evaluated_sources(candidates: Iterable[object]) -> dict[str, str] | None:+ """Merge every populated carrier of the identity into one, or raise.++ A card can carry the identity in more than two places -- the Tier 3 payload,+ that payload's summary, and the metadata of any validation result -- and the+ order those are visited is incidental to how the run was scheduled. Reducing+ them by "first valid wins" would therefore let result ordering decide which+ source a published card claims to describe, which is precisely the+ unverifiable provenance this contract exists to prevent.++ Every carrier is normalized and folded together field by field instead, so a+ disagreement anywhere among them fails closed rather than being silently+ resolved by position.+ """+ merged: dict[str, str] = {}+ for candidate in candidates:+ normalized = normalized_evaluated_source(candidate)+ if not normalized:+ continue+ conflicts = sorted(field for field in normalized.keys() & merged.keys() if normalized[field] != merged[field])+ if conflicts:+ detail = ", ".join(f"{field}: {merged[field]!r} vs {normalized[field]!r}" for field in conflicts)+ raise EvaluatedSourceConflict(f"conflicting evaluated source identity ({detail})")+ merged.update(normalized)+ return merged or None+++def evaluated_source_carriers(metadata: object) -> tuple[object, ...]:+ """Return every place one validation result can carry the identity.++ A result records the identity in its own metadata, and a Tier 3 result also+ carries the agent-eval payload, which the producer stamps at both its top+ level and inside its summary. Listing the three places once, here, keeps the+ card, the machine-readable report and the CLI reading the same set: a+ carrier honoured by one report and ignored by another is how a run ends up+ publishing two different answers about what it evaluated.++ A level that is missing or is not a mapping yields a ``None`` entry rather+ than being skipped, because ``merge_evaluated_sources`` already drops+ whatever does not validate and a caller should not have to know the shape of+ each nesting level. A metadata value that is not a mapping carries nothing,+ so it yields no entries at all.+ """+ if not isinstance(metadata, dict):+ return ()+ payload = metadata.get("agent_eval")+ payload = payload if isinstance(payload, dict) else {}+ summary = payload.get("summary")+ return (+ metadata.get("evaluated_source"),+ payload.get("evaluated_source"),+ summary.get("evaluated_source") if isinstance(summary, dict) else None,+ )+++def recorded_evaluated_source(metadatas: Iterable[object]) -> dict[str, str] | None:+ """Return the single identity a run recorded, or raise if it recorded two.++ Every carrier of every result is folded, rather than one nested agent-eval+ payload being selected and the rest of them ignored. Selecting one would let+ result ordering decide what a report claims to describe: a second agent-eval+ result naming a different source tree would be shadowed by whichever result+ the run happened to produce first, and reversing the list would publish the+ other identity instead of failing closed.++ Conflicts raise ``EvaluatedSourceConflict``, so a run that cannot say what it+ evaluated publishes nothing rather than a guess.+ """+ return merge_evaluated_sources(+ carrier for metadata in metadatas for carrier in evaluated_source_carriers(metadata)+ )+++def resolve_evaluated_source(+ explicit: object,+ fallback: object,+) -> dict[str, str] | None:+ """Merge the two orchestration channels into one identity.++ Fields are resolved individually rather than whole-dict, so a mistyped+ field in one channel cannot silently discard a good identity in the other.++ Publication has to fail closed when the recorded provenance "conflicts with+ the orchestration input", so a field the two channels both supply and+ disagree about raises rather than being resolved by precedence: a card that+ quietly picked one of two contradictory source revisions would be exactly+ the unverifiable provenance this contract exists to prevent.+ """+ return merge_evaluated_sources((fallback, explicit))diff --git a/src/skillevaluator/tier3/commands.py b/src/skillevaluator/tier3/commands.pyindex 6a7823a8..51b8157b 100644--- a/src/skillevaluator/tier3/commands.py+++ b/src/skillevaluator/tier3/commands.py@@ -624,6 +624,7 @@ def evaluate(override_cpus: int | None,override_memory_mb: int | None,override_storage_mb: int | None,+ evaluated_source: dict[str, str] | None = None,progress_reporter: ProgressReporter | None = None,) -> dict[str, Any]:"""Run Harbor live-agent evaluation for a skill."""@@ -688,6 +689,7 @@ def evaluate(env_mode=env_mode,env_mode_source="CLI",timeout_multiplier=timeout_multiplier,+ evaluated_source=evaluated_source,override_cpus=override_cpus,override_memory_mb=override_memory_mb,override_storage_mb=override_storage_mb,diff --git a/src/skillevaluator/tier3/harbor/runner.py b/src/skillevaluator/tier3/harbor/runner.pyindex ff6d1b7d..3c586775 100644--- a/src/skillevaluator/tier3/harbor/runner.py+++ b/src/skillevaluator/tier3/harbor/runner.py@@ -36,6 +36,7 @@ from skillevaluator.provider_config import (_normalize_anthropic_base_url,resolve_llm_provider,)+from skillevaluator.source_identity import normalized_evaluated_sourcefrom skillevaluator.tier3.evals_config import EvalsConfigError, load_evals_configfrom skillevaluator.tier3.harbor.adapter import (_VERIFIER_JUDGE_MODEL_ENV_VARS,@@ -1801,6 +1802,7 @@ def _run_harbor_eval_impl(override_cpus: int | None = None,override_memory_mb: int | None = None,override_storage_mb: int | None = None,+ evaluated_source: dict[str, str] | None = None,progress_reporter: ProgressReporter | None = None,_evaluator_skill_path: Path | None = None,_monotonic_start: float | None = None,@@ -2150,6 +2152,9 @@ def _run_harbor_eval_impl("task_source": task_source,"grading": {"mode": grading_mode},"agents": model_resolution,+ # Persisted so a card rendered later from this run directory records the+ # source tree that was evaluated, not the evaluator build that ran it.+ "evaluated_source": normalized_evaluated_source(evaluated_source),}verifier_env = {**configured_runtime_env, **provider_env}staged_verifier_env = {name: f"${{{name}}}" for name in verifier_env if name not in _VERIFIER_JUDGE_MODEL_ENV_VARS}diff --git a/tests/golden/benchmark_fail/BENCHMARK.md b/tests/golden/benchmark_fail/BENCHMARK.mdindex 23f18a11..4b06fa62 100644--- a/tests/golden/benchmark_fail/BENCHMARK.md+++ b/tests/golden/benchmark_fail/BENCHMARK.md@@ -9,6 +9,9 @@ The skill should be reviewed before publication. Address the blocking findings b- Skill: `demo-skill`- Evaluation date: 2026-07-24- Evaluator version: `0.8.2`+- Evaluated source: `NVIDIA/demo_skills`+- Evaluated source revision: `2263a2ebdab903e87f7e7c0a001d22c3a926a9cf`+- Evaluator container revision: `ghcr.io/nvidia/skillevaluator@sha256:0117bc2e3e54da4244a656466526c5b1b5a559ea0117bc2e3e54da4244a65646`- Agents: Claude Code (`claude-sonnet`), Codex (`gpt-codex`)- Tasks: 8 evaluation tasks (6 positive, 2 negative)- Dataset digest: `sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef` (skill-evaluator-dataset-snapshot/1)diff --git a/tests/golden/benchmark_pass/BENCHMARK.md b/tests/golden/benchmark_pass/BENCHMARK.mdindex e86f1b57..abbeb8e9 100644--- a/tests/golden/benchmark_pass/BENCHMARK.md+++ b/tests/golden/benchmark_pass/BENCHMARK.md@@ -11,6 +11,9 @@ Recommended for publication based on the completed evaluation evidence in this r- Skill: `demo-skill`- Evaluation date: 2026-07-24- Evaluator version: `0.8.2`+- Evaluated source: `NVIDIA/demo_skills`+- Evaluated source revision: `2263a2ebdab903e87f7e7c0a001d22c3a926a9cf`+- Evaluator container revision: `ghcr.io/nvidia/skillevaluator@sha256:0117bc2e3e54da4244a656466526c5b1b5a559ea0117bc2e3e54da4244a65646`- Agents: Claude Code (`claude-sonnet`), Codex (`gpt-codex`)- Tasks: 8 evaluation tasks (6 positive, 2 negative)- Dataset digest: `sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef` (skill-evaluator-dataset-snapshot/1)diff --git a/tests/golden/cli_surface.json b/tests/golden/cli_surface.jsonindex 0b598491..2dfa9efe 100644--- a/tests/golden/cli_surface.json+++ b/tests/golden/cli_surface.json@@ -540,6 +540,30 @@"param_type": "option","type": "integer"},+ {+ "name": "evaluated_source_repository",+ "opts": [+ "--evaluated-source-repository"+ ],+ "param_type": "option",+ "type": "text"+ },+ {+ "name": "evaluated_source_revision",+ "opts": [+ "--evaluated-source-revision"+ ],+ "param_type": "option",+ "type": "text"+ },+ {+ "name": "evaluator_container_revision",+ "opts": [+ "--evaluator-container-revision"+ ],+ "param_type": "option",+ "type": "text"+ },{"choices": ["auto",@@ -1758,6 +1782,30 @@"param_type": "option","type": "boolean"},+ {+ "name": "evaluated_source_repository",+ "opts": [+ "--evaluated-source-repository"+ ],+ "param_type": "option",+ "type": "text"+ },+ {+ "name": "evaluated_source_revision",+ "opts": [+ "--evaluated-source-revision"+ ],+ "param_type": "option",+ "type": "text"+ },+ {+ "name": "evaluator_container_revision",+ "opts": [+ "--evaluator-container-revision"+ ],+ "param_type": "option",+ "type": "text"+ },{"choices": ["cli",@@ -2450,6 +2498,30 @@"param_type": "option","type": "integer"},+ {+ "name": "evaluated_source_repository",+ "opts": [+ "--evaluated-source-repository"+ ],+ "param_type": "option",+ "type": "text"+ },+ {+ "name": "evaluated_source_revision",+ "opts": [+ "--evaluated-source-revision"+ ],+ "param_type": "option",+ "type": "text"+ },+ {+ "name": "evaluator_container_revision",+ "opts": [+ "--evaluator-container-revision"+ ],+ "param_type": "option",+ "type": "text"+ },{"choices": ["auto",@@ -3055,6 +3127,30 @@"param_type": "option","type": "boolean"},+ {+ "name": "evaluated_source_repository",+ "opts": [+ "--evaluated-source-repository"+ ],+ "param_type": "option",+ "type": "text"+ },+ {+ "name": "evaluated_source_revision",+ "opts": [+ "--evaluated-source-revision"+ ],+ "param_type": "option",+ "type": "text"+ },+ {+ "name": "evaluator_container_revision",+ "opts": [+ "--evaluator-container-revision"+ ],+ "param_type": "option",+ "type": "text"+ },{"choices": ["cli",diff --git a/tests/golden/test_benchmark_card.py b/tests/golden/test_benchmark_card.pyindex eb3b87c9..c47611b1 100644--- a/tests/golden/test_benchmark_card.py+++ b/tests/golden/test_benchmark_card.py@@ -109,6 +109,11 @@ def _live_tier3_result() -> ValidationResult:},"dataset_digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef","dataset_digest_algorithm": "skill-evaluator-dataset-snapshot/1",+ "evaluated_source": {+ "repository": "NVIDIA/demo_skills",+ "commit": "2263a2ebdab903e87f7e7c0a001d22c3a926a9cf",+ "evaluator_container_revision": "ghcr.io/nvidia/skillevaluator@sha256:0117bc2e3e54da4244a656466526c5b1b5a559ea0117bc2e3e54da4244a65646",+ },"attempt_policy": {"max_attempts": 3, "pass_threshold": 0.5},"agents": {"claude-code": {diff --git a/tests/test_benchmark_source_provenance.py b/tests/test_benchmark_source_provenance.pynew file mode 100644index 00000000..e24f2939--- /dev/null+++ b/tests/test_benchmark_source_provenance.py@@ -0,0 +1,1308 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Evaluated-source provenance in the published benchmark card.++A published card has to say which source tree was evaluated, separately from the+evaluator build that evaluated it. Without that separation two skills from+different repositories can publish cards whose only recorded revision is the+shared evaluator container tag.+"""++from __future__ import annotations++from pathlib import Path++import pytest+from scripts.ci import check_public_benchmarks as benchmark_gate++from skillevaluator import source_identity+from skillevaluator.evaluation.tier3_report import build_agent_eval_payload+from skillevaluator.models import ValidationResult+from skillevaluator.reporting import BenchmarkReporter+from skillevaluator.source_identity import (+ EvaluatedSourceConflict,+ normalized_evaluated_source,+ resolve_evaluated_source,+)++_COMMIT_A = "2263a2ebdab903e87f7e7c0a001d22c3a926a9cf"+_COMMIT_B = "134c829305918a3e9a84e819b42b54fafd125186"+_CONTAINER = "ghcr.io/nvidia/skillevaluator@sha256:" + "0117bc2e" * 8+_DIGEST = "sha256:" + "ab" * 32+_UNRECORDED = "not recorded (not supplied by the orchestration input)"++_SCORES = {+ "security": 0.9,+ "skill_execution": 0.9,+ "skill_efficiency": 0.9,+ "accuracy": 0.9,+ "goal_accuracy": 0.9,+ "behavior_check": 0.9,+}+_AGENTS = {+ "codex": {+ "model": "gpt-codex",+ "with_skill": _SCORES,+ "without_skill": dict.fromkeys(_SCORES, 0.5),+ "execution_status": "succeeded",+ "rewards": [],+ "num_trials": 1,+ }+}+++def _card(evaluated_source: dict[str, str] | None) -> str:+ result = ValidationResult(+ validator_name="AGENT_EVAL",+ validator_description="Run live agent evaluation",+ )+ result.metadata["agent_eval"] = {+ "skill_name": "demo-skill",+ "evaluator_version": "0.8.2",+ "evaluated_source": evaluated_source,+ "summary": {"environment": "Isolated sandbox"},+ "agents": {"codex": {"model": "gpt-codex"}},+ }+ return BenchmarkReporter().render_all([result])+++def _value(card: str, field: str) -> str | None:+ return benchmark_gate._metadata_field_value(card, field)+++def _payload(**kwargs: object) -> dict[str, object]:+ payload = build_agent_eval_payload("demo-skill", _AGENTS, use_llm_judge=False, **kwargs)+ assert payload is not None+ return payload+++class TestNormalization:+ """The identity comes from orchestration input, so it is validated, not trusted."""++ def test_canonical_identity_is_preserved(self) -> None:+ assert normalized_evaluated_source({"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A}) == {+ "repository": "NVIDIA/NVFlare",+ "commit": _COMMIT_A,+ }++ @pytest.mark.parametrize("field", ["commit", "content_digest"])+ def test_hex_revisions_are_case_folded(self, field: str) -> None:+ value = _COMMIT_A if field == "commit" else _DIGEST+ assert normalized_evaluated_source({field: value.upper()}) == {field: value}++ @pytest.mark.parametrize(+ "value",+ [+ None,+ {},+ "NVIDIA/NVFlare",+ {"repository": "not a repository"},+ {"repository": "NVIDIA/NVFlare`` injected"},+ {"repository": "-leading/dash"},+ {"repository": "a" * 40 + "/repo"},+ {"commit": "zzzz"},+ {"commit": "abc"},+ {"content_digest": "sha256:nothex"},+ {"content_digest": "totally-fake:" + "0" * 64},+ {"evaluator_container_revision": "has space"},+ {"repository": 42, "commit": ["list"]},+ ],+ )+ def test_malformed_identity_is_dropped(self, value: object) -> None:+ assert normalized_evaluated_source(value) is None++ def test_valid_fields_survive_an_invalid_sibling(self) -> None:+ assert normalized_evaluated_source({"repository": "NVIDIA/NVFlare", "commit": "nope"}) == {+ "repository": "NVIDIA/NVFlare"+ }++ @pytest.mark.parametrize(+ "value",+ [+ {"commit": _COMMIT_A[:7]},+ {"commit": _COMMIT_A[:12]},+ {"commit": "a" * 39},+ {"commit": "a" * 41},+ {"content_digest": "sha512:" + "ab" * 16},+ {"content_digest": "sha256:" + "ab" * 48},+ {"content_digest": "sha384:" + "ab" * 32},+ ],+ )+ def test_ambiguous_or_mismatched_revisions_are_dropped(self, value: object) -> None:+ """A short prefix can grow a second match, and a digest must match its own algorithm."""+ assert normalized_evaluated_source(value) is None++ @pytest.mark.parametrize("commit", ["a" * 40, "b" * 64])+ def test_full_git_object_ids_are_accepted(self, commit: str) -> None:+ """SHA-1 and SHA-256 object ids are both canonical Git revisions."""+ assert normalized_evaluated_source({"commit": commit}) == {"commit": commit}++ @pytest.mark.parametrize(("algorithm", "width"), [("sha256", 64), ("sha384", 96), ("sha512", 128)])+ def test_each_digest_algorithm_requires_its_own_width(self, algorithm: str, width: int) -> None:+ digest = f"{algorithm}:" + "a" * width+ assert normalized_evaluated_source({"content_digest": digest}) == {"content_digest": digest}++ @pytest.mark.parametrize(+ "repository",+ ["holgerroth/nvflare_examples", "NVIDIA/Megatron_LM", "some-org/repo.name"],+ )+ def test_real_world_repository_names_are_accepted(self, repository: str) -> None:+ """Underscores and dots are ordinary in forge names and must survive."""+ assert normalized_evaluated_source({"repository": repository}) == {"repository": repository}++ def test_digest_pinned_container_reference_is_accepted(self) -> None:+ """``@sha256:`` is the only immutable way to pin the evaluator build."""+ assert normalized_evaluated_source({"evaluator_container_revision": _CONTAINER}) == {+ "evaluator_container_revision": _CONTAINER+ }++ def test_a_long_repository_pinned_by_digest_is_accepted(self) -> None:+ """A cap over the whole reference charged the digest to the name and dropped ordinary references."""+ reference = "ghcr.io/nvidia/" + "e" * 53 + "@sha256:" + "0117bc2e" * 8+ assert len(reference.partition("@")[0]) == 68+ assert len(reference) == 140+ assert normalized_evaluated_source({"evaluator_container_revision": reference}) == {+ "evaluator_container_revision": reference+ }++ @pytest.mark.parametrize(("length", "accepted"), [(255, True), (256, False)])+ def test_the_path_is_bounded_at_the_oci_maximum(self, length: int, accepted: bool) -> None:+ """255 is RepositoryNameTotalLengthMax, and it bounds the path rather than the whole reference."""+ reference = "n" * length + "@sha256:" + "0117bc2e" * 8+ assert bool(normalized_evaluated_source({"evaluator_container_revision": reference})) is accepted++ @pytest.mark.parametrize(("length", "accepted"), [(255, True), (256, False)])+ def test_a_registry_host_does_not_spend_the_path_budget(self, length: int, accepted: bool) -> None:+ """The reference grammar bounds the path once the host is split off, not the two together."""+ reference = "ghcr.io/" + "n" * length + "@sha256:" + "0117bc2e" * 8+ assert bool(normalized_evaluated_source({"evaluator_container_revision": reference})) is accepted++ @pytest.mark.parametrize(+ "revision",+ [+ "NVIDIA/skillevaluator@sha256:" + "0117bc2e" * 8,+ "ghcr.io/NVIDIA/skillevaluator@sha256:" + "0117bc2e" * 8,+ ],+ )+ def test_an_uppercase_path_is_refused_with_and_without_a_registry(self, revision: str) -> None:+ """Reading any first label as a host took the bare form and refused the same name under a registry."""+ assert normalized_evaluated_source({"evaluator_container_revision": revision}) is None++ @pytest.mark.parametrize(+ "revision",+ [+ "localhost/team/image@sha256:" + "0117bc2e" * 8,+ "Registry.Example.COM/team/image@sha256:" + "0117bc2e" * 8,+ "myregistry:5000/team/image@sha256:" + "0117bc2e" * 8,+ ],+ )+ def test_a_registry_host_is_localhost_dotted_or_ported(self, revision: str) -> None:+ """Those are the three shapes the grammar reads as a host, and a host name may use any case."""+ assert normalized_evaluated_source({"evaluator_container_revision": revision}) == {+ "evaluator_container_revision": revision+ }++ @pytest.mark.parametrize(+ "revision",+ [+ "localhost:5000/team/image:1.2.3@sha256:" + "0117bc2e" * 8,+ "nvcr.io/nvidia/skillevaluator:1.3.2",+ _COMMIT_A,+ ],+ )+ def test_references_that_name_a_revision_are_accepted(self, revision: str) -> None:+ """A registry port, a tag and an implementation revision all name the build that ran."""+ assert normalized_evaluated_source({"evaluator_container_revision": revision}) == {+ "evaluator_container_revision": revision+ }++ @pytest.mark.parametrize(+ "revision",+ [+ "ghcr.io/nvidia/skillevaluator",+ "ghcr.io/nvidia/skillevaluator@sha512:" + "ab" * 16,+ "ghcr.io/nvidia/skillevaluator@md5:" + "ab" * 16,+ "ghcr.io/nvidia/skillevaluator:" + "t" * 129,+ "has space",+ "ghcr.io/nvidia/skillevaluator`` injected",+ ],+ )+ def test_references_that_name_no_build_are_dropped(self, revision: str) -> None:+ """A bare name is a repository, not a revision, and the rest are unpinnable or not inert in a code span."""+ assert normalized_evaluated_source({"evaluator_container_revision": revision}) is None+++class TestPayloadContract:+ """The identity travels through the result contract rather than being re-derived."""++ def test_run_config_supplies_the_identity(self) -> None:+ payload = _payload(run_config={"evaluated_source": {"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A}})+ assert payload["evaluated_source"] == {"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A}+ assert payload["summary"]["evaluated_source"] == payload["evaluated_source"]++ def test_the_two_channels_combine(self) -> None:+ payload = _payload(+ run_config={"evaluated_source": {"commit": _COMMIT_A}},+ evaluated_source={"repository": "NVIDIA/NVFlare"},+ )+ assert payload["evaluated_source"] == {"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A}++ def test_absent_identity_is_recorded_as_none(self) -> None:+ assert _payload()["evaluated_source"] is None++ def test_malformed_identity_does_not_reach_the_payload(self) -> None:+ assert _payload(evaluated_source={"repository": "not a repository"})["evaluated_source"] is None+++class TestRendering:+ """Source provenance and evaluator provenance get separate, unambiguous labels."""++ def test_card_records_the_evaluated_source(self) -> None:+ card = _card({"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A, "evaluator_container_revision": _CONTAINER})+ assert _value(card, "Evaluated source") == "`NVIDIA/NVFlare`"+ assert _value(card, "Evaluated source revision") == f"`{_COMMIT_A}`"+ assert _value(card, "Evaluator container revision") == f"`{_CONTAINER}`"++ def test_evaluator_version_is_not_the_source_revision(self) -> None:+ card = _card({"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A})+ assert _value(card, "Evaluator version") == "`0.8.2`"+ assert _value(card, "Evaluated source revision") == f"`{_COMMIT_A}`"++ def test_content_digest_stands_in_for_a_missing_commit(self) -> None:+ card = _card({"repository": "NVIDIA/NVFlare", "content_digest": _DIGEST})+ assert _value(card, "Evaluated source revision") == f"`{_DIGEST}`"++ def test_commit_wins_over_a_content_digest(self) -> None:+ card = _card({"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A, "content_digest": _DIGEST})+ assert _value(card, "Evaluated source revision") == f"`{_COMMIT_A}`"++ def test_missing_identity_names_the_orchestration_input(self) -> None:+ card = _card(None)+ for field in ("Evaluated source", "Evaluated source revision", "Evaluator container revision"):+ assert _value(card, field) == _UNRECORDED++ @pytest.mark.parametrize("repository", ["holgerroth/nvflare_examples", "NVIDIA/Megatron_LM"])+ def test_underscored_repository_is_published_verbatim(self, repository: str) -> None:+ """Markdown escaping would rewrite ``_`` and publish a repository that does not exist."""+ card = _card({"repository": repository, "commit": _COMMIT_A})+ assert _value(card, "Evaluated source") == f"`{repository}`"+ assert "\\_" not in card++ def test_digest_pinned_container_is_published_verbatim(self) -> None:+ """Escaping ``@`` to ``@`` would publish an unresolvable image reference."""+ card = _card({"repository": "NVIDIA/NVFlare", "evaluator_container_revision": _CONTAINER})+ assert _value(card, "Evaluator container revision") == f"`{_CONTAINER}`"+ assert "@" not in card++ @pytest.mark.parametrize(+ "hostile",+ [+ {"repository": "A/B` INJECTED **bold** `x"},+ {"repository": "A/B\n- Evaluated source: `spoofed/repo`"},+ {"evaluator_container_revision": "x` <script>alert(1)</script> `"},+ {"commit": "`evil`"},+ ],+ )+ def test_hostile_payload_is_dropped_not_escaped(self, hostile: dict[str, str]) -> None:+ """A card can be rendered from a metadata dict that never passed the producer."""+ card = _card(hostile)+ for field in ("Evaluated source", "Evaluated source revision", "Evaluator container revision"):+ assert _value(card, field) == _UNRECORDED+ assert "INJECTED" not in card+ assert "spoofed/repo" not in card+ assert "<script>" not in card++ def test_two_repositories_sharing_an_evaluator_keep_distinct_source_revisions(self) -> None:+ """The regression this feature exists to prevent.++ Both skills are evaluated by the same evaluator container. Before the+ evaluated-source contract the only revision on either card was that+ shared container tag, so the two cards were indistinguishable.+ """+ first = _card({"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A, "evaluator_container_revision": _CONTAINER})+ second = _card(+ {"repository": "NVIDIA/NeMo-Fabric", "commit": _COMMIT_B, "evaluator_container_revision": _CONTAINER}+ )++ assert _value(first, "Evaluated source revision") == f"`{_COMMIT_A}`"+ assert _value(second, "Evaluated source revision") == f"`{_COMMIT_B}`"+ assert _value(first, "Evaluated source") != _value(second, "Evaluated source")++ # The shared evaluator provenance stays identical, and is never mistaken+ # for the source revision -- the precise confusion issue #72 reported.+ shared = _value(first, "Evaluator container revision")+ assert shared == _value(second, "Evaluator container revision") == f"`{_CONTAINER}`"+ assert _value(first, "Evaluated source revision") != shared+ assert _value(second, "Evaluated source revision") != shared++ def test_a_run_without_a_commit_does_not_borrow_the_container_revision(self) -> None:+ """With no source revision the card says so, rather than showing the evaluator's."""+ card = _card({"repository": "NVIDIA/NVFlare", "evaluator_container_revision": _CONTAINER})+ assert _value(card, "Evaluated source revision") == _UNRECORDED+ assert _value(card, "Evaluator container revision") == f"`{_CONTAINER}`"+++_PASS_CARD = """\+# Skill Benchmark: demo++> **Overall verdict: PASS**++## Evaluation Metadata++- Evaluation date: 2026-07-25+- Evaluator version: `0.8.3`+{source_lines}- Agents: Codex (`gpt-codex`)+- Tasks: 4 evaluation tasks+- Dataset digest: `sha256:{digest}` (skill-evaluator-dataset-snapshot/1)+- Attempts per task: 1+- Environment: `Isolated sandbox`+- Tier 3 evidence: {tier3_evidence}++## Results at a Glance++| Measure | Codex (Baseline → Skill Uplift) |+|---|---:|+| Overall | 47% → 92% (+45 points) |++## Tier Status++| Tier | Purpose | Status | Evidence |+|---|---|---|---|+| Tier 1 | Static validation | **PASSED** | Complete |+| Tier 2 | Semantic deduplication | **PASSED** | Complete |+| Tier 3 | Live agent evaluation | **{tier3_status}** | Complete |++## Freshness++Regenerate after material inputs change.+"""++_RECORDED = (+ "- Evaluated source: `NVIDIA/demo_skills`\n"+ f"- Evaluated source revision: `{_COMMIT_A}`\n"+ f"- Evaluator container revision: `{_CONTAINER}`\n"+)+_NOT_RECORDED = (+ f"- Evaluated source: {_UNRECORDED}\n"+ f"- Evaluated source revision: {_UNRECORDED}\n"+ f"- Evaluator container revision: {_UNRECORDED}\n"+)+_ABSENT = ""+++def _pass_card(+ source_lines: str,+ *,+ tier3_status: str = "PASS",+ tier3_evidence: str = "required for publication",+) -> str:+ return _PASS_CARD.format(+ source_lines=source_lines,+ digest="0123456789abcdef" * 4,+ tier3_status=tier3_status,+ tier3_evidence=tier3_evidence,+ )+++def _scan(tmp_path: Path, card: str, *, require: bool) -> list[str]:+ path = tmp_path / "BENCHMARK.md"+ path.write_text(card, encoding="utf-8")+ return [offender.reason for offender in benchmark_gate.scan_file(path, require_source_provenance=require)]+++class TestPublicationGateIsOptIn:+ """The default scan stays byte-compatible with pre-contract behaviour."""++ def test_card_without_the_fields_passes_by_default(self, tmp_path: Path) -> None:+ assert _scan(tmp_path, _pass_card(_ABSENT), require=False) == []++ def test_card_without_the_fields_fails_when_required(self, tmp_path: Path) -> None:+ reasons = _scan(tmp_path, _pass_card(_ABSENT), require=True)+ assert "missing required section: - Evaluated source:" in reasons+ assert "missing required section: - Evaluated source revision:" in reasons++ def test_placeholder_provenance_is_tolerated_by_default(self, tmp_path: Path) -> None:+ assert _scan(tmp_path, _pass_card(_NOT_RECORDED), require=False) == []+++class TestPublicationGateFailsClosed:+ """``--require-source-provenance`` makes a published PASS name its source."""++ def test_recorded_provenance_passes(self, tmp_path: Path) -> None:+ assert _scan(tmp_path, _pass_card(_RECORDED), require=True) == []++ def test_missing_provenance_fails_a_pass_card(self, tmp_path: Path) -> None:+ reasons = _scan(tmp_path, _pass_card(_NOT_RECORDED), require=True)+ assert "publication PASS without recorded evaluated source" in reasons+ assert "publication PASS without recorded evaluated source revision" in reasons++ def test_optional_tier3_policy_cannot_dodge_the_check(self, tmp_path: Path) -> None:+ """A PASS published under an optional-Tier-3 policy still needs the identity."""+ card = _pass_card(_NOT_RECORDED, tier3_status="SKIPPED", tier3_evidence="optional by policy")+ reasons = _scan(tmp_path, card, require=True)+ assert "publication PASS without recorded evaluated source" in reasons++ @pytest.mark.parametrize(+ ("field", "value"),+ [+ ("Evaluated source", "`not a repository`"),+ ("Evaluated source", "`-leading/dash`"),+ ("Evaluated source revision", "`zzzz`"),+ ("Evaluator container revision", "`has space`"),+ ],+ )+ def test_malformed_values_are_rejected(self, tmp_path: Path, field: str, value: str) -> None:+ lines = []+ for line in _RECORDED.strip().splitlines():+ name = line.split(":", 1)[0][2:]+ lines.append(f"- {field}: {value}" if name == field else line)+ reasons = _scan(tmp_path, _pass_card("\n".join(lines) + "\n"), require=True)+ assert f"invalid metadata field: - {field}:" in reasons+++class TestGateEntryPoints:+ """The flag has to survive main() and find_offenders(), not just scan_file()."""++ def _write(self, tmp_path: Path, card: str) -> Path:+ (tmp_path / "BENCHMARK.md").write_text(card, encoding="utf-8")+ return tmp_path++ def test_main_accepts_a_recorded_card(self, tmp_path: Path) -> None:+ root = self._write(tmp_path, _pass_card(_RECORDED))+ assert benchmark_gate.main(["--require-source-provenance", str(root)]) == 0++ def test_main_rejects_a_card_without_provenance(self, tmp_path: Path) -> None:+ root = self._write(tmp_path, _pass_card(_NOT_RECORDED))+ assert benchmark_gate.main(["--require-source-provenance", str(root)]) == 1++ def test_main_tolerates_the_same_card_by_default(self, tmp_path: Path) -> None:+ root = self._write(tmp_path, _pass_card(_NOT_RECORDED))+ assert benchmark_gate.main([str(root)]) == 0++ def test_find_offenders_propagates_the_flag(self, tmp_path: Path) -> None:+ root = self._write(tmp_path, _pass_card(_NOT_RECORDED))+ _files, offenders = benchmark_gate.find_offenders([root], require_source_provenance=True)+ assert offenders+++class TestReporterGateRoundTrip:+ """Whatever the reporter can emit, the gate must accept."""++ @pytest.mark.parametrize(+ "source",+ [+ {"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A, "evaluator_container_revision": _CONTAINER},+ {+ "repository": "holgerroth/nvflare_examples",+ "commit": _COMMIT_A,+ "evaluator_container_revision": _CONTAINER,+ },+ {"repository": "NVIDIA/Megatron_LM", "commit": _COMMIT_A, "evaluator_container_revision": _CONTAINER},+ {"repository": "some-org/repo.name", "content_digest": _DIGEST, "evaluator_container_revision": _CONTAINER},+ ],+ )+ def test_rendered_card_satisfies_the_strict_gate(self, tmp_path: Path, source: dict[str, str]) -> None:+ result = ValidationResult(+ validator_name="AGENT_EVAL",+ validator_description="Run live agent evaluation",+ )+ result.metadata["agent_eval"] = {+ "skill_name": "demo-skill",+ "evaluated_source": source,+ "summary": {"environment": "Isolated sandbox"},+ "agents": {"codex": {"model": "gpt-codex"}},+ }+ path = tmp_path / "BENCHMARK.md"+ path.write_text(BenchmarkReporter().render_all([result]), encoding="utf-8")+ assert [o.reason for o in benchmark_gate.scan_file(path, require_source_provenance=True)] == []+++class TestIdentityResolution:+ """Two orchestration channels, resolved per field and never silently reconciled."""++ def test_fields_merge_across_channels(self) -> None:+ merged = resolve_evaluated_source({"repository": "NVIDIA/NVFlare"}, {"commit": _COMMIT_A})+ assert merged == {"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A}++ def test_an_invalid_field_does_not_discard_the_other_channel(self) -> None:+ merged = resolve_evaluated_source(+ {"repository": "not a repository"},+ {"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A},+ )+ assert merged == {"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A}++ def test_conflicting_channels_fail_closed(self) -> None:+ with pytest.raises(EvaluatedSourceConflict, match="commit"):+ resolve_evaluated_source({"commit": _COMMIT_A}, {"commit": _COMMIT_B})++ def test_agreeing_channels_do_not_conflict(self) -> None:+ assert resolve_evaluated_source({"commit": _COMMIT_A}, {"commit": _COMMIT_A}) == {"commit": _COMMIT_A}++ def test_conflict_propagates_out_of_the_payload_builder(self) -> None:+ with pytest.raises(EvaluatedSourceConflict):+ _payload(+ run_config={"evaluated_source": {"repository": "NVIDIA/stale"}},+ evaluated_source={"repository": "NVIDIA/NVFlare"},+ )+++class TestNonTier3CardsCanRecordTheIdentity:+ """A PASS can be published without a completed Tier 3 run, so those cards need a carrier."""++ def _card_from(self, *results: ValidationResult) -> str:+ return BenchmarkReporter().render_all(list(results))++ def test_result_metadata_carries_the_identity(self) -> None:+ tier1 = ValidationResult(validator_name="Schema", validator_description="d")+ tier1.add_success(check_name="ok", message="fine")+ tier1.metadata["evaluated_source"] = {"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A}+ card = self._card_from(tier1)+ assert _value(card, "Evaluated source") == "`NVIDIA/NVFlare`"+ assert _value(card, "Evaluated source revision") == f"`{_COMMIT_A}`"++ def test_carriers_that_agree_are_merged(self) -> None:+ tier1 = ValidationResult(validator_name="Schema", validator_description="d")+ tier1.metadata["evaluated_source"] = {"repository": "NVIDIA/NVFlare"}+ tier3 = ValidationResult(validator_name="AGENT_EVAL", validator_description="d")+ tier3.metadata["agent_eval"] = {+ "skill_name": "demo-skill",+ "evaluated_source": {"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A},+ "summary": {"environment": "Isolated sandbox"},+ "agents": {"codex": {"model": "gpt-codex"}},+ }+ card = self._card_from(tier3, tier1)+ assert _value(card, "Evaluated source") == "`NVIDIA/NVFlare`"+ assert _value(card, "Evaluated source revision") == f"`{_COMMIT_A}`"++ @pytest.mark.parametrize("reversed_order", [False, True])+ def test_contradictory_carriers_fail_closed(self, reversed_order: bool) -> None:+ """Result ordering must not decide which source tree a published card names."""+ tier1 = ValidationResult(validator_name="Schema", validator_description="d")+ tier1.metadata["evaluated_source"] = {"repository": "NVIDIA/stale", "commit": _COMMIT_B}+ tier3 = ValidationResult(validator_name="AGENT_EVAL", validator_description="d")+ tier3.metadata["agent_eval"] = {+ "skill_name": "demo-skill",+ "evaluated_source": {"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A},+ "summary": {"environment": "Isolated sandbox"},+ "agents": {"codex": {"model": "gpt-codex"}},+ }+ results = [tier1, tier3] if reversed_order else [tier3, tier1]+ with pytest.raises(EvaluatedSourceConflict):+ self._card_from(*results)++ def test_payload_conflicting_with_its_own_summary_fails_closed(self) -> None:+ tier3 = ValidationResult(validator_name="AGENT_EVAL", validator_description="d")+ tier3.metadata["agent_eval"] = {+ "skill_name": "demo-skill",+ "evaluated_source": {"repository": "NVIDIA/NVFlare"},+ "summary": {"environment": "Isolated sandbox", "evaluated_source": {"repository": "NVIDIA/stale"}},+ "agents": {"codex": {"model": "gpt-codex"}},+ }+ with pytest.raises(EvaluatedSourceConflict):+ self._card_from(tier3)++ def test_two_results_disagreeing_fail_closed(self) -> None:+ first = ValidationResult(validator_name="Schema", validator_description="d")+ first.metadata["evaluated_source"] = {"commit": _COMMIT_A}+ second = ValidationResult(validator_name="Lint", validator_description="d")+ second.metadata["evaluated_source"] = {"commit": _COMMIT_B}+ with pytest.raises(EvaluatedSourceConflict):+ self._card_from(first, second)++ def test_hostile_result_metadata_is_dropped(self) -> None:+ tier1 = ValidationResult(validator_name="Schema", validator_description="d")+ tier1.metadata["evaluated_source"] = {"repository": "A/B` INJECTED `x"}+ card = self._card_from(tier1)+ assert _value(card, "Evaluated source") == _UNRECORDED+ assert "INJECTED" not in card+++class TestPassDetection:+ """The verdict line is honoured whether or not the card blockquotes it."""++ _POLICY_BULLET = (+ "- Overall verdict: PASS only when every configured dimension passes for at least one supported agent."+ )+ _UNRECORDED_REASONS = (+ "publication PASS without recorded evaluated source",+ "publication PASS without recorded evaluated source revision",+ "publication PASS without recorded evaluator container revision",+ )++ @staticmethod+ def _unrecorded(reasons: list[str]) -> tuple[str, ...]:+ return tuple(reason for reason in reasons if reason.startswith("publication PASS without recorded"))++ @pytest.mark.parametrize(+ "verdict_line",+ [+ "> **Overall verdict: PASS**",+ "**Overall verdict: PASS**",+ # The callout the reporter renders, written with the em dash escaped.+ "> \u2705 **Overall verdict: PASS \u2014 Recommended for publication**",+ ],+ )+ def test_published_pass_needs_the_identity(self, tmp_path: Path, verdict_line: str) -> None:+ card = _pass_card(_NOT_RECORDED).replace("> **Overall verdict: PASS**", verdict_line, 1)+ reasons = _scan(tmp_path, card, require=True)+ assert "publication PASS without recorded evaluated source" in reasons++ @pytest.mark.parametrize("verdict", ["INCOMPLETE", "FAIL"])+ def test_the_policy_bullet_is_not_a_published_verdict(self, tmp_path: Path, verdict: str) -> None:+ """The policy section restates the rule as a bullet under every verdict.++ A card that publishes no PASS claims no source identity, so reading that+ bullet as the verdict charges it with three offences it never committed.+ """+ card = _pass_card(_ABSENT).replace("> **Overall verdict: PASS**", f"> **Overall verdict: {verdict}**", 1)+ card = f"{card}\n## Scoring Policy\n\n{self._POLICY_BULLET}\n"+ assert self._unrecorded(_scan(tmp_path, card, require=True)) == ()++ def test_the_published_card_carries_that_bullet_and_is_still_detected(self, tmp_path: Path) -> None:+ """Excluding bullets must not blind the check on the card the reporter writes."""+ card = (Path(__file__).parent / "golden" / "benchmark_pass" / "BENCHMARK.md").read_text(encoding="utf-8")+ assert self._POLICY_BULLET in card+ assert _scan(tmp_path, card, require=True) == []++ stripped = "\n".join(+ line for line in card.splitlines() if not line.startswith(benchmark_gate._SOURCE_PROVENANCE_MARKERS)+ )+ assert self._unrecorded(_scan(tmp_path, stripped, require=True)) == self._UNRECORDED_REASONS+++class TestStrictGateRevisionSyntax:+ """The stdlib-only gate mirrors the library rules, so neither side can drift."""++ _STRICT = "publication PASS without recorded evaluator container revision"+ _ADVISORY = "invalid metadata field: - Evaluator container revision:"++ @staticmethod+ def _container_reasons(tmp_path: Path, revision: str) -> list[str]:+ card = _pass_card(+ f"- Evaluated source: `NVIDIA/NVFlare`\n"+ f"- Evaluated source revision: `{_COMMIT_A}`\n"+ f"- Evaluator container revision: `{revision}`\n"+ )+ return _scan(tmp_path, card, require=True)++ def test_the_gate_mirrors_the_library_reference_grammar(self) -> None:+ """Two copies of one rule stay one rule only while their source text is identical."""+ assert benchmark_gate._CONTAINER_REFERENCE_PATTERN == source_identity._CONTAINER_REFERENCE_PATTERN+ assert benchmark_gate._CONTAINER_PATH_MAX == source_identity._CONTAINER_PATH_MAX+ # The gate's hand copy of the revision pattern went uncompared, so it+ # could drift from the library while the reference grammar still matched.+ assert benchmark_gate._GIT_OBJECT_ID.pattern == source_identity._SOURCE_COMMIT.pattern++ def test_a_long_repository_pinned_by_digest_satisfies_a_pass(self, tmp_path: Path) -> None:+ """The reference is 140 characters, which the old whole-reference cap discarded."""+ reference = "ghcr.io/nvidia/" + "e" * 53 + "@sha256:" + "0117bc2e" * 8+ assert len(reference) == 140+ assert self._container_reasons(tmp_path, reference) == []++ @pytest.mark.parametrize(("length", "accepted"), [(255, True), (256, False)])+ def test_the_path_is_bounded_at_the_oci_maximum(self, tmp_path: Path, length: int, accepted: bool) -> None:+ reference = "n" * length + "@sha256:" + "0117bc2e" * 8+ reasons = self._container_reasons(tmp_path, reference)+ assert (self._STRICT not in reasons) is accepted++ @pytest.mark.parametrize(("length", "accepted"), [(255, True), (256, False)])+ def test_a_registry_host_does_not_spend_the_path_budget(+ self, tmp_path: Path, length: int, accepted: bool+ ) -> None:+ """The bound measures the path once the host is split off, as the reference grammar does."""+ reference = "ghcr.io/" + "n" * length + "@sha256:" + "0117bc2e" * 8+ reasons = self._container_reasons(tmp_path, reference)+ assert (self._STRICT not in reasons) is accepted++ @pytest.mark.parametrize(+ "revision",+ [+ "NVIDIA/skillevaluator@sha256:" + "0117bc2e" * 8,+ "ghcr.io/NVIDIA/skillevaluator@sha256:" + "0117bc2e" * 8,+ ],+ )+ def test_an_uppercase_path_satisfies_no_pass_either_way(self, tmp_path: Path, revision: str) -> None:+ """One rule for both spellings: a first label is a host only where the grammar says so."""+ assert self._STRICT in self._container_reasons(tmp_path, revision)++ @pytest.mark.parametrize(+ "revision",+ [+ "localhost/team/image@sha256:" + "0117bc2e" * 8,+ "Registry.Example.COM/team/image@sha256:" + "0117bc2e" * 8,+ "myregistry:5000/team/image@sha256:" + "0117bc2e" * 8,+ ],+ )+ def test_a_registry_host_is_localhost_dotted_or_ported(self, tmp_path: Path, revision: str) -> None:+ """Those are the three shapes the grammar reads as a host, and a host name may use any case."""+ assert self._container_reasons(tmp_path, revision) == []++ def test_a_registry_port_and_tag_still_pin_by_digest(self, tmp_path: Path) -> None:+ reference = "localhost:5000/team/image:1.2.3@sha256:" + "0117bc2e" * 8+ assert self._container_reasons(tmp_path, reference) == []++ def test_an_implementation_revision_satisfies_a_pass(self, tmp_path: Path) -> None:+ """An evaluator built from a checkout has no image digest to record."""+ assert self._container_reasons(tmp_path, _COMMIT_A) == []++ def test_a_tagged_reference_is_recorded_but_not_publishable(self, tmp_path: Path) -> None:+ """Normalization keeps the tag the operator supplied; only publication demands the digest."""+ reasons = self._container_reasons(tmp_path, "nvcr.io/nvidia/skillevaluator:1.3.2")+ assert self._ADVISORY not in reasons+ assert self._STRICT in reasons++ @pytest.mark.parametrize(+ "revision",+ [+ "ghcr.io/nvidia/skillevaluator",+ "ghcr.io/nvidia/skillevaluator@sha512:" + "ab" * 16,+ "ghcr.io/nvidia/skillevaluator@md5:" + "ab" * 16,+ "ghcr.io/nvidia/skillevaluator:" + "t" * 129,+ "has space",+ "ghcr.io/nvidia/skillevaluator`` injected",+ ],+ )+ def test_references_that_name_no_build_do_not_satisfy_a_pass(self, tmp_path: Path, revision: str) -> None:+ assert self._STRICT in self._container_reasons(tmp_path, revision)++ @pytest.mark.parametrize("revision", [_COMMIT_A[:7], "sha512:" + "ab" * 16])+ def test_ambiguous_revision_does_not_satisfy_a_pass(self, tmp_path: Path, revision: str) -> None:+ card = _pass_card(+ f"- Evaluated source: `NVIDIA/NVFlare`\n"+ f"- Evaluated source revision: `{revision}`\n"+ f"- Evaluator container revision: `{_CONTAINER}`\n"+ )+ reasons = _scan(tmp_path, card, require=True)+ assert "publication PASS without recorded evaluated source revision" in reasons++ def test_pass_without_a_container_revision_is_flagged(self, tmp_path: Path) -> None:+ """#72 was a card whose only revision was the evaluator image, so strict mode needs it too."""+ card = _pass_card(+ f"- Evaluated source: `NVIDIA/NVFlare`\n"+ f"- Evaluated source revision: `{_COMMIT_A}`\n"+ f"- Evaluator container revision: {_UNRECORDED}\n"+ )+ reasons = _scan(tmp_path, card, require=True)+ assert "publication PASS without recorded evaluator container revision" in reasons++ def test_mutable_container_tag_does_not_satisfy_a_pass(self, tmp_path: Path) -> None:+ """A tag can be repointed after publication, so it cannot pin the build that ran."""+ card = _pass_card(+ f"- Evaluated source: `NVIDIA/NVFlare`\n"+ f"- Evaluated source revision: `{_COMMIT_A}`\n"+ f"- Evaluator container revision: `ghcr.io/nvidia/skillevaluator:latest`\n"+ )+ reasons = _scan(tmp_path, card, require=True)+ assert "publication PASS without recorded evaluator container revision" in reasons++ def test_digest_pinned_container_satisfies_a_pass(self, tmp_path: Path) -> None:+ card = _pass_card(+ f"- Evaluated source: `NVIDIA/NVFlare`\n"+ f"- Evaluated source revision: `{_COMMIT_A}`\n"+ f"- Evaluator container revision: `{_CONTAINER}`\n"+ )+ reasons = _scan(tmp_path, card, require=True)+ assert "publication PASS without recorded evaluator container revision" not in reasons+++class TestCliSurfacesTheConflict:+ """Failing closed must still read as a CLI error, not an unhandled traceback."""++ def test_validate_reports_a_conflicting_identity_cleanly(+ self,+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+ ) -> None:+ from click.testing import CliRunner++ from skillevaluator.cli import cli+ from skillevaluator.reporting import BenchmarkReporter+ from skillevaluator.validators.code_risk import CodeRiskValidator+ from skillevaluator.validators.secrets import SecretsValidator++ skill = tmp_path / "conflicting-source"+ skill.mkdir()+ (skill / "SKILL.md").write_text(+ "---\n"+ "name: conflicting-source\n"+ "description: A skill whose run records two different evaluated sources.\n"+ "---\n"+ "\n"+ "# Conflicting source\n",+ encoding="utf-8",+ )++ monkeypatch.setattr(CodeRiskValidator, "validate", lambda _self, _path: ValidationResult())+ monkeypatch.setattr(SecretsValidator, "validate", lambda _self, _path: ValidationResult())++ def _conflict(*_args: object, **_kwargs: object) -> None:+ raise EvaluatedSourceConflict("conflicting evaluated source identity (repository: 'A/b' vs 'C/d')")++ monkeypatch.setattr(BenchmarkReporter, "save", _conflict)++ result = CliRunner().invoke(+ cli,+ ["validate", str(skill), "--no-dedup", "-r", "cli", "-o", str(tmp_path / "out")],+ )++ assert result.exit_code != 0+ assert "evaluated source" in result.output+ assert "Traceback" not in result.output+++class TestCliSuppliesTheIdentity:+ """A normal validate run records the identity, which is what #72 asked for."""++ def _skill(self, tmp_path: Path) -> Path:+ skill = tmp_path / "demo-skill"+ skill.mkdir()+ (skill / "SKILL.md").write_text(+ "---\n"+ "name: demo-skill\n"+ "description: A minimal skill used to check the recorded source identity.\n"+ "---\n"+ "\n"+ "# Demo skill\n",+ encoding="utf-8",+ )+ return skill++ def _run(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *args: str):+ from click.testing import CliRunner++ from skillevaluator.cli import cli+ from skillevaluator.validators.code_risk import CodeRiskValidator+ from skillevaluator.validators.secrets import SecretsValidator++ # Exercise the public CLI without the independent scanner integrations.+ monkeypatch.setattr(CodeRiskValidator, "validate", lambda _self, _path: ValidationResult())+ monkeypatch.setattr(SecretsValidator, "validate", lambda _self, _path: ValidationResult())+ output_dir = tmp_path / "out"+ invocation = CliRunner().invoke(+ cli,+ ["validate", str(self._skill(tmp_path)), "--no-dedup", "-r", "cli", "-o", str(output_dir), *args],+ )+ return invocation, output_dir / "BENCHMARK.md"++ def test_supplied_identity_reaches_the_card(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:+ _invocation, card_path = self._run(+ tmp_path,+ monkeypatch,+ "--evaluated-source-repository",+ "NVIDIA/NVFlare",+ "--evaluated-source-revision",+ _COMMIT_A,+ "--evaluator-container-revision",+ _CONTAINER,+ )+ card = card_path.read_text(encoding="utf-8")+ assert _value(card, "Evaluated source") == "`NVIDIA/NVFlare`"+ assert _value(card, "Evaluated source revision") == f"`{_COMMIT_A}`"+ assert _value(card, "Evaluator container revision") == f"`{_CONTAINER}`"++ def test_a_content_digest_is_accepted_as_the_revision(+ self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ """A run with no upstream commit still records an immutable revision."""+ _invocation, card_path = self._run(tmp_path, monkeypatch, "--evaluated-source-revision", _DIGEST)+ assert _value(card_path.read_text(encoding="utf-8"), "Evaluated source revision") == f"`{_DIGEST}`"++ def test_without_the_options_the_card_says_not_recorded(+ self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ """Default behaviour is unchanged for a run with no orchestration input."""+ _invocation, card_path = self._run(tmp_path, monkeypatch)+ card = card_path.read_text(encoding="utf-8")+ assert _value(card, "Evaluated source") == _UNRECORDED+ assert _value(card, "Evaluated source revision") == _UNRECORDED++ @pytest.mark.parametrize(+ ("option", "value"),+ [+ ("--evaluated-source-revision", _COMMIT_A[:7]),+ ("--evaluated-source-revision", "sha512:" + "ab" * 16),+ ("--evaluated-source-repository", "not a repository"),+ ("--evaluator-container-revision", "has space"),+ ],+ )+ def test_non_canonical_values_are_refused_at_the_boundary(+ self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, option: str, value: str+ ) -> None:+ """Rejecting here beats rendering `not recorded` for a field the operator supplied."""+ invocation, card_path = self._run(tmp_path, monkeypatch, option, value)+ assert invocation.exit_code != 0+ assert "expected" in invocation.output+ assert not card_path.exists()+++class TestIdentityReachesTheRunArtifact:+ """The CLI identity has to survive every hop down to `run_config.json`."""++ def test_evaluation_options_forward_the_identity(self) -> None:+ from skillevaluator.evaluation import EvaluationOptions++ identity = {"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A}+ options = EvaluationOptions(skill_path=Path(), evaluated_source=identity)+ assert options.engine_kwargs()["evaluated_source"] == identity++ @pytest.mark.parametrize(+ ("module_path", "function_name"),+ [+ ("skillevaluator.cli", "_run_agent_eval_or_skip"),+ ("skillevaluator.tier3.commands", "evaluate"),+ ("skillevaluator.tier3.harbor.runner", "_run_harbor_eval_impl"),+ ],+ )+ def test_each_hop_accepts_the_identity(self, module_path: str, function_name: str) -> None:+ import importlib+ import inspect++ module = importlib.import_module(module_path)+ signature = inspect.signature(getattr(module, function_name))+ assert "evaluated_source" in signature.parameters++ def test_a_standalone_run_persists_the_identity_in_run_config(+ self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ """A signature alone does not prove the value is written to disk.++ The standalone ``tier3 evaluate`` producer owns its own run directory, so+ the file a card is later rendered from has to carry the identity rather+ than the ``null`` a dropped option leaves behind.+ """+ import json++ from skillevaluator.provider_config import ProviderConfig+ from skillevaluator.tier3 import commands as tier3_commands+ from skillevaluator.tier3.harbor import runner, runtime_preflight++ skill = tmp_path / "demo"+ (skill / "evals").mkdir(parents=True)+ (skill / "evals" / "evals.json").write_text("[]\n", encoding="utf-8")+ provider = ProviderConfig(+ provider="nv_build",+ model="meta/llama-3.1-8b-instruct",+ api_key="nvapi-test",+ base_url="https://integrate.api.nvidia.com/v1",+ litellm_model="openai/meta/llama-3.1-8b-instruct",+ )++ def emit(_skill, target, **_kwargs):+ task = target / "case-001"+ task.mkdir(parents=True)+ return [task]++ monkeypatch.setattr(tier3_commands, "resolve_llm_provider", lambda: provider)+ monkeypatch.setattr(runner, "resolve_llm_provider", lambda: provider)+ monkeypatch.setattr(+ runner,+ "load_evals_config",+ lambda _path: ({"harbor": {"task_source": "evals_json"}}, None),+ )+ monkeypatch.setattr(runner, "find_evals_file", lambda _path: skill / "evals" / "evals.json")+ monkeypatch.setattr(runner, "_check_prerequisites", lambda **_kwargs: [])+ monkeypatch.setattr(runner, "generate_harbor_tasks", emit)+ monkeypatch.setattr(+ runtime_preflight,+ "probe_model",+ lambda selected_provider: runtime_preflight.ModelProbeResult(+ True,+ selected_provider.provider,+ selected_provider.model,+ f"model {selected_provider.model} is available",+ ),+ )+ # The preflight is failed deliberately: the run stops before any agent+ # work, and `run_config.json` is still written, which is the artifact+ # under test.+ monkeypatch.setattr(+ runtime_preflight,+ "run_agent_runtime_preflight",+ lambda **_kwargs: runtime_preflight.PreflightResult(+ False,+ "opencode",+ provider.model,+ "401 Unauthorized",+ "runtime-preflight-opencode",+ ),+ )++ identity = {"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A}+ result = tier3_commands.evaluate(+ skill,+ agents="opencode",+ env_mode="docker",+ skip_baseline=True,+ n_attempts=None,+ pass_threshold=None,+ n_concurrent=None,+ max_agents=None,+ model=None,+ agent_model=(),+ custom_dockerfile_mode=None,+ skill_workspace_mode=None,+ include_skills=(),+ copy_repo=False,+ grading_mode=None,+ results_dir=tmp_path / "results",+ harbor_keep_jobs=False,+ agent_runtime_preflight=True,+ timeout_multiplier=None,+ override_cpus=None,+ override_memory_mb=None,+ override_storage_mb=None,+ evaluated_source=identity,+ )++ run_config_path = Path(result["run_dir"]) / "run_config.json"+ persisted = json.loads(run_config_path.read_text(encoding="utf-8"))+ assert persisted["evaluated_source"] == identity+++def _agent_eval_carrier(repository: str) -> dict[str, object]:+ """A complete agent-eval payload naming one repository in both of its carriers."""+ return {+ "skill_name": "demo-skill",+ "evaluated_source": {"repository": repository},+ "summary": {"environment": "Isolated sandbox", "evaluated_source": {"repository": repository}},+ "agents": {"codex": {"model": "gpt-codex"}},+ }+++def _agent_eval_result(payload: dict[str, object]) -> ValidationResult:+ result = ValidationResult(validator_name="AGENT_EVAL", validator_description="Run live agent evaluation")+ result.metadata["agent_eval"] = payload+ return result+++class TestEveryNestedAgentEvalCarrierIsFolded:+ """Selecting one payload let result ordering decide which source tree a card named."""++ @pytest.mark.parametrize("reversed_order", [False, True])+ def test_two_payloads_naming_different_repositories_fail_closed(self, reversed_order: bool) -> None:+ first = _agent_eval_result(_agent_eval_carrier("NVIDIA/NVFlare"))+ second = _agent_eval_result(_agent_eval_carrier("NVIDIA/stale"))+ results = [second, first] if reversed_order else [first, second]+ with pytest.raises(EvaluatedSourceConflict, match="repository"):+ BenchmarkReporter().render_all(results)++ @pytest.mark.parametrize("reversed_order", [False, True])+ def test_two_payloads_that_agree_still_render_the_identity(self, reversed_order: bool) -> None:+ """Folding must not turn a repeated identity into a conflict."""+ first = _agent_eval_result(_agent_eval_carrier("NVIDIA/NVFlare"))+ second = _agent_eval_result(_agent_eval_carrier("NVIDIA/NVFlare"))+ results = [second, first] if reversed_order else [first, second]+ assert _value(BenchmarkReporter().render_all(results), "Evaluated source") == "`NVIDIA/NVFlare`"++ @pytest.mark.parametrize("reversed_order", [False, True])+ def test_a_summary_only_carrier_on_a_second_result_still_conflicts(self, reversed_order: bool) -> None:+ """The deepest carrier of the second payload counts as much as the first payload's."""+ payload = _agent_eval_result(_agent_eval_carrier("NVIDIA/NVFlare"))+ summary_only = _agent_eval_result(+ {+ "skill_name": "demo-skill",+ "summary": {+ "environment": "Isolated sandbox",+ "evaluated_source": {"repository": "NVIDIA/stale"},+ },+ "agents": {"codex": {"model": "gpt-codex"}},+ }+ )+ results = [summary_only, payload] if reversed_order else [payload, summary_only]+ with pytest.raises(EvaluatedSourceConflict, match="repository"):+ BenchmarkReporter().render_all(results)+++class TestJsonReportRecordsTheIdentity:+ """CI parses the JSON report, so the provenance contract has to hold there too."""++ def _report(self, *results: ValidationResult) -> dict[str, object]:+ import json++ from skillevaluator.reporting.json_reporter import JSONReporter++ return json.loads(JSONReporter(include_timestamp=False).render_all(list(results)))++ def test_result_metadata_reaches_the_top_level(self) -> None:+ tier1 = ValidationResult(validator_name="Schema", validator_description="d")+ tier1.metadata["evaluated_source"] = {"repository": "NVIDIA/NVFlare", "commit": _COMMIT_A}+ assert self._report(tier1)["evaluated_source"] == {+ "repository": "NVIDIA/NVFlare",+ "commit": _COMMIT_A,+ }++ def test_a_nested_payload_carrier_reaches_the_top_level(self) -> None:+ """A Tier 3 run records the identity on the payload, not on the result."""+ report = self._report(_agent_eval_result(_agent_eval_carrier("NVIDIA/NVFlare")))+ assert report["evaluated_source"] == {"repository": "NVIDIA/NVFlare"}++ def test_a_run_with_no_identity_records_null(self) -> None:+ """The key is always present, so null means none was supplied, not an older reporter."""+ report = self._report(ValidationResult(validator_name="Schema", validator_description="d"))+ assert "evaluated_source" in report+ assert report["evaluated_source"] is None++ def test_conflicting_carriers_fail_closed(self) -> None:+ first = ValidationResult(validator_name="Schema", validator_description="d")+ first.metadata["evaluated_source"] = {"commit": _COMMIT_A}+ second = ValidationResult(validator_name="Lint", validator_description="d")+ second.metadata["evaluated_source"] = {"commit": _COMMIT_B}+ with pytest.raises(EvaluatedSourceConflict, match="commit"):+ self._report(first, second)+++def _minimal_skill(tmp_path: Path, name: str, description: str) -> Path:+ skill = tmp_path / name+ skill.mkdir()+ (skill / "SKILL.md").write_text(+ f"---\nname: {name}\ndescription: {description}\n---\n\n# {name}\n",+ encoding="utf-8",+ )+ return skill+++class TestTier1OnlyReportsCarryTheIdentity:+ """A Tier 1-only run has no Tier 3 payload, and its JSON report still has to say what ran."""++ def _run(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *args: str) -> Path:+ from click.testing import CliRunner++ from skillevaluator.cli import cli+ from skillevaluator.validators.code_risk import CodeRiskValidator+ from skillevaluator.validators.secrets import SecretsValidator++ # Exercise the public CLI without the independent scanner integrations.+ monkeypatch.setattr(CodeRiskValidator, "validate", lambda _self, _path: ValidationResult())+ monkeypatch.setattr(SecretsValidator, "validate", lambda _self, _path: ValidationResult())+ skill = _minimal_skill(tmp_path, "demo-skill", "A minimal skill used to check the recorded identity.")+ output_dir = tmp_path / "out"+ invocation = CliRunner().invoke(+ cli,+ ["validate", str(skill), "--no-dedup", "-r", "json", "-o", str(output_dir), *args],+ )+ assert "evaluated source" not in invocation.output+ return output_dir++ def test_the_json_report_and_the_card_record_the_same_identity(+ self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ import json++ output_dir = self._run(+ tmp_path,+ monkeypatch,+ "--evaluated-source-repository",+ "NVIDIA/NVFlare",+ "--evaluated-source-revision",+ _COMMIT_A,+ "--evaluator-container-revision",+ _CONTAINER,+ )++ report = json.loads(next(output_dir.glob("*.json")).read_text(encoding="utf-8"))+ assert report["evaluated_source"] == {+ "repository": "NVIDIA/NVFlare",+ "commit": _COMMIT_A,+ "evaluator_container_revision": _CONTAINER,+ }++ card = (output_dir / "BENCHMARK.md").read_text(encoding="utf-8")+ assert _value(card, "Evaluated source") == "`NVIDIA/NVFlare`"+ assert _value(card, "Evaluated source revision") == f"`{_COMMIT_A}`"+ assert _value(card, "Evaluator container revision") == f"`{_CONTAINER}`"+++class TestConflictAbortsBeforeAnyReportIsWritten:+ """Failing closed after emit left JSON and HTML on disk carrying one of two identities."""++ def test_no_report_file_survives_a_contradictory_identity(+ self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ from click.testing import CliRunner++ from skillevaluator import cli as cli_module+ from skillevaluator.cli import cli++ skill = _minimal_skill(tmp_path, "conflicting-source", "A skill whose run records two sources.")+ tier1 = ValidationResult(validator_name="SCHEMA")+ tier1.add_success("schema", "ok")+ tier1.metadata["evaluated_source"] = {"repository": "C/d"}+ monkeypatch.setattr(cli_module, "run_validation", lambda *_args, **_kwargs: [tier1])++ output_dir = tmp_path / "out"+ invocation = CliRunner().invoke(+ cli,+ [+ "validate",+ str(skill),+ "--no-dedup",+ "-r",+ "json",+ "-r",+ "html",+ "-o",+ str(output_dir),+ "--evaluated-source-repository",+ "A/b",+ ],+ )++ assert invocation.exit_code != 0+ assert "evaluated source" in invocation.output+ assert "Traceback" not in invocation.output+ written = sorted(path.name for path in output_dir.rglob("*")) if output_dir.exists() else []+ assert [name for name in written if name.endswith((".json", ".html")) or name == "BENCHMARK.md"] == []+++class TestPartialRecordedIdentityMergesWithTheInput:+ """A producer that recorded one field must not shadow the fields the operator supplied."""++ def test_a_partial_carrier_and_the_options_publish_one_union(+ self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch+ ) -> None:+ """Attaching the supplied identity only where none was recorded dropped the rest of it."""+ import json++ from click.testing import CliRunner++ from skillevaluator import cli as cli_module+ from skillevaluator.cli import cli++ skill = _minimal_skill(tmp_path, "partial-source", "A skill whose run records only the repository.")+ tier1 = ValidationResult(validator_name="SCHEMA")+ tier1.add_success("schema", "ok")+ # The documented result-metadata seam, carrying one field and no more.+ tier1.metadata["evaluated_source"] = {"repository": "NVIDIA/NVFlare"}+ monkeypatch.setattr(cli_module, "run_validation", lambda *_args, **_kwargs: [tier1])++ output_dir = tmp_path / "out"+ invocation = CliRunner().invoke(+ cli,+ [+ "validate",+ str(skill),+ "--no-dedup",+ "-r",+ "json",+ "-o",+ str(output_dir),+ "--evaluated-source-revision",+ _COMMIT_A,+ "--evaluator-container-revision",+ _CONTAINER,+ ],+ )++ assert "evaluated source" not in invocation.output+ report = json.loads(next(output_dir.glob("*.json")).read_text(encoding="utf-8"))+ assert report["evaluated_source"] == {+ "repository": "NVIDIA/NVFlare",+ "commit": _COMMIT_A,+ "evaluator_container_revision": _CONTAINER,+ }++ card = (output_dir / "BENCHMARK.md").read_text(encoding="utf-8")+ assert _value(card, "Evaluated source") == "`NVIDIA/NVFlare`"+ assert _value(card, "Evaluated source revision") == f"`{_COMMIT_A}`"+ assert _value(card, "Evaluator container revision") == f"`{_CONTAINER}`"diff --git a/tests/test_cli.py b/tests/test_cli.pyindex 1e1cf343..b6a9ed7d 100644--- a/tests/test_cli.py+++ b/tests/test_cli.py@@ -593,6 +593,84 @@ def test_live_eval_progress_is_presentation_only(monkeypatch: pytest.MonkeyPatchassert "progress" not in captured["options"].engine_kwargs()+_EVALUATED_REPOSITORY = "NVIDIA/NVFlare"+_EVALUATED_COMMIT = "2263a2ebdab903e87f7e7c0a001d22c3a926a9cf"+_EVALUATOR_CONTAINER = "ghcr.io/nvidia/skillevaluator@sha256:" + "0117bc2e" * 8+++def test_standalone_evaluate_forwards_the_evaluated_source(monkeypatch: pytest.MonkeyPatch) -> None:+ """The standalone producer records the identity instead of persisting null.++ ``tier3 evaluate`` writes its own ``run_config.json``, so an identity the+ operator supplied has to reach the engine here just as it does through+ ``validate``.+ """+ from skillevaluator.evaluation import EvaluationService++ captured: dict[str, object] = {}++ def _fake_evaluate(self, options, *, progress_reporter=None):+ captured["options"] = options+ return {"execution_status": "succeeded", "execution_errors": []}++ monkeypatch.setattr(EvaluationService, "evaluate", _fake_evaluate, raising=True)+ result = CliRunner().invoke(+ cli,+ [+ "evaluate",+ str(Path(__file__).parent / "fixtures" / "skills" / "simple"),+ "--progress",+ "plain",+ "--evaluated-source-repository",+ _EVALUATED_REPOSITORY,+ "--evaluated-source-revision",+ _EVALUATED_COMMIT,+ "--evaluator-container-revision",+ _EVALUATOR_CONTAINER,+ ],+ )++ assert result.exit_code == 0, result.output+ assert captured["options"].engine_kwargs()["evaluated_source"] == {+ "repository": _EVALUATED_REPOSITORY,+ "commit": _EVALUATED_COMMIT,+ "evaluator_container_revision": _EVALUATOR_CONTAINER,+ }+++def test_standalone_evaluate_refuses_a_non_canonical_revision(monkeypatch: pytest.MonkeyPatch) -> None:+ """Refusing at the boundary beats a card that says the field was never recorded."""+ from skillevaluator.evaluation import EvaluationService++ def _unreachable(self, options, *, progress_reporter=None):+ raise AssertionError("the engine must not run on a non-canonical identity")++ monkeypatch.setattr(EvaluationService, "evaluate", _unreachable, raising=True)+ result = CliRunner().invoke(+ cli,+ [+ "evaluate",+ str(Path(__file__).parent / "fixtures" / "skills" / "simple"),+ "--progress",+ "plain",+ "--evaluated-source-revision",+ _EVALUATED_COMMIT[:7],+ ],+ )++ assert result.exit_code != 0+ assert "expected" in result.output+++def test_tier3_evaluate_help_lists_the_provenance_options() -> None:+ result = CliRunner().invoke(cli, ["tier3", "evaluate", "--help"])++ assert result.exit_code == 0+ assert "--evaluated-source-repository" in result.output+ assert "--evaluated-source-revision" in result.output+ assert "--evaluator-container-revision" in result.output++def test_harbor_view_command_uses_the_skillevaluator_wrapper() -> None:command = format_harbor_view_command("/tmp/jobs")diff --git a/tests/test_commands.py b/tests/test_commands.pyindex 88d62766..54142000 100644--- a/tests/test_commands.py+++ b/tests/test_commands.py@@ -402,6 +402,116 @@ def test_validate_catalog_workers_runs_skills_in_parallel() -> None:assert any(Path("out/simple2").glob("*.html"))+_EVALUATED_REPOSITORY = "NVIDIA/NVFlare"+_EVALUATED_COMMIT = "2263a2ebdab903e87f7e7c0a001d22c3a926a9cf"+_EVALUATOR_CONTAINER = "ghcr.io/nvidia/skillevaluator@sha256:" + "0117bc2e" * 8+++def test_catalog_child_argv_carries_the_evaluated_source(tmp_path: Path) -> None:+ """A parallel child renders its own card, so it needs the parent's identity."""+ from skillevaluator.cli import _catalog_child_argv_from_ctx, validate++ skill_dir = tmp_path / "catalog" / "simple"+ skill_dir.mkdir(parents=True)+ output_dir = tmp_path / "out" / "simple"+ ctx = validate.make_context(+ "validate",+ [+ str(skill_dir),+ "--workers",+ "2",+ "--evaluated-source-repository",+ _EVALUATED_REPOSITORY,+ "--evaluated-source-revision",+ _EVALUATED_COMMIT,+ "--evaluator-container-revision",+ _EVALUATOR_CONTAINER,+ ],+ )++ argv = _catalog_child_argv_from_ctx(ctx, skill_dir, output_dir)++ for option, value in (+ ("--evaluated-source-repository", _EVALUATED_REPOSITORY),+ ("--evaluated-source-revision", _EVALUATED_COMMIT),+ ("--evaluator-container-revision", _EVALUATOR_CONTAINER),+ ):+ assert option in argv+ assert argv[argv.index(option) + 1] == value+ # The child validates one skill, so re-sending --workers would fan out again.+ assert "--workers" not in argv+++def test_catalog_child_argv_forwards_every_validate_option() -> None:+ """The allowlist is explicit, so a new option is silently dropped by default.++ Each parameter is either forwarded by name or listed below with the reason+ it must not reach a child.+ """+ import inspect++ from skillevaluator.cli import _catalog_child_argv_from_ctx, validate++ not_forwarded = {+ "target_path": "the child is given one skill directory, not the catalog root",+ "content_type": "the child re-detects the type of the single skill it is handed",+ "workers": "forwarding it would make each child fan out again",+ "output_dir": "each child is given its own per-skill output directory",+ }+ source = inspect.getsource(_catalog_child_argv_from_ctx)++ dropped = [+ param.name+ for param in validate.params+ if f'params.get("{param.name}"' not in source and f'params["{param.name}"]' not in source+ ]++ assert sorted(dropped) == sorted(not_forwarded)+++def test_validate_catalog_workers_records_the_evaluated_source_on_every_card() -> None:+ runner = CliRunner()+ with runner.isolated_filesystem():+ catalog = Path("catalog")+ for name in ("simple", "simple2"):+ shutil.copytree(FIXTURE, catalog / name)+ second = catalog / "simple2" / "SKILL.md"+ second.write_text(+ second.read_text(encoding="utf-8").replace("name: simple", "name: simple2"),+ encoding="utf-8",+ )+ result = runner.invoke(+ cli,+ [+ "validate",+ str(catalog.resolve()),+ "--workers",+ "2",+ "--no-llm",+ "--no-dedup",+ "--checks",+ "quality",+ "-r",+ "html",+ "--evaluated-source-repository",+ _EVALUATED_REPOSITORY,+ "--evaluated-source-revision",+ _EVALUATED_COMMIT,+ "--evaluator-container-revision",+ _EVALUATOR_CONTAINER,+ "-o",+ "out",+ ],+ )++ assert result.exit_code == 0, result.output+ for name in ("simple", "simple2"):+ card = Path("out") / name / "BENCHMARK.md"+ text = card.read_text(encoding="utf-8")+ assert f"- Evaluated source: `{_EVALUATED_REPOSITORY}`" in text+ assert f"- Evaluated source revision: `{_EVALUATED_COMMIT}`" in text++def test_validate_catalog_workers_accepts_options_before_target_path() -> None:runner = CliRunner()with runner.isolated_filesystem():