Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

Reuse standard scored PR comments without hosted reports (#528)

Christopher Kevinc19b5d70a55a
diff --git a/.github/workflows/comment-skill-checks.yml b/.github/workflows/comment-skill-checks.yml
new file mode 100644
index 00000000..2402184d
--- /dev/null
+++ b/.github/workflows/comment-skill-checks.yml
@@ -0,0 +1,98 @@
+name: Comment on skill checks
+on:
+ workflow_call:
+ inputs:
+ tools-ref:
+ description: Reviewed immutable evaluator commit containing the report tools
+ type: string
+ required: true
+ tools-repository:
+ description: Repository supplying the reviewed reusable workflows and report tools
+ type: string
+ default: NVIDIA/SkillEvaluator
+ run-id:
+ type: string
+ required: true
+ run-attempt:
+ description: Completed scan attempt that triggered this publication
+ type: string
+ required: true
+ trusted-evaluator-shas:
+ description: Approved immutable evaluator revisions; required unless delegated-evaluator-wrapper is set
+ type: string
+ default: ''
+ trusted-caller-blobs:
+ description: Approved Git blobs for the consumer scan workflow, separated by commas or newlines
+ type: string
+ required: true
+ trusted-wrapper-workflows:
+ description: Approved exact wrapper workflow paths at full SHAs or tags, separated by commas or newlines
+ type: string
+ default: ''
+ delegated-evaluator-wrapper:
+ description: Optional approved root wrapper allowed to select pinned evaluator revisions; requires a sole caller job
+ type: string
+ default: ''
+permissions: {}
+jobs:
+ comment:
+ if: >-
+ (github.event_name == 'workflow_dispatch' ||
+ (github.event_name == 'workflow_run' && github.event.workflow_run.status == 'completed' &&
+ github.event.workflow_run.event == 'pull_request' &&
+ github.event.workflow_run.head_repository.full_name == github.repository)) &&
+ github.ref == format('refs/heads/{0}', github.event.repository.default_branch)
+ runs-on: ubuntu-24.04
+ timeout-minutes: 15
+ concurrency:
+ group: skill-report-comment-${{ github.repository }}
+ cancel-in-progress: false
+ queue: max
+ permissions:
+ contents: read
+ actions: read
+ pull-requests: write
+ steps:
+ - name: Validate trusted tool identity
+ env:
+ TOOLS_REF: ${{ inputs.tools-ref }}
+ TOOLS_REPOSITORY: ${{ inputs.tools-repository }}
+ run: |
+ python3 - <<'PY'
+ import os, re
+ if not re.fullmatch(r"[0-9a-f]{40}", os.environ["TOOLS_REF"]):
+ raise SystemExit("tools-ref must be a reviewed full commit SHA")
+ if not re.fullmatch(r"[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+", os.environ["TOOLS_REPOSITORY"]):
+ raise SystemExit("tools-repository must be an owner/repository")
+ PY
+ - name: Checkout trusted report tools
+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0
+ with:
+ repository: ${{ inputs.tools-repository }}
+ ref: ${{ inputs.tools-ref }}
+ path: publisher
+ persist-credentials: false
+ # Cone mode also keeps every root file, including the package build inputs.
+ sparse-checkout: |
+ ci
+ src
+ - name: Install the trusted native report renderer
+ working-directory: publisher
+ # No dependency cache is restored: other default-branch workflows can write
+ # this job's cache scope, and uv does not recheck files from its cache.
+ run: |
+ python3 -m venv "$RUNNER_TEMP/uv"
+ "$RUNNER_TEMP/uv/bin/python" -m pip install --disable-pip-version-check \
+ --require-hashes --no-deps --only-binary :all: -r ci/requirements/uv.txt
+ "$RUNNER_TEMP/uv/bin/uv" sync --frozen --no-dev --no-editable --extra publisher
+ - name: Verify evidence and update the native PR comment
+ env:
+ GH_TOKEN: ${{ github.token }}
+ REPORT_RUN_ID: ${{ inputs.run-id }}
+ REPORT_RUN_ATTEMPT: ${{ inputs.run-attempt }}
+ TRUSTED_EVALUATOR_SHA: ${{ inputs.trusted-evaluator-shas }}
+ TRUSTED_CALLER_BLOBS: ${{ inputs.trusted-caller-blobs }}
+ TRUSTED_WRAPPER_WORKFLOWS: ${{ inputs.trusted-wrapper-workflows }}
+ DELEGATED_EVALUATOR_WRAPPER: ${{ inputs.delegated-evaluator-wrapper }}
+ EVALUATOR_REPOSITORY: ${{ inputs.tools-repository }}
+ run: publisher/.venv/bin/python publisher/ci/prepare_comments.py
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 92c6687d..ce9658ba 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,12 @@ All notable changes to SkillEvaluator are documented in this file.
### Added
+- Canonical `comment-skill-checks.yml` reusable workflow and source template for
+ the standard scored Tier 1 PR comment without hosted reports. It verifies the
+ existing signed receipts and native evidence, reuses the shared report and
+ comment helpers, and needs no Pages configuration or history branch. Future
+ generated bundles include the template; archived bundles and the released
+ onboarding CLI retain their existing hosted publisher setup.
- `harness` rules for frontmatter parsing: duplicate keys and tab indentation
(strict YAML parsers drop the skill), a UTF-8 BOM (Codex), no front matter
(Copilot CLI) and long custom frontmatter that never reaches the model. Tier 1
diff --git a/ci/README.md b/ci/README.md
index 8702d070..c61f44be 100644
--- a/ci/README.md
+++ b/ci/README.md
@@ -34,7 +34,7 @@ policy label. Profile names do not control network access or report visibility.
| `paths` | `skills`; up to 100 newline-separated repository-relative skill directories or roots |
| `scan-mode` | `auto`: affected skills on PRs, all configured skills on pushes/manual runs; `all` overrides selection |
| `llm` | `false`; static checks require no model key |
-| `artifact-name` | Keep `skill-evaluator-tier1` for the optional publisher |
+| `artifact-name` | Keep `skill-evaluator-tier1` for PR comments or the optional hosted publisher |
| `report-base-url` | Optional own Pages URL, only after hosted publication is configured |
| `scan-workers` | `4`; skills scanned at once, from 1 to 16. `1` scans one skill at a time |
| `enforcement-mode` | `strict` (default) blocks on failed verdicts; `advisory` keeps complete policy/score failures non-blocking but still blocks runtime/evidence/incomplete failures |
@@ -75,8 +75,8 @@ The short `summary.md` is the Actions landing view. `details.md`, original
per-skill JSON, native HTML, SARIF and scanner logs retain detailed evidence in
the scan artifact, named `skill-evaluator-tier1-attempt-N` for attempt `N`.
A second, small artifact, `skill-evaluator-tier1-evidence-attempt-N`, holds
-only `summary.json` and each skill's `skillevaluator-tier1.json`. The optional
-publisher downloads only that one. Artifacts expire after the configured retention period;
+only `summary.json` and each skill's `skillevaluator-tier1.json`. The comment and
+hosted publishing workflows download only that one. Artifacts expire after the configured retention period;
hosted history is an optional separate surface. Quality scores are not scanner
pass percentages or proof of useful live-agent behavior.
@@ -148,11 +148,57 @@ digest in `skills[].security_baseline`. The skill's native report records whethe
the baseline applied and what it suppressed. Policy files receive the same
credential preflight and report redaction as skill content.
+## PR comments without hosted reports
+
+Use the canonical
+[`comment-skill-checks.yml`](../.github/workflows/comment-skill-checks.yml) reusable
+workflow to update the existing scored Tier 1 PR comment without deploying a
+report site. It uses the same signed execution receipt, caller and evaluator
+approvals, artifact validation, report renderer and bot comment updater as the
+hosted publisher. The comment shows counts, status, quality score and grade, and
+the first action for each affected skill. The workflow run links to downloadable
+reports. No Pages settings, site history branch or report URL are needed.
+
+This is a source capability for a future release bundle. Existing archived
+bundles and the released `skillevaluator ci init` still configure hosted
+publication. Use a reviewed immutable revision containing the new workflow and
+helpers; do not add its filename to an older release pin.
+
+1. Keep `.github/workflows/skill-checks.yml` named **Skill checks**, with one
+ reusable scan job and the default **skill-evaluator-tier1** artifact name.
+ Enable `id-token: write` alongside `contents: read` for signed execution
+ receipts. Keep the reviewed evaluator workflow and image pins.
+2. Copy [`comment-skill-checks.yml`](templates/comment-skill-checks.yml) into
+ `.github/workflows/`. Pin its `uses:` and `tools-ref` to the reviewed comment
+ workflow revision. Set `trusted-evaluator-shas` to the approved scanner
+ revision; this may differ from the comment tools revision.
+3. After all scan-file edits, run
+ `git hash-object .github/workflows/skill-checks.yml` and set
+ `trusted-caller-blobs` to that exact value. Review and refresh the blob after
+ every later scan-file edit. For shared wrappers, keep the
+ [same wrapper approvals and delegation](#calling-through-a-shared-workflow).
+4. Merge both files into the default branch and open a PR changing a skill.
+ The comment workflow grants only `contents: read`, `actions: read` and
+ `pull-requests: write`. Organization policy must allow these permissions and
+ the referenced actions. Choose this workflow or the hosted publisher for PR
+ feedback; both update the same bot comment.
+
+The second workflow runs after **Skill checks** completes, including failed
+scans. Its manual trigger requires the completed `run-id` and `run-attempt`.
+It comments only on an open, current-head PR from the same repository. Failed
+scans with valid retained native evidence receive the standard failure comment;
+failures before evidence upload do not produce a synthetic result. No affected
+skills uses the existing **No skills changed** result. Fork PRs, superseded
+results, stale PR heads, expired artifacts and unverifiable evidence do not
+receive a comment. The signed token's expiry alone does not invalidate a
+historical receipt; its signing key and the attempt's artifacts must still be
+available.
+
## Optional hosted reports and PR feedback
This is additional setup, not a `comment: true` switch in the basic scan file.
-The current implementation couples comments to successful hosted deployment;
-Pages is not an inherent requirement for GitHub PR comments.
+This publisher couples comments to successful hosted deployment. To use the
+same comment without hosting, follow the [comments-only setup](#pr-comments-without-hosted-reports).
1. Decide where reports can live. **Pages sites are public unless explicitly
configured as private, even when the repository is private.** Private Pages
@@ -202,7 +248,7 @@ Pages is not an inherent requirement for GitHub PR comments.
7. The publisher caller must grant `contents: write`, `actions: read`,
`pages: write`, `id-token: write` and `pull-requests: write`. Reusable jobs
narrow these to history preparation, Pages deployment and PR feedback. The
- scan uses `contents: read`, plus `id-token: write` only when hosting is enabled.
+ scan uses `contents: read`, plus `id-token: write` when PR comments or hosting are enabled.
The automatic GitHub token is sufficient;
no personal access token or model credential is required for publication.
@@ -228,11 +274,16 @@ failed deploy still publishes and comments.
A publication skipped because its PR closed or moved after the scan neither
deploys nor comments.
Fork PR scans can run without provider secrets; fork PR bot comments and current
-PR aliases are not enabled. Comment-only operation and hosting alongside an
-arbitrary existing Pages deployment need further integration work.
+PR aliases are not enabled. Hosting alongside an arbitrary existing Pages
+deployment needs further integration work.
### Calling through a shared workflow
+The comments-only workflow accepts the same `trusted-wrapper-workflows` and
+`delegated-evaluator-wrapper` inputs below, with the same signed receipts and
+caller blob approvals. Its wrapper continues to forward scan settings and pins;
+the standard comment renderer stays in SkillEvaluator.
+
Direct consumers need no wrapper configuration. If the scan calls a shared
workflow that then calls SkillEvaluator, use a reviewed publisher revision that
exposes `trusted-wrapper-workflows`. The archived 0.7.1 publisher pin predates
diff --git a/ci/comment_report.py b/ci/comment_report.py
index 891ad900..be000b24 100644
--- a/ci/comment_report.py
+++ b/ci/comment_report.py
@@ -2,7 +2,7 @@
# SPDX-License-Identifier: Apache-2.0
# ruff: noqa: INP001, T201 -- standalone trusted workflow entrypoint with fixed diagnostics.
-"""Upsert verified current-head PR feedback after a trusted Pages deployment.
+"""Upsert verified current-head PR feedback from trusted scan evidence.
Only the configured bot's marked comments are eligible for update. This helper
does not check out PR code, execute artifacts, or print the comment body.
@@ -65,6 +65,30 @@ def run_marker(run_id: int, attempt: int) -> str:
return f"<!-- skillevaluator-tier1-run:{run_id}:{attempt} -->"
+def _verify_run(api, repository: str, run_id: int, attempt: int, sha: str | None = None) -> str | None:
+ """Reject superseded attempts, including reruns started while rendering feedback."""
+ run = api.get(f"/repos/{repository}/actions/runs/{run_id}")
+ if (
+ not isinstance(run, dict)
+ or type(run.get("id")) is not int
+ or run["id"] != run_id
+ or not isinstance(run.get("repository"), dict)
+ or run["repository"].get("full_name") != repository
+ or not isinstance(run.get("head_sha"), str)
+ or SHA_PATTERN.fullmatch(run["head_sha"]) is None
+ or (sha is not None and run["head_sha"] != sha)
+ or type(run.get("run_attempt")) is not int
+ or run["run_attempt"] < attempt
+ or run.get("status") not in {"completed", "queued", "in_progress", "waiting", "pending", "requested"}
+ ):
+ raise FeedbackError("The source run lacks trustworthy repository, revision, or attempt metadata.")
+ if run["run_attempt"] != attempt:
+ return "A newer scan attempt exists."
+ if run["status"] != "completed":
+ return "The scan attempt is not completed."
+ return None
+
+
def _verify_pr(api, repository: str, number: int, sha: str) -> str | None:
pr = api.get(f"/repos/{repository}/pulls/{number}")
if not isinstance(pr, dict) or pr.get("state") not in {"open", "closed"}:
@@ -172,7 +196,7 @@ def comment_report(api, repository: str, run_id: int, feedback_dir: Path, actor_
)
if len(body.encode("utf-8")) > MAX_BODY_BYTES:
raise FeedbackError("The complete feedback comment exceeds the 60000-byte limit.")
- reason = _verify_pr(api, repository, number, sha)
+ reason = _verify_run(api, repository, run_id, attempt, sha) or _verify_pr(api, repository, number, sha)
if reason:
return {"comment_action": "skipped", "comment_reason": reason}
owned = _owned_comments(api, repository, number, actor_bot_login)
@@ -180,7 +204,7 @@ def comment_report(api, repository: str, run_id: int, feedback_dir: Path, actor_
return {"comment_action": "skipped", "comment_reason": "A newer report comment already exists."}
canonical = max(owned, key=lambda item: item["id"]) if owned else None
# Recheck immediately before mutation after pagination and receipt validation.
- reason = _verify_pr(api, repository, number, sha)
+ reason = _verify_run(api, repository, run_id, attempt, sha) or _verify_pr(api, repository, number, sha)
if reason:
return {"comment_action": "skipped", "comment_reason": reason}
if canonical is not None and canonical["body"] == body:
diff --git a/ci/prepare_comments.py b/ci/prepare_comments.py
new file mode 100644
index 00000000..9e3f8825
--- /dev/null
+++ b/ci/prepare_comments.py
@@ -0,0 +1,104 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+# ruff: noqa: INP001, T201 -- standalone trusted workflow entrypoint with fixed diagnostics.
+
+"""Render and post native PR feedback from verified scan JSON, without hosting."""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+import tempfile
+import urllib.error
+from pathlib import Path
+
+try:
+ import comment_report
+ import publish_reports
+ import report_views
+except ModuleNotFoundError:
+ from ci import comment_report, publish_reports, report_views
+
+
+def prepare_comments(api, repository: str, run_id: int, run_attempt: int, output: Path) -> dict[str, str]:
+ """Reuse the publisher's evidence checks and the native Markdown/comment path."""
+ if (
+ not publish_reports.REPO_PATTERN.fullmatch(repository)
+ or not publish_reports._integer(run_id, 1)
+ or not publish_reports._integer(run_attempt, 1)
+ ):
+ raise publish_reports.PublicationError("Repository, run ID, and attempt must be explicit valid identities.")
+ reason = comment_report._verify_run(api, repository, run_id, run_attempt)
+ if reason:
+ return {"comment_action": "skipped", "comment_reason": reason}
+ evidence = publish_reports.load_report_evidence(api, repository, run_id, run_attempt)
+ run, summary = evidence["run"], evidence["summary"]
+ if run["event"] != "pull_request":
+ return {"comment_action": "skipped", "comment_reason": "The scan is not a pull request run."}
+ routes, number = publish_reports._current_routes(api, repository, run)
+ if not number or routes != [f"pr/{number}"]:
+ return {"comment_action": "skipped", "comment_reason": "No unique current same-repository pull request."}
+ if output.is_symlink():
+ raise publish_reports.PublicationError("The feedback directory must not be a symlink.")
+ output.mkdir(parents=True, exist_ok=True)
+ publish_reports._dump(output / "summary.json", summary)
+ for directory, report in evidence["reports"].items():
+ publish_reports._dump(output / directory / "skillevaluator-tier1.json", report)
+ run_url = f"https://github.com/{repository}/actions/runs/{run_id}"
+ report_views.render_reports(
+ output,
+ summary,
+ {
+ "run_url": run_url,
+ "commit": run["head_sha"],
+ "image": evidence["image"],
+ "llm_enabled": evidence["llm_enabled"],
+ "ci_conclusion": run["conclusion"],
+ },
+ )
+ publish_reports._dump(
+ output / "publication.json",
+ {
+ "run_url": run_url,
+ "pr_number": number,
+ "source_head_sha": run["head_sha"],
+ "is_current": "true",
+ "run_attempt": str(run_attempt),
+ },
+ )
+ return comment_report.comment_report(api, repository, run_id, output, "github-actions[bot]")
+
+
+def main() -> int:
+ try:
+ token = os.environ.get("GH_TOKEN", "").strip()
+ repository = os.environ.get("GITHUB_REPOSITORY", "")
+ run_id = comment_report._positive(os.environ.get("REPORT_RUN_ID"))
+ attempt = comment_report._positive(os.environ.get("REPORT_RUN_ATTEMPT"))
+ if not token:
+ raise publish_reports.PublicationError("GH_TOKEN is required to publish authorized PR feedback.")
+ with tempfile.TemporaryDirectory(prefix="skillevaluator-feedback-") as directory:
+ output = Path(directory)
+ result = prepare_comments(publish_reports.GitHubAPI(repository, token), repository, run_id, attempt, output)
+ if os.environ.get("GITHUB_STEP_SUMMARY") and (output / "summary.md").is_file():
+ with Path(os.environ["GITHUB_STEP_SUMMARY"]).open("a", encoding="utf-8") as handle:
+ handle.write((output / "summary.md").read_text(encoding="utf-8"))
+ if result["comment_action"] == "skipped":
+ print("SkillEvaluator PR feedback skipped: " + result["comment_reason"])
+ else:
+ print("SkillEvaluator PR feedback verified successfully.")
+ return 0
+ except publish_reports.StalePublication as exc:
+ print("SkillEvaluator PR feedback skipped: " + str(exc))
+ return 0
+ except (publish_reports.PublicationError, comment_report.FeedbackError) as exc:
+ print("SkillEvaluator PR feedback rejected: " + json.dumps(str(exc)[:1000]), file=sys.stderr)
+ return 1
+ except (OSError, KeyError, ValueError, TypeError, RecursionError, urllib.error.URLError):
+ print("SkillEvaluator PR feedback failed; inspect the trusted workflow configuration.", file=sys.stderr)
+ return 1
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/ci/publish_reports.py b/ci/publish_reports.py
index a647b8a6..12137878 100644
--- a/ci/publish_reports.py
+++ b/ci/publish_reports.py
@@ -937,6 +937,37 @@ def _validate_pages_privacy(api, repository: str, base_url: str) -> None:
)
+def load_report_evidence(api, repository: str, run_id: int, run_attempt: int, *, image="") -> dict:
+ """Load the signed, bounded JSON evidence for one completed scan attempt."""
+ if not REPO_PATTERN.fullmatch(repository) or not _integer(run_id, 1) or not _integer(run_attempt, 1):
+ raise PublicationError("Repository and report run ID must be explicit valid identities.")
+ run = api.get(f"/repos/{repository}/actions/runs/{run_id}/attempts/{run_attempt}")
+ _validate_run(run, repository, run_id)
+ if run["run_attempt"] != run_attempt:
+ raise PublicationError("GitHub returned a different workflow attempt from the requested one.")
+ provenance = _validate_workflow_provenance(api, repository, run)
+ artifact = _select_artifact(api, repository, run)
+ summary, reports = read_artifact(api.download_artifact(artifact["id"]))
+ successful = summary["overall_passed"] or summary["overall_status"] == "skipped" or advisory_downgraded(summary)
+ if (run["conclusion"] == "success") != successful:
+ raise PublicationError("The workflow conclusion contradicts the report artifact verdict.")
+ if summary["overall_status"] == "skipped" and summary["selection"]["head"] != run["head_sha"]:
+ raise PublicationError("The no-change selection does not match the workflow revision.")
+ selected_image = _recorded_image(summary, image)
+ recorded_llm = summary.get("execution", {}).get("llm_enabled")
+ if recorded_llm is not None and type(recorded_llm) is not bool:
+ raise PublicationError("The recorded LLM setting must be a boolean when present.")
+ return {
+ "run": run,
+ "provenance": provenance,
+ "artifact": artifact,
+ "summary": summary,
+ "reports": reports,
+ "image": selected_image,
+ "llm_enabled": recorded_llm,
+ }
+
+
def publish(
api,
repository: str,
@@ -967,22 +998,10 @@ def publish(
raise PublicationError("The site and output directories must not be symlinks.")
site.mkdir(parents=True, exist_ok=True)
output.mkdir(parents=True, exist_ok=True)
- run = api.get(f"/repos/{repository}/actions/runs/{run_id}/attempts/{run_attempt}")
- _validate_run(run, repository, run_id)
- if run["run_attempt"] != run_attempt:
- raise PublicationError("GitHub returned a different workflow attempt from the requested one.")
- provenance = _validate_workflow_provenance(api, repository, run)
- artifact = _select_artifact(api, repository, run)
- summary, reports = read_artifact(api.download_artifact(artifact["id"]))
- successful = summary["overall_passed"] or summary["overall_status"] == "skipped" or advisory_downgraded(summary)
- if (run["conclusion"] == "success") != successful:
- raise PublicationError("The workflow conclusion contradicts the report artifact verdict.")
- if summary["overall_status"] == "skipped" and summary["selection"]["head"] != run["head_sha"]:
- raise PublicationError("The no-change selection does not match the workflow revision.")
- selected_image = _recorded_image(summary, image)
- recorded_llm = summary.get("execution", {}).get("llm_enabled")
- if recorded_llm is not None and type(recorded_llm) is not bool:
- raise PublicationError("The recorded LLM setting must be a boolean when present.")
+ evidence = load_report_evidence(api, repository, run_id, run_attempt, image=image)
+ run, provenance, artifact = evidence["run"], evidence["provenance"], evidence["artifact"]
+ summary, reports = evidence["summary"], evidence["reports"]
+ selected_image, recorded_llm = evidence["image"], evidence["llm_enabled"]
attempt = run["run_attempt"]
relative = f"runs/{run_id}/{attempt}"
report_url = f"{base_url.rstrip('/')}/{relative}/"
diff --git a/ci/render_templates.py b/ci/render_templates.py
index efd722d9..1145946c 100644
--- a/ci/render_templates.py
+++ b/ci/render_templates.py
@@ -43,6 +43,7 @@ def render_templates(
rendered[source.name] = (
source.read_text(encoding="utf-8")
.replace("publish-skill-reports.yml@RELEASE_WORKFLOW_SHA", f"publish-skill-reports.yml@{publisher_ref}")
+ .replace("comment-skill-checks.yml@RELEASE_WORKFLOW_SHA", f"comment-skill-checks.yml@{publisher_ref}")
.replace("tools-ref: RELEASE_WORKFLOW_SHA", f"tools-ref: {publisher_ref}")
.replace("RELEASE_WORKFLOW_SHA", workflow_ref)
.replace(IMAGE_PLACEHOLDER, image)
@@ -71,11 +72,27 @@ def render_templates(
"## GitHub: choose your templates\n\n"
"- Checks and downloadable reports: copy `skill-checks.yml` to `.github/workflows/skill-checks.yml`. "
"Set `paths` to your skill roots and choose `min-score` (default 70).\n"
+ "- Standard PR comments: also copy `comment-skill-checks.yml` to `.github/workflows/` "
+ "and complete the comment setup below. No hosted reports are required.\n"
"- Hosted reports and PR comments: also copy `publish-skill-reports.yml` to `.github/workflows/` "
"and complete the one-time setup below.\n\n"
- "The workflow and image pins are already filled in. Both templates use GitHub's automatic token; "
+ "Choose one feedback workflow. The workflow and image pins are already filled in. "
+ "The templates use GitHub's automatic token; "
"no personal token or model key is needed for the default checks. "
"Your organization must allow the referenced actions and the provided permissions.\n\n"
+ "### PR comment setup\n\n"
+ "1. Uncomment `id-token: write` under `permissions` in `skill-checks.yml` for signed execution receipts. "
+ "Keep its single reusable scan job and customize your skill paths.\n"
+ "2. Run `git hash-object .github/workflows/skill-checks.yml` after all scan-file edits and put the "
+ "result in `trusted-caller-blobs` in `comment-skill-checks.yml`. Refresh it after every scan-file edit.\n"
+ "3. Merge both files into the default branch, preserving the supplied scan name, filename and artifact name. "
+ "Open a skill PR to trigger a new scan. The comment workflow needs only `contents: read`, "
+ "`actions: read` and `pull-requests: write`; no Pages or history branch is needed.\n\n"
+ "The standard scored Tier 1 comment is updated for eligible current-head, same-repository PRs, "
+ "including failed scans with valid evidence. Full reports remain downloadable from the scan run. "
+ "Manual retries require both the completed run ID and attempt number.\n\n"
+ f"[Full comment setup](https://github.com/{repository}/blob/{publisher_ref}/ci/README.md"
+ "#pr-comments-without-hosted-reports).\n\n"
"### One-time publisher setup\n\n"
"1. Enable Settings → Pages → Source: GitHub Actions. Reports can include source excerpts; "
"a private repository does not automatically make its Pages site private. "
@@ -108,7 +125,7 @@ def main() -> None:
parser.add_argument("--workflow-ref", required=True)
parser.add_argument(
"--publisher-ref",
- help="Reviewed publisher workflow and tools SHA; defaults to --workflow-ref. Does not change the scanner pin.",
+ help="Reviewed comment/publisher workflows and tools SHA; defaults to --workflow-ref. Keeps the scanner pin.",
)
parser.add_argument("--image", required=True)
parser.add_argument("--repository", default="NVIDIA/SkillEvaluator")
diff --git a/ci/templates/README.md b/ci/templates/README.md
index 6d41a483..cc826abb 100644
--- a/ci/templates/README.md
+++ b/ci/templates/README.md
@@ -8,6 +8,7 @@ together in this public NVIDIA/SkillEvaluator repository.
| Platform and result | Source template | Where to put it |
| --- | --- | --- |
| GitHub: checks and downloadable reports | [skill-checks.yml](skill-checks.yml) | `.github/workflows/skill-checks.yml` |
+| GitHub: also update the standard PR comment, without hosting | [comment-skill-checks.yml](comment-skill-checks.yml) | `.github/workflows/comment-skill-checks.yml`, alongside the checks file |
| GitHub: also host reports and update a PR comment | [publish-skill-reports.yml](publish-skill-reports.yml) | `.github/workflows/publish-skill-reports.yml`, alongside the checks file |
| GitHub: optional trusted manual LLM checks | [skill-checks-llm.yml](skill-checks-llm.yml) | `.github/workflows/skill-checks-llm.yml` |
| GitLab: checks and downloadable reports | [gitlab-skill-checks.yml](gitlab-skill-checks.yml) | `.gitlab-ci.yml`, or merge into your existing root configuration |
@@ -21,6 +22,10 @@ a separate demo repository.
**These source templates contain placeholders.** Copy the
[ready-to-use configurations](../bundles/0.9.0/README.md) instead; their reviewed
workflow commits and container image digest are already filled in.
+The comments-only workflow is a source capability for a future bundle; the
+archived 0.9.0 bundle and released onboarding CLI still offer the hosted publisher.
+For comments only, use a reviewed revision containing the new reusable workflow
+and follow the [comment setup](../README.md#pr-comments-without-hosted-reports).
Open a file in that folder and use GitHub's **Raw** view to copy its complete
contents. The generated files and their checksums are kept in this public
@@ -31,11 +36,13 @@ If you customize a source template instead, use the immutable workflow/image
identities in the generated bundle's `release.json`. Do not leave placeholders
or substitute `main` or `latest` for runtime pins.
-Maintainers can release a reviewed publisher update while keeping the verified
+Maintainers can release a reviewed feedback workflow update while keeping the verified
scanner workflow and image unchanged. Pass its full commit SHA as
`--publisher-ref` to `ci/render_templates.py`; otherwise it uses `--workflow-ref`.
-The release receipt records both revisions. This changes the publisher workflow
-and tools only; the scanner pin and approved evaluator identity stay unchanged.
+That revision must contain both `comment-skill-checks.yml` and
+`publish-skill-reports.yml`. The release receipt records both revisions. This
+changes the feedback workflows and tools only; the scanner pin and approved
+evaluator identity stay unchanged.
## Customize and commit
@@ -48,8 +55,9 @@ and tools only; the scanner pin and approved evaluator identity stay unchanged.
rules; preserve your other jobs and `workflow: rules`. The job uses `test` by
default, so override its stage if your pipeline does not define it. Follow the
[GitLab guide](../../docs/gitlab-ci.mdx).
-- **Optional GitHub comments:** the second file needs the
- [one-time hosting and trust setup](../../docs/github-actions.mdx#enable-pr-comments).
+- **Optional GitHub comments:** choose the
+ [comments-only setup](../README.md#pr-comments-without-hosted-reports) or the
+ [hosted publisher setup](../README.md#optional-hosted-reports-and-pr-feedback).
Uncomment `id-token: write` in the scan file for signed execution receipts,
then approve that file's new blob. Copying the second file alone does not
enable comments.
diff --git a/ci/templates/comment-skill-checks.yml b/ci/templates/comment-skill-checks.yml
new file mode 100644
index 00000000..2bdfd1c7
--- /dev/null
+++ b/ci/templates/comment-skill-checks.yml
@@ -0,0 +1,39 @@
+# SPDX-License-Identifier: Apache-2.0
+# Optional standard PR feedback without hosted reports or Pages configuration.
+# Use reviewed release pins that include the comment-skill-checks workflow.
+# Grant id-token: write in skill-checks.yml, then approve its exact Git blob below.
+name: Comment skill checks
+on:
+ workflow_run:
+ workflows: [Skill checks]
+ types: [completed]
+ workflow_dispatch:
+ inputs:
+ run-id:
+ description: Completed Skill checks run to comment on
+ type: string
+ required: true
+ run-attempt:
+ description: Completed attempt number for that run
+ type: string
+ required: true
+permissions:
+ contents: read
+ actions: read
+ pull-requests: write
+jobs:
+ comment:
+ uses: NVIDIA/SkillEvaluator/.github/workflows/comment-skill-checks.yml@RELEASE_WORKFLOW_SHA
+ with:
+ run-id: ${{ format('{0}', github.event.workflow_run.id || inputs.run-id) }}
+ run-attempt: ${{ format('{0}', github.event.workflow_run.run_attempt || inputs.run-attempt) }}
+ trusted-evaluator-shas: RELEASE_WORKFLOW_SHA
+ trusted-caller-blobs: YOUR_APPROVED_CALLER_BLOB
+ # Shared wrappers use the same approvals as the hosted publisher.
+ # trusted-wrapper-workflows: your-org/shared-ci/.github/workflows/skill-checks.yml@FULL_COMMIT_SHA
+ # To delegate evaluator pin selection to one reviewed shared wrapper:
+ # trusted-wrapper-workflows: your-org/shared-ci/.github/workflows/skill-checks.yml@v3
+ # delegated-evaluator-wrapper: your-org/shared-ci/.github/workflows/skill-checks.yml@v3
+ # Omit trusted-evaluator-shas when delegating; caller blob approval stays required.
+ tools-ref: RELEASE_WORKFLOW_SHA
+ tools-repository: NVIDIA/SkillEvaluator
diff --git a/ci/templates/skill-checks.yml b/ci/templates/skill-checks.yml
index 0f10170a..0380fec9 100644
--- a/ci/templates/skill-checks.yml
+++ b/ci/templates/skill-checks.yml
@@ -10,7 +10,7 @@ on:
workflow_dispatch:
permissions:
contents: read
- # Hosted reports/PR comments: uncomment this, then refresh trusted-caller-blobs.
+ # PR comments or hosted reports: uncomment this, then refresh trusted-caller-blobs.
# id-token: write
jobs:
skills:
diff --git a/docs/github-actions.mdx b/docs/github-actions.mdx
index 4796fe58..8785c690 100644
--- a/docs/github-actions.mdx
+++ b/docs/github-actions.mdx
@@ -63,8 +63,9 @@ reusable workflow and actions.
<Note>
The released template provides **checks and downloadable reports**.
- Hosted reports and PR comments need the [optional publisher setup](#enable-pr-comments).
- The same release also includes the optional publisher template.
+ The same release includes the [optional hosted publisher](#enable-hosted-reports).
+ [Comments without hosting](#enable-pr-comments) are available in the source
+ workflow for a future release bundle.
</Note>
<Accordion title="I prefer to inspect the source templates">
@@ -80,7 +81,7 @@ reusable workflow and actions.
| --- | --- |
| **PR → Checks** | The `skills / Tier 1 scan` result and its short summary. |
| **Actions → Skill checks** | The run and downloadable report artifact. |
-| **PR → Conversation**, with publishing enabled | The updated bot comment and **Open full report** link. |
+| **PR → Conversation**, with feedback enabled | The updated scored bot comment and workflow run; hosted publication also adds **Open full report**. |
Download `skill-evaluator-tier1-attempt-N`, extract it, and open `index.html`.
Failed scans retain their available reports too. The HTML separates the
@@ -134,10 +135,10 @@ required security or schema check.
### On the pull request
-With publishing enabled, one bot comment is updated for new results. Reviewers
+With PR feedback enabled, one bot comment is updated for new results. Reviewers
see the issue and next step without reading the full Actions log.
-<Frame caption="Shortened illustrative PR comment. Real comments link to your report and workflow run.">
+<Frame caption="Shortened illustrative PR comment. Both feedback workflows use this scored format; the hosted publisher also links to the deployed report.">
<div style={{ padding: "16px", width: "100%", minWidth: 0 }}>
**github-actions[bot]**
@@ -164,6 +165,41 @@ for more detail. No personal demo-repository access is needed.
## Enable PR comments
+Use the canonical [comment-skill-checks.yml workflow](https://github.com/NVIDIA/SkillEvaluator/blob/main/.github/workflows/comment-skill-checks.yml)
+for the same Tier 1 comment without hosted reports. It shows counts, status,
+quality score and grade, and the first action for each affected skill. Full
+reports remain downloadable from the scan run. Pages settings, a history branch
+and a report URL are not required.
+
+<Note>
+ This is a source capability for a future release bundle. The archived 0.9.0
+ bundle and released `skillevaluator ci init` still configure the hosted
+ publisher. Use a reviewed immutable revision containing the new comment
+ workflow and helpers; an older release pin does not include it.
+</Note>
+
+1. Keep the **Skill checks** scan as one reusable job in
+ `.github/workflows/skill-checks.yml`, with its reviewed evaluator and image
+ pins. Enable `id-token: write` for signed execution receipts.
+2. Add the [comment workflow template](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/templates/comment-skill-checks.yml)
+ as `.github/workflows/comment-skill-checks.yml`. Pin its workflow and tools to
+ the reviewed comment revision, approve the scanner revision, and set
+ `trusted-caller-blobs` to the output of
+ `git hash-object .github/workflows/skill-checks.yml` after all scan-file edits.
+3. Merge both files into the default branch, then open a skill PR. The comment
+ workflow needs `contents: read`, `actions: read` and `pull-requests: write`.
+ Choose this workflow or the hosted publisher for feedback.
+
+Comments run when the scan completes, including failures with valid retained
+evidence. No changed skills keeps the existing **No skills changed** result.
+Only open, current-head PRs from the same repository receive a comment; fork PRs,
+stale results, expired artifacts and failures without verifiable evidence do not.
+Manual retries require both the completed run ID and attempt number. Follow the
+[full comment setup](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/README.md#pr-comments-without-hosted-reports)
+for the exact inputs and shared-wrapper approvals.
+
+## Enable hosted reports
+
The optional publisher hosts the HTML and posts a comment **after deployment
succeeds**. It needs a second workflow plus one-time hosting and approval settings.
@@ -256,13 +292,19 @@ or let the [CLI](/skills/skillevaluator/cli-onboarding) handle it.
</Accordion>
<Accordion title="The scan finished, but the comment is missing">
Confirm setup is on the default branch and the PR is eligible for comments.
- Open **Publish skill reports** to see whether deployment completed. If
- hosting was skipped, check the scan's `id-token: write` permission and
+ Open **Comment skill checks**, or **Publish skill reports** if using hosting,
+ and inspect the evidence validation result. The hosted publisher also needs
+ successful deployment. Check the scan's `id-token: write` permission and
`trusted-caller-blobs` value. A new scan must use the updated workflow;
rerunning an old job keeps its original workflow. The scan's downloadable
report remains a separate way to inspect available findings.
</Accordion>
<Accordion title="Our organization uses a shared workflow wrapper">
+ The comments-only workflow accepts the same caller, wrapper and evaluator
+ approvals as the hosted publisher. Keep the standard rendering in
+ SkillEvaluator and use the [manual comment setup](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/README.md#pr-comments-without-hosted-reports).
+ The released CLI setup below configures hosted publication.
+
Keep the component reference in your existing caller, for example
`your-org/ci/.github/workflows/skill-checks.yml@v3`, then run
`skillevaluator ci init` in each consumer repository. The CLI recognizes
@@ -285,7 +327,7 @@ or let the [CLI](/skills/skillevaluator/cli-onboarding) handle it.
archived 0.7.1 publisher does not. Direct callers keep an immutable
evaluator approval. For manual configuration, follow the
[shared-workflow setup](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/README.md#calling-through-a-shared-workflow)
- guide. PR comments follow successful report deployment; private reports
+ guide. Hosted PR comments follow successful report deployment; private reports
require the private Pages setup described above.
</Accordion>
<Accordion title="Do the checks need an LLM key or network access?">
diff --git a/tests/ci/test_comment_report.py b/tests/ci/test_comment_report.py
index 4817058a..fbf00aab 100644
--- a/tests/ci/test_comment_report.py
+++ b/tests/ci/test_comment_report.py
@@ -48,9 +48,18 @@ class FakeAPI:
self.sha, self.state, self.head_repo = sha, state, head_repo
self.calls = []
self.readback = None
+ self.run = {
+ "id": 123,
+ "repository": {"full_name": REPOSITORY},
+ "head_sha": SHA,
+ "run_attempt": 1,
+ "status": "completed",
+ }
def get(self, path):
self.calls.append(("GET", path))
+ if path.endswith("/actions/runs/123"):
+ return self.run
if path.endswith("/pulls/1"):
return {
"state": self.state,
@@ -220,3 +229,44 @@ def test_oversized_comment_and_symlinked_feedback_are_rejected(tmp_path):
(tmp_path / "pr-comment.md").symlink_to(tmp_path / "publication.json")
with pytest.raises(feedback.FeedbackError):
invoke(FakeAPI(), tmp_path)
+
+
[email protected]("changes", [{"run_attempt": 2}, {"status": "in_progress"}, {"status": "queued"}])
[email protected]("during_listing", [False, True])
+def test_superseded_or_unfinished_run_never_posts(tmp_path, changes, during_listing):
+ prepare(tmp_path)
+
+ class ChangingRunAPI(FakeAPI):
+ def get(self, path):
+ response = super().get(path)
+ if "/comments?" in path:
+ self.run.update(changes)
+ return response
+
+ api = ChangingRunAPI()
+ if not during_listing:
+ api.run.update(changes)
+ assert invoke(api, tmp_path)["comment_action"] == "skipped"
+ assert not any(call[0] in {"POST", "PATCH"} for call in api.calls)
+
+
+ "changes",
+ [
+ {"id": 124},
+ {"repository": {"full_name": "other/repository"}},
+ {"head_sha": "b" * 40},
+ {"head_sha": "bad"},
+ {"run_attempt": True},
+ {"run_attempt": 0},
+ {"status": "unknown"},
+ {"repository": None},
+ ],
+)
+def test_malformed_or_mismatched_source_run_fails_closed(tmp_path, changes):
+ prepare(tmp_path)
+ api = FakeAPI()
+ api.run.update(changes)
+ with pytest.raises(feedback.FeedbackError, match="source run"):
+ invoke(api, tmp_path)
+ assert not any(call[0] in {"POST", "PATCH"} for call in api.calls)
diff --git a/tests/ci/test_prepare_comments.py b/tests/ci/test_prepare_comments.py
new file mode 100644
index 00000000..f20e27af
--- /dev/null
+++ b/tests/ci/test_prepare_comments.py
@@ -0,0 +1,238 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Comments-only publication must share native output and verified evidence."""
+
+import copy
+import re
+from pathlib import Path
+
+import pytest
+import test_comment_report as feedback_fixtures
+import test_publish_reports as evidence_fixtures
+import yaml
+from test_publish_reports import trusted_workflow_pins # noqa: F401 -- shared signed-receipt fixture
+
+from ci import prepare_comments as comments
+
+pytestmark = pytest.mark.usefixtures("trusted_workflow_pins")
+REPOSITORY = "owner/consumer"
+
+
+class FakeAPI(evidence_fixtures.FakeAPI):
+ def __init__(self, *, payload=None, record=None, **kwargs):
+ record = record or {
+ **evidence_fixtures.run_record(),
+ "event": "pull_request",
+ "pull_requests": [{"number": 1}],
+ }
+ super().__init__(record=record, **kwargs)
+ self.latest = copy.deepcopy(self.record)
+ self.payload = payload or evidence_fixtures.valid_archive()
+ self.feedback = feedback_fixtures.FakeAPI()
+ self.downloads = []
+
+ def get(self, path):
+ if path.endswith("/actions/runs/123"):
+ self.paths.append(path)
+ return self.latest
+ if "/issues/" in path:
+ self.paths.append(path)
+ return self.feedback.get(path)
+ return super().get(path)
+
+ def post(self, path, payload):
+ return self.feedback.post(path, payload)
+
+ def patch(self, path, payload):
+ return self.feedback.patch(path, payload)
+
+ def download_artifact(self, artifact_id):
+ self.downloads.append(artifact_id)
+ if artifact_id == 456:
+ return self.payload
+ return super().download_artifact(artifact_id)
+
+
+def invoke(api, output):
+ return comments.prepare_comments(api, REPOSITORY, 123, 1, output)
+
+
+def assert_no_writes(api):
+ assert not any(call[0] in {"POST", "PATCH"} for call in api.feedback.calls)
+
+
[email protected]("status", ["passed", "failed", "skipped", "advisory"])
+def test_comment_is_byte_identical_to_native_renderer_with_scores_fixes_and_noop(tmp_path, status):
+ from ci import native_reports
+
+ summary = evidence_fixtures.summary()
+ report = evidence_fixtures.report()
+ if status in {"failed", "advisory"}:
+ summary = evidence_fixtures.failed_summary(advisory_downgraded=status == "advisory")
+ report = evidence_fixtures.failed_report()
+ if status == "skipped":
+ summary = evidence_fixtures.skipped_summary()
+ report["quality_summary"] = [{"overall_score": 85.8, "grade": "B", "dimensions": {}}]
+ files = [("summary.json", summary)]
+ if status != "skipped":
+ files.append(("skill-001/skillevaluator-tier1.json", report))
+ # The adapter never trusts artifact Markdown or executes artifact code.
+ files.extend([("pr-comment.md", b"forged score @everyone"), ("payload.py", b"raise RuntimeError('executed')")])
+ api = FakeAPI(payload=evidence_fixtures.archive(files))
+ if status == "failed":
+ api.record["conclusion"] = api.latest["conclusion"] = "failure"
+ output = tmp_path / "feedback"
+ assert invoke(api, output)["comment_action"] == "created"
+ expected = tmp_path / "native"
+ comments.publish_reports._dump(expected / "summary.json", summary)
+ if status != "skipped":
+ comments.publish_reports._dump(expected / "skill-001/skillevaluator-tier1.json", report)
+ native_reports.render_native_reports(
+ expected,
+ summary,
+ {
+ "run_url": f"https://github.com/{REPOSITORY}/actions/runs/123",
+ "commit": "a" * 40,
+ "image": "",
+ "llm_enabled": None,
+ "ci_conclusion": api.record["conclusion"],
+ },
+ )
+ body = (output / "pr-comment.md").read_bytes()
+ assert body == (expected / "pr-comment.md").read_bytes()
+ assert body.decode() in api.feedback.comments[0]["body"]
+ assert b"forged score" not in body
+ assert not list(output.rglob("*.html"))
+ assert not (output / "payload.py").exists()
+ assert not any("/pages" in path for path in api.paths)
+ if status == "skipped":
+ assert b"NO SKILLS CHANGED" in body and b"did not run" in body
+ assert b"85.8" not in body
+ else:
+ assert b"85.8 / B" in body
+ assert (b"What to fix" in body) is (status in {"failed", "advisory"})
+ # A second invocation follows the same existing upsert protocol.
+ assert invoke(api, output)["comment_action"] == "unchanged"
+
+
[email protected]("changes", [{"run_attempt": 2}, {"status": "in_progress"}])
+def test_stale_attempt_skips_before_artifact_download(tmp_path, changes):
+ api = FakeAPI()
+ api.latest.update(changes)
+ assert invoke(api, tmp_path)["comment_action"] == "skipped"
+ assert api.downloads == []
+ assert_no_writes(api)
+
+
[email protected]("failure", ["closed", "old_head", "fork", "ambiguous", "push"])
+def test_only_unique_current_same_repository_pr_gets_feedback(tmp_path, failure):
+ api = FakeAPI()
+ if failure == "old_head":
+ api.branch_sha = "e" * 40
+ elif failure == "fork":
+ api.record["head_repository"] = {"full_name": "attacker/fork"}
+ elif failure == "ambiguous":
+ api.record["pull_requests"] = [{"number": 1}, {"number": 2}]
+ elif failure == "push":
+ api.record["event"] = "push"
+ elif failure == "closed":
+ original_get = api.get
+
+ def get(path):
+ result = original_get(path)
+ if "/pulls/" in path:
+ result["state"] = "closed"
+ return result
+
+ api.get = get
+ assert invoke(api, tmp_path)["comment_action"] == "skipped"
+ assert_no_writes(api)
+
+
+ "failure", ["missing_json", "malformed", "bad_caller", "bad_evaluator", "bad_receipt", "verdict", "noop_head"]
+)
+def test_untrusted_or_inconsistent_evidence_never_posts(tmp_path, failure, monkeypatch):
+ api = FakeAPI()
+ if failure == "missing_json":
+ api.payload = evidence_fixtures.archive([("pr-comment.md", b"PASS")])
+ elif failure == "malformed":
+ api.payload = b"not a ZIP"
+ elif failure == "bad_caller":
+ monkeypatch.setenv("TRUSTED_CALLER_BLOBS", "e" * 40)
+ elif failure == "bad_evaluator":
+ monkeypatch.setenv("TRUSTED_EVALUATOR_SHA", "e" * 40)
+ elif failure == "bad_receipt":
+ original = api.download_artifact
+ api.download_artifact = lambda identifier: b"forged" if identifier == 457 else original(identifier)
+ elif failure == "verdict":
+ api.record["conclusion"] = "failure"
+ elif failure == "noop_head":
+ summary = evidence_fixtures.skipped_summary()
+ summary["selection"]["head"] = "e" * 40
+ api.payload = evidence_fixtures.archive([("summary.json", summary)])
+ with pytest.raises(comments.publish_reports.PublicationError):
+ invoke(api, tmp_path)
+ assert_no_writes(api)
+
+
+def test_rerun_started_during_rendering_suppresses_comment(tmp_path, monkeypatch):
+ api = FakeAPI()
+ render = comments.report_views.render_reports
+
+ def changing_render(*args, **kwargs):
+ render(*args, **kwargs)
+ api.latest.update(run_attempt=2, status="in_progress")
+
+ monkeypatch.setattr(comments.report_views, "render_reports", changing_render)
+ assert invoke(api, tmp_path)["comment_action"] == "skipped"
+ assert_no_writes(api)
+
+
+def test_comments_workflow_privileges_and_trusted_code_boundary():
+ root = Path(__file__).parents[2]
+ workflow = yaml.load((root / ".github/workflows/comment-skill-checks.yml").read_text(), Loader=yaml.BaseLoader)
+ assert workflow["permissions"] == {}
+ assert set(workflow["on"]) == {"workflow_call"}
+ inputs = workflow["on"]["workflow_call"]["inputs"]
+ for name in ("run-id", "run-attempt", "tools-ref", "trusted-caller-blobs"):
+ assert inputs[name]["required"] == "true"
+ assert "report-base-url" not in inputs and "retention-runs" not in inputs
+ assert set(workflow["jobs"]) == {"comment"}
+ job = workflow["jobs"]["comment"]
+ assert job["permissions"] == {"contents": "read", "actions": "read", "pull-requests": "write"}
+ for guard in (
+ "workflow_run",
+ "workflow_dispatch",
+ "github.event.workflow_run.status == 'completed'",
+ "github.event.repository.default_branch",
+ "github.event.workflow_run.event == 'pull_request'",
+ "github.event.workflow_run.head_repository.full_name == github.repository",
+ ):
+ assert guard in job["if"]
+ assert job["concurrency"] == {
+ "group": "skill-report-comment-${{ github.repository }}",
+ "cancel-in-progress": "false",
+ "queue": "max",
+ }
+ checkouts = [step for step in job["steps"] if "uses" in step]
+ assert len(checkouts) == 1
+ assert re.fullmatch(r"actions/checkout@[0-9a-f]{40}", checkouts[0]["uses"])
+ assert checkouts[0]["with"]["repository"] == "${{ inputs.tools-repository }}"
+ assert checkouts[0]["with"]["ref"] == "${{ inputs.tools-ref }}"
+ assert checkouts[0]["with"]["persist-credentials"] == "false"
+ hosted = yaml.load((root / ".github/workflows/publish-skill-reports.yml").read_text(), Loader=yaml.BaseLoader)
+ installs = [step for step in job["steps"] if step.get("working-directory") == "publisher"]
+ original = next(step for step in hosted["jobs"]["prepare"]["steps"] if step.get("working-directory") == "publisher")
+ assert installs == [original]
+ assert hosted["jobs"]["comment"]["permissions"]["actions"] == "read"
+ last = job["steps"][-1]
+ assert last["run"] == "publisher/.venv/bin/python publisher/ci/prepare_comments.py"
+ for name in (
+ "TRUSTED_EVALUATOR_SHA",
+ "TRUSTED_CALLER_BLOBS",
+ "TRUSTED_WRAPPER_WORKFLOWS",
+ "DELEGATED_EVALUATOR_WRAPPER",
+ ):
+ assert last["env"][name] == hosted["jobs"]["prepare"]["steps"][4]["env"][name]
diff --git a/tests/ci/test_render_templates.py b/tests/ci/test_render_templates.py
index e71c3e95..55dbb279 100644
--- a/tests/ci/test_render_templates.py
+++ b/tests/ci/test_render_templates.py
@@ -29,6 +29,15 @@ def test_release_templates_are_copyable_and_bind_the_actual_caller(tmp_path):
publisher = yaml.load((output / "publish-skill-reports.yml").read_text(), Loader=yaml.BaseLoader)
assert publisher["jobs"]["reports"]["with"]["trusted-caller-blobs"] == blob
assert publisher["jobs"]["reports"]["with"]["tools-ref"] == "b" * 40
+ comments = yaml.load((output / "comment-skill-checks.yml").read_text(), Loader=yaml.BaseLoader)
+ assert comments["permissions"] == {"contents": "read", "actions": "read", "pull-requests": "write"}
+ assert comments["on"]["workflow_run"] == {"workflows": ["Skill checks"], "types": ["completed"]}
+ assert set(comments["on"]["workflow_dispatch"]["inputs"]) == {"run-id", "run-attempt"}
+ assert all(value["required"] == "true" for value in comments["on"]["workflow_dispatch"]["inputs"].values())
+ assert comments["jobs"]["comment"]["with"]["trusted-caller-blobs"] == blob
+ assert comments["jobs"]["comment"]["with"]["trusted-evaluator-shas"] == "b" * 40
+ assert comments["jobs"]["comment"]["with"]["tools-ref"] == "b" * 40
+ assert "report-base-url" not in comments["jobs"]["comment"]["with"]
for template in output.glob("*.yml"):
assert "RELEASE_" not in template.read_text()
assert "YOUR_REPORT_BASE_URL" in (output / "publish-skill-reports.yml").read_text()
@@ -65,6 +74,11 @@ def test_separate_publisher_revision_preserves_scanner_and_caller_trust(tmp_path
assert publisher["with"]["tools-ref"] == publisher_ref
assert publisher["with"]["trusted-evaluator-shas"] == scanner_ref
assert publisher["with"]["trusted-caller-blobs"] == blob
+ comments = yaml.load((output / "comment-skill-checks.yml").read_text(), Loader=yaml.BaseLoader)["jobs"]["comment"]
+ assert comments["uses"].endswith("@" + publisher_ref)
+ assert comments["with"]["tools-ref"] == publisher_ref
+ assert comments["with"]["trusted-evaluator-shas"] == scanner_ref
+ assert comments["with"]["trusted-caller-blobs"] == blob
assert publisher_ref in (output / "README.md").read_text()
for name in ("skill-checks.yml", "skill-checks-llm.yml", "gitlab-skill-checks.yml"):
content = (output / name).read_text()
@@ -100,9 +114,10 @@ def test_unpinned_or_invalid_release_identity_is_rejected(tmp_path, ref, image,
def test_fork_verification_templates_point_to_the_fork(tmp_path):
output = tmp_path / "out"
release.render_templates(output, "b" * 40, "ghcr.io/example/scan@sha256:" + "a" * 64, "example/scan")
- text = (output / "publish-skill-reports.yml").read_text()
- assert "example/scan/.github/workflows/" in text
- assert "tools-repository: example/scan" in text
+ for filename in ("publish-skill-reports.yml", "comment-skill-checks.yml"):
+ text = (output / filename).read_text()
+ assert "example/scan/.github/workflows/" in text
+ assert "tools-repository: example/scan" in text
def test_existing_release_output_is_preserved(tmp_path):