Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
---
title: "GitHub Actions"
description: "Bring SkillEvaluator to your GitHub repository with a reusable workflow, clear results and optional PR feedback."
---
Bring quality and security checks to your repository. No model key required.
Optional LLM checks are available.
Choose your skill folders; pull requests check changed skills automatically.
The shared image includes **SkillSpector security scanning**.
<CardGroup cols={2}>
<Card title="Use the CLI" icon="terminal" href="/skills/skillevaluator/cli-onboarding">
Answer a few questions. The CLI prepares checks, optional hosted feedback and a setup PR.
</Card>
<Card title="Copy the workflows" icon="file-code" href="#get-started">
Copy the ready-to-use templates. No local CLI installation required.
</Card>
</CardGroup>
## Get started
The scanner uses one **default validation policy** for public and private
repositories. There is no `internal`/`external` workflow input. Older images call
this same policy `external`; that label does not publish your repository or reports.
<Steps>
### Copy the ready-to-use template
Open [skill-checks.yml](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/bundles/1.0.1/skill-checks.yml)
from the [ready-to-use templates](https://github.com/NVIDIA/SkillEvaluator/tree/main/ci/bundles/1.0.1).
Use GitHub's **Raw** view to copy it into `.github/workflows/skill-checks.yml`.
The template already supplies pinned workflow and image versions.
### Point it at your skills
Edit these inputs in the downloaded file:
```yaml title="Your choices in skill-checks.yml"
with:
paths: |-
skills
.agents/skills
min-score: 70
scan-mode: auto
llm: false
```
Use your own skill folders. Keep `auto` to scan changed skills in PRs;
`all` checks the configured collection. If your default branch is not `main`,
update the template's push branch.
The 1.0.1 template supports `enforcement-mode`. Omit it for strict merge
gating, or choose advisory enforcement as described below.
### Commit the workflow and open a skill PR
Merge the workflow onto your default branch, then open a PR changing a selected
skill. GitHub Actions must be enabled and allowed to use the referenced
reusable workflow and actions.
</Steps>
<Note>
The released template provides **checks and downloadable reports**.
The same release includes the [optional hosted publisher](#enable-hosted-reports).
[Comments without hosting](#enable-pr-comments) are available in the source
workflow for a future release bundle.
</Note>
<Accordion title="I prefer to inspect the source templates">
Browse the public [template catalog](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/templates/README.md)
and [skill-checks.yml source](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/templates/skill-checks.yml).
Source templates contain placeholders. Use the release-generated file when
onboarding; do not substitute `main` or `latest` for its immutable versions.
</Accordion>
## See what happened
| Start here | Then open |
| --- | --- |
| **PR → Checks** | The `skills / Tier 1 scan` result and its short summary. |
| **Actions → Skill checks** | The run and downloadable report artifact. |
| **PR → Conversation**, with feedback enabled | The updated scored bot comment and workflow run; hosted publication also adds **Open full report**. |
Download `skill-evaluator-tier1-attempt-N`, extract it, and open `index.html`.
Failed scans retain their available reports too. The HTML separates the
**verdict** from the **quality score**: a high score does not override a failed
required security or schema check.
<AccordionGroup>
<Accordion title="Use advisory enforcement">
The 1.0.1 bundle pins a matching workflow and scanner image with
`enforcement-mode` support. For an existing 0.8.0 workflow, review both
pins together before adding this input; the older workflow does not
accept it.
With compatible pins, omit `enforcement-mode` or set it to `strict` for
merge gating. Set `enforcement-mode: advisory` for visibility runs:
complete policy or score failures retain their failed verdict and full
artifacts, emit a workflow warning, and leave the job successful.
Configuration errors, incomplete scanners, missing or malformed evidence,
and runtime failures still fail the job.
Hosted publication and PR comments run for advisory downgrades recorded as
`enforcement: {mode: advisory, downgraded: true}`; strict failures still
require a failing workflow conclusion. An older image without advisory
support rejects the explicit advisory flag; strict mode continues to work
with existing images.
</Accordion>
<Accordion title="Why is an author email reported as PII?">
The `metadata.author` attribution is accepted. Extra copies in contact
fields, body text, or reference documents are checked separately. Keep the
author field and use a handle or contact-directory link for extra contact
information. See [author attribution and approved exceptions](/skills/skillevaluator/tier1-reference#author-attribution-and-contact-emails).
</Accordion>
<Accordion title="Use reviewed security baselines">
This requires a workflow and image version that include baseline support;
existing released templates do not gain it automatically.
Set `security-baselines: true` in the reusable scan job and add a reviewed
`.skillevaluator/skillspector-baselines.yaml` mapping each skill to its
baseline. For PRs, CI reads both the mapping and baseline from the target
branch, so merge the approved policy before expecting it to apply.
Regenerate and review fingerprint exceptions when the scanner version
changes. Baselines do not waive PII checks or incomplete analysis. The [baseline setup guide](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/README.md#reviewed-security-baselines)
covers the file format and shared-component forwarding.
</Accordion>
</AccordionGroup>
<Frame caption="Report example: the skill scored 98.2 but did not meet a configured 100-point threshold. The shared renderer is used by both GitHub and GitLab.">
<img src="./assets/repository-ci-results-example.png" alt="SkillEvaluator report table showing the skill verdict, score and link to its detailed report" />
</Frame>
### On the pull request
With PR feedback enabled, one bot comment is updated for new results. Reviewers
see the issue and next step without reading the full Actions log.
<Frame caption="Shortened illustrative PR comment. Both feedback workflows use this scored format; the hosted publisher also links to the deployed report.">
<div style={{ padding: "16px", width: "100%", minWidth: 0 }}>
**github-actions[bot]**
**Tier 1 scan result: FAILED**
1 skill · 0 passed · 1 failed · 0 incomplete
| Skill | Status | Quality / 100 |
| --- | --- | --- |
| `skills/glossary-builder` | FAIL | 98.2 / A |
**Problem:** Quality score 98.2/100 is below the required 100/100.
**Next step:** Open the skill report and address its quality findings.
**Open full report** · **View workflow run**
</div>
</Frame>
Browse the [public report and log examples](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/examples/README.md)
for more detail. No personal demo-repository access is needed.
## Enable PR comments
Use the canonical [comment-skill-checks.yml workflow](https://github.com/NVIDIA/SkillEvaluator/blob/main/.github/workflows/comment-skill-checks.yml)
for the same Tier 1 comment without hosted reports. It shows counts, status,
quality score and grade, and the first action for each affected skill. Full
reports remain downloadable from the scan run. Pages settings, a history branch
and a report URL are not required.
<Note>
The 1.0.1 bundle includes the comments-only workflow; the archived 0.9.0
bundle does not. Default `skillevaluator ci init` feedback still configures
the hosted publisher. For comments without hosting, use the versioned
template below and approve the exact scan caller after your edits.
</Note>
1. Keep the **Skill checks** scan as one reusable job in
`.github/workflows/skill-checks.yml`, with its reviewed evaluator and image
pins. Enable `id-token: write` for signed execution receipts.
2. Add the [comment workflow template](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/bundles/1.0.1/comment-skill-checks.yml)
as `.github/workflows/comment-skill-checks.yml`. Pin its workflow and tools to
the reviewed comment revision, approve the scanner revision, and set
`trusted-caller-blobs` to the output of
`git hash-object .github/workflows/skill-checks.yml` after all scan-file edits.
3. Merge both files into the default branch, then open a skill PR. The comment
workflow needs `contents: read`, `actions: read` and `pull-requests: write`.
Choose this workflow or the hosted publisher for feedback.
Comments run when the scan completes, including failures with valid retained
evidence. No changed skills keeps the existing **No skills changed** result.
Only open, current-head PRs from the same repository receive a comment; fork PRs,
stale results, expired artifacts and failures without verifiable evidence do not.
Manual retries require both the completed run ID and attempt number. Follow the
[full comment setup](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/README.md#pr-comments-without-hosted-reports)
for the exact inputs and shared-wrapper approvals.
## Enable hosted reports
The optional publisher hosts the HTML and posts a comment **after deployment
succeeds**. It needs a second workflow plus one-time hosting and approval settings.
<Card title="Try the guided setup" icon="terminal" href="/skills/skillevaluator/cli-onboarding">
The CLI recognizes an existing single-job scan, keeps its settings and
component version tag, and adds reporting in a setup PR. New repositories
get both workflows.
</Card>
Copy [publish-skill-reports.yml](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/bundles/1.0.1/publish-skill-reports.yml)
from the same template folder as your scan workflow. Complete the one-time setup below,
or let the [CLI](/skills/skillevaluator/cli-onboarding) handle it.
<AccordionGroup>
<Accordion title="What does manual publisher setup involve?">
Start with the filled-in publisher template and follow the
[publisher setup guide](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/README.md#optional-hosted-reports-and-pr-feedback).
The main steps are:
1. Enable **Settings → Pages → Source: GitHub Actions** and initialize the
`gh-pages` report-history branch. For private reports, an administrator
must also select **Private** site visibility in GitHub's settings; see
the private-hosting requirements below. Preserve any existing site or history.
2. Add `publish-skill-reports.yml` with your Pages URL. Keep the evaluator
and publisher versions supplied by the release.
3. Grant the scan workflow `id-token: write`, then approve its exact contents
in the publisher's `trusted-caller-blobs` input. Compute the value with
`git hash-object .github/workflows/skill-checks.yml`; refresh it whenever
that scan file changes.
4. Merge both workflow files into the default branch. Open a new skill PR
to test checks, hosting and feedback together.
Keep the **Skill checks** workflow name, scan filename and default artifact
name. The publisher uses GitHub's automatic token; organization policies
must allow its permissions and referenced actions. Copying a second file
alone does not finish setup.
</Accordion>
<Accordion title="Who receives a comment, and who can read the report?">
Comments are posted on open, current-head PRs from branches in the same
repository. Fork-origin PRs keep their available reports in Actions.
Pages is public unless explicitly configured as private; repository privacy
alone does not make its Pages site private. Reports may contain source
excerpts and unmerged findings. Use downloadable artifacts when public
hosting is unsuitable.
</Accordion>
<Accordion title="Can our private repository publish private reports?">
Access-controlled Pages requires a private or internal project repository
owned by a GitHub Enterprise Cloud organization. An administrator first
selects **GitHub Actions** and **Private** in **Settings → Pages**. The
REST API cannot set Pages visibility. Personal repositories cannot host
private Pages.
The CLI verifies the existing private site and uses its assigned
`*.pages.github.io` URL. It accepts an existing SkillEvaluator report site
or a newly configured empty private site, and preserves unrelated websites.
If private hosting is unavailable, choose checks only. See
[private report hosting](/skills/skillevaluator/cli-onboarding#private-report-hosting) for the
guided path and [GitHub's access-control requirements](https://docs.github.com/en/enterprise-cloud@latest/pages/getting-started-with-github-pages/changing-the-visibility-of-your-github-pages-site).
</Accordion>
</AccordionGroup>
## Fit it into your workflow
<AccordionGroup>
<Accordion title="Will this wait for my existing workflows?">
This workflow runs independently. Existing workflows remain separate; it
does not automatically wait for them to finish. Configure explicit workflow
dependencies if your pipeline needs that ordering.
</Accordion>
<Accordion title="Which changes trigger a scan?">
`auto` selects affected skills on pull requests. Pushes to the default
branch and manual runs scan the configured collection. `all` scans that
collection each time. No changed skills is a successful **Skipped** result,
not a scanner pass. Changes to shared files outside the skill roots are not
inferred dependencies; use `all` when they should trigger a full scan.
Selection is capped at 100 skills. Timeouts retain completed results and
mark unfinished work incomplete. The workflow scans up to four skills at
once; `scan-workers` accepts 1–16.
</Accordion>
<Accordion title="How do I make this a merge gate?">
After the first run, select its scan check in GitHub branch rules. The
current template emits `skills / Tier 1 scan`; use the actual check name
shown in your repository. `min-score` sets the quality threshold; required
security, schema and other checks remain independent. The reusable scanner
runs quality and security checks only.
Keep `enforcement-mode: strict` for a required check. Advisory checks stay
green on failed policy or score verdicts, so do not use them as required
merge gates.
</Accordion>
<Accordion title="The scan finished, but the comment is missing">
Confirm setup is on the default branch and the PR is eligible for comments.
Open **Comment skill checks**, or **Publish skill reports** if using hosting,
and inspect the evidence validation result. The hosted publisher also needs
successful deployment. Check the scan's `id-token: write` permission and
`trusted-caller-blobs` value. A new scan must use the updated workflow;
rerunning an old job keeps its original workflow. The scan's downloadable
report remains a separate way to inspect available findings.
</Accordion>
<Accordion title="Our organization uses a shared workflow wrapper">
The comments-only workflow accepts the same caller, wrapper and evaluator
approvals as the hosted publisher. Keep the standard rendering in
SkillEvaluator and use the [manual comment setup](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/README.md#pr-comments-without-hosted-reports).
The released CLI setup below configures hosted publication.
Keep the component reference in your existing caller, for example
`your-org/ci/.github/workflows/skill-checks.yml@v3`, then run
`skillevaluator ci init` in each consumer repository. The CLI recognizes
`.github/workflows/skill-checks.yml` with the name **Skill checks** and one
reusable scan job. It preserves inputs, triggers and the version tag while
adding the publisher and any needed scan-job `id-token: write` permission.
The component maintainer must first grant `id-token: write` in the job
calling the evaluator. The CLI checks that prerequisite before applying
reporting setup; a consumer permission alone cannot grant it downstream.
With delegation, the publisher approves the exact component path and tag
in `trusted-wrapper-workflows` and `delegated-evaluator-wrapper`. Updating
`v3` then updates future scans centrally, with no consumer workflow edit.
Published execution receipts record the resolved component and evaluator
commit SHAs for each run. The component still pins its evaluator and image
immutably. Use a full component SHA when you want to hold that version fixed.
Use SkillEvaluator 0.7.2 or later and its reviewed publisher for these delegation inputs; the
archived 0.7.1 publisher does not. Direct callers keep an immutable
evaluator approval. For manual configuration, follow the
[shared-workflow setup](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/README.md#calling-through-a-shared-workflow)
guide. Hosted PR comments follow successful report deployment; private reports
require the private Pages setup described above.
</Accordion>
<Accordion title="Do the checks need an LLM key or network access?">
Default checks use no model key. Dependency audits can contact
vulnerability services and submit package metadata; packages are not
installed and build scripts are not run. Optional LLM checks use a separate
[manual workflow](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/templates/skill-checks-llm.yml),
your provider key and provider charges. They send selected content to that
provider and are outside the default publisher allowlist.
</Accordion>
</AccordionGroup>
<CardGroup cols={2}>
<Card title="Set up with the CLI" icon="terminal" href="/skills/skillevaluator/cli-onboarding">
See the guided setup and try your first test PR.
</Card>
<Card title="GitLab CI/CD" icon="code-branch" href="/skills/skillevaluator/gitlab-ci">
Use the shared scanner image with a GitLab pipeline.
</Card>
</CardGroup>
<div id="three-tier-checks-and-feedback">
For the explicit all-tier opt-in and its prerequisites, see [Advanced CI](/skills/skillevaluator/advanced-ci).
The [GitHub setup reference](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/templates/all-tiers/README.md#github-checks-and-feedback) preserves the detailed contracts.
</div>