Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

18.3 KB
Raw Blame
---
title: "GitHub Actions"
description: "Bring SkillEvaluator to your GitHub repository with a reusable workflow, clear results and optional PR feedback."
---

Bring quality and security checks to your repository. No model key required.
Optional LLM checks are available.
Choose your skill folders; pull requests check changed skills automatically.
The shared image includes **SkillSpector security scanning**.

<CardGroup cols={2}>
  <Card title="Use the CLI" icon="terminal" href="/skills/skillevaluator/cli-onboarding">
    Answer a few questions. The CLI prepares checks, optional hosted feedback and a setup PR.
  </Card>
  <Card title="Copy the workflows" icon="file-code" href="#get-started">
    Copy the ready-to-use templates. No local CLI installation required.
  </Card>
</CardGroup>

## Get started

The scanner uses one **default validation policy** for public and private
repositories. There is no `internal`/`external` workflow input. Older images call
this same policy `external`; that label does not publish your repository or reports.

<Steps>

### Copy the ready-to-use template

Open [skill-checks.yml](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/bundles/1.0.1/skill-checks.yml)
from the [ready-to-use templates](https://github.com/NVIDIA/SkillEvaluator/tree/main/ci/bundles/1.0.1).
Use GitHub's **Raw** view to copy it into `.github/workflows/skill-checks.yml`.
The template already supplies pinned workflow and image versions.

### Point it at your skills

Edit these inputs in the downloaded file:

```yaml title="Your choices in skill-checks.yml"
with:
  paths: |-
    skills
    .agents/skills
  min-score: 70
  scan-mode: auto
  llm: false
```

Use your own skill folders. Keep `auto` to scan changed skills in PRs;
`all` checks the configured collection. If your default branch is not `main`,
update the template's push branch.

The 1.0.1 template supports `enforcement-mode`. Omit it for strict merge
gating, or choose advisory enforcement as described below.

### Commit the workflow and open a skill PR

Merge the workflow onto your default branch, then open a PR changing a selected
skill. GitHub Actions must be enabled and allowed to use the referenced
reusable workflow and actions.

</Steps>

<Note>
  The released template provides **checks and downloadable reports**.
  The same release includes the [optional hosted publisher](#enable-hosted-reports).
  [Comments without hosting](#enable-pr-comments) are available in the source
  workflow for a future release bundle.
</Note>

<Accordion title="I prefer to inspect the source templates">
  Browse the public [template catalog](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/templates/README.md)
  and [skill-checks.yml source](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/templates/skill-checks.yml).
  Source templates contain placeholders. Use the release-generated file when
  onboarding; do not substitute `main` or `latest` for its immutable versions.
</Accordion>

## See what happened

| Start here | Then open |
| --- | --- |
| **PR → Checks** | The `skills / Tier 1 scan` result and its short summary. |
| **Actions → Skill checks** | The run and downloadable report artifact. |
| **PR → Conversation**, with feedback enabled | The updated scored bot comment and workflow run; hosted publication also adds **Open full report**. |

Download `skill-evaluator-tier1-attempt-N`, extract it, and open `index.html`.
Failed scans retain their available reports too. The HTML separates the
**verdict** from the **quality score**: a high score does not override a failed
required security or schema check.

<AccordionGroup>
  <Accordion title="Use advisory enforcement">
    The 1.0.1 bundle pins a matching workflow and scanner image with
    `enforcement-mode` support. For an existing 0.8.0 workflow, review both
    pins together before adding this input; the older workflow does not
    accept it.

    With compatible pins, omit `enforcement-mode` or set it to `strict` for
    merge gating. Set `enforcement-mode: advisory` for visibility runs:
    complete policy or score failures retain their failed verdict and full
    artifacts, emit a workflow warning, and leave the job successful.
    Configuration errors, incomplete scanners, missing or malformed evidence,
    and runtime failures still fail the job.

    Hosted publication and PR comments run for advisory downgrades recorded as
    `enforcement: {mode: advisory, downgraded: true}`; strict failures still
    require a failing workflow conclusion. An older image without advisory
    support rejects the explicit advisory flag; strict mode continues to work
    with existing images.
  </Accordion>
  <Accordion title="Why is an author email reported as PII?">
    The `metadata.author` attribution is accepted. Extra copies in contact
    fields, body text, or reference documents are checked separately. Keep the
    author field and use a handle or contact-directory link for extra contact
    information. See [author attribution and approved exceptions](/skills/skillevaluator/tier1-reference#author-attribution-and-contact-emails).
  </Accordion>
  <Accordion title="Use reviewed security baselines">
    This requires a workflow and image version that include baseline support;
    existing released templates do not gain it automatically.

    Set `security-baselines: true` in the reusable scan job and add a reviewed
    `.skillevaluator/skillspector-baselines.yaml` mapping each skill to its
    baseline. For PRs, CI reads both the mapping and baseline from the target
    branch, so merge the approved policy before expecting it to apply.

    Regenerate and review fingerprint exceptions when the scanner version
    changes. Baselines do not waive PII checks or incomplete analysis. The [baseline setup guide](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/README.md#reviewed-security-baselines)
    covers the file format and shared-component forwarding.
  </Accordion>
</AccordionGroup>

<Frame caption="Report example: the skill scored 98.2 but did not meet a configured 100-point threshold. The shared renderer is used by both GitHub and GitLab.">
  <img src="./assets/repository-ci-results-example.png" alt="SkillEvaluator report table showing the skill verdict, score and link to its detailed report" />
</Frame>

### On the pull request

With PR feedback enabled, one bot comment is updated for new results. Reviewers
see the issue and next step without reading the full Actions log.

<Frame caption="Shortened illustrative PR comment. Both feedback workflows use this scored format; the hosted publisher also links to the deployed report.">
<div style={{ padding: "16px", width: "100%", minWidth: 0 }}>

**github-actions[bot]**

**Tier 1 scan result: FAILED**

1 skill · 0 passed · 1 failed · 0 incomplete

| Skill | Status | Quality / 100 |
| --- | --- | --- |
| `skills/glossary-builder` | FAIL | 98.2 / A |

**Problem:** Quality score 98.2/100 is below the required 100/100.

**Next step:** Open the skill report and address its quality findings.

**Open full report** · **View workflow run**

</div>
</Frame>

Browse the [public report and log examples](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/examples/README.md)
for more detail. No personal demo-repository access is needed.

## Enable PR comments

Use the canonical [comment-skill-checks.yml workflow](https://github.com/NVIDIA/SkillEvaluator/blob/main/.github/workflows/comment-skill-checks.yml)
for the same Tier 1 comment without hosted reports. It shows counts, status,
quality score and grade, and the first action for each affected skill. Full
reports remain downloadable from the scan run. Pages settings, a history branch
and a report URL are not required.

<Note>
  The 1.0.1 bundle includes the comments-only workflow; the archived 0.9.0
  bundle does not. Default `skillevaluator ci init` feedback still configures
  the hosted publisher. For comments without hosting, use the versioned
  template below and approve the exact scan caller after your edits.
</Note>

1. Keep the **Skill checks** scan as one reusable job in
   `.github/workflows/skill-checks.yml`, with its reviewed evaluator and image
   pins. Enable `id-token: write` for signed execution receipts.
2. Add the [comment workflow template](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/bundles/1.0.1/comment-skill-checks.yml)
   as `.github/workflows/comment-skill-checks.yml`. Pin its workflow and tools to
   the reviewed comment revision, approve the scanner revision, and set
   `trusted-caller-blobs` to the output of
   `git hash-object .github/workflows/skill-checks.yml` after all scan-file edits.
3. Merge both files into the default branch, then open a skill PR. The comment
   workflow needs `contents: read`, `actions: read` and `pull-requests: write`.
   Choose this workflow or the hosted publisher for feedback.

Comments run when the scan completes, including failures with valid retained
evidence. No changed skills keeps the existing **No skills changed** result.
Only open, current-head PRs from the same repository receive a comment; fork PRs,
stale results, expired artifacts and failures without verifiable evidence do not.
Manual retries require both the completed run ID and attempt number. Follow the
[full comment setup](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/README.md#pr-comments-without-hosted-reports)
for the exact inputs and shared-wrapper approvals.

## Enable hosted reports

The optional publisher hosts the HTML and posts a comment **after deployment
succeeds**. It needs a second workflow plus one-time hosting and approval settings.

<Card title="Try the guided setup" icon="terminal" href="/skills/skillevaluator/cli-onboarding">
  The CLI recognizes an existing single-job scan, keeps its settings and
  component version tag, and adds reporting in a setup PR. New repositories
  get both workflows.
</Card>

Copy [publish-skill-reports.yml](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/bundles/1.0.1/publish-skill-reports.yml)
from the same template folder as your scan workflow. Complete the one-time setup below,
or let the [CLI](/skills/skillevaluator/cli-onboarding) handle it.

<AccordionGroup>
  <Accordion title="What does manual publisher setup involve?">
    Start with the filled-in publisher template and follow the
    [publisher setup guide](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/README.md#optional-hosted-reports-and-pr-feedback).
    The main steps are:

    1. Enable **Settings → Pages → Source: GitHub Actions** and initialize the
       `gh-pages` report-history branch. For private reports, an administrator
       must also select **Private** site visibility in GitHub's settings; see
       the private-hosting requirements below. Preserve any existing site or history.
    2. Add `publish-skill-reports.yml` with your Pages URL. Keep the evaluator
       and publisher versions supplied by the release.
    3. Grant the scan workflow `id-token: write`, then approve its exact contents
       in the publisher's `trusted-caller-blobs` input. Compute the value with
       `git hash-object .github/workflows/skill-checks.yml`; refresh it whenever
       that scan file changes.
    4. Merge both workflow files into the default branch. Open a new skill PR
       to test checks, hosting and feedback together.

    Keep the **Skill checks** workflow name, scan filename and default artifact
    name. The publisher uses GitHub's automatic token; organization policies
    must allow its permissions and referenced actions. Copying a second file
    alone does not finish setup.
  </Accordion>
  <Accordion title="Who receives a comment, and who can read the report?">
    Comments are posted on open, current-head PRs from branches in the same
    repository. Fork-origin PRs keep their available reports in Actions.
    Pages is public unless explicitly configured as private; repository privacy
    alone does not make its Pages site private. Reports may contain source
    excerpts and unmerged findings. Use downloadable artifacts when public
    hosting is unsuitable.
  </Accordion>
  <Accordion title="Can our private repository publish private reports?">
    Access-controlled Pages requires a private or internal project repository
    owned by a GitHub Enterprise Cloud organization. An administrator first
    selects **GitHub Actions** and **Private** in **Settings → Pages**. The
    REST API cannot set Pages visibility. Personal repositories cannot host
    private Pages.

    The CLI verifies the existing private site and uses its assigned
    `*.pages.github.io` URL. It accepts an existing SkillEvaluator report site
    or a newly configured empty private site, and preserves unrelated websites.
    If private hosting is unavailable, choose checks only. See
    [private report hosting](/skills/skillevaluator/cli-onboarding#private-report-hosting) for the
    guided path and [GitHub's access-control requirements](https://docs.github.com/en/enterprise-cloud@latest/pages/getting-started-with-github-pages/changing-the-visibility-of-your-github-pages-site).
  </Accordion>
</AccordionGroup>

## Fit it into your workflow

<AccordionGroup>
  <Accordion title="Will this wait for my existing workflows?">
    This workflow runs independently. Existing workflows remain separate; it
    does not automatically wait for them to finish. Configure explicit workflow
    dependencies if your pipeline needs that ordering.
  </Accordion>
  <Accordion title="Which changes trigger a scan?">
    `auto` selects affected skills on pull requests. Pushes to the default
    branch and manual runs scan the configured collection. `all` scans that
    collection each time. No changed skills is a successful **Skipped** result,
    not a scanner pass. Changes to shared files outside the skill roots are not
    inferred dependencies; use `all` when they should trigger a full scan.
    Selection is capped at 100 skills. Timeouts retain completed results and
    mark unfinished work incomplete. The workflow scans up to four skills at
    once; `scan-workers` accepts 1–16.
  </Accordion>
  <Accordion title="How do I make this a merge gate?">
    After the first run, select its scan check in GitHub branch rules. The
    current template emits `skills / Tier 1 scan`; use the actual check name
    shown in your repository. `min-score` sets the quality threshold; required
    security, schema and other checks remain independent. The reusable scanner
    runs quality and security checks only.

    Keep `enforcement-mode: strict` for a required check. Advisory checks stay
    green on failed policy or score verdicts, so do not use them as required
    merge gates.
  </Accordion>
  <Accordion title="The scan finished, but the comment is missing">
    Confirm setup is on the default branch and the PR is eligible for comments.
    Open **Comment skill checks**, or **Publish skill reports** if using hosting,
    and inspect the evidence validation result. The hosted publisher also needs
    successful deployment. Check the scan's `id-token: write` permission and
    `trusted-caller-blobs` value. A new scan must use the updated workflow;
    rerunning an old job keeps its original workflow. The scan's downloadable
    report remains a separate way to inspect available findings.
  </Accordion>
  <Accordion title="Our organization uses a shared workflow wrapper">
    The comments-only workflow accepts the same caller, wrapper and evaluator
    approvals as the hosted publisher. Keep the standard rendering in
    SkillEvaluator and use the [manual comment setup](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/README.md#pr-comments-without-hosted-reports).
    The released CLI setup below configures hosted publication.

    Keep the component reference in your existing caller, for example
    `your-org/ci/.github/workflows/skill-checks.yml@v3`, then run
    `skillevaluator ci init` in each consumer repository. The CLI recognizes
    `.github/workflows/skill-checks.yml` with the name **Skill checks** and one
    reusable scan job. It preserves inputs, triggers and the version tag while
    adding the publisher and any needed scan-job `id-token: write` permission.

    The component maintainer must first grant `id-token: write` in the job
    calling the evaluator. The CLI checks that prerequisite before applying
    reporting setup; a consumer permission alone cannot grant it downstream.

    With delegation, the publisher approves the exact component path and tag
    in `trusted-wrapper-workflows` and `delegated-evaluator-wrapper`. Updating
    `v3` then updates future scans centrally, with no consumer workflow edit.
    Published execution receipts record the resolved component and evaluator
    commit SHAs for each run. The component still pins its evaluator and image
    immutably. Use a full component SHA when you want to hold that version fixed.

    Use SkillEvaluator 0.7.2 or later and its reviewed publisher for these delegation inputs; the
    archived 0.7.1 publisher does not. Direct callers keep an immutable
    evaluator approval. For manual configuration, follow the
    [shared-workflow setup](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/README.md#calling-through-a-shared-workflow)
    guide. Hosted PR comments follow successful report deployment; private reports
    require the private Pages setup described above.
  </Accordion>
  <Accordion title="Do the checks need an LLM key or network access?">
    Default checks use no model key. Dependency audits can contact
    vulnerability services and submit package metadata; packages are not
    installed and build scripts are not run. Optional LLM checks use a separate
    [manual workflow](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/templates/skill-checks-llm.yml),
    your provider key and provider charges. They send selected content to that
    provider and are outside the default publisher allowlist.
  </Accordion>
</AccordionGroup>

<CardGroup cols={2}>
  <Card title="Set up with the CLI" icon="terminal" href="/skills/skillevaluator/cli-onboarding">
    See the guided setup and try your first test PR.
  </Card>
  <Card title="GitLab CI/CD" icon="code-branch" href="/skills/skillevaluator/gitlab-ci">
    Use the shared scanner image with a GitLab pipeline.
  </Card>
</CardGroup>

<div id="three-tier-checks-and-feedback">

For the explicit all-tier opt-in and its prerequisites, see [Advanced CI](/skills/skillevaluator/advanced-ci).
The [GitHub setup reference](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/templates/all-tiers/README.md#github-checks-and-feedback) preserves the detailed contracts.

</div>