Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

24.8 KB
Raw Blame
---
title: Gate Your CI
description: Make SkillEvaluator a merge gate — exit codes, a copy-paste GitHub Actions workflow, collection validation, progressive adoption, and machine-readable results.
---

Turn SkillEvaluator into a merge gate for your skill repository. Quality and
security checks need no model API key; full scanner-backed checks can still
contact vulnerability services. Adopt the CLI in any CI system and add
model-backed checks separately.

For the prebuilt runtime and one-file consumer setup, see
[GitHub Actions](/skills/skillevaluator/github-actions). The recipe below installs tools explicitly;
its PR-comment step is separate from the reusable publisher's hosted-feedback
setup.

## Exit codes

`skillevaluator validate` communicates pass/fail through its exit code, so a plain `run:` step is already a gate — no output parsing required.

| Exit code | Meaning | What to do |
| --- | --- | --- |
| `0` | Success — every gating check passed | Merge |
| `1` | Validation failed — a gating check failed, the run could not complete, or an unexpected internal error occurred | Read the report or log; fix the skill, or re-run and file an issue if the run itself failed |
| `2` | Usage error — an unknown flag, a value outside an option's type or choices, or a target path that does not exist | Fix the pipeline, not the skill |

Quality and security checks always gate the exit code. Deduplication findings gate when dedup runs unless you pass `--no-block-on-dedup`; the recommended keyless gate below selects `--tiers 1` to exclude deduplication and live agent evaluation. Live-evaluation results are advisory inside `validate` unless you pass `--block-on-agent-eval`. Reports record the effective choice for each stage. Without an explicit scope, `validate` runs all three stages for skills, including live-evaluation autopilot.

## The recommended gate

```bash title="Keyless CI gate"
skillevaluator validate ./skills --profile default --tiers 1 -r json,markdown,sarif -o reports --min-score 70
```

Each flag earns its place:

- **`--profile default`** pins the standard validation policy explicitly instead of relying on the environment, so the gate behaves identically on every runner. The active profile name is stamped into the HTML and JSON reports (and printed in the run banner with `--verbose`), so a run always records which gate was applied.
- **`--tiers 1`** runs static checks without provider-backed deduplication or live agent evaluation. Use `--tiers 1,2` once you add a provider secret, or omit the option to include live evaluation after configuring its runtime. `--no-dedup` alone skips only deduplication and still runs live evaluation.
- **`-r json,markdown,sarif`** writes machine-readable JSON, a Markdown PR comment, and a SARIF file for GitHub Code Scanning.
- **`-o reports`** collects everything in one directory for artifact upload.
- **`--min-score 70`** is the default quality bar, written out explicitly so raising it later is a visible one-line diff.

## GitHub Actions recipe

A complete workflow: install with uv, add the external Semgrep, SkillSpector,
and Gitleaks scanners for a complete security result, validate, upload the
reports, and post the Markdown report on the pull request.

```yaml title="ci.yml"
name: Validate skills

on:
  pull_request:
    paths:
      - "skills/**"

jobs:
  validate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
      security-events: write
    steps:
      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0

      - name: Install uv
        uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0
        with:
          python-version: "3.13"

      - name: Install SkillEvaluator
        run: uv tool install --python 3.13 "skillevaluator @ git+https://github.com/NVIDIA/SkillEvaluator.git"

      - name: Install external Python scanners
        run: |
          uv tool install semgrep
          uv tool install git+https://github.com/NVIDIA/SkillSpector.git@54824463814a0be01360792523ec7245a69190fc # v2.13.4

      - name: Install Gitleaks
        run: |
          GITLEAKS_VERSION=8.30.0
          # SHA-256 of the linux_x64 archive, from the release's checksums.txt.
          # Update it together with GITLEAKS_VERSION.
          GITLEAKS_SHA256=79a3ab579b53f71efd634f3aaf7e04a0fa0cf206b7ed434638d1547a2470a66e
          base="https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}"
          archive="gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz"
          cd "$RUNNER_TEMP"
          curl -sSfL --retry 3 "${base}/${archive}" -o "${archive}"
          printf '%s  %s\n' "${GITLEAKS_SHA256}" "${archive}" | sha256sum -c -
          tar -xzf "${archive}" gitleaks
          sudo mv gitleaks /usr/local/bin/

      - name: Validate skills
        run: |
          skillevaluator validate ./skills --profile default --tiers 1 \
            -r json,markdown,sarif -o reports --min-score 70

      - name: Upload reports
        if: always()
        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
        with:
          name: skillevaluator-reports
          path: reports/

      - name: Find SARIF reports
        if: always()
        id: sarif
        run: |
          mapfile -t files < <(find reports -name '*.sarif.json' -type f 2>/dev/null | sort)
          if [ "${#files[@]}" -eq 0 ]; then
            echo "count=0" >> "$GITHUB_OUTPUT"
            exit 0
          fi
          skillevaluator_python="$(dirname "$(readlink -f "$(command -v skillevaluator)")")/python"
          "$skillevaluator_python" - <<'PY'
          import json
          import pathlib

          from skillevaluator.reporting.sarif_reporter import merge_catalog_sarif_documents

          documents = []
          for raw in sorted(pathlib.Path("reports").rglob("*.sarif.json")):
              if raw.name == "catalog-sarif.json":
                  continue
              documents.append(json.loads(raw.read_text(encoding="utf-8")))
          if not documents:
              raise SystemExit(0)
          merged = merge_catalog_sarif_documents(documents)
          out = pathlib.Path("reports/catalog-sarif.json")
          out.write_text(json.dumps(merged, indent=2), encoding="utf-8")
          print(f"merged {len(documents)} child report(s) into one run at {out}")
          PY
          echo "count=${#files[@]}" >> "$GITHUB_OUTPUT"
          echo "path=reports/catalog-sarif.json" >> "$GITHUB_OUTPUT"

      - name: Upload SARIF to GitHub Code Scanning
        if: always() && steps.sarif.outputs.count != '0'
        uses: github/codeql-action/upload-sarif@33d7b85da1a04171e211befe2c739303b9a4c066 # v4.39.3
        with:
          sarif_file: ${{ steps.sarif.outputs.path }}
          category: skillevaluator

      - name: Post the Markdown report as a PR comment
        if: always() && github.event_name == 'pull_request'
        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
        with:
          script: |
            const fs = require("fs");
            const [latest] = fs.readdirSync("reports")
              .filter((f) => f.endsWith(".md"))
              .sort()
              .reverse();
            if (!latest) return;
            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: fs.readFileSync(`reports/${latest}`, "utf8"),
            });
```

Each action is pinned to a commit SHA, with its release in a comment, as in this
repository's own workflows, so a moved tag cannot change what the job runs.
To update an action, replace the SHA with the commit of the newer release.
The same holds for the scanners: SkillSpector installs from the commit behind
its release tag, and the Gitleaks archive is checked against the SHA-256 from
the release's `gitleaks_<version>_checksums.txt` before it is unpacked. To
update either, take the new commit or checksum from the release page.

SkillEvaluator installs Bandit, and the `security` extra adds pip-audit. Semgrep and SkillSpector
stay in separate uv tool environments, while Gitleaks is installed from its
official [releases](https://github.com/gitleaks/gitleaks/releases). Report
filenames embed a sortable timestamp (`skillevaluator-output-<timestamp>.md`),
which is why a lexicographic sort finds the newest one. The `if: always()`
conditions keep reports flowing even when validation fails — which is exactly
when you want them.

The same shape ports to any CI system: install, run `validate`, gate on the exit code, archive `reports/`.

## Validating collections

Point `validate` at a folder of skills and it validates each one:

```bash title="Validate a whole collection"
skillevaluator validate ./skills --profile default --tiers 1 -c -r json,markdown -o reports
```

Two flags control how a multi-skill run behaves:

| Flag | Behavior | Use it when |
| --- | --- | --- |
| `-c`, `--continue-on-failure` | Runs the full pipeline without stopping early and records every issue; for folder targets, keeps scanning every skill past a CRITICAL finding | You want one complete report per CI run (recommended for gates) |
| `--fail-fast` | Stops on the first failing check | You want the fastest possible red signal |

`-c` overrides `--fail-fast` if both are passed. For a merge gate, prefer `-c`: contributors get the complete picture in one run instead of a fix-push-repeat loop.

## Adopt progressively

You do not have to turn on the strictest gate on day one. A path that works well:

<Steps>
  ### Start advisory

  Run the gate but never fail the build. Contributors see reports and get used to the findings before anything blocks a merge.

  ```bash title="Report-only mode"
  skillevaluator validate ./skills --profile default --tiers 1 -r markdown -o reports || true
  ```

  ### Require quality and security checks

  <div id="gate-on-tier-1">

  Remove the `|| true`. Exit code `1` now blocks merges on schema, security, PII, license, code-integrity, Unicode, and quality failures — all still keyless. (The `lint` check also runs, but its findings are advisory warnings and never fail validation.)

  </div>

  ### Raise the quality bar

  Bump `--min-score` above the default `70` once your collection consistently clears it. The score and grade in each report tell you where the collection stands before you move the line.

  ### Add LLM-backed checks

  Add a provider key as a masked CI secret, set `SKILL_EVAL_LLM_PROVIDER`, then add `--llm` (and `--llm-verify` to suppress false positives) and select `--tiers 1,2` so overlap checks run too. Once dedup runs, its findings gate alongside quality and security checks by default; add `--no-block-on-dedup` if you want evidence without enforcing it yet. See [Providers & Credentials](/skills/skillevaluator/configuration) for provider setup.

  ### Add an advisory Tier 3 job

  Run live agent evaluation as a separate job with `skillevaluator tier3 ./my-skill`, or remove `--tiers 1,2` from `validate` to run all three tiers. Both workflows reuse existing evaluation sources and generate one starter case when none exists. Attached Tier 3 is advisory by default; add `--block-on-agent-eval` when the team is ready for its findings or invalid source evidence to fail the merge gate. Standalone Tier 3 exits non-zero when the run itself fails to complete. Expose agent credentials as CI secrets in the job environment — the `env:` block of the workflow step — because SkillEvaluator reads operator credentials from the host environment only. With the `nv_build` provider and `--env-mode docker`, a single `NVIDIA_API_KEY` secret covers the evaluator and all three agents. See [Tier 3: Live Agent Evaluation](/skills/skillevaluator/tier3-live-evaluation).
</Steps>

<Note>
Keep provider and agent keys as masked CI secrets in the job environment — never in the repo or in `evals/config.yml`. SkillEvaluator enforces this: a `harbor.runtime_env` entry that names or `${VAR}`-references an operator-owned credential (such as `OPENAI_API_KEY` or `NVIDIA_API_KEY`) fails the Tier 3 run with a hard error.
</Note>

## Custom policy in CI

If the bundled `default` policy is too loose or too strict for your repository, commit a policy overlay next to your skills and pin it in the gate. Severity decisions then get reviewed like code:

```yaml title="policy.yaml"
profile: community-strict
identity:
  author_email_regex: '<[^>]+@example\.org>'
severity_overrides:
  SCHEMA.author_missing: critical
  LICENSE.*: critical
```

```bash title="Gate with a pinned policy"
skillevaluator validate ./skills --profile default --policy ./policy.yaml --tiers 1 -r json,markdown -o reports
```

The policy overlays the default rules, and the resulting profile name is stamped into the HTML and JSON reports (and printed in the run banner with `--verbose`) — so a run always tells you which gate produced it. See the [validation policy reference](/skills/skillevaluator/tier1-validation) for the full overlay syntax.

## Parse the JSON

When the exit code is not enough — dashboards, custom thresholds, badge generation — read `reports/skillevaluator-output-<timestamp>.json`. The fields CI cares about sit at the top level:

| Field | Type | Contents |
| --- | --- | --- |
| `version` | string | SkillEvaluator version that wrote the report |
| `target_path` | string or null | The validated target. `validate` records its absolute path, or its browsable URL when it is in a Git checkout with an `origin` remote. `tier1` and `tier2` record the path as given on the command line; in a report that CI retains, the path is the skill's path relative to the repository root, never the runner's absolute path. The single-check commands such as `security-scan` give `null` |
| `overall_passed` | boolean | `true` when every check passed |
| `overall_status` | string | `passed`, `failed`, or `incomplete` (a required scanner produced no evidence) |
| `publication_status` | string | `pass`, `neutral`, `fail`, or `incomplete`; use this field for publication eligibility |
| `publication` | object | Publication decision, eligibility, reasons, and Tier 3 evidence status |
| `benchmark_policy` | object | Persisted `tier2_required` and `tier3_required` evidence policy used for the publication decision |
| `results[].publication_target` | object | Exact NFC filesystem-entry name, versioned source-tree digest, and digest algorithm used to bind split-tier evidence |
| `results[].publication_evidence` | object | Validated built-in Tier 1 or Tier 2 producer contract: schema version, producer, tier, and canonical check ID |
| `results[].skillspector_baseline` | object | Single-skill targets with `--skillspector-baseline`, when SkillSpector is installed: `status`, baseline file name and `sha256:` digest (no local path), scanner versions, suppressed (per stage) and active finding counts, and a message. Absent from merged `security-scan` folder reports; `validate` on a folder records it in each skill's report. See [Audited SkillSpector baselines](/skills/skillevaluator/tier1-reference#audited-skillspector-baselines) |
| `tier3.run_id` | string | Run-owned Tier 3 identity; duplicated in the Tier 3 summary for consistency checks |
| `severity_counts` | object | Totals for `critical`, `high`, `medium`, `low` |
| `total_errors`, `total_warnings` | number | Raw error and warning records across all validators; a finding that two validators report counts twice |
| `finding_counts` | object | Reconciled counts: `raw_occurrences`, `distinct_occurrences`, `unique_findings`, `duplicates_removed`, `occurrences_grouped`, `raw_by_severity`, `unique_by_severity` |
| `finding_groups` | array | One entry per unique finding: `finding_id`, highest `severity`, `message`, `validators`, and every location in `occurrences[]` with its `occurrence_id`, including each location a finding lists in `metadata.occurrences` when its check folds repeats of one value into it (the PII scan does) |
| `message_counts` | object | `independent_errors` and `independent_warnings`: error and warning strings with no finding behind them, such as `gate_failures` and scanner or execution errors (strings that mirror a finding, and the `Workflow scope:` note for a stage that configuration skips, are not counted); `informational`: informational messages, not counting the `[OK]` line recorded for each passed check |
| `results[].findings[].finding_id`, `results[].findings[].occurrence_id` | string | Stable finding and location identities (see below) |
| `skills` | array | Per-skill `{ name, passed, issue_count }`; `issue_count` counts raw finding records. `validate`, `tier1` and `tier2` on one skill folder list that skill once, under its folder name, also when a narrowed run such as `--checks schema` reports no finding |
| `top_issues` | array | Top issues across every tier in the report, most severe first: `message` (a security finding is grouped under the text before its first `: `), highest `severity`, `total_count` of raw finding records (the unit of `skills[].issue_count`, so a PII finding that folds repeats of one value counts once although `finding_groups` lists each of its locations), `skills_affected`, `category` and `suggestion`. The HTML report data carries the same rows; the HTML summary panel lists the Tier 1 rows only |
| `gating.tiers` | object | `validate`, `tier1` and `tier2` runs: one entry per tier that ran (`"1"`, `"2"`, `"3"`). `blocking` is `true` when any of the tier's checks gates the exit code, `validators` lists the checks, and `validator_policies` gives each check's own `{ validator, blocking }` |
| `quality_summary` | array | Quality details with `overall_score` (0–100) and `grade` (A–F); a folder target reports the collection average plus a `skill_count` |

Every finding carries two stable IDs. `finding_id` (`SEF-` plus 20 hex
characters) hashes the finding's category, check name, message, and suggestion,
ignoring case and extra whitespace. It does not change when the finding moves
to another line or file, or when a policy changes its severity, so you can
track a finding across runs. Location text that a check also writes into its
message is left out of the hash: the `in <file>:<line>` in a secret detection
message, also when a folder or plugin run puts the skill or member name in
front of the finding's file path, and the `— N occurrences (…)` list that the
PII scan appends when it folds repeats of one value into one finding. Other
message and suggestion text stays part of the ID: script lint names the script
in its messages and secret detection names the file in its suggestion, so the
same problem in two files has two `finding_id` values. `occurrence_id`
(`SEO-…`) adds the file path, line number, and line content, so it identifies
one location; the source line itself stays on the finding in
`results[].findings[]` and is not repeated in `finding_groups`.
`finding_groups` groups findings by `finding_id` across validators, and each
result in `results[]` carries its own `finding_counts`. Identical findings in
different files or skills share one `finding_id`, so they count as one unique
finding with one occurrence per location. When the CI templates redact a
detected credential from a report, they recompute the IDs and groups from the
redacted text and render the HTML, Markdown, and SARIF copies from that JSON,
so no ID is derived from a removed value; such a finding's IDs differ from a
local run's.

The Markdown summary and the HTML report count findings the same way: each
unique finding once, at its highest severity. Markdown's Total Issues also
shows distinct occurrences, raw records, and the `independent_errors` and
`independent_warnings`, so a run that fails a gate or leaves a scan
incomplete without any finding is not reported as zero issues. In HTML, the
tier cards, the Tier 1 Findings and Blocking critical + high numbers, the
verdict sentence, the skill finding badges and the gate policy line count
unique findings, and the Findings number shows the same breakdown beneath it
when it differs from the unique count. The HTML report's embedded data, which
Export JSON saves, counts `summary.total_issues` the same way: unique findings
plus `independent_errors` and `independent_warnings`, where it used to add up
raw errors and warnings. Its raw finding count is in
`summary.raw_finding_occurrences` and `finding_counts.raw_occurrences`.
`total_errors`, `total_warnings`, `severity_counts`, `skills[].issue_count` and
contributor totals still count raw records.

`overall_status` and `overall_passed` describe the configured command gate. They
can be green when an advisory tier did not run, so do not use them as a
publication signal. Require `publication_status == "pass"` (or
`publication.eligible == true`) before publishing a benchmark.

A step that enforces a stricter score than the built-in gate:

```bash title="Gate on quality score with jq"
mapfile -t reports < <(find reports -name 'skillevaluator-output-*.json' -type f)
if [ "${#reports[@]}" -eq 0 ]; then
  echo "No SkillEvaluator JSON report under reports/" >&2
  exit 1
fi
jq -se --argjson n "${#reports[@]}" \
  'length == $n and all(.[]; [.quality_summary[]?.overall_score] | length > 0 and min >= 80)' "${reports[@]}"
```

A single-skill target writes its report at the top of `reports/`, and a folder target writes one per skill under `reports/<skill>/`, so the step checks every report it finds; write them to an empty `reports/` folder. The gate fails closed: a run with no report, an empty report file, or a report without a quality score fails it. `jq -e` exits non-zero when the expression is false, so the step fails the job on its own. For Tier 3 payloads and the full results-on-disk layout, see [Reports & Results](/skills/skillevaluator/reports).

## Troubleshooting

<AccordionGroup>
  <Accordion title="The run exits 2 instead of 1">
    Exit code `2` is a configuration error — a bad flag or an unreadable target path. Fix the pipeline; the skill was never judged. (Passing `--llm` without a configured provider does not exit `2`: the LLM-backed scanner is marked incomplete and the run fails with `1`.)
  </Accordion>
  <Accordion title="overall_status is 'incomplete'">
    A required scanner is missing or returned no evidence. Check the report for
    the named scanner: Semgrep, SkillSpector, and Gitleaks are all separate
    executables. When SkillSpector or Gitleaks is not installed, the first
    Tier 1 run downloads its pinned, checksum-verified release (Gitleaks on
    Linux and macOS only), so the runner needs network access to GitHub, and
    to PyPI for SkillSpector. Semgrep is optional and never downloaded; a
    missing Semgrep makes the run incomplete only when
    `SKILLEVALUATOR_REQUIRE_SEMGREP=1` is set. The scanners fail closed:
    `validate` exits non-zero and the report stays non-green until every
    required scanner runs and returns valid evidence. See the workflow above or
    [Installation: System tools](/skills/skillevaluator/installation-options#system-tools).
  </Accordion>
  <Accordion title="Claude answers HTTP 400 'is not a valid fallback target'">
    The reusable SkillEvaluator workflows pass only their documented inputs
    (provider, model, base URL, and keys) to the provider-backed tiers, so a
    `SKILL_EVAL_LLM_REFUSAL_FALLBACK_MODEL` or `LLM_JUDGE_FALLBACK_MODELS` set
    in the calling job does not reach them. Choose a Claude model that accepts
    the default `claude-opus-4-8` fallback there. A pipeline that runs the CLI
    directly, as in the recipe above, can set
    `SKILL_EVAL_LLM_REFUSAL_FALLBACK_MODEL` in the step environment, or `none`
    to send no fallback. See
    [Environment variables](/skills/skillevaluator/environment-variables).
  </Accordion>
  <Accordion title="The report artifact is empty">
    Make sure the artifact `path` matches your `-o` directory and that the upload step carries `if: always()` — otherwise a failed gate skips the upload, and you lose the report exactly when you need it.
  </Accordion>
</AccordionGroup>

## Next steps

<CardGroup cols={2}>
  <Card title="Reports & Results" icon="book" href="/skills/skillevaluator/reports">
    What every report format contains and how to read scores, severities, and Tier 3 payloads.
  </Card>
  <Card title="Quality and security checks" icon="shield" href="/skills/skillevaluator/tier1-validation">
    Every check behind the gate — what it inspects and how to fix findings.
  </Card>
  <Card title="Providers & Credentials" icon="flask" href="/skills/skillevaluator/configuration">
    Add an LLM provider to unlock `--llm`, Tier 2 dedup, and Tier 3.
  </Card>
  <Card title="CLI Reference" icon="terminal" href="/skills/skillevaluator/cli-reference">
    The complete flag reference for `validate` and every other command.
  </Card>
</CardGroup>

<div id="managed-three-tier-ci">

For the explicit all-tier opt-in and its prerequisites, see [Advanced CI](/skills/skillevaluator/advanced-ci).
The [managed CI reference](https://github.com/NVIDIA/SkillEvaluator/blob/main/ci/templates/all-tiers/README.md#managed-three-tier-ci) preserves the detailed contracts.

</div>