Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Copy a CI template
Bring SkillEvaluator to your repository by copying a configuration file. You do not need to install SkillEvaluator locally, use an onboarding CLI, or copy evaluation code into your project. GitHub and GitLab templates are maintained together in this public NVIDIA/SkillEvaluator repository.
| Platform and result | Source template | Where to put it |
|---|---|---|
| GitHub: checks and downloadable reports | skill-checks.yml | .github/workflows/skill-checks.yml |
| GitHub: also update the standard PR comment, without hosting | comment-skill-checks.yml | .github/workflows/comment-skill-checks.yml, alongside the checks file |
| GitHub: also host reports and update a PR comment | publish-skill-reports.yml | .github/workflows/publish-skill-reports.yml, alongside the checks file |
| GitHub: optional trusted manual LLM checks | skill-checks-llm.yml | .github/workflows/skill-checks-llm.yml |
| GitLab: checks and downloadable reports | gitlab-skill-checks.yml | .gitlab-ci.yml, or merge into your existing root configuration |
Open a source file to inspect it; GitHub's Raw view lets you copy its full contents. See example results without needing access to a separate demo repository.
Get a version you can run
These source templates contain placeholders. Copy the ready-to-use configurations instead; their reviewed workflow commits and container image digest are already filled in. The 1.0.1 bundle includes the comments-only workflow; the archived 0.9.0 bundle does not. Default CLI feedback still configures the hosted publisher. For comments only, copy the versioned workflow and follow the comment setup.
Open a file in that folder and use GitHub's Raw view to copy its complete contents. The generated files and their checksums are kept in this public repository. Future releases attach the same files to the single SkillEvaluator product release, alongside its Python packages.
If you customize a source template instead, use the immutable workflow/image
identities in the generated bundle's release.json. Do not leave placeholders
or substitute main or latest for runtime pins.
Maintainers can release a reviewed feedback workflow update while keeping the verified
scanner workflow and image unchanged. Pass its full commit SHA as
--publisher-ref to ci/render_templates.py; otherwise it uses --workflow-ref.
That revision must contain both comment-skill-checks.yml and
publish-skill-reports.yml. The release receipt records both revisions. This
changes the feedback workflows and tools only; the scanner pin and approved
evaluator identity stay unchanged.
Customize and commit
- GitHub: set
pathsand optionallymin-score(default70). Changebranches: [main]if needed, then commit the workflow to your default branch. Existing workflows remain separate. Follow the GitHub guide. - GitLab: set
SKILL_EVAL_PATHSand optionallySKILL_EVAL_MIN_SCORE(default70). For an existing pipeline, merge the include, variables and job rules; preserve your other jobs andworkflow: rules. The job usestestby default, so override its stage if your pipeline does not define it. Follow the GitLab guide. - Optional GitHub comments: choose the
comments-only setup or the
hosted publisher setup.
Uncomment
id-token: writein the scan file for signed execution receipts, then approve that file's new blob. Copying the second file alone does not enable comments. - Optional LLM checks: configure a provider key in the separate manual workflow. The default checks need no model key.
Open a pull request or merge request that changes a skill to see the checks. Enable the platform's CI service, allow the referenced workflows or includes, and provide a compatible runner. The platform guides cover results and merge-gating settings.
Advanced CI
The explicitly enabled all-tier CLI, already included in 0.9.0, has separate setup and maintainer instructions. Released templates above provide quality and security checks without a model key.