Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

docs(ci): say the first Tier 1 run installs SkillSpector and Gitleaks (#669)

Narendran Raghavan2d0660ae8b9c
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 18ad0f4a..b81728a7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -323,6 +323,11 @@ All notable changes to SkillEvaluator are documented in this file.
(it passed before) and checks every skill's report in a folder run.
- The CI guide's GitHub Actions recipe pins each action to a commit SHA, as the
repository's own workflows do.
+- The CI guide's troubleshooting note for an `incomplete` status no longer
+ says SkillEvaluator never installs the scanners. The first Tier 1 run
+ downloads pinned SkillSpector and Gitleaks releases when they are missing, so
+ the note now points to network access, and it says a missing Semgrep counts
+ only when `SKILLEVALUATOR_REQUIRE_SEMGREP=1` is set.
- "Evaluate Skills from Your Agent" is shorter and visual: a how-it-works diagram,
install steps that start with `npx skills add NVIDIA/SkillEvaluator` (with Claude
diff --git a/docs/ci-integration.mdx b/docs/ci-integration.mdx
index 5ea06026..a622971f 100644
--- a/docs/ci-integration.mdx
+++ b/docs/ci-integration.mdx
@@ -347,10 +347,14 @@ A single-skill target writes its report at the top of `reports/`, and a folder t
<Accordion title="overall_status is 'incomplete'">
A required scanner is missing or returned no evidence. Check the report for
the named scanner: Semgrep, SkillSpector, and Gitleaks are all separate
- executables, and installing SkillEvaluator alone does not install them. The
- scanners fail closed: `validate` exits non-zero and the report stays
- non-green until every required scanner is installed and returns valid
- evidence. See the workflow above or
+ executables. When SkillSpector or Gitleaks is not installed, the first
+ Tier 1 run downloads its pinned, checksum-verified release (Gitleaks on
+ Linux and macOS only), so the runner needs network access to GitHub, and
+ to PyPI for SkillSpector. Semgrep is optional and never downloaded; a
+ missing Semgrep makes the run incomplete only when
+ `SKILLEVALUATOR_REQUIRE_SEMGREP=1` is set. The scanners fail closed:
+ `validate` exits non-zero and the report stays non-green until every
+ required scanner runs and returns valid evidence. See the workflow above or
[Installation: System tools](/skills/skillevaluator/installation#system-tools).
</Accordion>
<Accordion title="Claude answers HTTP 400 'is not a valid fallback target'">