Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

Pin SkillSpector and verify the Gitleaks download in the CI recipe (#630)

Christopher Kevinbcbd770f7c53
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 94c0ca58..679df936 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -158,6 +158,9 @@ All notable changes to SkillEvaluator are documented in this file.
### Changed
+- The CI guide's recipe installs SkillSpector from the commit behind its release
+ tag and checks the Gitleaks archive against the published SHA-256 before
+ unpacking it, matching the repository's own workflows.
- The `nv_build` default model moves from `nvidia/nemotron-3-super-120b-a12b` to
`nvidia/nemotron-3.5-lightning-30b-a3b`. NVIDIA Build retired the hosted endpoint
for the Super model, so a run with no `llm-model` set answered HTTP 404 and the
diff --git a/docs/ci-integration.mdx b/docs/ci-integration.mdx
index 9dae469e..5ea06026 100644
--- a/docs/ci-integration.mdx
+++ b/docs/ci-integration.mdx
@@ -74,13 +74,20 @@ jobs:
- name: Install external Python scanners
run: |
uv tool install semgrep
- uv tool install git+https://github.com/NVIDIA/[email protected]
+ uv tool install git+https://github.com/NVIDIA/SkillSpector.git@54824463814a0be01360792523ec7245a69190fc # v2.13.4
- name: Install Gitleaks
run: |
GITLEAKS_VERSION=8.30.0
- curl -sSfL "https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz" \
- | tar -xz gitleaks
+ # SHA-256 of the linux_x64 archive, from the release's checksums.txt.
+ # Update it together with GITLEAKS_VERSION.
+ GITLEAKS_SHA256=79a3ab579b53f71efd634f3aaf7e04a0fa0cf206b7ed434638d1547a2470a66e
+ base="https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}"
+ archive="gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz"
+ cd "$RUNNER_TEMP"
+ curl -sSfL --retry 3 "${base}/${archive}" -o "${archive}"
+ printf '%s %s\n' "${GITLEAKS_SHA256}" "${archive}" | sha256sum -c -
+ tar -xzf "${archive}" gitleaks
sudo mv gitleaks /usr/local/bin/
- name: Validate skills
@@ -155,6 +162,10 @@ jobs:
Each action is pinned to a commit SHA, with its release in a comment, as in this
repository's own workflows, so a moved tag cannot change what the job runs.
To update an action, replace the SHA with the commit of the newer release.
+The same holds for the scanners: SkillSpector installs from the commit behind
+its release tag, and the Gitleaks archive is checked against the SHA-256 from
+the release's `gitleaks_<version>_checksums.txt` before it is unpacked. To
+update either, take the new commit or checksum from the release page.
SkillEvaluator installs Bandit, and the `security` extra adds pip-audit. Semgrep and SkillSpector
stay in separate uv tool environments, while Gitleaks is installed from its
diff --git a/tests/test_oss_packaging.py b/tests/test_oss_packaging.py
index 90b9d821..1725d3b0 100644
--- a/tests/test_oss_packaging.py
+++ b/tests/test_oss_packaging.py
@@ -987,7 +987,18 @@ def test_skillspector_install_instructions_pin_one_release() -> None:
flags=re.MULTILINE,
)
- assert install_specs == {f"git+https://github.com/NVIDIA/SkillSpector.git@{SKILLSPECTOR_RELEASE}"}
+ tag_spec = f"git+https://github.com/NVIDIA/SkillSpector.git@{SKILLSPECTOR_RELEASE}"
+ commit_spec = f"git+https://github.com/NVIDIA/SkillSpector.git@{SKILLSPECTOR_RELEASE_COMMIT}"
+ ci_recipe_specs = set(
+ re.findall(
+ r"git\+https://github\.com/NVIDIA/SkillSpector\.git[^\s`\"]*",
+ (REPO_ROOT / "docs" / "ci-integration.mdx").read_text(encoding="utf-8"),
+ )
+ )
+ # The copy-paste CI recipe pins the release commit so a moved tag cannot change what a job installs;
+ # everything else pins the release tag. Both name the same release.
+ assert ci_recipe_specs == {commit_spec}
+ assert install_specs == {tag_spec, commit_spec}
assert archive_refs == {SKILLSPECTOR_RELEASE_COMMIT}
assert notice_rows == [(SKILLSPECTOR_RELEASE_COMMIT[:7], SKILLSPECTOR_RELEASE_COMMIT)]