Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
docs: clarify README launch overview and NVIDIA Build quickstart (#33)
Christopher Kevin82e1194ee627
diff --git a/.github/workflows/dco.yml b/.github/workflows/dco.ymlnew file mode 100644index 00000000..d0348701--- /dev/null+++ b/.github/workflows/dco.yml@@ -0,0 +1,56 @@+# SPDX-License-Identifier: Apache-2.0+# Copyright (c) 2026 NVIDIA Corporation. All rights reserved.+#+# Verify every commit in a pull request carries a Signed-off-by+# trailer per the project's DCO requirement (see CONTRIBUTING.md).++name: DCO Check++on:+ pull_request:+ types: [opened, synchronize, reopened]++permissions:+ contents: read++jobs:+ dco:+ name: DCO+ runs-on: ubuntu-latest+ steps:+ - name: Checkout PR branch with full history+ uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0+ with:+ fetch-depth: 0+ ref: ${{ github.event.pull_request.head.sha }}++ - name: Verify DCO sign-off on every commit+ env:+ BASE_SHA: ${{ github.event.pull_request.base.sha }}+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}+ run: |+ set -euo pipefail++ unsigned=()+ while read -r sha; do+ if ! git show --format='%(trailers:key=Signed-off-by,valueonly)' -s "$sha" | grep -q .; then+ subject=$(git show --format='%s' -s "$sha")+ unsigned+=("$sha $subject")+ fi+ done < <(git log --no-merges --format='%H' "$BASE_SHA..$HEAD_SHA")++ if [ ${#unsigned[@]} -gt 0 ]; then+ echo "::error::Some commits in this PR are missing a Signed-off-by trailer."+ echo ""+ echo "Unsigned commits:"+ printf ' %s\n' "${unsigned[@]}"+ echo ""+ echo "To fix, run on your branch:"+ echo ""+ echo " git rebase --signoff origin/main && git push --force-with-lease"+ echo ""+ echo "See CONTRIBUTING.md for the full DCO requirement."+ exit 1+ fi++ echo "All commits signed off."diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 11e05e9b..e6c80918 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -6,6 +6,8 @@ All notable changes to SkillEvaluator are documented in this file.### Added+- CI DCO check that fails pull requests whose commits lack a `Signed-off-by`+ trailer, matching the sign-off requirement in `CONTRIBUTING.md`.- Initial public release candidate.- Enabled optional semantic-version validation in the default Tier 1 pipeline,including a public `--previous-version` monotonic-bump bound.diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 05fd69ca..911ae97a 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -25,7 +25,8 @@ you confirm that you have the right to contribute it under theunder the same license, or a compatible license.- Any contribution which contains commits that are not Signed-Off will not be- accepted.+ accepted. Pull requests are checked in CI by the **DCO** workflow; unsigned+ commits fail the check.- To sign off on a commit you simply use the `--signoff` (or `-s`) option whencommitting your changes:diff --git a/README.md b/README.mdindex 862bb061..c6700b83 100644--- a/README.md+++ b/README.md@@ -4,6 +4,7 @@[](LICENSE)[](https://www.python.org/)+[](https://docs.nvidia.com/skills/skillevaluator/)SkillEvaluator is an open-source, multi-tier framework for evaluating AI agentartifacts, starting with agent skills: deterministic quality gates, semantic@@ -12,11 +13,26 @@ overlap detection, synthetic eval dataset generation, and live agent evaluation.Agent skills are folders of instructions and supporting files that extend AIagents, as defined by the [Agent Skills specification](https://agentskills.io/).SkillEvaluator is part of the-[NVIDIA Verified Skills pipeline](https://docs.nvidia.com/skills/), with-[SkillSpector](https://github.com/NVIDIA/SkillSpector) providing the specialized-security-scanning capability used by Tier 1 and+[NVIDIA Verified Skills pipeline](https://github.com/NVIDIA/skills).++## Three-tier overview++++Tiers are independent entry points; nothing requires running earlier ones first.++| Tier | Purpose | Representative commands | Requires |+| --- | --- | --- | --- |+| Tier 1: Validation | Safe & well-formed? | `validate`, `quality-check`, `security-scan`, `pii-scan`, `lint-scripts`, `rubric-eval` | No API key for deterministic checks; the `security` extra plus external Semgrep, SkillSpector, and Gitleaks for full scanner coverage; a provider key for LLM checks |+| Tier 2: Deduplication | Overlap with what exists? | `context-optimization-check`, `similarity-check` | An embeddings provider; intra-skill analysis also needs a chat LLM — local OpenAI-compatible endpoints work |+| Tier 3: Live Evaluation | Does it help the agent? | `create-eval-dataset`, `tier3 evaluate`, `compare` | No credential for keyless templates and report inspection; a provider key for LLM generation and grading; live evaluation also needs the agent CLI with its credential and a Docker, local OS, or cloud sandbox |++[SkillSpector](https://github.com/NVIDIA/SkillSpector) provides specialized+security scanning for Tier 1 validation.[Harbor](https://github.com/harbor-framework/harbor), the open-source agent-evaluation framework, powering Tier 3 sandboxed agent evaluation.+evaluation framework, powers the sandboxed agent runs in Tier 3 live+evaluation. Full tier guides live in the+[documentation](https://docs.nvidia.com/skills/skillevaluator/).## Quickstart@@ -41,9 +57,14 @@ If your shell cannot find the command after installation, run## LLM provider setup-LLM-backed security analysis, rubric judging, Tier 2 deduplication, dataset-generation, and Tier 3 grading need a configured provider. For NVIDIA Build,-one API Catalog key covers both chat and embeddings:+No OpenAI or Anthropic key yet? Create a free API key at+[build.nvidia.com](https://build.nvidia.com) — NVIDIA Build offers free+inferencing, and NVIDIA Build defaults to the open-source Nemotron model+`nvidia/nemotron-3-nano-30b-a3b` for a quick try. Prefer a different model?+Pick any free model on [build.nvidia.com](https://build.nvidia.com) and set+`SKILL_EVAL_LLM_MODEL`. Once that key is set, the same provider works+seamlessly across Tier 1 LLM checks, Tier 2, and Tier 3 (chat plus embeddings+with one credential):```bashexport SKILL_EVAL_LLM_PROVIDER=nv_builddiff --git a/config/oss_boundary_allowlist.json b/config/oss_boundary_allowlist.jsonindex c292e7df..93cf47db 100644--- a/config/oss_boundary_allowlist.json+++ b/config/oss_boundary_allowlist.json@@ -52,7 +52,7 @@{"path": "tests/test_oss_packaging.py","rule": "harbor-upload-integration",- "line": 455,+ "line": 456,"reason": "Negative upload boundary assertion","expires": "2027-07-08","review_owner": "oss-release-reviewers"@@ -60,7 +60,7 @@{"path": "tests/test_oss_packaging.py","rule": "internal-nvidia-credential",- "line": 496,+ "line": 497,"reason": "Negative documentation boundary assertion","expires": "2027-07-08","review_owner": "oss-release-reviewers"diff --git a/docs/assets/three-tier-overview.svg b/docs/assets/three-tier-overview.svgnew file mode 100644index 00000000..69ba6924--- /dev/null+++ b/docs/assets/three-tier-overview.svg@@ -0,0 +1,116 @@+<?xml version="1.0" encoding="UTF-8"?>+<svg viewBox="0 0 1040 300" width="1040" height="300" xmlns="http://www.w3.org/2000/svg" role="img" aria-labelledby="tiers-title tiers-desc">+ <title id="tiers-title">SkillEvaluator three-tier evaluation pipeline</title>+ <desc id="tiers-desc">A skill flows through Tier 1 validation, Tier 2 deduplication, and Tier 3 live agent evaluation, producing reports and skill lift.</desc>+ <defs>+ <linearGradient id="bg" x1="0" y1="0" x2="1" y2="1">+ <stop offset="0%" stop-color="#0d1117"/>+ <stop offset="50%" stop-color="#101812"/>+ <stop offset="100%" stop-color="#0d1117"/>+ </linearGradient>+ <linearGradient id="card" x1="0" y1="0" x2="0" y2="1">+ <stop offset="0%" stop-color="#1a261c"/>+ <stop offset="100%" stop-color="#121a14"/>+ </linearGradient>+ <linearGradient id="accent1" x1="0" y1="0" x2="1" y2="0">+ <stop offset="0%" stop-color="#76b900"/>+ <stop offset="100%" stop-color="#76b900" stop-opacity="0.15"/>+ </linearGradient>+ <linearGradient id="accent2" x1="0" y1="0" x2="1" y2="0">+ <stop offset="0%" stop-color="#9bdb34"/>+ <stop offset="100%" stop-color="#9bdb34" stop-opacity="0.15"/>+ </linearGradient>+ <linearGradient id="accent3" x1="0" y1="0" x2="1" y2="0">+ <stop offset="0%" stop-color="#76b900"/>+ <stop offset="100%" stop-color="#b6ef4a" stop-opacity="0.35"/>+ </linearGradient>+ <linearGradient id="outCard" x1="0" y1="0" x2="0" y2="1">+ <stop offset="0%" stop-color="#243528"/>+ <stop offset="100%" stop-color="#162016"/>+ </linearGradient>+ <marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto">+ <path d="M1.2 1.2 L8.5 5 L1.2 8.8" fill="none" stroke="#6f9264" stroke-width="1.6" stroke-linecap="round" stroke-linejoin="round"/>+ </marker>+ </defs>++ <rect width="1040" height="300" rx="24" fill="url(#bg)" stroke="#273129" stroke-width="1"/>+ <circle cx="220" cy="160" r="100" fill="#76b900" fill-opacity="0.045"/>+ <circle cx="520" cy="130" r="120" fill="#76b900" fill-opacity="0.05"/>+ <circle cx="820" cy="170" r="110" fill="#9bdb34" fill-opacity="0.04"/>++ <text x="36" y="38" font-family="Inter, ui-sans-serif, -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif" font-size="12" font-weight="600" letter-spacing="2.6" fill="#6f9264">THREE-TIER PIPELINE</text>+ <text x="36" y="62" font-family="Inter, ui-sans-serif, -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif" font-size="20" font-weight="700" letter-spacing="-0.3" fill="#f0f4ee">From static gates to measured agent lift</text>++ <!-- Skill -->+ <g transform="translate(36 88)">+ <rect width="100" height="168" rx="16" fill="#152117" stroke="#304631" stroke-width="1"/>+ <circle cx="50" cy="44" r="17" fill="none" stroke="#39523a" stroke-width="3"/>+ <circle cx="50" cy="44" r="10.5" fill="none" stroke="#6f9264" stroke-width="2.4"/>+ <circle cx="50" cy="44" r="4.5" fill="#76b900"/>+ <text x="50" y="84" text-anchor="middle" font-family="Inter, ui-sans-serif, sans-serif" font-size="15" font-weight="700" fill="#ffffff">Skill</text>+ <text x="50" y="106" text-anchor="middle" font-family="ui-monospace, SFMono-Regular, Menlo, monospace" font-size="11" fill="#9bb08f">SKILL.md</text>+ <text x="50" y="128" text-anchor="middle" font-family="Inter, ui-sans-serif, sans-serif" font-size="10" fill="#6f7f68">+ scripts</text>+ </g>++ <line x1="144" y1="172" x2="170" y2="172" stroke="#6f9264" stroke-width="1.6" marker-end="url(#arrow)"/>++ <!-- Tier 1 -->+ <g transform="translate(178 88)">+ <rect width="210" height="168" rx="16" fill="url(#card)" stroke="#304631" stroke-width="1"/>+ <rect x="0" y="0" width="210" height="3" rx="1.5" fill="url(#accent1)"/>+ <rect x="16" y="18" width="48" height="22" rx="11" fill="#76b900" fill-opacity="0.16" stroke="#76b900" stroke-opacity="0.45" stroke-width="1"/>+ <text x="40" y="33" text-anchor="middle" font-family="Inter, ui-sans-serif, sans-serif" font-size="11" font-weight="700" letter-spacing="0.8" fill="#9bdb34">TIER 1</text>+ <text x="16" y="64" font-family="Inter, ui-sans-serif, sans-serif" font-size="17" font-weight="700" fill="#f3f7f0">Validation</text>+ <text x="16" y="88" font-family="Inter, ui-sans-serif, sans-serif" font-size="12.5" font-weight="600" fill="#dbe8d6">Safe & well-formed?</text>+ <text x="16" y="110" font-family="Inter, ui-sans-serif, sans-serif" font-size="11" fill="#8fa088">Schema · security · PII</text>+ <text x="16" y="126" font-family="Inter, ui-sans-serif, sans-serif" font-size="11" fill="#8fa088">quality · offline gates</text>+ <line x1="16" y1="136" x2="194" y2="136" stroke="#304631" stroke-width="1"/>+ <text x="16" y="155" font-family="ui-monospace, SFMono-Regular, Menlo, monospace" font-size="10" fill="#6f9264">validate · quality-check</text>+ </g>++ <line x1="396" y1="172" x2="422" y2="172" stroke="#6f9264" stroke-width="1.6" marker-end="url(#arrow)"/>++ <!-- Tier 2 -->+ <g transform="translate(430 88)">+ <rect width="210" height="168" rx="16" fill="url(#card)" stroke="#304631" stroke-width="1"/>+ <rect x="0" y="0" width="210" height="3" rx="1.5" fill="url(#accent2)"/>+ <rect x="16" y="18" width="48" height="22" rx="11" fill="#76b900" fill-opacity="0.16" stroke="#76b900" stroke-opacity="0.45" stroke-width="1"/>+ <text x="40" y="33" text-anchor="middle" font-family="Inter, ui-sans-serif, sans-serif" font-size="11" font-weight="700" letter-spacing="0.8" fill="#9bdb34">TIER 2</text>+ <text x="16" y="64" font-family="Inter, ui-sans-serif, sans-serif" font-size="17" font-weight="700" fill="#f3f7f0">Deduplication</text>+ <text x="16" y="88" font-family="Inter, ui-sans-serif, sans-serif" font-size="12.5" font-weight="600" fill="#dbe8d6">Already covered?</text>+ <text x="16" y="110" font-family="Inter, ui-sans-serif, sans-serif" font-size="11" fill="#8fa088">Embedding overlap</text>+ <text x="16" y="126" font-family="Inter, ui-sans-serif, sans-serif" font-size="11" fill="#8fa088">within & across skills</text>+ <line x1="16" y1="136" x2="194" y2="136" stroke="#304631" stroke-width="1"/>+ <text x="16" y="150" font-family="ui-monospace, SFMono-Regular, Menlo, monospace" font-size="9.5" fill="#6f9264">context-optimization-check</text>+ <text x="16" y="163" font-family="ui-monospace, SFMono-Regular, Menlo, monospace" font-size="9.5" fill="#6f9264">similarity-check</text>+ </g>++ <line x1="648" y1="172" x2="674" y2="172" stroke="#6f9264" stroke-width="1.6" marker-end="url(#arrow)"/>++ <!-- Tier 3 -->+ <g transform="translate(682 88)">+ <rect width="210" height="168" rx="16" fill="url(#card)" stroke="#39523a" stroke-width="1"/>+ <rect x="0" y="0" width="210" height="3" rx="1.5" fill="url(#accent3)"/>+ <rect x="16" y="18" width="48" height="22" rx="11" fill="#76b900" fill-opacity="0.22" stroke="#76b900" stroke-opacity="0.65" stroke-width="1"/>+ <text x="40" y="33" text-anchor="middle" font-family="Inter, ui-sans-serif, sans-serif" font-size="11" font-weight="700" letter-spacing="0.8" fill="#b6ef4a">TIER 3</text>+ <text x="16" y="64" font-family="Inter, ui-sans-serif, sans-serif" font-size="17" font-weight="700" fill="#f3f7f0">Live Evaluation</text>+ <text x="16" y="88" font-family="Inter, ui-sans-serif, sans-serif" font-size="12.5" font-weight="600" fill="#dbe8d6">Does it help?</text>+ <text x="16" y="110" font-family="Inter, ui-sans-serif, sans-serif" font-size="11" fill="#8fa088">Sandboxed A/B runs</text>+ <text x="16" y="126" font-family="Inter, ui-sans-serif, sans-serif" font-size="11" fill="#8fa088">with skill vs without</text>+ <line x1="16" y1="136" x2="194" y2="136" stroke="#304631" stroke-width="1"/>+ <text x="16" y="150" font-family="ui-monospace, SFMono-Regular, Menlo, monospace" font-size="9.5" fill="#6f9264">create-eval-dataset</text>+ <text x="16" y="163" font-family="ui-monospace, SFMono-Regular, Menlo, monospace" font-size="9.5" fill="#6f9264">tier3 evaluate · compare</text>+ </g>++ <line x1="900" y1="172" x2="922" y2="172" stroke="#6f9264" stroke-width="1.6" marker-end="url(#arrow)"/>++ <!-- Reports -->+ <g transform="translate(930 121)">+ <rect width="78" height="102" rx="14" fill="url(#outCard)" stroke="#76b900" stroke-opacity="0.5" stroke-width="1"/>+ <text x="39" y="40" text-anchor="middle" font-family="Inter, ui-sans-serif, sans-serif" font-size="12" font-weight="700" fill="#9bdb34">Reports</text>+ <text x="39" y="62" text-anchor="middle" font-family="Inter, ui-sans-serif, sans-serif" font-size="11" fill="#9bb08f">verdict</text>+ <text x="39" y="80" text-anchor="middle" font-family="Inter, ui-sans-serif, sans-serif" font-size="11" fill="#9bb08f">+ lift</text>+ </g>++ <text x="520" y="282" text-anchor="middle" font-family="Inter, ui-sans-serif, sans-serif" font-size="12" fill="#6f7f68">Each tier is an independent entry point — run any command directly</text>+</svg>diff --git a/docs/index.mdx b/docs/index.mdxindex fd666d77..6a79f50c 100644--- a/docs/index.mdx+++ b/docs/index.mdx@@ -6,7 +6,7 @@ layout: overviewSkillEvaluator is an open-source, multi-tier framework for evaluating AI agent artifacts, starting with agent skills: deterministic quality gates, semantic overlap detection, synthetic eval dataset generation, and live agent evaluation.-Agent skills are folders of instructions and supporting files that extend AI agents, as defined by the [Agent Skills specification](https://agentskills.io/). SkillEvaluator is part of the [NVIDIA Verified Skills pipeline](https://docs.nvidia.com/skills/). Tier 1 integrates [SkillSpector](https://github.com/NVIDIA/SkillSpector) for specialized security scanning; see [Installation](installation.mdx#system-tools) for the security setup.+Agent skills are folders of instructions and supporting files that extend AI agents, as defined by the [Agent Skills specification](https://agentskills.io/). SkillEvaluator is part of the [NVIDIA Verified Skills pipeline](https://github.com/NVIDIA/skills). Tier 1 integrates [SkillSpector](https://github.com/NVIDIA/SkillSpector) for specialized security scanning; see [Installation](installation.mdx#system-tools) for the security setup.<Note>SkillEvaluator's support level is **Experimental**: community-supported on a best-effort basis through [GitHub Issues](https://github.com/NVIDIA/SkillEvaluator/issues), with no SLA. See [SUPPORT.md](https://github.com/NVIDIA/SkillEvaluator/blob/main/SUPPORT.md).diff --git a/tests/test_oss_packaging.py b/tests/test_oss_packaging.pyindex 32ef42b3..8a06a0ee 100644--- a/tests/test_oss_packaging.py+++ b/tests/test_oss_packaging.py@@ -245,9 +245,10 @@ def test_public_readme_is_a_concise_docs_landing_page() -> None:)assert positioning in normalizedassert "https://docs.nvidia.com/skills/skillevaluator/" in readme- assert "https://docs.nvidia.com/skills/" in readme+ assert "https://github.com/NVIDIA/skills" in readmeassert "https://github.com/NVIDIA/SkillSpector" in readme- assert "\n## Three-tier overview\n" not in readme+ assert "\n## Three-tier overview\n" in readme+ assert "docs/assets/three-tier-overview.svg" in readmeassert "\n## Quickstart\n" in readmeassert "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git" in readmeassert (@@ -271,7 +272,7 @@ def test_public_readme_is_a_concise_docs_landing_page() -> None:assert "\n## Tier 1:" not in readmeassert "Skill Evaluator" not in readmeassert "Skillevaluator" not in readme- assert len(readme.split()) <= 750+ assert len(readme.split()) <= 1100def test_release_metadata_is_public_facing_and_version_consistent() -> None:@@ -547,7 +548,7 @@ def test_fern_docs_use_the_verified_skills_basepath_and_launch_positioning() ->assert "custom-domain: docs.nvidia.com/skills/skillevaluator" in fernassert "docs.nvidia.com/skillevaluator" not in fernassert positioning in normalized_overview- assert "https://docs.nvidia.com/skills/" in overview+ assert "https://github.com/NVIDIA/skills" in overviewassert "https://github.com/NVIDIA/SkillSpector" in overview