Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Documentation
Long-form documentation for SkillEvaluator. Pages are authored in MDX and
published to https://docs.nvidia.com/skills/skillevaluator. This page is the human
quickstart; AGENTS.md holds the detailed authoring rules (MDX
gotchas, link conventions, page-adding checklist) for both AI agents and
contributors making non-trivial edits.
Layout
index.mdx— landing page and three-tier overviewquickstart.mdx— check a bundled example with the installed CLIinstallation.mdx— shared pip, uv, and GitHub installation paths and verificationinstallation-options.mdx— upgrades, Docker, release differences, extras, and scanner setupconfiguration.mdx— providers, embeddings, and Tier 3 credential rolestier1-validation.mdx— first static run, report, and next stepstier1-reference.mdx— detailed checks, policies, scanners, and exit semanticstier2-deduplication.mdx— choose a scope, run deduplication, and read findingstier2-reference.mdx— chunking, catalogs, thresholds, algorithms, and limitstier3-live-evaluation.mdx— first live comparison, from model setup to reportprepare-dataset.mdx— author, generate, review, and validate cases before trialstier3-run-options.mdx— choose an agent and attempt count for reviewed casestier3-run-reference.mdx— staging contracts, run settings, budgets, and repeat comparisonsnemo-gym.mdx— NeMo Gym setup, first evaluation, supported agents, and optional referencetier3-runtime.mdx— runtime contracts, backend compatibility, and troubleshootingtier3-grading.mdx— judges, evidence budgets, retries, and fallback modelstier3-results.mdx— scores, applicability, verdicts, and incomplete resultstier3-experiments.mdx— skill interactions and activation validationeval-datasets.mdx— theevals/contract, dataset generation, run policyagents-and-sandboxes.mdx— agents, environment backends, local mode, readinesscustom-graders.mdx— bring-your-own graders and native Harbor tasksreports.mdx— report formats, scores, and the results-on-disk contractperformance-dashboard.mdx— compare saved Tier 3 scores and resource usebenchmark-rollout.mdx— regenerating and promotingBENCHMARK.mdcardsgithub-actions.mdx— reusable skill checks and optional hosted PR feedbackcli-onboarding.mdx— install the CLI, automate GitHub setup and open an optional test PRrepository-integration.mdx— choose GitHub or GitLab for skill checks; pre-commit hooksgitlab-ci.mdx— reusable skill checks and downloadable GitLab reportsadvanced-ci.mdx— secondary opt-in prerequisites and all-tier referenceci-integration.mdx— exit codes, GitHub Actions recipe, progressive adoptioncli-reference.mdx— every command and flagenvironment-variables.mdx— variable-by-variable inventorydeveloper-guide.mdx— contributor setup, tests, and docs workflow
Navigation order and slugs are defined in ../fern/docs.yml.
Build Locally
You only need to build locally if you are editing the documentation. The site is
published automatically from main via the Fern GitHub integration.
Prerequisites: Node.js 22+ and npm 10+ (the versions the Fern CLI requires).
# Install the pinned Fern CLI from the committed lockfile
npm ci --prefix fern --ignore-scripts --omit=optional
# From the repo root, preview the site with live reload
./fern/node_modules/.bin/fern docs dev
# Validate the docs configuration and links
./fern/node_modules/.bin/fern check
fern docs dev serves the site at http://localhost:3000 and reloads on changes
to .mdx files or fern/docs.yml.
Authoring Notes
- New pages must be added to the
navigationsection of../fern/docs.yml; otherwise they will not appear in the site. - In published MDX pages, use site paths such as
/skills/skillevaluator/configurationfor other documentation pages. File-relative.mdxlinks are not resolved by Fern; see the official link format. The Fern 5.121.0 local preview returned HTTP 400 for those.mdxpage URLs. Keep images, YAMLpath:values and repository-only Markdown links relative to their source files. - The Fern version is pinned in
../fern/fern.config.json; bump it intentionally rather than as a side effect.