Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
AGENTS.md — docs/
Guidance for AI agents editing SkillEvaluator's long-form documentation. This
folder holds MDX pages published to https://docs.nvidia.com/skills/skillevaluator via
Fern. Human-facing notes live in
README.md; this file is the machine-facing companion.
What lives here
index.mdx— landing page and three-tier overview (Fern slug/)quickstart.mdx— check a bundled example with the installed CLIstarter-demo.mdx— reproducible public terminal starter, output checks and adaptationinstallation.mdx— shared pip, uv, and GitHub installation paths and verificationinstallation-options.mdx— upgrades, Docker, release differences, extras, and scanner setupconfiguration.mdx— providers, embeddings, and Tier 3 credential rolesagent-skill.mdx— installing and using theskills/skillevaluatoragent skilltier1-validation.mdx— first static run, report, and next stepstier1-reference.mdx— detailed checks, policies, scanners, and exit semanticstier2-deduplication.mdx— choose a scope, run deduplication, and read findingstier2-reference.mdx— chunking, catalogs, thresholds, algorithms, and limitstier3-live-evaluation.mdx— first Tier 3 comparison; keep this task-focusedprepare-dataset.mdx— author, generate, review, and validate cases before trialstier3-run-options.mdx— choose an agent and attempt count for reviewed casestier3-run-reference.mdx— staging contracts, run settings, budgets, and repeat comparisonsnemo-gym.mdx— NeMo Gym setup, first evaluation, supported agents, and optional referencetier3-runtime.mdx— backend compatibility, runtime contracts, and troubleshootingtier3-grading.mdx— judges, evidence budgets, retries, and fallback policiestier3-results.mdx— scores, applicability, verdicts, and incomplete evidencetier3-experiments.mdx— skill interaction and activation experimentsgrader-validation.mdx— independent-label qualification of the shipped verifiereval-datasets.mdx— theevals/contract, dataset generation, run policyagents-and-sandboxes.mdx— agents, environment backends, local mode, readinesscustom-graders.mdx— bring-your-own graders and native Harbor tasksreports.mdx— report formats, scores, and the results-on-disk contractmetrics-reference.mdx— every metric's formula, range, units, evidence, and validation statusperformance-dashboard.mdx— compare saved Tier 3 scores and resource usebenchmark-rollout.mdx— regenerating and promotingBENCHMARK.mdcardsgithub-actions.mdx— skill check workflow setup and optional hosted PR feedbackcli-onboarding.mdx— released GitHub setup wizardrepository-integration.mdx— choose GitHub or GitLab for skill checks; pre-commit hooksgitlab-ci.mdx— reusable skill checks and downloadable GitLab reportsadvanced-ci.mdx— secondary opt-in prerequisites and all-tier referenceci-integration.mdx— exit codes, GitHub Actions recipe, progressive adoptioncli-reference.mdx— every command and flagenvironment-variables.mdx— variable-by-variable inventorydeveloper-guide.mdx— contributor setup, tests, and docs workflowassets/— images referenced by the pagesREADME.md(human authoring guide) and thisAGENTS.mdare not published pages — do not add them to the navigation
Navigation order, page titles, and slugs are defined in
../fern/docs.yml. The Fern CLI version is pinned in
../fern/fern.config.json.
Rules for editing
Keep entry pages focused on the reader's next action: setup, a runnable command, the expected output, and a next step. Put advanced contracts and edge cases in the linked reference page. Use Steps for ordered actions, Tabs for exclusive choices, and Accordions for optional detail; do not put a reference chapter ahead of the first runnable example.
Keep pip, uv, uv bootstrap, and direct repository installation in
installation.mdx; keep upgrades and advanced setup in
installation-options.mdx. Default to the lightweight skillevaluator install;
keep skillevaluator[all] as an explicit opt-in for all tiers. Distinguish the published PyPI release from
unreleased GitHub features and dependencies. Tier guides and the
quickstart link there and use the installed skillevaluator command. Avoid
copying package-manager recipes into each tier or mixing uv run, uv tool install, and pip in a single first-run workflow. Contributor checkout commands
belong in the development guide.
Every page needs YAML frontmatter with
titleanddescription. Thetitlerenders as the page H1 — do not also add a# Headingin the body, or the title appears twice.--- title: "Configuration" description: "One-sentence summary used for SEO and the nav preview." ---File names use URL best practices: lowercase, hyphen-separated, no spaces or underscores (
tier1-validation.mdx, notTIER1_VALIDATION.mdx). Keep the file name equal to the pageslugindocs.yml.Adding a page: create the
.mdxfile, then register it undernavigationin../fern/docs.yml. A page not listed there will not appear on the site. Rename viagit mvto preserve history, and update every reference (see rule 5).This is MDX, not Markdown. A bare
<starts a JSX tag and{starts an expression. Keep angle brackets and braces (e.g.<placeholder>,${VAR}) inside backtick code spans or fenced code blocks, where MDX leaves them literal. Do not use<https://...>autolink syntax — write a normal[text](url)link.Links:
- In published MDX, use site paths such as
/skills/skillevaluator/configurationor/skills/skillevaluator/tier1-validation#anchor. The pinned Fern 5.121.0 preview returns HTTP 400 for file-relative.mdxpage URLs; it does not resolve them to published slugs. - For repository-only files outside
docs/, use full GitHub source URLs in published MDX. Keep repository Markdown, images and YAMLpath:entries relative to their source files. - When renaming a page, grep the whole repo and fix references in the sibling
pages, the root
../README.md,README.md, and../fern/docs.yml.
- In published MDX, use site paths such as
Verify before committing
Requires Node.js 22+. Install the pinned CLI from the committed lockfile with
npm ci --prefix fern --ignore-scripts --omit=optional.
./fern/node_modules/.bin/fern check # validate docs.yml config and all links — must pass
./fern/node_modules/.bin/fern docs dev # optional live preview at http://localhost:3000
fern check must pass; it is the same gate the site build relies on. Publishing
is automatic from main via ../.github/workflows/publish-docs.yml
— do not run fern generate locally.
Repo conventions that still apply
- Sign off commits (
git commit -s); the DCO check rejects unsigned commits (see../CONTRIBUTING.md). - Update
../CHANGELOG.mdwhen a docs change is user-visible.