Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
---
title: "Contributing"
description: "Set up a SkillEvaluator development environment, run the test suite, preview the docs locally, and submit a pull request that passes review."
---
This guide gets you from a fresh clone to a mergeable pull request. You need Python 3.12, 3.13, or 3.14 and Git; no API keys are required for the default development loop.
## Set up a development environment
<Steps>
### Clone the repository
```bash title="Clone SkillEvaluator"
git clone https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator
```
### Install with all extras
```bash title="Sync the dev environment"
uv sync --python 3.13 --all-extras
```
Install [uv](https://docs.astral.sh/uv/getting-started/installation/) first.
uv creates `.venv`; either activate it using the command for your shell, or
run Make through uv so `python3` resolves to that environment:
```bash
uv run make lint
uv run make test
uv run make build
```
Make is an additional dependency. Without Make, the equivalent commands are `uv run python -m ruff check src tests`,
`uv run python -m pytest`, and `uv run python -m build` instead.
If you prefer pip, `make install` runs the equivalent editable install:
```bash title="pip alternative"
pip install -e '.[all,dev]'
```
### Run the contribution gate
These three commands must pass locally before you open a pull request:
```bash title="The contribution gate"
uv run make lint && uv run make test && uv run make build
```
</Steps>
The Makefile wraps the standard tools, so you can run the underlying commands directly if you prefer:
| Target | Runs | Purpose |
| --- | --- | --- |
| `make install` | `pip install -e '.[all,dev]'` | Editable install with every extra plus dev tooling |
| `make test` | `pytest` | Default (fast) test suite |
| `make lint` | `ruff check src tests` | Lint the source and tests |
| `make build` | `python -m build` | Build the wheel and sdist |
| `make clean` | removes `build`, `dist`, and tool caches | Reset build state |
## Running tests
`make test` runs the fast suite. Two pytest markers are excluded by default, so slow and credentialed tests never block the everyday loop:
[Harbor](https://github.com/harbor-framework/harbor) is the open-source agent
evaluation framework used by Tier 3; the `live` marker exercises its in-process
backend.
The POSIX same-inode read-race test explicitly advances the modified file's
mtime: two same-size writes can otherwise share a filesystem clock tick.
| Marker | What it covers | Run it with |
| --- | --- | --- |
| *(none)* | Fast unit tests — the default selection | `pytest` |
| `integration` | End-to-end integration tests; may need fixtures or longer runtime | `pytest -m integration` |
| `live` | Live tests against the in-process Harbor backend; requires the `tier3` extra and `NVIDIA_API_KEY` | `pytest -m live` |
The MCP-staging and legacy-summary restored-mtime regressions model an advanced
ctime explicitly, so their source-change assertions do not depend on the host
clock ticking.
The directory-snapshot race regression explicitly advances directory mtime when
inserting its late entry, rather than assuming creation crosses a clock tick.
Git URL-rewrite tests point `GIT_CONFIG_GLOBAL` at their temporary configuration,
so caller-level Git isolation does not suppress the rewrite being tested.
The `dev` extra includes pytest-xdist, so you can parallelize any of these runs:
```bash title="Run tests in parallel"
pytest -n auto
```
### Check the HTML report in a browser
The development extra includes Playwright. The dedicated browser suite checks
all three tiers, Tier 3 subtabs, keyboard navigation, themes, JSON downloads,
offline charts, print, and narrow layouts using synthetic reports. Regular
pytest runs skip browser checks unless explicitly enabled:
```bash title="Run the HTML report browser checks"
python -m playwright install chromium
SKILLEVALUATOR_BROWSER_TESTS=1 pytest tests/reporting/test_report_browser.py -q
```
Set `SKILLEVALUATOR_BROWSER_SCREENSHOTS` to a directory to retain screenshots.
To use an installed Chrome instead of Playwright's Chromium, set
`SKILLEVALUATOR_TEST_BROWSER` to its executable. The convenience command
`python tests/reporting/browser_navigation_checks.py browser-checks` runs the
same suite and retains screenshots plus a test-result XML file. The dedicated
HTML browser CI job builds a wheel and runs the suite outside the checkout
against that installed wheel, so missing packaged templates and chart assets
fail the same checks.
## Documentation and starter changes
The [terminal starter](/skills/skillevaluator/starter-demo) uses a public skill, task and input with
real CLI output checks. For changes to it, run:
```bash
uv run python scripts/demo_starter.py --output reports/contribution-demo
uv run pytest tests/test_demo_starter.py
uv run ruff check scripts/demo_starter.py tests/test_demo_starter.py
```
Choose a new output directory for each replay. To verify the released route,
also run the pinned command in the starter guide. Report the evaluator version,
platform and exact commands tested in your pull request. Keep generated reports
out of commits. Describe seeded failures and incomplete evidence honestly; do
not reuse historical scores as a fresh run or silently change expectations to
make a check green. A fresh-user review should ask someone unfamiliar with the
tool to explain the passing, regressed and incomplete outputs before showing
the walkthrough's answers, and record what confused them in the pull request.
The focused test intentionally runs the checkout's evaluator. When the
evaluator is the checkout's own source, the runner checks the `checkout` scores
in `examples/starter/expected.json` instead of the pinned release's scores. A
scoring change on main updates those `checkout` scores; the release scores and
the starter guide change only when the pinned release does. Review both results
before changing expectations.
## Code style and dependencies
Ruff owns linting — `make lint` is the project's lint gate. CI runs `ruff check` over the whole repository, so lint the files you touch even outside `src` and `tests`.
Linting is only the first gate. CI also runs an OSS source-boundary scan (`python scripts/check_oss_boundary.py --root . --allowlist config/oss_boundary_allowlist.json`), which fails closed if the source tree or built distributions contain private integrations. The packaging job repeats that scan over the wheel and sdist, installs the wheel into a clean environment, and smoke-tests the CLI — including `skillevaluator models --help` and a `doctor` run with placeholder `nv_build` credentials. Platform jobs round out the matrix: a macOS runner executes the Tier 3 local-mode, progress, and preflight test files, and a native-Windows runner asserts local mode fails closed, with `doctor` required to exit non-zero for every `SKILLEVALUATOR_LOCAL_SANDBOX` value.
Follow the existing Python typing and test patterns, keep changes scoped, and add regression coverage for behavior changes.
For dependency changes: update `pyproject.toml`, `uv.lock`, and [THIRD_PARTY_NOTICES.md](https://github.com/NVIDIA/SkillEvaluator/blob/main/THIRD_PARTY_NOTICES.md) together — the notices file lists every direct dependency and its license, and `uv.lock` records the exact resolved set for the release.
## Work on the docs
The documentation site is built with [Fern](https://buildwithfern.com/) from the MDX pages in `docs/`. You only need a local build when you are editing docs; publishing happens automatically from `main`.
<Steps>
### Install the Fern CLI
Requires Node.js 22+ and npm 10+. `fern/package-lock.json` pins the CLI and every package
below it; `npm ci` reproduces that tree exactly, and the flags keep npm from running any
install-time scripts. This is the same install CI performs.
```bash title="Install the pinned fern-api"
npm ci --prefix fern --ignore-scripts --omit=optional
```
### Preview with live reload
```bash title="Local docs preview"
./fern/node_modules/.bin/fern docs dev
```
The site serves at `http://localhost:3000` and reloads on changes to `.mdx` files or `fern/docs.yml`.
### Validate before pushing
```bash title="Check config and links"
./fern/node_modules/.bin/fern check
```
`fern check` must pass — it is the same gate the site build relies on.
</Steps>
New pages must be registered under `navigation` in [fern/docs.yml](https://github.com/NVIDIA/SkillEvaluator/blob/main/fern/docs.yml), or they will not appear on the site. The full authoring rules — MDX gotchas, link conventions, the page-adding checklist — live in [docs/AGENTS.md](https://github.com/NVIDIA/SkillEvaluator/blob/main/docs/AGENTS.md).
## Submit a change
Contributions flow through GitHub pull requests:
1. **Discuss substantial changes in an issue first**, before you invest in an implementation.
2. **Keep each pull request focused.** Use the pull request template and explain the user-visible behavior and the verification you performed.
3. **Add focused tests** for behavioral changes.
4. **Update [CHANGELOG.md](https://github.com/NVIDIA/SkillEvaluator/blob/main/CHANGELOG.md)** when the change affects users.
5. **Keep provider credentials, customer data, and private benchmark material out of the repository.** Tier 3 live evaluation reads provider and agent credentials from your environment, never from the repo (with NVIDIA Build, one `NVIDIA_API_KEY` covers both roles) — see [Providers & Credentials](/skills/skillevaluator/configuration).
<Warning>
Every commit needs a Developer Certificate of Origin sign-off — commit with `git commit -s`. Pull requests containing unsigned commits are not accepted. By contributing, you confirm you have the right to submit the work under the Apache License 2.0.
</Warning>
## Project resources
| Resource | What it covers |
| --- | --- |
| [CONTRIBUTING.md](https://github.com/NVIDIA/SkillEvaluator/blob/main/CONTRIBUTING.md) | Full contribution policy, including the DCO text |
| [CODE_OF_CONDUCT.md](https://github.com/NVIDIA/SkillEvaluator/blob/main/CODE_OF_CONDUCT.md) | Contributor Covenant code of conduct |
| [GOVERNANCE.md](https://github.com/NVIDIA/SkillEvaluator/blob/main/GOVERNANCE.md) | Project governance — maintained by NVIDIA |
| [SECURITY.md](https://github.com/NVIDIA/SkillEvaluator/blob/main/SECURITY.md) | Report vulnerabilities privately via NVIDIA PSIRT — the [submission form](https://www.nvidia.com/object/submit-security-vulnerability.html) or [email protected] — never through public GitHub issues |
| [SUPPORT.md](https://github.com/NVIDIA/SkillEvaluator/blob/main/SUPPORT.md) | Support level: Experimental — best-effort via GitHub Issues, no SLA |
| [LICENSE](https://github.com/NVIDIA/SkillEvaluator/blob/main/LICENSE) | Apache License 2.0 |
| [THIRD_PARTY_NOTICES.md](https://github.com/NVIDIA/SkillEvaluator/blob/main/THIRD_PARTY_NOTICES.md) | Third-party license notices |
| [CHANGELOG.md](https://github.com/NVIDIA/SkillEvaluator/blob/main/CHANGELOG.md) | User-visible changes per release |
| [CITATION.cff](https://github.com/NVIDIA/SkillEvaluator/blob/main/CITATION.cff) | How to cite SkillEvaluator |
If you use SkillEvaluator in academic or technical work, cite the methodology
paper listed under `preferred-citation` in `CITATION.cff`, and record the
SkillEvaluator version you used for reproducibility:
```yaml title="CITATION.cff (excerpt)"
cff-version: 1.2.0
title: "SkillEvaluator"
type: software
version: "1.0.1"
preferred-citation:
type: article
title: "Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills"
year: 2026
doi: "10.48550/arXiv.2608.20614"
url: "https://arxiv.org/abs/2608.20614"
```