Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

Install Bandit with the base package (#487)

Christopher Kevin2bd0e7927605
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f3710154..76f8f0d4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -40,6 +40,14 @@ All notable changes to SkillEvaluator are documented in this file.
beside it and the failure is reported. After a timeout, the failed run still keeps
the scores of the arms that finished.
+### Changed
+
+- Install Bandit with the base package, so `pip install skillevaluator` or
+ `uv tool install skillevaluator` runs every default Tier 1 check without an
+ extra. pip-audit, used only by the opt-in dependency audit
+ (`--check dependency`), stays in the `security` extra. The Bandit install hint
+ now says to reinstall SkillEvaluator.
+
### Documentation
- A new Metrics Reference page defines every number the CLI, HTML report, JSON
diff --git a/docs/ci-integration.mdx b/docs/ci-integration.mdx
index 3c69f57f..2052f8d0 100644
--- a/docs/ci-integration.mdx
+++ b/docs/ci-integration.mdx
@@ -69,7 +69,7 @@ jobs:
python-version: "3.13"
- name: Install SkillEvaluator
- run: uv tool install --python 3.13 "skillevaluator[security] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
+ run: uv tool install --python 3.13 "skillevaluator @ git+https://github.com/NVIDIA/SkillEvaluator.git"
- name: Install external Python scanners
run: |
@@ -152,7 +152,7 @@ jobs:
});
```
-The `security` extra installs Bandit and pip-audit. Semgrep and SkillSpector
+SkillEvaluator installs Bandit, and the `security` extra adds pip-audit. Semgrep and SkillSpector
stay in separate uv tool environments, while Gitleaks is installed from its
official [releases](https://github.com/gitleaks/gitleaks/releases). Report
filenames embed a sortable timestamp (`skillevaluator-output-<timestamp>.md`),
@@ -327,7 +327,7 @@ jq -e '([.quality_summary[]?.overall_score] | min // 100) >= 80' "$report"
<Accordion title="overall_status is 'incomplete'">
A required scanner is missing or returned no evidence. Check the report for
the named scanner: Semgrep, SkillSpector, and Gitleaks are all separate
- executables, and the `security` extra alone does not install them. The
+ executables, and installing SkillEvaluator alone does not install them. The
scanners fail closed: `validate` exits non-zero and the report stays
non-green until every required scanner is installed and returns valid
evidence. See the workflow above or
diff --git a/docs/index.mdx b/docs/index.mdx
index 4f0198bd..075d42b7 100644
--- a/docs/index.mdx
+++ b/docs/index.mdx
@@ -77,7 +77,7 @@ Every tier is an independent entry point — run any command directly; nothing r
| Tier | Representative commands | Requires |
| --- | --- | --- |
-| [Tier 1](tier1-validation.mdx) | `tier1 PATH`, `quality-check`, `security-scan`, `pii-scan`, `lint-scripts`, `rubric-eval` | No API key for deterministic checks; the `security` extra plus external Semgrep, SkillSpector, and Gitleaks executables for full scanner coverage; a provider key for LLM checks |
+| [Tier 1](tier1-validation.mdx) | `tier1 PATH`, `quality-check`, `security-scan`, `pii-scan`, `lint-scripts`, `rubric-eval` | No API key for deterministic checks; external Semgrep, SkillSpector, and Gitleaks executables for full scanner coverage; a provider key for LLM checks |
| [Tier 2](tier2-deduplication.mdx) | `context-optimization-check`, `similarity-check` | An embeddings provider; intra-skill analysis also needs a chat LLM — local OpenAI-compatible endpoints work |
| [Tier 3](tier3-live-evaluation.mdx) | `create-eval-dataset`, `tier3 evaluate`, `compare` | No credential for keyless templates and report inspection; a provider key for LLM generation and grading; live evaluation also needs the agent CLI with its credential and a Docker, local OS, or cloud sandbox |
diff --git a/docs/installation.mdx b/docs/installation.mdx
index cd5a4c61..9e3ce0bb 100644
--- a/docs/installation.mdx
+++ b/docs/installation.mdx
@@ -23,8 +23,8 @@ This project will download and install additional third-party open source softwa
- Python 3.12, 3.13, or 3.14. `uv tool install`, `uv run`, and `uv sync` provision a supported interpreter automatically when passed `--python 3.13` (or `--python 3.14`); only the plain-pip path needs one on PATH.
- Git for installs from the repository and for the bundled-example checkout.
- Docker, only for the container install and Tier 3 Docker-mode evaluation.
-- No API key is needed for deterministic Tier 1 checks. Full default scanner
- coverage requires the `security` extra; Tier 1 provisions SkillSpector and
+- No API key is needed for deterministic Tier 1 checks. The base package
+ includes Bandit; Tier 1 provisions SkillSpector and
Gitleaks itself, and Semgrep is optional (see [System tools](#system-tools)). LLM-backed checks and Tiers 2–3 use a configured provider — see
[Providers & Credentials](configuration.mdx).
@@ -161,7 +161,7 @@ The image does not bundle the external Semgrep, SkillSpector, or Gitleaks execut
## Choosing extras
-The base package completes the schema, PII, license, quality, Unicode, script-lint, and hygiene checks on its own. `quality-check` needs no scanner setup. When you run the full static suite, scanner-backed checks report INCOMPLETE until the `security` extra and the required external scanner executables are installed. Running without an LLM or API key does not remove those scanner requirements. Everything else lives behind an extra:
+The base package completes the schema, PII, license, quality, Unicode, script-lint, and hygiene checks on its own. `quality-check` needs no scanner setup. When you run the full static suite, the base package also runs Bandit, and scanner-backed checks report INCOMPLETE until the required external scanners are available (see [System tools](#system-tools)). Running without an LLM or API key does not remove those scanner requirements. Everything else lives behind an extra:
[Harbor](https://github.com/harbor-framework/harbor) is the open-source agent
evaluation framework that the `tier3` extra installs for live task execution
@@ -172,7 +172,7 @@ and environment backends.
| `llm` | Shared LLM and embedding clients used by LLM-backed checks, Tier 2, and Tier 3 |
| `tier2` | Intra-skill deduplication and local-catalog inter-skill similarity; includes `llm` |
| `tier3` | Docker, local, and cloud live-agent evaluation through Harbor (pins `harbor==0.25.0` and requires `python-dotenv>=1.2.0`); includes `llm` |
-| `security` | Bandit and pip-audit |
+| `security` | pip-audit, for the opt-in dependency audit (Bandit is in the base package) |
| `all` | `tier2`, `tier3`, and `security` in one bundle |
| `dashboard` | Local Streamlit tables for comparing saved reports and retained runs (`skillevaluator dashboard`); not included in `all` |
| `dev` | Build, test, coverage, and formatting tools for contributors |
diff --git a/docs/quickstart.mdx b/docs/quickstart.mdx
index 74bd810b..1bc47be0 100644
--- a/docs/quickstart.mdx
+++ b/docs/quickstart.mdx
@@ -112,12 +112,12 @@ uv run --python 3.13 skillevaluator quality-check ./my-skill \
### Add full static validation
-Full static Tier 1 coverage needs the `security` extra and the separate
+Full static Tier 1 coverage needs the separate
Semgrep, SkillSpector, and Gitleaks executables. Follow
[scanner setup](installation.mdx#system-tools), then run:
```bash title="Run the static Tier 1 suite"
-uv run --python 3.13 --extra security skillevaluator tier1 ./my-skill --no-llm
+uv run --python 3.13 skillevaluator tier1 ./my-skill --no-llm
```
`--no-llm` disables model-backed stages; it does not remove scanner
diff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdx
index c014ebd9..b8cebc59 100644
--- a/docs/tier1-validation.mdx
+++ b/docs/tier1-validation.mdx
@@ -80,7 +80,7 @@ for skills. Its Tier 1 LLM checks still run only when requested.
skillevaluator validate ./my-skill --tiers 1
```
- With `skillevaluator[security]`, Semgrep, SkillSpector, and Gitleaks installed, every required scanner produces evidence and the run is deterministic — the same input yields the same exit code, with nothing leaving your machine. Run `skillevaluator doctor --tier 1` first to confirm that each scanner resolves and reports a supported version.
+ With Semgrep, SkillSpector, and Gitleaks installed, every required scanner produces evidence and the run is deterministic — the same input yields the same exit code, with nothing leaving your machine. Run `skillevaluator doctor --tier 1` first to confirm that each scanner resolves and reports a supported version.
### Narrow to specific checks
@@ -673,7 +673,7 @@ The command line rejects a malformed waiver file before the run starts, with exi
## Scanner setup
-Install `skillevaluator[security]` for the bundled Python scanners (Bandit and pip-audit). Install Semgrep and SkillSpector in separate tool environments, and install Gitleaks separately — it is a Go binary, not a pip package. See [Installation](installation.mdx) for commands and the extras table.
+Bandit installs with SkillEvaluator; install `skillevaluator[security]` for pip-audit, which the opt-in dependency audit uses. Install Semgrep and SkillSpector in separate tool environments, and install Gitleaks separately — it is a Go binary, not a pip package. See [Installation](installation.mdx) for commands and the extras table.
Semgrep uses a policy packaged with the installed distribution, with metrics and version checks disabled; Tier 1 never fetches a registry policy at runtime. The other public Tier 1 configuration also ships inside the package under `skillevaluator/config/`: `pii_patterns.yaml`, `unicode_smuggle_patterns.yaml`, `license_config.yaml`, and `profiles/default.yaml`.
@@ -773,7 +773,7 @@ Tier 1 checks always gate the exit code inside `validate`. Tier 2 gates when it
Install it with `brew install gitleaks` or download a binary from the official [Gitleaks releases](https://github.com/gitleaks/gitleaks/releases).
</Accordion>
<Accordion title="A scanner is missing">
- Install Bandit and pip-audit with `skillevaluator[security]`. Install Semgrep separately with `brew install semgrep` or `uv tool install semgrep`, and SkillSpector with `uv tool install git+https://github.com/NVIDIA/[email protected]`. Required scanner failures remain `INCOMPLETE` until the scanner is installed and returns valid evidence. `skillevaluator doctor --tier 1` lists every scanner the selected checks need, with its resolved path and version.
+ Bandit installs with SkillEvaluator; reinstall it if Bandit is missing. Install pip-audit with `skillevaluator[security]`. Install Semgrep separately with `brew install semgrep` or `uv tool install semgrep`, and SkillSpector with `uv tool install git+https://github.com/NVIDIA/[email protected]`. Required scanner failures remain `INCOMPLETE` until the scanner is installed and returns valid evidence. `skillevaluator doctor --tier 1` lists every scanner the selected checks need, with its resolved path and version.
</Accordion>
<Accordion title="SkillSpector reports incomplete analysis for a valid helper script">
SkillSpector releases before v2.13.1 can stop inspecting a valid Python or Perl helper early and record a `static_parse_limit`, `obfuscated_instruction_text`, or `runtime_limit` ledger exception. The security check is then `INCOMPLETE`, often with a High **Incomplete referenced artifact analysis (AE1)** finding at each line that names the helper. The finding's suggestion names the helper and the reason, such as `static_parse_limit (static_patterns_tool_misuse)`. Check the installed release with `skillspector --version` and upgrade with `uv tool install git+https://github.com/NVIDIA/[email protected]`. If the exception remains after the upgrade, investigate it like any other incomplete scan.
diff --git a/pyproject.toml b/pyproject.toml
index 1b22d585..4e1a28e8 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -38,6 +38,8 @@ license-files = [
# Base install is static-validation capable (Tier 1) and must NOT pull Harbor
# or LLM clients. Those live behind the extras below.
dependencies = [
+ # Tier 1 Code Risk Analysis scanner, which validate runs by default.
+ "bandit>=1.9.4",
"click>=8.3.3,<9",
"idna>=3.10,<4",
"jinja2>=3.1",
@@ -75,9 +77,8 @@ tier3 = [
# PYTHON_DOTENV_DISABLED, which keeps Harbor from loading .env files, needs 1.2.0.
"python-dotenv>=1.2.0",
]
-# Static security scanners (floors raised to the established release baseline).
+# Scanner for the opt-in Dependency Vulnerability Audit (`--check dependency`).
security = [
- "bandit>=1.9.4",
"pip-audit>=2.10.0",
]
# Verify GitHub-signed execution receipts for the optional CI report publisher.
diff --git a/src/skillevaluator/doctor.py b/src/skillevaluator/doctor.py
index 96b60b8f..7be5aeed 100644
--- a/src/skillevaluator/doctor.py
+++ b/src/skillevaluator/doctor.py
@@ -38,7 +38,7 @@ _SCANNERS_BY_CHECK: dict[str, tuple[str, ...]] = {
"dependency": ("pip_audit",),
}
# Older releases lack the CLI or report contract the validators parse: the
-# Bandit floor matches the security extra, and Gitleaks 8 introduced the
+# Bandit floor matches the core dependency, and Gitleaks 8 introduced the
# ``detect`` command the secrets scan runs.
_MINIMUM_SCANNER_VERSIONS: dict[str, tuple[int, int, int]] = {
"bandit": (1, 9, 4),
diff --git a/src/skillevaluator/utils/tool_runner.py b/src/skillevaluator/utils/tool_runner.py
index 4c73eb4e..f9c0ef7c 100644
--- a/src/skillevaluator/utils/tool_runner.py
+++ b/src/skillevaluator/utils/tool_runner.py
@@ -237,6 +237,10 @@ class ExternalTool:
if self._configuration_error:
return self._configuration_error
bundled_scanner_hint = (
+ "Bundled with SkillEvaluator — reinstall with: "
+ 'uv tool install --reinstall "skillevaluator @ git+https://github.com/NVIDIA/SkillEvaluator.git"'
+ )
+ security_extra_hint = (
"Bundled with the security extra — reinstall with: "
'uv tool install "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"'
)
@@ -244,7 +248,7 @@ class ExternalTool:
"gitleaks": (
"Install with: brew install gitleaks (macOS) or go install github.com/gitleaks/gitleaks/v8@latest"
),
- "pip-audit": bundled_scanner_hint,
+ "pip-audit": security_extra_hint,
"safety": "Optional secondary scanner; install separately with: uv tool install safety",
"bandit": bundled_scanner_hint,
"semgrep": "Optional; install separately with: brew install semgrep (macOS) or uv tool install semgrep",
diff --git a/tests/test_doctor.py b/tests/test_doctor.py
index 172899be..9271e55c 100644
--- a/tests/test_doctor.py
+++ b/tests/test_doctor.py
@@ -160,7 +160,7 @@ def test_tier1_scanner_failure_does_not_hide_other_scanners(scanners, monkeypatc
assert status == "fail"
assert "secret-canary" not in detail
expected = {
- "missing": "not installed. Bundled with the security extra",
+ "missing": "not installed. Bundled with SkillEvaluator",
"malformed override": "SKILLEVALUATOR_BANDIT_PATH must name an absolute executable file",
"timeout": "Bandit timed out after 4 seconds",
"exit status": "version probe exited with status 2",
diff --git a/tests/test_oss_packaging.py b/tests/test_oss_packaging.py
index 975b9df3..3aef8650 100644
--- a/tests/test_oss_packaging.py
+++ b/tests/test_oss_packaging.py
@@ -399,6 +399,7 @@ def test_security_extra_uses_pip_audit_without_bundling_safety() -> None:
assert any(requirement.startswith("pip-audit") for requirement in security)
assert not any(requirement.startswith("safety") for requirement in security)
+ assert any(requirement.startswith("bandit") for requirement in project["project"]["dependencies"])
assert "safety" not in lock_names
assert "safety-schemas" not in lock_names
assert "nltk" not in lock_names
diff --git a/tests/utils/test_tool_runner.py b/tests/utils/test_tool_runner.py
index 9855604d..414f8a03 100644
--- a/tests/utils/test_tool_runner.py
+++ b/tests/utils/test_tool_runner.py
@@ -418,17 +418,19 @@ class TestExternalTool:
)
assert "[security]" not in tool.get_install_hint()
- def test_bundled_scanner_hints_point_at_security_extra(self):
- """Scanners shipped via the security extra hint at reinstalling with extras.
+ def test_bundled_scanner_hints_point_at_reinstalling_skillevaluator(self):
+ """Bundled scanners hint at reinstalling SkillEvaluator, not a bare pip install.
- Bare ``pip install <scanner>`` is misleading for the documented
- install paths (uv tool / pipx), where the fix is reinstalling
- skillevaluator with the extras enabled.
+ Bandit is a core dependency, so the fix needs no extra; pip-audit
+ ships with the security extra.
"""
with patch("shutil.which", return_value=None):
- for command in ("bandit", "pip-audit"):
- hint = ExternalTool(command, command).get_install_hint()
- assert "[all]" in hint or "[security]" in hint, f"{command}: {hint}"
+ bandit_hint = ExternalTool("bandit", "bandit").get_install_hint()
+ pip_audit_hint = ExternalTool("pip-audit", "pip-audit").get_install_hint()
+
+ assert "uv tool install --reinstall" in bandit_hint
+ assert "[security]" not in bandit_hint and "[all]" not in bandit_hint
+ assert "[all]" in pip_audit_hint or "[security]" in pip_audit_hint
def test_semgrep_hint_uses_a_separate_tool_install(self):
with patch("shutil.which", return_value=None):
diff --git a/uv.lock b/uv.lock
index 934a6d4e..b5a49caf 100644
--- a/uv.lock
+++ b/uv.lock
@@ -2560,6 +2560,7 @@ name = "skillevaluator"
version = "0.9.0"
source = { editable = "." }
dependencies = [
+ { name = "bandit" },
{ name = "click" },
{ name = "idna" },
{ name = "jinja2" },
@@ -2574,7 +2575,6 @@ dependencies = [
[package.optional-dependencies]
all = [
{ name = "anthropic" },
- { name = "bandit" },
{ name = "boto3" },
{ name = "harbor", extra = ["gke"] },
{ name = "litellm" },
@@ -2607,7 +2607,6 @@ publisher = [
{ name = "pyjwt", extra = ["crypto"] },
]
security = [
- { name = "bandit" },
{ name = "pip-audit" },
]
tier2 = [
@@ -2631,7 +2630,7 @@ tier3 = [
[package.metadata]
requires-dist = [
{ name = "anthropic", marker = "extra == 'llm'", specifier = ">=0.83.0" },
- { name = "bandit", marker = "extra == 'security'", specifier = ">=1.9.4" },
+ { name = "bandit", specifier = ">=1.9.4" },
{ name = "boto3", marker = "extra == 'llm'", specifier = ">=1.34.0" },
{ name = "build", marker = "extra == 'dev'", specifier = ">=1.2" },
{ name = "click", specifier = ">=8.3.3,<9" },