Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

Set up Tier 1 scanners on first use and add --skip (#318)

Christopher Kevina705bb6a8ec8
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f421a86c..a0b29f9b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -135,6 +135,12 @@ All notable changes to SkillEvaluator are documented in this file.
rules, and is not accepted by `validate`, so a baseline-only run cannot pass
a Tier 3 gate.
+- Tier 1 sets up SkillSpector 2.13.3 and Gitleaks 8.30.0 on first use from
+ pinned, SHA-256-verified releases, without a prompt; Semgrep is optional
+ unless `SKILLEVALUATOR_REQUIRE_SEMGREP=1`. `--skip <check>` on `validate` and
+ `tier1` leaves a check out and reports it as skipped. A `scanners` row and the
+ JSON report record which scanner versions ran.
+
### Changed
- Shared LLM calls and the Harbor verifier retry HTTP 408, 429, and every 5xx
diff --git a/README.md b/README.md
index 4c096ba7..62cedf55 100644
--- a/README.md
+++ b/README.md
@@ -107,10 +107,10 @@ Each tier is independent. Add the relevant extra to your checkout command:
| Tier 2: Deduplication | Does guidance repeat or overlap? | `uv run --extra tier2 skillevaluator tier2 ./my-skill` | Embeddings and chat; add `--catalog ./skill-catalog.json` for inter-skill comparison |
| Tier 3: Live Evaluation | Does the skill help the agent? | `uv run --extra tier3 skillevaluator tier3 ./my-skill` | Provider access, a supported agent, and a running sandbox (Docker by default) |
-Full static Tier 1 needs Semgrep, SkillSpector, and Gitleaks in addition to the
-`security` extra. Follow [scanner installation](docs/installation.mdx#system-tools).
-`--no-llm` disables model-backed stages; it does not remove scanner requirements.
-Missing required scanner evidence is reported as `INCOMPLETE` and exits nonzero.
+Tier 1 fetches pinned, verified SkillSpector and Gitleaks releases when they
+are missing; Semgrep is optional. See [scanners](docs/installation.mdx#system-tools).
+`--no-llm` does not remove scanner requirements; a scanner that cannot run is
+`INCOMPLETE` and exits nonzero.
[SkillSpector](https://github.com/NVIDIA/SkillSpector) provides specialized skill
security scanning; [Harbor](https://github.com/harbor-framework/harbor), the
open-source agent evaluation framework, powers sandboxed agent runs.
diff --git a/ci/Dockerfile.tier1 b/ci/Dockerfile.tier1
index 84399087..6bdcaa8a 100644
--- a/ci/Dockerfile.tier1
+++ b/ci/Dockerfile.tier1
@@ -70,6 +70,7 @@ COPY --from=build /usr/local/bin/gitleaks /usr/local/bin/gitleaks
COPY --from=build /usr/local/share/licenses/gitleaks /usr/local/share/licenses/gitleaks
COPY --from=build /opt/skillevaluator/.venv /opt/skillevaluator/.venv
ENV PATH="/opt/skillevaluator/.venv/bin:/opt/semgrep/bin:/opt/skillspector/bin:${PATH}"
+ENV SKILLEVALUATOR_REQUIRE_SEMGREP=1
RUN bandit --version && pip-audit --version && semgrep --version && skillspector --version \
&& gitleaks version && /opt/skillspector/bin/python -c 'import yara'
RUN useradd --uid 10001 --create-home --shell /usr/sbin/nologin evaluator \
diff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdx
index b8e2a628..bcb87c1f 100644
--- a/docs/cli-reference.mdx
+++ b/docs/cli-reference.mdx
@@ -127,6 +127,7 @@ errors, such as an option the workflow does not accept, exit `2`.
| Flag | Default | Effect |
| --- | --- | --- |
| `--checks TEXT` | All applicable static checks, including dependency auditing | Comma-separated subset of static checks |
+| `--skip CHECKS` | none | Comma-separated checks to leave out, e.g. `secrets` for the Gitleaks scan. Skipped checks are shown and reported as skipped, nothing is installed for them, and the run cannot certify Tier 1 for publication |
| `--llm / --no-llm` | Enabled when a provider is configured | Enable or skip rubric, LLM security, and finding verification; explicit `--llm` requires valid provider configuration |
| `--min-score INTEGER` | `70` | Minimum passing quality and rubric score, from 0 to 100; quality severity overrides cannot waive the minimum |
| `--quality-config PATH` | packaged `quality-default.yaml` | YAML file that replaces the quality check's expected-heading rules. See [Tier 1: Validation](tier1-validation.mdx#the-checks). |
@@ -233,6 +234,7 @@ Static checks; LLM-free by default. Tier 1 gates the exit code and always runs.
| Flag | Default | Effect |
| --- | --- | --- |
| `--checks, --tier1-checks TEXT` | all applicable | Comma-separated subset of Tier 1 checks. Default choices: `schema`, `version`, `security`, `pii`, `license`, `code-integrity`, `unicode`, `quality`, `lint`; opt-in (not run by default): `dependency`. `quality`/`lint`/`version` are skill-only and skipped for rules and workflows. |
+| `--skip CHECKS` | none | Comma-separated Tier 1 checks to leave out: any check above or `secrets` (the Gitleaks scan inside `code-integrity`). Skipped checks show as skipped in the terminal and reports, nothing is installed for them, and the run cannot certify Tier 1 for publication. |
| `--previous-version VERSION` | `SKILLEVALUATOR_PREVIOUS_VERSION`, else unset | Previous released version; the skill's version must be a strictly greater SemVer. Single-skill targets only. |
| `--fail-fast` | off | Stop on the first failing check instead of collecting all issues. |
| `-c, --continue-on-failure` | off | Run the full pipeline without stopping early; record all issues in the reports. Overrides `--fail-fast`, and for folder validation keeps scanning every skill past a CRITICAL finding. |
diff --git a/docs/installation.mdx b/docs/installation.mdx
index 1fff4c76..4f55e6d1 100644
--- a/docs/installation.mdx
+++ b/docs/installation.mdx
@@ -24,8 +24,8 @@ This project will download and install additional third-party open source softwa
- Git for installs from the repository and for the bundled-example checkout.
- Docker, only for the container install and Tier 3 Docker-mode evaluation.
- No API key is needed for deterministic Tier 1 checks. Full default scanner
- coverage requires the `security` extra plus separate Semgrep, SkillSpector,
- and Gitleaks executables. LLM-backed checks and Tiers 2–3 use a configured provider — see
+ coverage requires the `security` extra; Tier 1 provisions SkillSpector and
+ Gitleaks itself, and Semgrep is optional (see [System tools](#system-tools)). LLM-backed checks and Tiers 2–3 use a configured provider — see
[Providers & Credentials](configuration.mdx).
## Install
@@ -184,16 +184,20 @@ and environment backends.
## System tools
<Note>
-Install the required security scanners separately so their dependency stacks do not constrain the SkillEvaluator distribution:
+Tier 1 provisions the external security scanners in their own environments, so their dependency stacks do not constrain the SkillEvaluator distribution:
-```bash title="External security scanners"
-brew install semgrep gitleaks # macOS
-uv tool install semgrep # isolated Python tool environment
+- **SkillSpector:** when no executable is found, Tier 1 downloads the 2.13.3 release wheel once into the SkillEvaluator runtime folder, verifies its SHA-256 checksum, and runs it with `uv tool run`. The first run needs uv and network access; later runs use the uv cache.
+- **Semgrep:** optional. When it is installed, Code Risk Analysis adds its rules to Bandit; when it is not, the check notes it was skipped. Set `SKILLEVALUATOR_REQUIRE_SEMGREP=1` to report a missing Semgrep as `INCOMPLETE` (the Tier 1 CI image does).
+- **Gitleaks:** on first use, SkillEvaluator downloads the pinned 8.30.0 release into its runtime folder and verifies its SHA-256 checksum. There is no prompt, so CI runs install it too. Linux and macOS only.
+
+Installed executables take precedence. To use your own, put them on `PATH` or set `SKILLEVALUATOR_SEMGREP_PATH` and `SKILLEVALUATOR_SKILLSPECTOR_PATH`:
+
+```bash title="Optional: preinstall the scanners"
+uv tool install semgrep
uv tool install git+https://github.com/NVIDIA/[email protected]
+brew install gitleaks # or a binary from https://github.com/gitleaks/gitleaks/releases
```
-On other systems, install Gitleaks with `go install github.com/gitleaks/gitleaks/v8@latest` or a binary from the [Gitleaks releases](https://github.com/gitleaks/gitleaks/releases). Semgrep and SkillSpector must be on `PATH`, or configured with `SKILLEVALUATOR_SEMGREP_PATH` and `SKILLEVALUATOR_SKILLSPECTOR_PATH`. RHEL 8 and compatible enterprise Linux systems can supply compatible executables independently of the SkillEvaluator Python environment.
-
Docker is required only for Tier 3 Docker-mode evaluation. Agent CLIs and their credentials are user-supplied — see [Agents & Sandboxes](agents-and-sandboxes.mdx).
`skillevaluator health-check --tier 1` reports the path and version of each installed scanner and exits non-zero when one is missing, misconfigured, or older than the supported release. Add `--checks security,code-integrity,dependency` to include pip-audit.
@@ -217,7 +221,7 @@ skillevaluator health-check
uv installs tools to `~/.local/bin`, which may not be on PATH yet. Run `uv tool update-shell` and open a new terminal.
</Accordion>
<Accordion title="Validation reports INCOMPLETE and exits non-zero">
-Install any missing required scanner named in the report. The `security` extra supplies Bandit and pip-audit; Semgrep, SkillSpector, and Gitleaks are separate executables. Missing required evidence leaves validation `INCOMPLETE` and non-zero.
+A required scanner could not run. Check that uv is on `PATH` and GitHub and PyPI are reachable (SkillSpector), or that GitHub is reachable for the Gitleaks download, or install the scanner named in the report. Missing required evidence leaves validation `INCOMPLETE` and non-zero. To run without that scanner, pass `--skip` with the check named in the fix list (e.g. `--skip secrets`); the check is then reported as skipped and the run cannot certify Tier 1.
</Accordion>
</AccordionGroup>
diff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.py
index 2d0b2dac..178e2b7c 100644
--- a/src/skillevaluator/cli.py
+++ b/src/skillevaluator/cli.py
@@ -794,6 +794,58 @@ def _evaluated_source_from_options(
return identity or None
+def _validate_skip(_ctx: click.Context, _param: click.Parameter, value: str | None) -> str | None:
+ """Reject ``--skip`` values that name no known Tier 1 check."""
+ from skillevaluator.tier1.commands import SKIPPABLE_CHECKS, skipped_checks
+
+ if value is None:
+ return None
+ names = skipped_checks(value)
+ if not names or names - SKIPPABLE_CHECKS:
+ unknown = ", ".join(sorted(names - SKIPPABLE_CHECKS)) or repr(value)
+ raise click.BadParameter(f"unknown check {unknown}; choose from {', '.join(sorted(SKIPPABLE_CHECKS))}")
+ return value
+
+
+def _offer_scanner_install(checks: str | None = None, skip: str | None = None) -> None:
+ """Before Tier 1, install the pinned, checksum-verified SkillSpector and Gitleaks when missing."""
+ from skillevaluator.tier1.commands import runs_secrets_scan, runs_security_scan
+ from skillevaluator.utils.scanner_install import (
+ GITLEAKS_VERSION,
+ SKILLSPECTOR_VERSION,
+ ensure_gitleaks,
+ managed_skillspector_wheel,
+ prepare_skillspector,
+ )
+ from skillevaluator.utils.tool_runner import Tools
+
+ steps = []
+ wheel = managed_skillspector_wheel()
+ if (
+ runs_security_scan(checks, skip)
+ and Tools.skillspector.resolution_source == "uv tool run"
+ and wheel is not None
+ and not wheel.is_file()
+ ):
+ uv = Tools.skillspector.path
+ steps.append((f"SkillSpector {SKILLSPECTOR_VERSION}", lambda: prepare_skillspector(uv)))
+ path = Tools.gitleaks.path
+ if (
+ runs_secrets_scan(checks, skip)
+ and Tools.gitleaks.resolution_source == "managed"
+ and path is not None
+ and not Path(path).is_file()
+ ):
+ steps.append((f"Gitleaks {GITLEAKS_VERSION}", ensure_gitleaks))
+ for label, install in steps:
+ with console.status(f"Installing {label}…"):
+ error = install()
+ if error:
+ console.print(f"[red]✗ {error[:1].upper()}{error[1:]}[/red]", highlight=False)
+ else:
+ console.print(f"[dim]✓ {label} installed (checksum-verified)[/dim]", highlight=False)
+
+
def _run_agent_eval_or_skip(
target_path: Path,
*,
@@ -1153,6 +1205,8 @@ def _catalog_child_argv_from_ctx(
argv.extend(["--tiers", str(params["tiers"])])
if params.get("checks"):
argv.extend(["--checks", str(params["checks"])])
+ if params.get("skip"):
+ argv.extend(["--skip", str(params["skip"])])
if params.get("previous_version"):
argv.extend(["--previous-version", str(params["previous_version"])])
if params.get("fail_fast"):
@@ -1563,6 +1617,11 @@ def _finish_pipeline_view(
headline = "gating tiers passed · advisory tier reported findings (see report)"
else:
headline = f"all {ran} tier{'s' if ran != 1 else ''} passed"
+ skipped = sum(
+ 1 for r in tier_gate_results if isinstance(r.metadata, dict) and r.metadata.get("skipped_by_user")
+ )
+ if skipped:
+ headline += f" · {skipped} check{'s' if skipped != 1 else ''} skipped"
lift = summary.get("overall_lift")
if agent_eval and isinstance(lift, (int, float)):
headline += f" · skill lift {lift:+.2f}"
@@ -1663,6 +1722,16 @@ def _print_run_banner(target_path: Path, content_type: str, profile: str | None)
"opt-in (not run by default): dependency. "
"quality/lint/version are skill-only and skipped for rules/workflows.",
)
+ "--skip",
+ default=None,
+ metavar="CHECKS",
+ callback=_validate_skip,
+ cls=GroupedOption,
+ help_group=_TIER1_GROUP,
+ help="Comma-separated Tier 1 checks to leave out, e.g. secrets (the Gitleaks scan). "
+ "Skipped checks are shown and recorded as skipped in the report, and nothing is installed for them.",
+)
@click.option(
"--previous-version",
default=None,
@@ -1977,6 +2046,7 @@ def validate(
full: bool,
verbose: bool,
checks: str | None,
+ skip: str | None,
previous_version: str | None,
fail_fast: bool,
continue_on_failure: bool,
@@ -2180,6 +2250,7 @@ def validate(
# for the later Tier 3 skip result and continue through Tier 1/2.
autopilot_error: str | None = None
autopilot_dataset_note: str | None = None
+ _offer_scanner_install(checks, skip)
view.start()
if autopilot:
# Generation now precedes Tier 1, so surface it on the first live block
@@ -2220,6 +2291,7 @@ def validate(
fail_fast=fail_fast,
continue_on_failure=continue_on_failure,
on_check=_on_check if quiet else None,
+ skip=skip,
)
# The raw pass/fail signal drives --fail-fast identically in both modes;
# the DISPLAYED tier summary must reflect policy-finalized severities or
@@ -2773,6 +2845,13 @@ def _workflow_report_options(func):
@click.command(cls=cli.command_class, context_settings=CONTEXT_SETTINGS)
@click.argument("skill_path", metavar="PATH", type=click.Path(exists=True, file_okay=False, path_type=Path))
@click.option("--checks", default=None, help="Select static checks; by default all checks include dependency auditing.")
+ "--skip",
+ default=None,
+ metavar="CHECKS",
+ callback=_validate_skip,
+ help="Leave out these checks, e.g. secrets; they are reported as skipped.",
+)
@click.option(
"--llm/--no-llm", default=None, help="Include rubric, LLM security, and verification [default: when configured]."
)
@@ -2789,6 +2868,7 @@ def _workflow_report_options(func):
def _tier1_workflow(
skill_path: Path,
checks: str | None,
+ skip: str | None,
llm: bool | None,
min_score: int,
quality_config: Path | None,
@@ -2805,9 +2885,11 @@ def _tier1_workflow(
policy = resolve_policy(profile=profile)
except (FileNotFoundError, ValueError) as exc:
raise click.ClickException(str(exc)) from exc
+ _offer_scanner_install(checks, skip)
results = run_tier1_workflow(
skill_path,
checks=checks,
+ skip=skip,
llm=llm,
min_score=min_score,
quality_config=quality_config,
diff --git a/src/skillevaluator/models/result.py b/src/skillevaluator/models/result.py
index ede11b9f..3d43fa2a 100644
--- a/src/skillevaluator/models/result.py
+++ b/src/skillevaluator/models/result.py
@@ -494,6 +494,16 @@ class ValidationResult:
scans.append(tool_name)
self.passed = False
+ @property
+ def scanners(self) -> dict[str, str]:
+ """External scanners that produced this result, mapped to their versions."""
+ value = self.metadata.get("scanners")
+ return dict(value) if isinstance(value, dict) else {}
+
+ def record_scanner(self, name: str, version: str | None) -> None:
+ """Record that scanner *name* ran, so reports show which scanner versions produced the result."""
+ self.metadata["scanners"] = {**self.scanners, name: version or "unknown"}
+
def merge(self, other: ValidationResult) -> None:
"""Merge another result into this one, combining all fields."""
if not other.passed:
@@ -508,9 +518,12 @@ class ValidationResult:
# another's (e.g. bandit and semgrep both failing); concatenate.
incomplete = list(dict.fromkeys([*self.incomplete_scans, *other.incomplete_scans]))
gate_failures = [*self.gate_failures, *other.gate_failures]
+ scanners = {**self.scanners, **other.scanners}
self.metadata.update(other.metadata)
if incomplete:
self.metadata["incomplete_scans"] = incomplete
+ if scanners:
+ self.metadata["scanners"] = scanners
if gate_failures:
self.metadata["gate_failures"] = gate_failures
@@ -564,6 +577,8 @@ class ValidationResult:
gate_failures = [*self.gate_failures, *(f"[{prefix}] {message}" for message in other.gate_failures)]
if gate_failures:
self.metadata["gate_failures"] = gate_failures
+ if scanners := {**self.scanners, **other.scanners}:
+ self.metadata["scanners"] = scanners
# Merge summary
self.summary.errors += other.summary.errors
@@ -583,4 +598,6 @@ class ValidationResult:
}
if self.gate_failures:
result["gate_failures"] = self.gate_failures
+ if self.scanners:
+ result["scanners"] = self.scanners
return result
diff --git a/src/skillevaluator/reporting/base.py b/src/skillevaluator/reporting/base.py
index 848adbce..a03536d5 100644
--- a/src/skillevaluator/reporting/base.py
+++ b/src/skillevaluator/reporting/base.py
@@ -1976,7 +1976,15 @@ def missing_tier1_publication_checks(results: list[ValidationResult]) -> tuple[s
if not declarations:
return ()
required = frozenset().union(*(declared_tier1_required_checks(value) for value in declarations))
- covered = {producer.check_id for _result, producer in executed}
+ user_skipped = {
+ producer.check_id
+ for result in results
+ if isinstance(result.metadata, dict)
+ and result.metadata.get("skipped_by_user")
+ and (producer := result_publication_evidence(result)) is not None
+ and producer.tier == 1
+ }
+ covered = {producer.check_id for _result, producer in executed} - user_skipped
return tuple(sorted(required - covered))
diff --git a/src/skillevaluator/reporting/console_ui.py b/src/skillevaluator/reporting/console_ui.py
index 9a0f3009..df9625f9 100644
--- a/src/skillevaluator/reporting/console_ui.py
+++ b/src/skillevaluator/reporting/console_ui.py
@@ -791,6 +791,35 @@ def stage_hint_row(label: str, hint: str, *, lead: str = "") -> TierRow:
# ── summaries: turn ValidationResult lists into tier rows ──────────────
+_SCANNER_NAMES = {"skillspector": "SkillSpector", "bandit": "Bandit", "semgrep": "Semgrep", "gitleaks": "Gitleaks"}
+_CHECK_SCANNERS = {"Security Scan": "skillspector", "Code Risk Analysis": "bandit", "Secrets Detection": "gitleaks"}
+
+
+def _scanners_row(results: list[ValidationResult]) -> list[TierRow]:
+ """The external scanners behind the Tier 1 checks, with their versions or why they did not run."""
+ states: dict[str, tuple[str, str]] = {}
+ for r in results:
+ if _is_user_skipped(r):
+ if scanner := _CHECK_SCANNERS.get(r.validator_name):
+ states.setdefault(scanner, ("skipped", MUTED))
+ continue
+ for scanner, version in r.scanners.items():
+ states[scanner] = (version, MUTED)
+ for scan in r.incomplete_scans:
+ if (scanner := scan.removesuffix("-llm")) in _SCANNER_NAMES:
+ states[scanner] = ("didn't run", f"bold {RED}")
+ segments: list[tuple[str, str]] = []
+ for scanner in [name for name in _SCANNER_NAMES if name in states]:
+ state, style = states[scanner]
+ segments += [*([(" · ", FAINT)] if segments else []), (_SCANNER_NAMES[scanner], TEXT), (f" {state}", style)]
+ return [TierRow("scanners", segments)] if segments else []
+
+
+def _is_user_skipped(result: ValidationResult) -> bool:
+ """True when the user left this check out with ``--skip``."""
+ return isinstance(result.metadata, dict) and bool(result.metadata.get("skipped_by_user"))
+
+
def _is_skipped(result: ValidationResult) -> bool:
"""True when a validator was recorded without executing (OSS conventions)."""
if bool(getattr(result, "skipped", False)):
@@ -935,8 +964,11 @@ def _check_key(validator_name: str) -> str | None:
def final_check_ticker_row(lineup: list[str], results: list[ValidationResult]) -> TierRow:
"""The check lineup with final per-check states: ✓ passed, ✗ failed, dim skipped."""
status: dict[str, str] = {}
+ user_skipped: set[str] = set()
for result in results:
key = _check_key(result.validator_name)
+ if key is not None and _is_user_skipped(result):
+ user_skipped.add(key)
if key is None or _is_skipped(result):
continue
if not result.passed:
@@ -952,6 +984,8 @@ def final_check_ticker_row(lineup: list[str], results: list[ValidationResult]) -
segments.append((f"✗ {name}", f"bold {RED}"))
elif state == PASS:
segments.append((f"✓ {name}", GREEN))
+ elif name in user_skipped:
+ segments.append((f"\N{EN DASH} {name}", MUTED))
else:
segments.append((name, FAINT))
return TierRow("checks", segments)
@@ -996,7 +1030,8 @@ def summarize_tier1(results: list[ValidationResult], lineup: list[str] | None =
When *lineup* is given, the per-check ticker stays in the completed
section with each check's final state (failed checks in red).
"""
- total = len(results)
+ user_skipped = [r for r in results if _is_user_skipped(r)]
+ total = len(results) - len(user_skipped)
failed = [r for r in results if not r.passed and not _is_skipped(r)]
incomplete = [r for r in failed if r.is_incomplete]
broken = [r for r in failed if not r.is_incomplete]
@@ -1009,16 +1044,25 @@ def summarize_tier1(results: list[ValidationResult], lineup: list[str] | None =
advisory[_short_name(r.validator_name)] = advisory.get(_short_name(r.validator_name), 0) + len(r.findings)
rows: list[TierRow] = []
+ scanners = _scanners_row(results)
if lineup:
rows.append(final_check_ticker_row(lineup, results))
+ rows += scanners
checks_segments: list[tuple[str, str]] = [(f"{total - len(failed)}/{total} passed", TEXT)]
if broken:
checks_segments += [(" · ", FAINT), (f"{len(broken)} failed", f"bold {RED}")]
if incomplete:
checks_segments += [(" · ", FAINT), (f"{len(incomplete)} didn't run", f"bold {RED}")]
+ if user_skipped:
+ checks_segments += [(" · ", FAINT), (f"{len(user_skipped)} skipped", MUTED)]
if warnings:
checks_segments += [(" · ", FAINT), (f"{warnings} warning{'s' if warnings != 1 else ''}", MUTED)]
rows.append(TierRow("summary" if lineup else "checks", checks_segments))
+ if not lineup:
+ rows += scanners
+ if user_skipped:
+ names = ", ".join(dict.fromkeys(_short_name(r.validator_name) for r in user_skipped))
+ rows.append(TierRow("skipped", [(names, MUTED), (" with --skip, not checked", FAINT)]))
if advisory:
detail = " · ".join(f"{name} {count}" for name, count in sorted(advisory.items(), key=lambda kv: -kv[1]))
@@ -1047,18 +1091,39 @@ def _tier1_fix(result: ValidationResult) -> FixItem:
_INSTALL_HINT = re.compile(r"\b[Ii]nstall(?: [\w-]+)*? with: (.+?)\.?$")
+_DOWNLOAD_FAILED = re.compile(r"couldn't (?:download|set up) .+? \((?:[^()]|\([^()]*\))*\)")
+
+
def _setup_fix(results: list[ValidationResult]) -> FixItem:
- """One item for checks that could not run because a scanner is not installed."""
+ """One item for checks that could not run because a scanner is missing or failed to download."""
scanners = list(dict.fromkeys(name for r in results for name in r.incomplete_scans))
- hints = []
+ hints: list[str] = []
+ failed_downloads: list[str] = []
for r in results:
- for text in r.warnings:
- match = _INSTALL_HINT.search(str(text))
+ for text in map(str, [*r.warnings, *r.errors]):
+ if (download := _DOWNLOAD_FAILED.search(text)) and download.group(0) not in failed_downloads:
+ failed_downloads.append(download.group(0))
+ match = _INSTALL_HINT.search(text)
if match and match.group(1) not in hints:
hints.append(match.group(1))
checks = ", ".join(dict.fromkeys(_short_name(r.validator_name) for r in results))
- missing = ", ".join(scanners) or "a required scanner"
- return FixItem("setup", f"{checks} didn't run: {missing} not installed", hints)
+ skip = ",".join(dict.fromkeys(_skip_name(r.validator_name) for r in results))
+ if failed_downloads:
+ hosts = "github.com and pypi.org" if any("set up" in d for d in failed_downloads) else "github.com"
+ actions = [f"check network access to {hosts}, then rerun"]
+ actions += [f"or install it yourself: {hint}" for hint in hints]
+ problem = f"{checks} didn't run: {'; '.join(failed_downloads)}"
+ else:
+ actions = list(hints)
+ problem = f"{checks} didn't run: {', '.join(scanners) or 'a required scanner'} not installed"
+ return FixItem("setup", problem, [*actions, f"or skip it: --skip {skip}"])
+
+
+def _skip_name(validator_name: str) -> str:
+ """The ``--skip`` value that leaves out the check producing *validator_name*."""
+ if validator_name == "Secrets Detection":
+ return "secrets"
+ return _check_key(validator_name) or _short_name(validator_name)
def summarize_tier2(results: list[ValidationResult]) -> tuple[bool, bool, list[TierRow], str]:
diff --git a/src/skillevaluator/reporting/json_reporter.py b/src/skillevaluator/reporting/json_reporter.py
index caf32512..936766d1 100644
--- a/src/skillevaluator/reporting/json_reporter.py
+++ b/src/skillevaluator/reporting/json_reporter.py
@@ -269,6 +269,7 @@ class JSONReporter(ReporterBase):
"skipped": skipped,
"skip_reason": get_skip_reason(result) if skipped else None,
"incomplete_scans": result.incomplete_scans,
+ **({"scanners": result.scanners} if result.scanners else {}),
"summary": {
"files_scanned": result.summary.files_scanned,
"checks_performed": result.summary.checks_performed,
diff --git a/src/skillevaluator/reporting/retained.py b/src/skillevaluator/reporting/retained.py
index f62f18d4..0038f86c 100644
--- a/src/skillevaluator/reporting/retained.py
+++ b/src/skillevaluator/reporting/retained.py
@@ -208,6 +208,13 @@ def retained_results(payload: dict, *, copy_metadata: bool = True) -> list[Valid
for key in ("incomplete_scans", "gate_failures"):
if key in row:
metadata[key] = _strings(row[key], key)
+ if "scanners" in row:
+ _require(
+ isinstance(row["scanners"], dict)
+ and all(isinstance(k, str) and isinstance(v, str) for k, v in row["scanners"].items()),
+ "scanners must map scanner names to versions",
+ )
+ metadata["scanners"] = dict(row["scanners"])
if "dependency_coverage" in row:
_require(
isinstance(row["dependency_coverage"], list)
diff --git a/src/skillevaluator/tier1/commands.py b/src/skillevaluator/tier1/commands.py
index 68d4871d..3451569b 100644
--- a/src/skillevaluator/tier1/commands.py
+++ b/src/skillevaluator/tier1/commands.py
@@ -80,6 +80,33 @@ OPTIONAL_CHECKS = ("dependency",)
# Every canonical check name ``run_validation`` understands after alias
# resolution (the default run plus the opt-in checks).
RECOGNIZED_CHECKS = frozenset(DEFAULT_CHECKS) | frozenset(OPTIONAL_CHECKS)
+# ``--skip`` accepts every check plus the Gitleaks-backed secrets scan inside code-integrity.
+SKIPPABLE_CHECKS = RECOGNIZED_CHECKS | {"secrets"}
+SKIP_REASON = "skipped with --skip"
+_CHECK_ALIASES = {
+ "code": "code-integrity",
+ "code-risk": "code-integrity",
+ "scripts": "lint",
+ "script-lint": "lint",
+ "dependencies": "dependency",
+ "deps": "dependency",
+ "dependency-audit": "dependency",
+ "licence": "license",
+ "license-check": "license",
+}
+# Validator names recorded for a skipped check, matching the names its run would produce.
+_SKIPPED_VALIDATORS = {
+ "schema": ("Schema & Repository Governance",),
+ "version": ("Semantic Version Validation",),
+ "security": ("Security Scan",),
+ "pii": ("PII Scan",),
+ "license": ("License Compliance",),
+ "code-integrity": ("Code Risk Analysis", "Secrets Detection", "Code Integrity & Hygiene"),
+ "dependency": ("Dependency Vulnerability Audit",),
+ "unicode": ("Unicode Smuggling Detection",),
+ "quality": ("Quality Score (4-Dimension Analysis)",),
+ "lint": ("Script Lint (Advisory)",),
+}
REPORTERS = {
"json": JSONReporter,
"html": HTMLReporter,
@@ -121,26 +148,27 @@ def _as_result(
return result
+def _parse_checks(value: str | None) -> set[str]:
+ return {_CHECK_ALIASES.get(check, check) for raw in (value or "").split(",") if (check := raw.strip().lower())}
+
+
def _enabled_checks(checks: str | None) -> set[str]:
- if not checks:
- return set(DEFAULT_CHECKS)
- aliases = {
- "code": "code-integrity",
- "code-risk": "code-integrity",
- "scripts": "lint",
- "script-lint": "lint",
- "dependencies": "dependency",
- "deps": "dependency",
- "dependency-audit": "dependency",
- "licence": "license",
- "license-check": "license",
- }
- enabled = set()
- for raw in checks.split(","):
- check = raw.strip().lower()
- if check:
- enabled.add(aliases.get(check, check))
- return enabled
+ return _parse_checks(checks) if checks else set(DEFAULT_CHECKS)
+
+
+def skipped_checks(skip: str | None) -> set[str]:
+ """Resolve ``--skip`` names (aliases included) to canonical check names."""
+ return _parse_checks(skip)
+
+
+def runs_secrets_scan(checks: str | None, skip: str | None) -> bool:
+ """True when a run with these ``--checks``/``--skip`` values executes the Gitleaks scan."""
+ return "code-integrity" in _enabled_checks(checks) and not {"code-integrity", "secrets"} & skipped_checks(skip)
+
+
+def runs_security_scan(checks: str | None, skip: str | None) -> bool:
+ """True when a run with these ``--checks``/``--skip`` values executes the SkillSpector scan."""
+ return "security" in _enabled_checks(checks) and "security" not in skipped_checks(skip)
def _schema_validator_for(content_type: str | None, policy: ValidationPolicy | None):
@@ -184,9 +212,13 @@ def run_validation(
fail_fast: bool = False,
continue_on_failure: bool = False,
on_check: Callable[[str], None] | None = None,
+ skip: str | None = None,
) -> list[ValidationResult]:
"""Run selected Tier 1 validators and return structured results.
+ Checks named in *skip* do not run; each is recorded as a user-skipped
+ result, which reports show as skipped and which never certifies Tier 1.
+
*on_check* is invoked with each canonical check name just before it runs;
when provided it replaces the stderr ``[n/total]`` progress lines (the
caller owns presentation, e.g. the quiet pipeline view).
@@ -204,6 +236,7 @@ def run_validation(
applied centrally in :func:`emit_reports` via the policy.
"""
enabled = _enabled_checks(checks)
+ skipped = skipped_checks(skip)
results: list[ValidationResult] = []
initial_target = publication_target_from_path(target_path)
skill_like = content_type in (None, CONTENT_TYPE_SKILL, CONTENT_TYPE_UNKNOWN)
@@ -224,6 +257,16 @@ def run_validation(
skill_like=skill_like,
)
+ def _skipped_result(check_id: str, name: str) -> ValidationResult:
+ result = ValidationResult(
+ validator_name=name,
+ validator_description=SKIP_REASON,
+ metadata={"skipped": True, "skipped_by_user": True, "skip_reason": SKIP_REASON},
+ )
+ stamp_publication_evidence([result], tier=1, check_id=check_id)
+ stamp_tier1_required_checks([result], skill_like=skill_like)
+ return result
+
def _schema_results() -> list[ValidationResult]:
v = _schema_validator_for(content_type, policy)
return [_run_result("schema", v.name, v.description, v.validate)]
@@ -238,7 +281,9 @@ def run_validation(
def _code_integrity_results() -> list[ValidationResult]:
return [
- _run_result("code-integrity", v.name, v.description, v.validate)
+ _skipped_result("code-integrity", v.name)
+ if "secrets" in skipped and isinstance(v, SecretsValidator)
+ else _run_result("code-integrity", v.name, v.description, v.validate)
for v in (CodeRiskValidator(), SecretsValidator(), HygieneValidator())
]
@@ -286,6 +331,9 @@ def run_validation(
active = [
(check_name, builder) for check_name, builder, applicable in steps if check_name in enabled and applicable
]
+ for check_name, _builder in [step for step in active if step[0] in skipped]:
+ active.remove((check_name, _builder))
+ results.extend(_skipped_result(check_name, name) for name in _SKIPPED_VALIDATORS[check_name])
with continue_on_failure_scope(continue_on_failure):
for step_number, (check_name, builder) in enumerate(active, 1):
if on_check is not None:
@@ -317,7 +365,7 @@ def run_validation(
finalize_publication_target(results, target_path, initial_target)
return results
- unknown = enabled - RECOGNIZED_CHECKS
+ unknown = (enabled - RECOGNIZED_CHECKS) | (skipped - SKIPPABLE_CHECKS)
if unknown:
result = ValidationResult(
validator_name="Tier 1 option validation",
diff --git a/src/skillevaluator/tier_workflows.py b/src/skillevaluator/tier_workflows.py
index 11c540c6..a6bf264b 100644
--- a/src/skillevaluator/tier_workflows.py
+++ b/src/skillevaluator/tier_workflows.py
@@ -72,6 +72,7 @@ def run_tier1_workflow(
quality_config: Path | None,
profile: ValidationPolicy,
previous_version: str | None,
+ skip: str | None = None,
) -> list[ValidationResult]:
require_skill(skill_path)
if checks is not None and not checks.strip(", \t\n"):
@@ -98,6 +99,7 @@ def run_tier1_workflow(
policy=profile,
content_type="skill",
continue_on_failure=True,
+ skip=skip,
)
apply_policy(results, profile)
stages = {"validation": _stage(results)}
diff --git a/src/skillevaluator/utils/scanner_install.py b/src/skillevaluator/utils/scanner_install.py
new file mode 100644
index 00000000..b898ccb0
--- /dev/null
+++ b/src/skillevaluator/utils/scanner_install.py
@@ -0,0 +1,182 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Pinned, checksum-verified provisioning of the external Tier 1 scanners.
+
+SkillSpector is not on PyPI, so when it is not installed its pinned release
+wheel is downloaded once into SkillEvaluator's runtime directory and run with
+``uv tool run`` in uv's own cached environment. Gitleaks is a native binary,
+downloaded once from its official release into the same directory.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import io
+import os
+import platform
+import subprocess
+import sys
+import tarfile
+import tempfile
+import urllib.request
+from pathlib import Path
+from types import MappingProxyType
+
+SKILLSPECTOR_VERSION = "2.13.3"
+SKILLSPECTOR_WHEEL = f"skillspector-{SKILLSPECTOR_VERSION}-py3-none-any.whl"
+SKILLSPECTOR_WHEEL_URL = (
+ f"https://github.com/NVIDIA/SkillSpector/releases/download/v{SKILLSPECTOR_VERSION}/{SKILLSPECTOR_WHEEL}"
+)
+SKILLSPECTOR_WHEEL_SHA256 = "a421dffba5e0f966bf1baf8acd77f4750a8ece0afc5520ecf49252b4004a8577"
+GITLEAKS_VERSION = "8.30.0"
+
+_GITLEAKS_SHA256 = MappingProxyType(
+ {
+ ("linux", "x64"): "79a3ab579b53f71efd634f3aaf7e04a0fa0cf206b7ed434638d1547a2470a66e",
+ ("linux", "arm64"): "b4cbbb6ddf7d1b2a603088cd03a4e3f7ce48ee7fd449b51f7de6ee2906f5fa2f",
+ ("darwin", "x64"): "ca221d012d247080c2f6f61f4b7a83bffa2453806b0c195c795bbe9a8c775ed5",
+ ("darwin", "arm64"): "b251ab2bcd4cd8ba9e56ff37698c033ebf38582b477d21ebd86586d927cf87e7",
+ }
+)
+_ARCH_ALIASES = MappingProxyType({"x86_64": "x64", "amd64": "x64", "aarch64": "arm64", "arm64": "arm64"})
+DOWNLOAD_TIMEOUT_SECONDS = 120
+_MAX_DOWNLOAD_BYTES = 64 * 1024 * 1024
+
+
+class _InstallError(Exception):
+ pass
+
+
+def _runtime_path(*parts: str) -> Path | None:
+ from skillevaluator.tier3.harbor.local_runtime import default_runtime_root
+
+ try:
+ return default_runtime_root().joinpath(*parts)
+ except ValueError:
+ return None
+
+
+def _download_verified(url: str, sha256: str, name: str) -> bytes:
+ try:
+ with urllib.request.urlopen(url, timeout=DOWNLOAD_TIMEOUT_SECONDS) as response:
+ payload = response.read(_MAX_DOWNLOAD_BYTES + 1)
+ except OSError as exc:
+ reason = getattr(exc, "reason", exc)
+ detail = getattr(reason, "strerror", None) or str(reason)
+ raise _InstallError(f"couldn't download {name} ({detail.lower()})") from exc
+ if len(payload) > _MAX_DOWNLOAD_BYTES:
+ raise _InstallError(f"{name} download exceeded the size limit")
+ if hashlib.sha256(payload).hexdigest() != sha256:
+ raise _InstallError(f"{name} download failed checksum verification")
+ return payload
+
+
+def _write_atomic(target: Path, data: bytes, mode: int) -> None:
+ target.parent.mkdir(parents=True, exist_ok=True)
+ fd, tmp = tempfile.mkstemp(dir=target.parent, prefix=f".{target.name}-")
+ try:
+ with os.fdopen(fd, "wb") as out:
+ out.write(data)
+ Path(tmp).chmod(mode)
+ Path(tmp).replace(target)
+ except OSError:
+ Path(tmp).unlink(missing_ok=True)
+ raise
+
+
+def managed_skillspector_wheel() -> Path | None:
+ """Where the pinned SkillSpector wheel lives; None when the runtime root is invalid."""
+ return _runtime_path("skillspector", SKILLSPECTOR_VERSION, SKILLSPECTOR_WHEEL)
+
+
+def ensure_skillspector_wheel() -> str | None:
+ """Make sure the verified SkillSpector wheel is present; return an error or None."""
+ target = managed_skillspector_wheel()
+ if target is None:
+ return "SkillEvaluator runtime directory is invalid"
+ try:
+ if target.is_file() and hashlib.sha256(target.read_bytes()).hexdigest() == SKILLSPECTOR_WHEEL_SHA256:
+ return None
+ payload = _download_verified(
+ SKILLSPECTOR_WHEEL_URL, SKILLSPECTOR_WHEEL_SHA256, f"SkillSpector {SKILLSPECTOR_VERSION}"
+ )
+ _write_atomic(target, payload, 0o644)
+ except _InstallError as exc:
+ return str(exc)
+ except OSError as exc:
+ return f"SkillSpector install failed: {exc}"
+ return None
+
+
+def prepare_skillspector(uv: str) -> str | None:
+ """Download the SkillSpector wheel and build its uv environment; return an error or None."""
+ error = ensure_skillspector_wheel()
+ if error:
+ return error
+ try:
+ proc = subprocess.run(
+ [uv, "tool", "run", "--quiet", "--from", str(managed_skillspector_wheel()), "skillspector", "--version"],
+ capture_output=True,
+ text=True,
+ timeout=900,
+ check=False,
+ )
+ except (OSError, subprocess.TimeoutExpired) as exc:
+ return f"couldn't set up SkillSpector {SKILLSPECTOR_VERSION} ({exc})"
+ if proc.returncode != 0:
+ lines = [line.strip() for line in (proc.stderr or proc.stdout).splitlines() if line.strip()]
+ causes = [line.split("Caused by:", 1)[1] for line in lines if "Caused by:" in line]
+ reason = (causes[-1] if causes else lines[0] if lines else f"exit {proc.returncode}").strip()
+ reason = reason.removeprefix("error:").lstrip("\N{MULTIPLICATION SIGN} ").rstrip(":").strip()
+ return f"couldn't set up SkillSpector {SKILLSPECTOR_VERSION} ({reason[:1].lower()}{reason[1:]})"
+ return None
+
+
+def gitleaks_platform() -> tuple[str, str] | None:
+ """Return the release (os, arch) for this host, or None when no pinned binary exists."""
+ key = (sys.platform, _ARCH_ALIASES.get(platform.machine().lower(), ""))
+ return key if key in _GITLEAKS_SHA256 else None
+
+
+def gitleaks_url(key: tuple[str, str]) -> str:
+ os_name, arch = key
+ return (
+ f"https://github.com/gitleaks/gitleaks/releases/download/v{GITLEAKS_VERSION}/"
+ f"gitleaks_{GITLEAKS_VERSION}_{os_name}_{arch}.tar.gz"
+ )
+
+
+def managed_gitleaks_path() -> Path | None:
+ """Where the managed Gitleaks binary lives; None when the runtime root is invalid."""
+ return _runtime_path("gitleaks", GITLEAKS_VERSION, "gitleaks")
+
+
+def ensure_gitleaks() -> str | None:
+ """Make sure the verified Gitleaks binary is installed; return an error or None."""
+ target = managed_gitleaks_path()
+ if target is not None and target.is_file() and os.access(target, os.X_OK):
+ return None
+ return install_gitleaks()
+
+
+def install_gitleaks() -> str | None:
+ """Download, verify and install the pinned Gitleaks binary; return an error or None."""
+ key = gitleaks_platform()
+ target = managed_gitleaks_path()
+ if key is None or target is None:
+ return f"no pinned Gitleaks {GITLEAKS_VERSION} binary for this platform"
+ try:
+ payload = _download_verified(gitleaks_url(key), _GITLEAKS_SHA256[key], f"Gitleaks {GITLEAKS_VERSION}")
+ with tarfile.open(fileobj=io.BytesIO(payload), mode="r:gz") as archive:
+ member = archive.getmember("gitleaks")
+ stream = archive.extractfile(member) if member.isfile() else None
+ if stream is None:
+ return "Gitleaks archive has no gitleaks binary"
+ binary = stream.read()
+ _write_atomic(target, binary, 0o755)
+ except _InstallError as exc:
+ return str(exc)
+ except (OSError, KeyError, tarfile.TarError) as exc:
+ return f"Gitleaks install failed: {exc}"
+ return None
diff --git a/src/skillevaluator/utils/tool_runner.py b/src/skillevaluator/utils/tool_runner.py
index 544c14c0..4c73eb4e 100644
--- a/src/skillevaluator/utils/tool_runner.py
+++ b/src/skillevaluator/utils/tool_runner.py
@@ -11,16 +11,27 @@ from __future__ import annotations
import json
import os
+import re
import shutil
import subprocess
import sys
import tempfile
-from collections.abc import Mapping
+from collections.abc import Callable, Mapping
from dataclasses import dataclass
from pathlib import Path
from skillevaluator.logging_config import get_logger
from skillevaluator.models.result import Severity
+from skillevaluator.utils.scanner_install import (
+ GITLEAKS_VERSION,
+ SKILLSPECTOR_VERSION,
+ SKILLSPECTOR_WHEEL_URL,
+ ensure_gitleaks,
+ ensure_skillspector_wheel,
+ gitleaks_platform,
+ managed_gitleaks_path,
+ managed_skillspector_wheel,
+)
logger = get_logger(__name__)
@@ -115,6 +126,7 @@ class ExternalTool:
*,
prefer_interpreter_sibling: bool = False,
override_env: str | None = None,
+ managed: bool = False,
):
"""Initialize tool wrapper.
@@ -126,12 +138,26 @@ class ExternalTool:
override_env: Optional environment variable containing an explicit
absolute executable path. Invalid overrides fail closed rather
than silently falling back to another executable.
+ managed: When the tool is not installed, fall back to SkillEvaluator's
+ pinned provisioning: the verified SkillSpector wheel through
+ ``uv tool run``, or the downloaded Gitleaks binary.
"""
self.name = name
self.command = command or name
self.override_env = override_env
+ self._prefer_interpreter_sibling = prefer_interpreter_sibling
+ self._managed = managed
+ self.refresh()
+
+ def refresh(self) -> None:
+ """Resolve the executable again, e.g. after a managed install."""
self.resolution_source = "automatic"
self._configuration_error: str | None = None
+ self._version: str | None = None
+ self._uv_run: tuple[str, tuple[str, ...]] | None = None
+ self._ensure: tuple[str, Callable[[], str | None]] | None = None
+ override_env = self.override_env
+ name = self.name
override = os.environ.get(override_env, "").strip() if override_env else ""
if override:
@@ -146,8 +172,31 @@ class ExternalTool:
else:
self._path = resolve_tool_path(
self.command,
- prefer_interpreter_sibling=prefer_interpreter_sibling,
+ prefer_interpreter_sibling=self._prefer_interpreter_sibling,
)
+ if self._path is None and self._managed:
+ self._resolve_managed()
+
+ def _resolve_managed(self) -> None:
+ if self.command == "skillspector":
+ uv = resolve_tool_path("uv")
+ wheel = managed_skillspector_wheel()
+ if uv is not None and wheel is not None:
+ self._path = uv
+ self._uv_run = (uv, ("tool", "run", "--quiet", "--from", str(wheel), "skillspector"))
+ self._ensure = (uv, ensure_skillspector_wheel)
+ self.resolution_source = "uv tool run"
+ elif self.command == "gitleaks":
+ binary = managed_gitleaks_path()
+ if binary is not None and gitleaks_platform() is not None:
+ self._path = str(binary)
+ self._ensure = (self._path, ensure_gitleaks)
+ self.resolution_source = "managed"
+
+ def _launch_args(self) -> tuple[str, ...]:
+ if self._uv_run is not None and self._path == self._uv_run[0]:
+ return self._uv_run[1]
+ return ()
@property
def is_available(self) -> bool:
@@ -159,6 +208,25 @@ class ExternalTool:
"""Get full path to tool binary."""
return self._path
+ @property
+ def version(self) -> str | None:
+ """The pinned release for managed installs, otherwise what the tool's version command reports."""
+ if self.resolution_source == "uv tool run":
+ return SKILLSPECTOR_VERSION
+ if self.resolution_source == "managed":
+ return GITLEAKS_VERSION
+ if self._path is None:
+ return None
+ if self._version is None:
+ argv = [self._path, "version" if self.command == "gitleaks" else "--version"]
+ try:
+ proc = subprocess.run(argv, capture_output=True, text=True, timeout=30, check=False)
+ except (OSError, subprocess.SubprocessError):
+ proc = None
+ match = proc and re.search(r"\d+\.\d+(?:\.\d+)?", proc.stdout or proc.stderr or "")
+ self._version = match.group(0) if match else ""
+ return self._version or None
+
@property
def configuration_error(self) -> str | None:
"""Return why an explicit executable override was rejected, if it was."""
@@ -179,11 +247,8 @@ class ExternalTool:
"pip-audit": bundled_scanner_hint,
"safety": "Optional secondary scanner; install separately with: uv tool install safety",
"bandit": bundled_scanner_hint,
- "semgrep": (
- "Install Semgrep separately with: brew install semgrep (macOS) "
- "or uv tool install semgrep"
- ),
- "skillspector": "Install with: uv tool install git+https://github.com/NVIDIA/[email protected]",
+ "semgrep": "Optional; install separately with: brew install semgrep (macOS) or uv tool install semgrep",
+ "skillspector": f"Install with: uv tool install {SKILLSPECTOR_WHEEL_URL}",
"skillevaluator": (
'Install with: uv tool install "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"'
),
@@ -225,7 +290,11 @@ class ExternalTool:
error_message=f"{self.name} not installed. {self.get_install_hint()}",
)
- cmd = [self._path, *args]
+ launch_args = self._launch_args()
+ if self._ensure is not None and self._path == self._ensure[0] and (error := self._ensure[1]()):
+ return ToolResult(success=False, stdout="", stderr="", exit_code=-1, error_message=error)
+
+ cmd = [self._path, *launch_args, *args]
if log_command:
logger.info(f"Running {self.name}: {' '.join(cmd[:5])}...")
@@ -327,7 +396,7 @@ class Tools:
tools and evaluation tools used by SkillEvaluator validators.
"""
- gitleaks = ExternalTool("Gitleaks", "gitleaks")
+ gitleaks = ExternalTool("Gitleaks", "gitleaks", managed=True)
pip_audit = ExternalTool("pip-audit", "pip-audit", prefer_interpreter_sibling=True)
safety = ExternalTool("Safety", "safety")
bandit = ExternalTool(
@@ -347,4 +416,5 @@ class Tools:
"skillspector",
prefer_interpreter_sibling=True,
override_env="SKILLEVALUATOR_SKILLSPECTOR_PATH",
+ managed=True,
)
diff --git a/src/skillevaluator/validators/code_risk.py b/src/skillevaluator/validators/code_risk.py
index 76faa195..f6dea895 100644
--- a/src/skillevaluator/validators/code_risk.py
+++ b/src/skillevaluator/validators/code_risk.py
@@ -9,6 +9,7 @@ code quality issues, and potential bugs via CWE/OWASP patterns.
from __future__ import annotations
+import os
from pathlib import Path
from skillevaluator.config import CONFIG_DIR
@@ -30,6 +31,9 @@ def _semgrep_file_excludes(skill_path: Path) -> list[str]:
return sorted(excludes)
+REQUIRE_SEMGREP_ENV = "SKILLEVALUATOR_REQUIRE_SEMGREP"
+
+
class CodeRiskValidator(ValidatorBase):
"""Static code analysis for security risks.
@@ -147,6 +151,7 @@ class CodeRiskValidator(ValidatorBase):
result.mark_scan_incomplete("bandit")
elif tool_result.exit_code in (0, 1):
# Exit code 0 = clean, 1 = findings; both emit JSON on stdout
+ result.record_scanner("bandit", Tools.bandit.version)
self._process_bandit_output(tool_result.stdout, result, exit_code=tool_result.exit_code)
else:
error_msg = tool_result.stderr or tool_result.stdout
@@ -264,8 +269,11 @@ class CodeRiskValidator(ValidatorBase):
result = ValidationResult()
if not Tools.semgrep.is_available:
- result.add_warning(f"semgrep not installed. {Tools.semgrep.get_install_hint()}")
- result.mark_scan_incomplete("semgrep")
+ if os.environ.get(REQUIRE_SEMGREP_ENV, "").strip().lower() in {"1", "true", "yes"}:
+ result.add_warning(f"semgrep not installed. {Tools.semgrep.get_install_hint()}")
+ result.mark_scan_incomplete("semgrep")
+ else:
+ result.add_message(f"Semgrep not installed; skipped. {Tools.semgrep.get_install_hint()}")
return result
rules_path = (CONFIG_DIR / "semgrep_rules.yaml").resolve()
@@ -296,6 +304,7 @@ class CodeRiskValidator(ValidatorBase):
result.mark_scan_incomplete("semgrep")
elif tool_result.exit_code in (0, 1):
# Exit code 0 = clean, 1 = findings; both emit JSON on stdout
+ result.record_scanner("semgrep", Tools.semgrep.version)
self._process_semgrep_output(tool_result.stdout, result, exit_code=tool_result.exit_code)
else:
error_msg = tool_result.stderr or tool_result.stdout or "no diagnostic output"
diff --git a/src/skillevaluator/validators/secrets.py b/src/skillevaluator/validators/secrets.py
index 418bc186..bb6bb645 100644
--- a/src/skillevaluator/validators/secrets.py
+++ b/src/skillevaluator/validators/secrets.py
@@ -130,9 +130,11 @@ tags = ["nvidia", "api-key"]
)
if tool_result.error_message:
- result.add_warning(tool_result.error_message)
+ hint = Tools.gitleaks.get_install_hint() if tool_result.error_message.startswith("couldn't ") else ""
+ result.add_warning(f"{tool_result.error_message}. {hint}".rstrip(". "))
result.mark_scan_incomplete("gitleaks")
return result
+ result.record_scanner("gitleaks", Tools.gitleaks.version)
# The custom findings code disambiguates findings from Gitleaks'
# default exit 1, which is also used for operational failures.
diff --git a/src/skillevaluator/validators/security.py b/src/skillevaluator/validators/security.py
index e862c689..5151732b 100644
--- a/src/skillevaluator/validators/security.py
+++ b/src/skillevaluator/validators/security.py
@@ -51,6 +51,7 @@ from skillevaluator.provider_config import (
from skillevaluator.skillspector_provenance import sanitize_skillspector_llm_provenance
from skillevaluator.spdx import is_spdx_only_html_comment
from skillevaluator.utils.path_security import matches_filesystem_name
+from skillevaluator.utils.scanner_install import prepare_skillspector
from skillevaluator.utils.secure_fs import SecurePathError, SecureRoot
from skillevaluator.utils.tool_runner import Tools, parse_json_output
from skillevaluator.validators.base import (
@@ -1299,9 +1300,17 @@ class SecurityValidator(ValidatorBase):
result.metadata["skillspector_llm_provenance"] = provenance
if tool_result.exit_code not in _SKILLSPECTOR_POLICY_EXIT_CODES:
- detail = tool_result.error_message or (
- f"{stage_name} failed with unexpected exit code {tool_result.exit_code}: "
- "scanner diagnostics were redacted"
+ error_message = tool_result.error_message
+ if not error_message and Tools.skillspector.resolution_source == "uv tool run":
+ error_message = prepare_skillspector(Tools.skillspector.path)
+ hint = Tools.skillspector.get_install_hint() if (error_message or "").startswith("couldn't ") else ""
+ detail = (
+ f"{error_message}. {hint}".rstrip(". ")
+ if error_message
+ else (
+ f"{stage_name} failed with unexpected exit code {tool_result.exit_code}: "
+ "scanner diagnostics were redacted"
+ )
)
result.add_error(detail)
result.mark_scan_incomplete(stage_name)
@@ -1325,6 +1334,7 @@ class SecurityValidator(ValidatorBase):
result.add_error(f"{stage_name} JSON output was not an object; security scan did not complete")
result.mark_scan_incomplete(stage_name)
return result
+ result.record_scanner("skillspector", Tools.skillspector.version)
if original_root is not None:
try:
diff --git a/tests/golden/cli_surface.json b/tests/golden/cli_surface.json
index 6a8181f6..ca2603d5 100644
--- a/tests/golden/cli_surface.json
+++ b/tests/golden/cli_surface.json
@@ -1638,6 +1638,14 @@
"param_type": "option",
"type": "text"
},
+ {
+ "name": "skip",
+ "opts": [
+ "--skip"
+ ],
+ "param_type": "option",
+ "type": "text"
+ },
{
"is_flag": true,
"name": "llm",
@@ -1733,6 +1741,14 @@
"param_type": "option",
"type": "text"
},
+ {
+ "name": "skip",
+ "opts": [
+ "--skip"
+ ],
+ "param_type": "option",
+ "type": "text"
+ },
{
"is_flag": true,
"name": "llm",
@@ -3284,6 +3300,14 @@
"param_type": "option",
"type": "text"
},
+ {
+ "name": "skip",
+ "opts": [
+ "--skip"
+ ],
+ "param_type": "option",
+ "type": "text"
+ },
{
"name": "previous_version",
"opts": [
diff --git a/tests/reporting/test_tier3_validate_cases.py b/tests/reporting/test_tier3_validate_cases.py
index b26186c5..15f89154 100644
--- a/tests/reporting/test_tier3_validate_cases.py
+++ b/tests/reporting/test_tier3_validate_cases.py
@@ -169,3 +169,46 @@ def test_reporter_pushes_case_rows_once_run_dir_is_known(tmp_path):
assert "cases 1 test case · k=2" in _text(pushed[-1])
assert "with skill · pdf-001 0 passed · 0 failed · 1 to go" in _text(pushed[-1])
assert "without" not in _text(pushed[-1])
+
+
+def test_setup_fix_names_failed_download_and_recovery():
+ from skillevaluator.models.result import ValidationResult
+ from skillevaluator.reporting.console_ui import _setup_fix
+
+ secrets = ValidationResult(
+ validator_name="Secrets Detection", passed=False, metadata={"incomplete_scans": ["gitleaks"]}
+ )
+ secrets.warnings.append(
+ "couldn't download Gitleaks 8.30.0 (connection refused). Install with: brew install gitleaks"
+ )
+ item = _setup_fix([secrets])
+ assert item.problem == "secrets didn't run: couldn't download Gitleaks 8.30.0 (connection refused)"
+ assert item.actions == [
+ "check network access to github.com, then rerun",
+ "or install it yourself: brew install gitleaks",
+ "or skip it: --skip secrets",
+ ]
+
+
+def test_setup_fix_names_failed_skillspector_setup():
+ from skillevaluator.models.result import ValidationResult
+ from skillevaluator.reporting.console_ui import _setup_fix
+
+ security = ValidationResult(
+ validator_name="Security Scan",
+ passed=False,
+ metadata={"incomplete_scans": ["skillspector"]},
+ )
+ security.errors.append(
+ "couldn't set up SkillSpector 2.13.3 (network is unreachable (os error 101)). "
+ "Install with: uv tool install skillspector.whl"
+ )
+ item = _setup_fix([security])
+ assert item.problem == (
+ "security didn't run: couldn't set up SkillSpector 2.13.3 (network is unreachable (os error 101))"
+ )
+ assert item.actions == [
+ "check network access to github.com and pypi.org, then rerun",
+ "or install it yourself: uv tool install skillspector.whl",
+ "or skip it: --skip security",
+ ]
diff --git a/tests/test_skip_checks.py b/tests/test_skip_checks.py
new file mode 100644
index 00000000..1a12457f
--- /dev/null
+++ b/tests/test_skip_checks.py
@@ -0,0 +1,106 @@
+"""``--skip``: user-skipped Tier 1 checks do not run, stay visible, and never certify Tier 1."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import TYPE_CHECKING
+from unittest.mock import patch
+
+from click.testing import CliRunner
+
+from skillevaluator.cli import cli
+from skillevaluator.reporting.base import is_cleanly_skipped, missing_tier1_publication_checks
+from skillevaluator.reporting.console_ui import summarize_tier1
+from skillevaluator.tier1.commands import run_validation, runs_secrets_scan
+
+if TYPE_CHECKING:
+ from skillevaluator.models.result import ValidationResult
+
+SKILL = Path(__file__).resolve().parents[1] / "src/skillevaluator/tier3/reference_skills/text-analyzer"
+
+
+def _by_name(results: list[ValidationResult]) -> dict[str, ValidationResult]:
+ return {r.validator_name: r for r in results}
+
+
+def test_skip_secrets_does_not_run_gitleaks_and_blocks_certification():
+ with patch("skillevaluator.validators.secrets.SecretsValidator.validate") as validate:
+ results = run_validation(SKILL, checks="code-integrity", skip="secrets")
+ validate.assert_not_called()
+ secrets = _by_name(results)["Secrets Detection"]
+ assert is_cleanly_skipped(secrets)
+ assert secrets.metadata["skip_reason"] == "skipped with --skip"
+ assert not _by_name(results)["Code Risk Analysis"].metadata.get("skipped")
+ assert "code-integrity" in missing_tier1_publication_checks(results)
+
+
+def test_skip_whole_check_records_skipped_result():
+ results = run_validation(SKILL, checks="unicode,schema", skip="unicode")
+ unicode = _by_name(results)["Unicode Smuggling Detection"]
+ assert is_cleanly_skipped(unicode)
+ assert "unicode" in missing_tier1_publication_checks(results)
+
+
+def test_skipped_check_is_visible_in_tier1_summary():
+ results = run_validation(SKILL, checks="unicode,schema", skip="unicode")
+ _ok, rows = summarize_tier1(results, lineup=["schema", "unicode"])
+ text = "\n".join("".join(seg for seg, _style in row.segments) for row in rows)
+ assert "\N{EN DASH} unicode" in text
+ assert "1/1 passed · 1 skipped" in text
+ assert "unicode with --skip, not checked" in text
+
+
+def test_runs_secrets_scan():
+ assert runs_secrets_scan(None, None)
+ assert not runs_secrets_scan(None, "secrets")
+ assert not runs_secrets_scan(None, "code")
+ assert not runs_secrets_scan("schema,security", None)
+
+
+def test_unknown_skip_is_a_usage_error():
+ result = CliRunner().invoke(cli, ["validate", str(SKILL), "--tiers", "1", "--skip", "nope"])
+ assert result.exit_code == 2
+ assert "unknown check nope" in result.output
+
+
+def test_skip_skips_gitleaks_setup():
+ with patch("skillevaluator.utils.scanner_install.ensure_gitleaks") as ensure:
+ from skillevaluator.cli import _offer_scanner_install
+
+ _offer_scanner_install(None, "secrets")
+ ensure.assert_not_called()
+
+
+def test_skip_security_skips_skillspector_setup():
+ with patch("skillevaluator.utils.scanner_install.prepare_skillspector") as prepare:
+ from skillevaluator.cli import _offer_scanner_install
+
+ _offer_scanner_install(None, "security,secrets")
+ prepare.assert_not_called()
+
+
+def test_tier1_summary_names_scanners_and_versions():
+ from skillevaluator.models.result import ValidationResult
+
+ security = ValidationResult(validator_name="Security Scan")
+ security.record_scanner("skillspector", "2.13.3")
+ code_risk = ValidationResult(validator_name="Code Risk Analysis")
+ code_risk.record_scanner("bandit", "1.9.4")
+ other = ValidationResult(validator_name="Code Risk Analysis")
+ other.record_scanner("semgrep", None)
+ code_risk.merge(other)
+ secrets = ValidationResult(validator_name="Secrets Detection", passed=False)
+ secrets.mark_scan_incomplete("gitleaks")
+ _ok, rows = summarize_tier1([security, code_risk, secrets])
+ row = next(row for row in rows if row.label == "scanners")
+ assert "".join(seg for seg, _style in row.segments) == (
+ "SkillSpector 2.13.3 · Bandit 1.9.4 · Semgrep unknown · Gitleaks didn't run"
+ )
+ assert code_risk.to_dict()["scanners"] == {"bandit": "1.9.4", "semgrep": "unknown"}
+
+
+def test_skipped_scanner_is_shown_as_skipped():
+ results = run_validation(SKILL, checks="code-integrity", skip="secrets")
+ _ok, rows = summarize_tier1(results)
+ row = next(row for row in rows if row.label == "scanners")
+ assert "Gitleaks skipped" in "".join(seg for seg, _style in row.segments)
diff --git a/tests/utils/test_scanner_install.py b/tests/utils/test_scanner_install.py
new file mode 100644
index 00000000..a29dd154
--- /dev/null
+++ b/tests/utils/test_scanner_install.py
@@ -0,0 +1,227 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Managed scanner provisioning: verified SkillSpector wheel, optional Semgrep, pinned Gitleaks."""
+
+import hashlib
+import io
+import os
+import tarfile
+from subprocess import CompletedProcess
+from unittest.mock import patch
+
+import pytest
+
+from skillevaluator.utils import scanner_install, tool_runner
+from skillevaluator.utils.tool_runner import ExternalTool, Tools
+from skillevaluator.validators.code_risk import REQUIRE_SEMGREP_ENV, CodeRiskValidator
+
+
+def _resolver(found: dict[str, str]):
+ return lambda command, **_kw: found.get(command)
+
+
+def runtime_dir(tmp_path, monkeypatch):
+ root = tmp_path / "runtimes"
+ monkeypatch.setenv("SKILLEVALUATOR_RUNTIME_DIR", str(root))
+ return root
+
+
+def _gitleaks_tarball(content: bytes = b"#!/bin/sh\necho gitleaks\n") -> bytes:
+ buf = io.BytesIO()
+ with tarfile.open(fileobj=buf, mode="w:gz") as archive:
+ info = tarfile.TarInfo("gitleaks")
+ info.size = len(content)
+ archive.addfile(info, io.BytesIO(content))
+ return buf.getvalue()
+
+
+class TestSkillSpectorWheel:
+ @pytest.mark.usefixtures("runtime_dir")
+ def test_prepare_downloads_wheel_and_builds_uv_environment(self, monkeypatch):
+ payload = b"wheel"
+ monkeypatch.setattr(scanner_install, "SKILLSPECTOR_WHEEL_SHA256", hashlib.sha256(payload).hexdigest())
+ with (
+ patch("urllib.request.urlopen", return_value=io.BytesIO(payload)),
+ patch("subprocess.run", return_value=CompletedProcess([], 0, "SkillSpector v2.13.3", "")) as run,
+ ):
+ assert scanner_install.prepare_skillspector("/opt/uv") is None
+ wheel = scanner_install.managed_skillspector_wheel()
+ assert wheel.read_bytes() == payload
+ assert run.call_args.args[0][-2:] == ["skillspector", "--version"]
+ assert str(wheel) in run.call_args.args[0]
+
+ @pytest.mark.usefixtures("runtime_dir")
+ def test_prepare_reports_environment_failure(self, monkeypatch):
+ monkeypatch.setattr(scanner_install, "ensure_skillspector_wheel", lambda: None)
+ with patch(
+ "subprocess.run",
+ return_value=CompletedProcess([], 2, "", "error: resolution failed\n Caused by: Network is unreachable\n"),
+ ):
+ error = scanner_install.prepare_skillspector("/opt/uv")
+ assert error == f"couldn't set up SkillSpector {scanner_install.SKILLSPECTOR_VERSION} (network is unreachable)"
+
+ def test_missing_skillspector_runs_verified_wheel_through_uv(self, runtime_dir, monkeypatch):
+ payload = b"wheel"
+ monkeypatch.setattr(scanner_install, "SKILLSPECTOR_WHEEL_SHA256", hashlib.sha256(payload).hexdigest())
+ with patch.object(tool_runner, "resolve_tool_path", _resolver({"uv": "/opt/uv"})):
+ tool = ExternalTool("skillspector", "skillspector", managed=True)
+ assert tool.is_available
+ assert tool.resolution_source == "uv tool run"
+ wheel = runtime_dir / "skillspector" / scanner_install.SKILLSPECTOR_VERSION / scanner_install.SKILLSPECTOR_WHEEL
+ with (
+ patch("urllib.request.urlopen", return_value=io.BytesIO(payload)) as urlopen,
+ patch("subprocess.run", return_value=CompletedProcess([], 0, "{}", "")) as run,
+ ):
+ tool.run(["scan", "x"])
+ tool.run(["scan", "y"])
+ urlopen.assert_called_once()
+ assert urlopen.call_args.args[0] == scanner_install.SKILLSPECTOR_WHEEL_URL
+ assert wheel.read_bytes() == payload
+ assert run.call_args.args[0] == [
+ "/opt/uv",
+ "tool",
+ "run",
+ "--quiet",
+ "--from",
+ str(wheel),
+ "skillspector",
+ "scan",
+ "y",
+ ]
+
+ @pytest.mark.usefixtures("runtime_dir")
+ def test_checksum_mismatch_never_runs(self):
+ with patch.object(tool_runner, "resolve_tool_path", _resolver({"uv": "/opt/uv"})):
+ tool = ExternalTool("skillspector", "skillspector", managed=True)
+ with (
+ patch("urllib.request.urlopen", return_value=io.BytesIO(b"tampered")),
+ patch("subprocess.run") as run,
+ ):
+ result = tool.run(["scan", "x"])
+ assert result.error_message == "SkillSpector 2.13.3 download failed checksum verification"
+ run.assert_not_called()
+
+ def test_pinned_wheel_url_matches_version(self):
+ assert scanner_install.SKILLSPECTOR_WHEEL_URL.endswith(
+ f"/v{scanner_install.SKILLSPECTOR_VERSION}/{scanner_install.SKILLSPECTOR_WHEEL}"
+ )
+ assert len(scanner_install.SKILLSPECTOR_WHEEL_SHA256) == 64
+
+ def test_installed_scanner_wins_over_fallback(self):
+ found = {"uv": "/opt/uv", "skillspector": "/usr/bin/skillspector"}
+ with patch.object(tool_runner, "resolve_tool_path", _resolver(found)):
+ tool = ExternalTool("skillspector", "skillspector", managed=True)
+ with patch("subprocess.run", return_value=CompletedProcess([], 0, "", "")) as run:
+ tool.run(["--version"])
+ assert run.call_args.args[0] == ["/usr/bin/skillspector", "--version"]
+
+ @pytest.mark.usefixtures("runtime_dir")
+ def test_patched_path_does_not_inherit_uv_arguments(self):
+ with patch.object(tool_runner, "resolve_tool_path", _resolver({"uv": "/opt/uv"})):
+ tool = ExternalTool("skillspector", "skillspector", managed=True)
+ with (
+ patch.object(tool, "_path", "/usr/bin/skillspector"),
+ patch("subprocess.run", return_value=CompletedProcess([], 0, "", "")) as run,
+ ):
+ tool.run(["--version"])
+ assert run.call_args.args[0] == ["/usr/bin/skillspector", "--version"]
+
+ @pytest.mark.parametrize("managed", [True, False])
+ def test_unavailable_without_uv_or_when_unmanaged(self, managed):
+ found = {} if managed else {"uv": "/opt/uv"}
+ with patch.object(tool_runner, "resolve_tool_path", _resolver(found)):
+ assert not ExternalTool("skillspector", "skillspector", managed=managed).is_available
+
+ def test_invalid_override_never_falls_back(self, monkeypatch):
+ monkeypatch.setenv("SKILLEVALUATOR_SKILLSPECTOR_PATH", "relative/skillspector")
+ with patch.object(tool_runner, "resolve_tool_path", _resolver({"uv": "/opt/uv"})):
+ tool = ExternalTool(
+ "skillspector", "skillspector", override_env="SKILLEVALUATOR_SKILLSPECTOR_PATH", managed=True
+ )
+ assert not tool.is_available
+ assert "SKILLEVALUATOR_SKILLSPECTOR_PATH" in tool.get_install_hint()
+
+
+class TestOptionalSemgrep:
+ @pytest.fixture
+ def skill_dir(self, tmp_path):
+ (tmp_path / "SKILL.md").write_text("---\nname: s\n---\n")
+ return tmp_path
+
+ def test_missing_semgrep_is_a_note_by_default(self, skill_dir, monkeypatch):
+ monkeypatch.delenv(REQUIRE_SEMGREP_ENV, raising=False)
+ with patch.object(Tools.semgrep, "_path", None):
+ result = CodeRiskValidator()._run_semgrep(skill_dir)
+ assert "incomplete_scans" not in result.metadata
+ assert not result.warnings
+ assert "Semgrep not installed; skipped" in result.messages[0]
+
+ def test_missing_semgrep_is_incomplete_when_required(self, skill_dir, monkeypatch):
+ monkeypatch.setenv(REQUIRE_SEMGREP_ENV, "1")
+ with patch.object(Tools.semgrep, "_path", None):
+ result = CodeRiskValidator()._run_semgrep(skill_dir)
+ assert result.metadata["incomplete_scans"] == ["semgrep"]
+
+
+class TestManagedGitleaks:
+ def test_missing_binary_installs_on_first_run_and_is_reused(self, runtime_dir, monkeypatch):
+ payload = _gitleaks_tarball()
+ key = ("linux", "x64")
+ monkeypatch.setattr(scanner_install, "gitleaks_platform", lambda: key)
+ monkeypatch.setattr(tool_runner, "gitleaks_platform", lambda: key)
+ monkeypatch.setattr(scanner_install, "_GITLEAKS_SHA256", {key: hashlib.sha256(payload).hexdigest()})
+ binary = runtime_dir / "gitleaks" / scanner_install.GITLEAKS_VERSION / "gitleaks"
+ with patch.object(tool_runner, "resolve_tool_path", _resolver({})):
+ tool = ExternalTool("Gitleaks", "gitleaks", managed=True)
+ assert tool.is_available
+ assert tool.path == str(binary)
+ assert tool.resolution_source == "managed"
+ assert not binary.exists()
+ with patch("urllib.request.urlopen", return_value=io.BytesIO(payload)) as urlopen:
+ result = tool.run(["version"])
+ tool.run(["version"])
+ assert result.success
+ assert urlopen.call_count == 1
+ assert urlopen.call_args.args[0] == scanner_install.gitleaks_url(key)
+ assert os.access(binary, os.X_OK)
+
+ @pytest.mark.usefixtures("runtime_dir")
+ def test_failed_first_run_install_is_reported_not_launched(self, monkeypatch):
+ monkeypatch.setattr(scanner_install, "gitleaks_platform", lambda: ("linux", "x64"))
+ monkeypatch.setattr(tool_runner, "gitleaks_platform", lambda: ("linux", "x64"))
+ with patch.object(tool_runner, "resolve_tool_path", _resolver({})):
+ tool = ExternalTool("Gitleaks", "gitleaks", managed=True)
+ with (
+ patch("urllib.request.urlopen", side_effect=OSError("network unreachable")),
+ patch("subprocess.run") as run,
+ ):
+ result = tool.run(["version"])
+ assert not result.success
+ assert result.error_message == "couldn't download Gitleaks 8.30.0 (network unreachable)"
+ run.assert_not_called()
+
+ @pytest.mark.usefixtures("runtime_dir")
+ def test_unsupported_platform_is_unavailable(self, monkeypatch):
+ monkeypatch.setattr(tool_runner, "gitleaks_platform", lambda: None)
+ with patch.object(tool_runner, "resolve_tool_path", _resolver({})):
+ assert not ExternalTool("Gitleaks", "gitleaks", managed=True).is_available
+
+ def test_checksum_mismatch_installs_nothing(self, runtime_dir, monkeypatch):
+ monkeypatch.setattr(scanner_install, "gitleaks_platform", lambda: ("linux", "x64"))
+ with patch("urllib.request.urlopen", return_value=io.BytesIO(_gitleaks_tarball())):
+ error = scanner_install.install_gitleaks()
+ assert error == "Gitleaks 8.30.0 download failed checksum verification"
+ assert not runtime_dir.exists()
+
+ @pytest.mark.usefixtures("runtime_dir")
+ def test_download_error_is_reported(self, monkeypatch):
+ monkeypatch.setattr(scanner_install, "gitleaks_platform", lambda: ("linux", "x64"))
+ with patch("urllib.request.urlopen", side_effect=OSError("network unreachable")):
+ assert scanner_install.install_gitleaks() == "couldn't download Gitleaks 8.30.0 (network unreachable)"
+
+ @pytest.mark.usefixtures("runtime_dir")
+ def test_unsupported_platform(self, monkeypatch):
+ monkeypatch.setattr(scanner_install, "gitleaks_platform", lambda: None)
+ assert "no pinned Gitleaks" in scanner_install.install_gitleaks()
diff --git a/tests/utils/test_tool_runner.py b/tests/utils/test_tool_runner.py
index a0ac20b5..9855604d 100644
--- a/tests/utils/test_tool_runner.py
+++ b/tests/utils/test_tool_runner.py
@@ -408,13 +408,13 @@ class TestExternalTool:
hint = tool.get_install_hint()
assert "gitleaks" in hint.lower()
- def test_skillspector_install_hint_uses_public_source_distribution(self):
+ def test_skillspector_install_hint_uses_release_wheel(self):
with patch("shutil.which", return_value=None):
tool = ExternalTool("SkillSpector", "skillspector")
assert (
- tool.get_install_hint()
- == "Install with: uv tool install git+https://github.com/NVIDIA/[email protected]"
+ tool.get_install_hint() == "Install with: uv tool install "
+ "https://github.com/NVIDIA/SkillSpector/releases/download/v2.13.3/skillspector-2.13.3-py3-none-any.whl"
)
assert "[security]" not in tool.get_install_hint()
@@ -496,3 +496,22 @@ class TestToolResult:
)
assert result.success is False
assert result.error_message == "Tool failed"
+
+
+def test_version_reports_pinned_release_for_managed_tools(tmp_path, monkeypatch):
+ from skillevaluator.utils.scanner_install import GITLEAKS_VERSION
+ from skillevaluator.utils.tool_runner import ExternalTool
+
+ monkeypatch.setenv("PATH", str(tmp_path))
+ monkeypatch.setenv("SKILLEVALUATOR_RUNTIME_DIR", str(tmp_path / "rt"))
+ assert ExternalTool("Gitleaks", "gitleaks", managed=True).version == GITLEAKS_VERSION
+
+
+def test_version_parses_tool_output(tmp_path, monkeypatch):
+ from skillevaluator.utils.tool_runner import ExternalTool
+
+ tool = tmp_path / "bandit"
+ tool.write_text("#!/bin/sh\necho 'bandit 1.9.4'\n")
+ tool.chmod(0o755)
+ monkeypatch.setenv("PATH", str(tmp_path))
+ assert ExternalTool("Bandit", "bandit").version == "1.9.4"
diff --git a/tests/validators/test_scan_incomplete.py b/tests/validators/test_scan_incomplete.py
index 99b594e1..3bc27073 100644
--- a/tests/validators/test_scan_incomplete.py
+++ b/tests/validators/test_scan_incomplete.py
@@ -70,7 +70,8 @@ class TestScanIncompleteMarking:
("gitleaks", lambda path: SecretsValidator()._validate_single_skill(path)),
],
)
- def test_missing_scanner_is_incomplete(self, tool_name, run_scan, skill_dir):
+ def test_missing_scanner_is_incomplete(self, tool_name, run_scan, skill_dir, monkeypatch):
+ monkeypatch.setenv("SKILLEVALUATOR_REQUIRE_SEMGREP", "1")
tool = getattr(Tools, tool_name)
with patch.object(tool, "_path", None):
result = run_scan(skill_dir)
diff --git a/tests/validators/test_security_validators.py b/tests/validators/test_security_validators.py
index bc5f7582..81971555 100644
--- a/tests/validators/test_security_validators.py
+++ b/tests/validators/test_security_validators.py
@@ -473,8 +473,9 @@ class TestCodeRiskValidator:
assert result.status == "incomplete"
assert not result.passed
- assert result.metadata["incomplete_scans"] == ["bandit", "semgrep"]
+ assert result.metadata["incomplete_scans"] == ["bandit"]
assert any("bandit not installed" in w for w in result.warnings)
+ assert any("Semgrep not installed; skipped" in m for m in result.messages)
def test_validate_bandit_clean(self, skill_with_security_issues: Path, mock_tool_available):
"""Test Bandit analysis with no findings."""