Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
feat(doctor): add Tier 1 scanner and Tier 2 provider readiness checks (#293)
Narendran Raghavan77fdf740197d
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 603dc915..3cd0df23 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -89,6 +89,21 @@ All notable changes to SkillEvaluator are documented in this file.expected-case checks cover only the cases that ran, `run_config.json` recordsthe selection as `case_selection`, and a run that skipped cases is reportedas publication-incomplete in JSON, HTML, and BENCHMARK.md.+- `doctor` and `health-check` take `--tier 1|2|3|all`, so Tier 1 and Tier 2+ readiness can be checked before a run instead of surfacing mid-validation.+ Tier 1 reports each scanner that the `--checks` selection runs: SkillSpector,+ Bandit, Semgrep, Gitleaks, and pip-audit for `dependency`. Each row shows the+ resolved path, any executable override, and the version from a bounded,+ shell-free probe. A scanner fails when it is missing, its override is+ malformed, its probe fails or exceeds `--timeout` (default 60 seconds, at+ most 3600), or it is older than Bandit 1.9.4 or Gitleaks 8. Tier 2 resolves+ the embedding and LLM providers offline. `doctor --tier 2 --verify-models`+ also sends one embedding request and one chat request, each bounded by+ `--timeout` and sent without retries, and reports only the HTTP status and a+ remedy. `--tier all` checks every tier, and a failed tier does not stop the+ next. Without `--tier`, both commands still check Tier 3 only. An option for+ an unselected tier, or `--timeout` for an offline Tier 2 check, is a usage+ error. `tier3 doctor` remains Tier 3-only.- Tier 1 `--llm` security scans of a single skill keep the LLM provenance thatSkillSpector v2.12.0 and later report: the provider adapter, the model anddiff --git a/docs/agents-and-sandboxes.mdx b/docs/agents-and-sandboxes.mdxindex 174626be..9c423799 100644--- a/docs/agents-and-sandboxes.mdx+++ b/docs/agents-and-sandboxes.mdx@@ -637,7 +637,10 @@ cleaner experiment.`doctor` and `health-check` run the same readiness checks and exit nonzero whenany check fails, so either can gate a script or CI job. Warnings from`doctor --verify-models` are nonblocking because the catalog result is-inconclusive rather than a definitive rejection.+inconclusive rather than a definitive rejection. Both commands check Tier 3 by+default. `--tier 1` checks the Tier 1 scanners, `--tier 2` the embedding and LLM+providers, and `--tier all` every tier; see the+[CLI Reference](cli-reference.mdx#doctor).### doctordiff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdxindex 3a854b9f..94f12cde 100644--- a/docs/cli-reference.mdx+++ b/docs/cli-reference.mdx@@ -38,8 +38,8 @@ commands referenced below.| [compare](#compare) | Compare live-evaluation results across agents | A previous `tier3 evaluate` run || [dashboard](#dashboard) | Filter quantitative tables and paired usage differences | Saved JSON reports or retained runs; dashboard extra || [harbor-view](#harbor-view) | Browse retained Harbor job artifacts | A run made with `--harbor-keep-jobs` |-| [doctor](#doctor) | Live-evaluation readiness check | Nothing |-| [health-check](#health-check) | Quick CLI + backend readiness check | Nothing |+| [doctor](#doctor) | Live-evaluation readiness check, or Tier 1 scanners and Tier 2 providers with `--tier` | Nothing |+| [health-check](#health-check) | Quick CLI + backend readiness check, or another tier with `--tier` | Nothing || [models](#models) | List models visible through the selected provider's catalog | Provider configuration/key || [tier3 validate](#tier3-validate) | Validate `evals/` and the optional Harbor contract | Nothing |@@ -582,26 +582,47 @@ Check live-evaluation runtime readiness: the CLI install, the configured provideskillevaluator doctor --env-mode docker```+`--tier` selects other readiness checks. Without it, `doctor` checks Tier 3 only, as above.++```bash title="Tier 1 scanners and Tier 2 providers"+skillevaluator doctor --tier 1 --checks security,code-integrity,dependency+skillevaluator doctor --tier 2 --verify-models+skillevaluator doctor --tier all+```++- **Tier 1** reports each external scanner the selected checks run: SkillSpector for `security`, Bandit, Semgrep and Gitleaks for `code-integrity`, and pip-audit for `dependency`. Each row shows the resolved executable, the override variable when one selected it, and the version from a bounded version probe. The probe never starts a shell. A scanner fails when it is missing, its override is malformed, or the probe exits nonzero or times out. It also fails when it is older than the release the validators parse: Bandit 1.9.4 or Gitleaks 8. An unreported version is a warning.+- **Tier 2** checks that the embedding provider and the LLM provider used for duplicate analysis resolve, without a network request. With `--verify-models`, it also sends one embedding request and one chat request, each bounded by `--timeout` and sent without retries. Failures show only the HTTP status and a remedy, never the provider's response body.+- **all** runs Tiers 1, 2 and 3 in order. A failed tier does not stop the next, and the exit code is nonzero when any tier fails.++An option for a tier that is not selected exits with a usage error before any check runs. For example, `--checks` without `--tier 1` or `all` is an error, and so is `--timeout` with `--tier 2` but no `--verify-models`, because offline Tier 2 sends no request.+| Flag | Default | Effect || --- | --- | --- |+| `--tier [1\|2\|3\|all]` | `3` | Readiness to check: Tier 1 scanners, Tier 2 providers, Tier 3 live evaluation, or all three in order. |+| `--checks TEXT` | `validate`'s default checks | Tier 1 only. Comma-separated checks whose scanners to probe, with the same names as `validate --checks`; add `dependency` to include pip-audit. |+| `--timeout FLOAT` | `60` | Tier 1, and Tier 2 with `--verify-models`. Seconds allowed for each scanner version probe and each Tier 2 live request, including a Bedrock chat request; must be finite, greater than 0 and at most 3600. || `-a, --agents TEXT` | provider-native | Comma-separated agents to check. Defaults: NVIDIA Build=`opencode`, OpenAI=`codex`, Anthropic=`claude-code`. || `--env-mode` | `docker` | Backend to check (same 27 modes as [tier3 evaluate](#tier3-evaluate)). || `--environment-kwarg`, `--ek` | none | `KEY=VALUE` non-secret constructor option for a Harbor-native non-Docker backend (repeatable). Pass the same values intended for evaluation. Docker and local reject all kwargs, and Kata accepts only `kata_runtime` and `kata_dns`; policy-owned fields are rejected for every eligible backend. || `--agent-model TEXT` | unset | Per-agent model override, `AGENT=MODEL` (repeatable) — check readiness with the model each agent will actually run. |-| `--verify-models` | off | Live per-agent catalog probe. Prints pass for verified access, warn for an inconclusive or non-authoritative result (success or failure), and fail only for a definitive credential, model, or configuration rejection. |+| `--verify-models` | off | Tier 3: live per-agent catalog probe. Prints pass for verified access, warn for an inconclusive or non-authoritative result (success or failure), and fail only for a definitive credential, model, or configuration rejection. Tier 2: one embedding and one chat request. |-`skillevaluator tier3 doctor` takes the same flags, including `--agent-model` and `--verify-models`.+`-a`, `--env-mode`, `--environment-kwarg` and `--agent-model` apply to Tier 3 only. `skillevaluator tier3 doctor` checks only Tier 3. It takes the Tier 3 flags, including `--agent-model` and `--verify-models`, and has no `--tier`, `--checks` or `--timeout`.## health-check-Quick readiness check for the CLI and the selected live-eval backend. Use it as an install smoke test or a fast CI preflight. It differs from [doctor](#doctor) by two flags: `health-check` has no `--agent-model` and no `--verify-models`.+Quick readiness check for the CLI and the selected live-eval backend. Use it as an install smoke test or a fast CI preflight. It differs from [doctor](#doctor) by two flags: `health-check` has no `--agent-model` and no `--verify-models`, so it never sends a live provider request. `--tier` and `--checks` work as in `doctor`, and `--timeout` bounds only the Tier 1 scanner probes.```bash title="Quick readiness check"skillevaluator health-check+skillevaluator health-check --tier 1 # CI preflight for the Tier 1 scanners```| Flag | Default | Effect || --- | --- | --- |+| `--tier [1\|2\|3\|all]` | `3` | Readiness to check, as in [doctor](#doctor). Tier 2 resolves providers offline. |+| `--checks TEXT` | `validate`'s default checks | Tier 1 only. Checks whose scanners to probe. |+| `--timeout FLOAT` | `60` | Tier 1 only. Seconds allowed for each scanner version probe; must be finite, greater than 0 and at most 3600. || `-a, --agents TEXT` | provider-native | Comma-separated agents to check. Defaults: NVIDIA Build=`opencode`, OpenAI=`codex`, Anthropic=`claude-code`. || `--env-mode` | `docker` | Backend to check (same 27 modes as [tier3 evaluate](#tier3-evaluate)). || `--environment-kwarg`, `--ek` | none | `KEY=VALUE` non-secret constructor option for a Harbor-native non-Docker backend (repeatable). Pass the same values intended for evaluation. Docker and local reject all kwargs, and Kata accepts only `kata_runtime` and `kata_dns`; policy-owned fields are rejected for every eligible backend. |diff --git a/docs/configuration.mdx b/docs/configuration.mdxindex 08c92feb..0e80a553 100644--- a/docs/configuration.mdx+++ b/docs/configuration.mdx@@ -55,7 +55,7 @@ contact package and vulnerability services.| `doctor`, `health-check` (offline readiness probes by default; missing credentials are reported as not ready) | `--llm-verify` on `validate`, `security-scan`, `pii-scan` — LLM false-positive suppression || | All Tier 2 commands — embeddings API; `context-optimization-check` and `dedup-scan` also use a chat LLM (`similarity-check` is embeddings-only). A local OpenAI-compatible server works too — see [Fully local setup](#fully-local-setup) || Tier 3 scaffolding and inspection: `create-eval-dataset --no-llm`, `init-custom-grader`, `init-harbor-task`, `tier3 validate`, `view`, `harbor-view`, and `compare` | Tier 3 LLM dataset generation needs a provider key; `tier3 evaluate` needs a provider key plus a credential the selected live agent can use — with NVIDIA Build in `--env-mode docker` or `local`, one `NVIDIA_API_KEY` covers both roles |-| | `models` and `doctor --verify-models` — provider-catalog reachability and membership checks; public catalogs do not prove that an inference key works |+| | `models` and `doctor --verify-models` — provider-catalog reachability and membership checks; public catalogs do not prove that an inference key works. `doctor --tier 2 --verify-models` sends one embedding and one chat request |`validate PATH` runs all three tiers for skills and needs the provider andTier 3 runtime setup. Use `--tiers 1` for keyless static checks. `--no-dedup`diff --git a/docs/installation.mdx b/docs/installation.mdxindex d81ffcb4..1fff4c76 100644--- a/docs/installation.mdx+++ b/docs/installation.mdx@@ -195,6 +195,8 @@ uv tool install git+https://github.com/NVIDIA/[email protected]On other systems, install Gitleaks with `go install github.com/gitleaks/gitleaks/v8@latest` or a binary from the [Gitleaks releases](https://github.com/gitleaks/gitleaks/releases). Semgrep and SkillSpector must be on `PATH`, or configured with `SKILLEVALUATOR_SEMGREP_PATH` and `SKILLEVALUATOR_SKILLSPECTOR_PATH`. RHEL 8 and compatible enterprise Linux systems can supply compatible executables independently of the SkillEvaluator Python environment.Docker is required only for Tier 3 Docker-mode evaluation. Agent CLIs and their credentials are user-supplied — see [Agents & Sandboxes](agents-and-sandboxes.mdx).++`skillevaluator health-check --tier 1` reports the path and version of each installed scanner and exits non-zero when one is missing, misconfigured, or older than the supported release. Add `--checks security,code-integrity,dependency` to include pip-audit.</Note>## Verify your installdiff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdxindex e511bf44..2f2d7dc8 100644--- a/docs/tier1-validation.mdx+++ b/docs/tier1-validation.mdx@@ -63,7 +63,7 @@ for skills. Its Tier 1 LLM checks still run only when requested.skillevaluator validate ./my-skill --tiers 1```- With `skillevaluator[security]`, Semgrep, SkillSpector, and Gitleaks installed, every required scanner produces evidence and the run is deterministic — the same input yields the same exit code, with nothing leaving your machine.+ With `skillevaluator[security]`, Semgrep, SkillSpector, and Gitleaks installed, every required scanner produces evidence and the run is deterministic — the same input yields the same exit code, with nothing leaving your machine. Run `skillevaluator doctor --tier 1` first to confirm that each scanner resolves and reports a supported version.### Narrow to specific checks@@ -406,7 +406,7 @@ Tier 1 checks always gate the exit code inside `validate`. Tier 2 gates when itInstall it with `brew install gitleaks` or download a binary from the official [Gitleaks releases](https://github.com/gitleaks/gitleaks/releases).</Accordion><Accordion title="A scanner is missing">- Install Bandit and pip-audit with `skillevaluator[security]`. Install Semgrep separately with `brew install semgrep` or `uv tool install semgrep`, and SkillSpector with `uv tool install git+https://github.com/NVIDIA/[email protected]`. Required scanner failures remain `INCOMPLETE` until the scanner is installed and returns valid evidence.+ Install Bandit and pip-audit with `skillevaluator[security]`. Install Semgrep separately with `brew install semgrep` or `uv tool install semgrep`, and SkillSpector with `uv tool install git+https://github.com/NVIDIA/[email protected]`. Required scanner failures remain `INCOMPLETE` until the scanner is installed and returns valid evidence. `skillevaluator doctor --tier 1` lists every scanner the selected checks need, with its resolved path and version.</Accordion><Accordion title="SkillSpector reports incomplete analysis for a valid helper script">SkillSpector releases before v2.13.1 can stop inspecting a valid Python or Perl helper early and record a `static_parse_limit`, `obfuscated_instruction_text`, or `runtime_limit` ledger exception. The security check is then `INCOMPLETE`, often with a High **Incomplete referenced artifact analysis (AE1)** finding at each line that names the helper. Check the installed release with `skillspector --version` and upgrade with `uv tool install git+https://github.com/NVIDIA/[email protected]`. If the exception remains after the upgrade, investigate it like any other incomplete scan.diff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.pyindex 99ec28e7..83a04678 100644--- a/src/skillevaluator/cli.py+++ b/src/skillevaluator/cli.py@@ -24,6 +24,7 @@ from skillevaluator.constants import (SIMILARITY_DEFAULT_MAX_SCALAR_COMPARISONS,SIMILARITY_MAX_ENTRIES,)+from skillevaluator.doctor import DEFAULT_PROBE_TIMEOUT_SEC, MAX_PROBE_TIMEOUT_SECfrom skillevaluator.logging_config import setup_loggingfrom skillevaluator.models.result import ValidationResultfrom skillevaluator.quality_config import QualityConfigError, load_quality_config@@ -47,6 +48,7 @@ from skillevaluator.reporting.naming import report_basename# imported lazily inside the command callbacks. This keeps `import skillevaluator.cli`# and the CLI surface available on a base install without those extras.from skillevaluator.tier1.commands import (+ RECOGNIZED_CHECKS,console,emit_reports,enabled_check_lineup,@@ -73,6 +75,16 @@ CONTEXT_SETTINGS = {"help_option_names": ["-h", "--help"]}ENV_MODE_CHOICE = click.Choice(list(all_env_modes()))+class _FiniteFloatRange(click.FloatRange):+ """Click float range that also rejects NaN and infinities."""++ def convert(self, value, param, ctx):+ converted = super().convert(value, param, ctx)+ if not math.isfinite(converted):+ self.fail(f"{value!r} is not a finite number", param, ctx)+ return converted++class _AliasChoice(click.Choice):"""Expose current names in help while accepting retired aliases."""@@ -3193,53 +3205,161 @@ def dashboard_command(paths: tuple[Path, ...], port: int | None, no_browser: booraise SystemExit(launch_dashboard(paths, port=port, no_browser=no_browser))+_DOCTOR_TIERS = ("1", "2", "3", "all")+# Options that apply only to some tiers. An explicit value for a tier that was+# not selected is rejected instead of being silently ignored.+_DOCTOR_OPTION_TIERS = {+ "checks": frozenset({"1"}),+ "timeout": frozenset({"1", "2"}),+ "verify_models": frozenset({"2", "3"}),+ "agents": frozenset({"3"}),+ "env_mode": frozenset({"3"}),+ "environment_kwarg": frozenset({"3"}),+ "agent_model": frozenset({"3"}),+}+++def _doctor_tier_options(*, timeout_help: str):+ """Add the tier selection and Tier 1/2 options shared by doctor and health-check."""++ def decorate(func):+ func = click.option(+ "--timeout",+ default=DEFAULT_PROBE_TIMEOUT_SEC,+ show_default=True,+ type=_FiniteFloatRange(min=0, min_open=True, max=MAX_PROBE_TIMEOUT_SEC),+ help=timeout_help,+ )(func)+ func = click.option(+ "--checks",+ default=None,+ help="Tier 1 checks whose scanners to probe, named as in validate --checks (default: validate's checks).",+ )(func)+ return click.option(+ "--tier",+ type=click.Choice(_DOCTOR_TIERS),+ default="3",+ show_default=True,+ help="Readiness to check: 1 scanners, 2 embedding and LLM providers, 3 live evaluation, or all.",+ )(func)++ return decorate+++def _tier3_doctor_options(func):+ """Add the Tier 3 options shared by doctor and tier3 doctor."""+ func = click.option("--agent-model", multiple=True, help="Per-agent model override, AGENT=MODEL.")(func)+ func = click.option(+ "--environment-kwarg",+ "--ek",+ multiple=True,+ help="Harbor environment constructor kwarg, KEY=VALUE. Repeat for multiple values; never pass secrets.",+ )(func)+ func = click.option("--env-mode", default="docker", show_default=True, type=ENV_MODE_CHOICE, metavar="MODE")(func)+ return click.option(+ "-a",+ "--agents",+ default=None,+ help=(+ "Comma-separated Harbor agents (claude aliases claude-code). Default follows the provider: "+ "NVIDIA Build=opencode, OpenAI=codex, Anthropic=claude-code."+ ),+ )(func)+++def _run_doctor(+ ctx: click.Context,+ *,+ tier: str,+ agents: str | None,+ env_mode: str,+ environment_kwarg: tuple[str, ...],+ checks: str | None = None,+ timeout: float = DEFAULT_PROBE_TIMEOUT_SEC,+ agent_model: tuple[str, ...] = (),+ verify_models: bool = False,+) -> int:+ """Validate every option before any probe, then check each selected tier in order."""+ selected_tiers = ("1", "2", "3") if tier == "all" else (tier,)++ def explicit(name: str) -> bool:+ return ctx.get_parameter_source(name) not in (None, click.core.ParameterSource.DEFAULT)++ for name, tiers in _DOCTOR_OPTION_TIERS.items():+ if explicit(name) and tiers.isdisjoint(selected_tiers):+ raise click.UsageError(f"--{name.replace('_', '-')} does not apply to --tier {tier}.")+ if explicit("timeout") and selected_tiers == ("2",) and not verify_models:+ # Tier 2 stays offline unless --verify-models sends live requests, so+ # there is nothing for --timeout to bound.+ condition = " without --verify-models" if "verify_models" in ctx.params else ""+ raise click.UsageError(f"--timeout does not apply to --tier 2{condition}.")+ if "1" in selected_tiers:+ unknown = sorted(set(enabled_check_lineup(checks)) - RECOGNIZED_CHECKS)+ if unknown:+ raise click.BadParameter(f"unknown Tier 1 check(s): {', '.join(unknown)}", param_hint="'--checks'")+ parsed_environment_kwargs = (+ _parse_environment_kwargs_cli(environment_kwarg, env_mode=env_mode) if "3" in selected_tiers else {}+ )++ # Tiers are independent: a failed tier never hides the readiness of the next.+ exit_codes = []+ for selected in selected_tiers:+ if selected == "3":+ from skillevaluator.tier3.commands import doctor as tier3_doctor++ exit_codes.append(+ tier3_doctor(+ agents=agents,+ env_mode=env_mode,+ environment_kwarg=environment_kwarg,+ verify_models=verify_models,+ agent_model=agent_model,+ environment_kwargs=parsed_environment_kwargs,+ )+ )+ else:+ from skillevaluator.doctor import doctor as lower_tier_doctor++ exit_codes.append(+ lower_tier_doctor(tier=selected, checks=checks, verify_models=verify_models, timeout=timeout)+ )+ return 1 if any(exit_codes) else 0++@cli.command()+@_doctor_tier_options(+ timeout_help="Seconds allowed for each Tier 1 scanner probe and, with --verify-models, each Tier 2 live request."+)+@_tier3_doctor_options@click.option(- "-a",- "--agents",- default=None,+ "--verify-models",+ is_flag=True,help=(- "Comma-separated Harbor agents (claude aliases claude-code). Default follows the provider: "- "NVIDIA Build=opencode, OpenAI=codex, Anthropic=claude-code."+ "Make live credential-bearing requests: Tier 3 checks resolved agent-model catalog reachability; "+ "Tier 2 sends one embedding and one chat request."),)[email protected]("--env-mode", default="docker", show_default=True, type=ENV_MODE_CHOICE, metavar="MODE")- "--environment-kwarg",- "--ek",- multiple=True,- help="Harbor environment constructor kwarg, KEY=VALUE. Repeat for multiple values; never pass secrets.",-)[email protected]("--agent-model", multiple=True, help="Per-agent model override, AGENT=MODEL.")[email protected]_context+def doctor(ctx: click.Context, **options: Any) -> None:+ """Check live-evaluation readiness, or select another tier with --tier."""+ raise SystemExit(_run_doctor(ctx, **options))++[email protected]("doctor", cls=cli.command_class, context_settings=CONTEXT_SETTINGS)+@_tier3_doctor_options@click.option("--verify-models",is_flag=True,help="Check resolved agent-model catalog reachability with a live credential-bearing request.",)-def doctor(- agents: str | None,- env_mode: str,- environment_kwarg: tuple[str, ...],- agent_model: tuple[str, ...],- verify_models: bool,-) -> None:[email protected]_context+def tier3_doctor_command(ctx: click.Context, **options: Any) -> None:"""Check live-evaluation runtime readiness."""- from skillevaluator.tier3.commands import doctor as tier3_doctor-- parsed_environment_kwargs = _parse_environment_kwargs_cli(environment_kwarg, env_mode=env_mode)- raise SystemExit(- tier3_doctor(- agents=agents,- env_mode=env_mode,- environment_kwarg=environment_kwarg,- verify_models=verify_models,- agent_model=agent_model,- environment_kwargs=parsed_environment_kwargs,- )- )+ raise SystemExit(_run_doctor(ctx, tier="3", **options))@cli.command("health-check")+@_doctor_tier_options(timeout_help="Seconds allowed for each Tier 1 scanner probe.")@click.option("-a","--agents",@@ -3253,21 +3373,10 @@ def doctor(multiple=True,help="Harbor environment constructor kwarg, KEY=VALUE. Repeat for multiple values; never pass secrets.",)-def health_check(agents: str | None, env_mode: str, environment_kwarg: tuple[str, ...]) -> None:- """Quick readiness check for the CLI and selected live-eval backend."""- from skillevaluator.tier3.commands import doctor as tier3_doctor-- parsed_environment_kwargs = _parse_environment_kwargs_cli(environment_kwarg, env_mode=env_mode)- raise SystemExit(- tier3_doctor(- agents=agents,- env_mode=env_mode,- environment_kwarg=environment_kwarg,- verify_models=False,- agent_model=(),- environment_kwargs=parsed_environment_kwargs,- )- )[email protected]_context+def health_check(ctx: click.Context, **options: Any) -> None:+ """Quick readiness check for the CLI and selected live-eval backend, or another tier with --tier."""+ raise SystemExit(_run_doctor(ctx, **options))@tier3.command("validate")@@ -3318,7 +3427,7 @@ tier3.add_command(init_custom_grader, "init-custom-grader")tier3.add_command(init_harbor_task, "init-harbor-task")tier3.add_command(compare, "compare")tier3.add_command(view, "view")-tier3.add_command(doctor, "doctor")+tier3.add_command(tier3_doctor_command, "doctor")from skillevaluator.tier3.workflow import build_tier3_workflowdiff --git a/src/skillevaluator/doctor.py b/src/skillevaluator/doctor.pynew file mode 100644index 00000000..96b60b8f--- /dev/null+++ b/src/skillevaluator/doctor.py@@ -0,0 +1,248 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Tier 1 and Tier 2 readiness checks for ``doctor`` and ``health-check``.++Tier 3 readiness lives in :func:`skillevaluator.tier3.commands.doctor`. This+module never imports Harbor, and Tier 2 loads provider SDKs only for an+explicit live probe, so Tier 1 readiness works on a base install.+"""++from __future__ import annotations++import importlib.util+import math+import re+from collections.abc import Callable+from typing import Any++from rich.box import SIMPLE+from rich.console import Console+from rich.table import Table++from skillevaluator import __version__+from skillevaluator.utils.rich_markup import escape_markup++Row = tuple[str, str, str]++DEFAULT_PROBE_TIMEOUT_SEC = 60.0+# A huge timeout overflows the subprocess timer, and no readiness probe needs+# more than an hour.+MAX_PROBE_TIMEOUT_SEC = 3600.0++# Tier 1 checks that run an external scanner, with the ``Tools`` entries they+# launch, in the order the checks run them.+_SCANNERS_BY_CHECK: dict[str, tuple[str, ...]] = {+ "security": ("skillspector",),+ "code-integrity": ("bandit", "semgrep", "gitleaks"),+ "dependency": ("pip_audit",),+}+# Older releases lack the CLI or report contract the validators parse: the+# Bandit floor matches the security extra, and Gitleaks 8 introduced the+# ``detect`` command the secrets scan runs.+_MINIMUM_SCANNER_VERSIONS: dict[str, tuple[int, int, int]] = {+ "bandit": (1, 9, 4),+ "gitleaks": (8, 0, 0),+}+_SCANNER_LABELS = {"skillspector": "SkillSpector"}+_VERSION_RE = re.compile(r"(\d+)\.(\d+)\.(\d+)")+_PROBE_TEXT = "SkillEvaluator readiness check"++# emoji=False: ":name:" codes in configuration values must print literally.+console = Console(emoji=False)+++def _probe_run_options(command: str) -> dict[str, Any]:+ """Launch a version probe with the environment the scanner's validator uses."""+ if command == "skillspector":+ # The static SkillSpector scan runs without provider credentials.+ from skillevaluator.validators.security import _skillspector_process_env++ return {"env": _skillspector_process_env(), "replace_env": True}+ if command == "semgrep":+ # The scan passes --disable-version-check; keep the probe offline too.+ return {"env": {"SEMGREP_ENABLE_VERSION_CHECK": "0"}}+ return {}+++def _scanner_row(tool: Any, timeout: float) -> Row:+ """Report one scanner's resolved executable and version without a shell."""+ label = _SCANNER_LABELS.get(tool.command, tool.name)+ if tool.configuration_error:+ return (label, "fail", tool.configuration_error)+ if not tool.is_available:+ return (label, "fail", f"not installed. {tool.get_install_hint()}")+ location = tool.path+ if tool.resolution_source != "automatic":+ location += f" (from {tool.override_env})"+ version_args = ["version"] if tool.command == "gitleaks" else ["--version"]+ result = tool.run(version_args, timeout=timeout, log_command=False, **_probe_run_options(tool.command))+ if result.error_message:+ return (label, "fail", result.error_message)+ if result.exit_code != 0:+ return (label, "fail", f"version probe exited with status {result.exit_code}; {location}")+ match = _VERSION_RE.search(result.stdout) or _VERSION_RE.search(result.stderr)+ if match is None:+ return (label, "warn", f"version not reported; {location}")+ version = tuple(int(part) for part in match.groups())+ minimum = _MINIMUM_SCANNER_VERSIONS.get(tool.command)+ if minimum is not None and version < minimum:+ required = ".".join(str(part) for part in minimum)+ return (+ label,+ "fail",+ f"{match.group(0)} at {location} is older than the supported {required}. {tool.get_install_hint()}",+ )+ return (label, "pass", f"{match.group(0)} at {location}")+++def _tier1_rows(checks: str | None, timeout: float) -> list[Row]:+ from skillevaluator.tier1.commands import RECOGNIZED_CHECKS, enabled_check_lineup+ from skillevaluator.utils.tool_runner import Tools++ lineup = enabled_check_lineup(checks)+ unknown = sorted(set(lineup) - RECOGNIZED_CHECKS)+ if unknown:+ raise ValueError(f"Unknown Tier 1 check(s): {', '.join(unknown)}")+ names = dict.fromkeys(name for check in lineup for name in _SCANNERS_BY_CHECK.get(check, ()))+ if not names:+ return [("Tier 1 scanners", "pass", "the selected checks run no external scanner")]+ return [_scanner_row(getattr(Tools, name), timeout) for name in names]+++def _module_available(module: str) -> bool:+ # find_spec does not import the module, and may raise when a meta-path+ # blocker is active; either way the extra is unavailable.+ try:+ return importlib.util.find_spec(module) is not None+ except (ImportError, ValueError):+ return False+++def _live_row(name: str, probe: Callable[[], str], diagnose: Callable[[Exception], str]) -> Row:+ """Run one live request; report only a bounded diagnostic, never a provider body."""+ try:+ detail = probe()+ except Exception as exc:+ return (name, "fail", diagnose(exc))+ return (name, "pass", detail)+++def _probe_embedding(http_client: Any, timeout: float) -> str:+ from skillevaluator.embedding.client import EmbeddingClient++ client = EmbeddingClient(max_retries=0, http_client=http_client, timeout=timeout)+ vector = client.embed_single(_PROBE_TEXT)+ return f"returned a {len(vector)}-dimension vector"+++def _embedding_failure(exc: Exception) -> str:+ from skillevaluator.embedding.client import SimilarityConfigError++ if isinstance(exc, SimilarityConfigError):+ # EmbeddingClient builds these messages without provider response bodies.+ return str(exc)+ return f"Embedding request failed ({type(exc).__name__})."+++def _probe_chat(http_client: Any, timeout: float) -> str:+ from skillevaluator.constants import CONTENT_DEDUP_LLM_REASONING_EFFORT+ from skillevaluator.inference.client import LLMClient++ # The same client settings as Tier 2 duplicate analysis, without retries.+ client = LLMClient(+ reasoning_effort=CONTENT_DEDUP_LLM_REASONING_EFFORT,+ max_retries=0,+ http_client=http_client,+ timeout=timeout,+ )+ client.completions("Reply with the single word OK.", _PROBE_TEXT)+ return "returned a chat completion"+++def _tier2_rows(*, verify_models: bool, timeout: float) -> list[Row]:+ if not _module_available("openai"):+ return [("Tier 2 extra", "fail", "not installed. Reinstall SkillEvaluator with the tier2 or all extra.")]++ from skillevaluator.inference.diagnostics import llm_failure_diagnostic, safe_llm_labels+ from skillevaluator.provider_config import (+ ProviderConfigurationError,+ resolve_embedding_provider,+ resolve_llm_provider,+ )++ rows: list[Row] = []+ resolved: set[str] = set()+ for label, resolve in (("Embedding provider", resolve_embedding_provider), ("LLM provider", resolve_llm_provider)):+ try:+ config = resolve()+ except ProviderConfigurationError as exc:+ rows.append((label, "fail", str(exc)))+ continue+ provider, model = safe_llm_labels(config.provider, config.model)+ rows.append((label, "pass", f"{provider} / {model}"))+ resolved.add(label)+ if not verify_models:+ return rows++ import httpx++ # Each SDK client gets the timeout explicitly and never retries; the shared+ # HTTP client only scopes the connections to this check.+ with httpx.Client(timeout=timeout) as http_client:+ if "Embedding provider" in resolved:+ rows.append(+ _live_row("Embedding model", lambda: _probe_embedding(http_client, timeout), _embedding_failure)+ )+ if "LLM provider" in resolved:+ rows.append(+ _live_row("Deduplication model", lambda: _probe_chat(http_client, timeout), llm_failure_diagnostic)+ )+ return rows+++def readiness_rows(+ *,+ tier: str,+ checks: str | None = None,+ verify_models: bool = False,+ timeout: float = DEFAULT_PROBE_TIMEOUT_SEC,+) -> list[Row]:+ """Return ``(check, pass|warn|fail, detail)`` rows for Tier 1 or Tier 2.++ Tier 1 reports the external scanner each selected check runs. Tier 2+ resolves the embedding and LLM providers offline; ``verify_models`` also+ sends one embedding and one chat request. ``timeout`` bounds each scanner+ probe and each live request.+ """+ if tier not in {"1", "2"}:+ raise ValueError("Expected tier 1 or 2")+ if not math.isfinite(timeout) or not 0 < timeout <= MAX_PROBE_TIMEOUT_SEC:+ raise ValueError(f"Timeout must be finite, positive and at most {MAX_PROBE_TIMEOUT_SEC:g} seconds")+ rows: list[Row] = [("CLI package", "pass", f"skillevaluator {__version__}")]+ if tier == "1":+ rows.extend(_tier1_rows(checks, timeout))+ else:+ rows.extend(_tier2_rows(verify_models=verify_models, timeout=timeout))+ return rows+++def doctor(+ *,+ tier: str,+ checks: str | None = None,+ verify_models: bool = False,+ timeout: float = DEFAULT_PROBE_TIMEOUT_SEC,+) -> int:+ """Print Tier 1 or Tier 2 readiness and return 1 when any check failed."""+ rows = readiness_rows(tier=tier, checks=checks, verify_models=verify_models, timeout=timeout)+ table = Table(title=f"SkillEvaluator Doctor (Tier {tier})", box=SIMPLE, show_edge=False)+ table.add_column("Check", style="bold")+ table.add_column("Status")+ table.add_column("Details")+ status_styles = {"pass": "green", "warn": "yellow", "fail": "red"}+ for name, status, detail in rows:+ style = status_styles[status]+ table.add_row(escape_markup(name), f"[{style}]{status}[/{style}]", escape_markup(detail))+ console.print(table)+ return 1 if any(status == "fail" for _, status, _ in rows) else 0diff --git a/src/skillevaluator/embedding/client.py b/src/skillevaluator/embedding/client.pyindex 711527f1..22b0b961 100644--- a/src/skillevaluator/embedding/client.py+++ b/src/skillevaluator/embedding/client.py@@ -107,10 +107,20 @@ class EmbeddingClient:model: str | None = None,base_url: str | None = None,api_key: str | None = None,+ *,+ max_retries: int | None = None,+ http_client: Any = None,+ timeout: float | None = None,) -> None:self._model = modelself._base_url = base_urlself._api_key = api_key+ # None keeps the SDK defaults. The SDK ignores an http_client's own+ # timeout when it equals httpx's 5-second default, so readiness probes+ # pass their bound explicitly.+ self._max_retries = max_retries+ self._http_client = http_client+ self._timeout = timeoutself._client: Any = Noneself._provider_config: ProviderConfig | None = None@@ -157,7 +167,14 @@ class EmbeddingClient:"The 'openai' package is required for similarity checks. Install it with: pip install openai") from exc- self._client = OpenAI(api_key=config.api_key, base_url=config.base_url)+ client_kwargs: dict[str, Any] = {"api_key": config.api_key, "base_url": config.base_url}+ if self._max_retries is not None:+ client_kwargs["max_retries"] = self._max_retries+ if self._http_client is not None:+ client_kwargs["http_client"] = self._http_client+ if self._timeout is not None:+ client_kwargs["timeout"] = self._timeout+ self._client = OpenAI(**client_kwargs)return self._clientdef embed(self, texts: list[str]) -> list[list[float]]:diff --git a/src/skillevaluator/inference/client.py b/src/skillevaluator/inference/client.pyindex 32b68301..83c2ea40 100644--- a/src/skillevaluator/inference/client.py+++ b/src/skillevaluator/inference/client.py@@ -386,6 +386,7 @@ class LLMClient:retry_base_delay: float | None = None,retry_max_delay: float | None = None,http_client: Any = None,+ timeout: float | None = None,) -> None:self._model = modelself._base_url = base_url@@ -401,6 +402,11 @@ class LLMClient:max_delay=retry_max_delay if retry_max_delay is not None else retry_cfg.max_delay,)self._http_client = http_client+ # Seconds for each provider request, Bedrock's litellm call included;+ # None keeps the provider default. The OpenAI and Anthropic SDKs ignore+ # an http_client's timeout that equals httpx's 5-second default, so the+ # bound is passed explicitly instead of inherited from the client.+ self._timeout = timeoutself._client: Any = Noneself._provider_config: ProviderConfig | None = Noneself.last_failure: str | None = None@@ -518,6 +524,8 @@ class LLMClient:client_kwargs["base_url"] = config.base_urlif self._http_client is not None:client_kwargs["http_client"] = self._http_client+ if self._timeout is not None:+ client_kwargs["timeout"] = self._timeoutself._client = Anthropic(**client_kwargs)return self._client@@ -535,6 +543,8 @@ class LLMClient:}if self._http_client is not None:client_kwargs["http_client"] = self._http_client+ if self._timeout is not None:+ client_kwargs["timeout"] = self._timeoutclient = OpenAI(**client_kwargs)if (config.base_url is not None@@ -620,6 +630,7 @@ class LLMClient:aws_region_name=config.region,**_temperature_kwargs(config.model, self._temperature),**({"max_tokens": max_tokens} if max_tokens is not None else {}),+ **({"timeout": self._timeout} if self._timeout is not None else {}),)return _extract_choice_content(response)diff --git a/src/skillevaluator/utils/tool_runner.py b/src/skillevaluator/utils/tool_runner.pyindex 315ac99c..544c14c0 100644--- a/src/skillevaluator/utils/tool_runner.py+++ b/src/skillevaluator/utils/tool_runner.py@@ -159,6 +159,11 @@ class ExternalTool:"""Get full path to tool binary."""return self._path+ @property+ def configuration_error(self) -> str | None:+ """Return why an explicit executable override was rejected, if it was."""+ return self._configuration_error+def get_install_hint(self) -> str:"""Get installation hint for this tool."""if self._configuration_error:diff --git a/tests/embedding/test_client.py b/tests/embedding/test_client.pyindex 9aa2b67f..d586aeac 100644--- a/tests/embedding/test_client.py+++ b/tests/embedding/test_client.py@@ -107,6 +107,23 @@ class TestEmbed:mock_cls.assert_called_once_with(api_key="test-key", base_url="https://api.openai.com/v1")+ def test_explicit_retry_http_client_and_timeout_reach_the_sdk(self, monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai")+ monkeypatch.setenv("SKILL_EVAL_EMBEDDING_PROVIDER", "openai")+ monkeypatch.setenv("OPENAI_API_KEY", "test-key")+ http_client = object()++ with patch("openai.OpenAI", return_value=MagicMock()) as mock_cls:+ EmbeddingClient(max_retries=0, http_client=http_client, timeout=5.0)._get_client()++ mock_cls.assert_called_once_with(+ api_key="test-key",+ base_url="https://api.openai.com/v1",+ max_retries=0,+ http_client=http_client,+ timeout=5.0,+ )+def test_embed_returns_vectors(self, monkeypatch: pytest.MonkeyPatch) -> None:monkeypatch.setenv("NVIDIA_API_KEY", "test-key")client = EmbeddingClient()diff --git a/tests/golden/cli_surface.json b/tests/golden/cli_surface.jsonindex 293cf70d..64fa19fe 100644--- a/tests/golden/cli_surface.json+++ b/tests/golden/cli_surface.json@@ -270,6 +270,37 @@},"doctor": {"params": [+ {+ "choices": [+ "1",+ "2",+ "3",+ "all"+ ],+ "default": "3",+ "name": "tier",+ "opts": [+ "--tier"+ ],+ "param_type": "option"+ },+ {+ "name": "checks",+ "opts": [+ "--checks"+ ],+ "param_type": "option",+ "type": "text"+ },+ {+ "default": "60.0",+ "name": "timeout",+ "opts": [+ "--timeout"+ ],+ "param_type": "option",+ "type": "float range"+ },{"name": "agents","opts": [@@ -683,6 +714,37 @@},"health-check": {"params": [+ {+ "choices": [+ "1",+ "2",+ "3",+ "all"+ ],+ "default": "3",+ "name": "tier",+ "opts": [+ "--tier"+ ],+ "param_type": "option"+ },+ {+ "name": "checks",+ "opts": [+ "--checks"+ ],+ "param_type": "option",+ "type": "text"+ },+ {+ "default": "60.0",+ "name": "timeout",+ "opts": [+ "--timeout"+ ],+ "param_type": "option",+ "type": "float range"+ },{"name": "agents","opts": [diff --git a/tests/inference/test_llm_client.py b/tests/inference/test_llm_client.pyindex 7dae183f..c40ecdc3 100644--- a/tests/inference/test_llm_client.py+++ b/tests/inference/test_llm_client.py@@ -95,6 +95,27 @@ class TestLLMClientGetClient:mock_cls.assert_called_once_with(api_key="test-key", base_url="https://api.openai.com/v1", max_retries=0)+ @pytest.mark.parametrize(+ ("provider", "key_env", "sdk_class"),+ [("openai", "OPENAI_API_KEY", "openai.OpenAI"), ("anthropic", "ANTHROPIC_API_KEY", "anthropic.Anthropic")],+ )+ def test_explicit_timeout_reaches_the_sdk(+ self, monkeypatch: pytest.MonkeyPatch, provider: str, key_env: str, sdk_class: str+ ) -> None:+ # The SDKs replace an http_client timeout equal to httpx's 5-second+ # default with their own 600 seconds, so the bound must be explicit.+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", provider)+ monkeypatch.setenv(key_env, "test-key")+ for name in ("SKILL_EVAL_LLM_BASE_URL", "OPENAI_BASE_URL", "ANTHROPIC_BASE_URL"):+ monkeypatch.delenv(name, raising=False)+ http_client = object()++ with patch(sdk_class, return_value=MagicMock()) as mock_cls:+ LLMClient(http_client=http_client, timeout=5.0)._get_client()++ assert mock_cls.call_args.kwargs["timeout"] == 5.0+ assert mock_cls.call_args.kwargs["http_client"] is http_client+def test_missing_api_key_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai")monkeypatch.delenv("OPENAI_API_KEY", raising=False)@@ -535,6 +556,18 @@ class TestCompletions:assert call_kwargs["max_tokens"] == 0assert "max_completion_tokens" not in call_kwargs+ @pytest.mark.parametrize(("timeout", "expected"), [(None, {}), (5.0, {"timeout": 5.0})])+ def test_bedrock_forwards_only_an_explicit_timeout(+ self, monkeypatch: pytest.MonkeyPatch, timeout: float | None, expected: dict+ ) -> None:+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "bedrock")+ response = SimpleNamespace(choices=[SimpleNamespace(message=SimpleNamespace(content="Done"))])++ with patch("litellm.completion", return_value=response) as completion:+ LLMClient(timeout=timeout).completions("system", "user")++ assert {key: value for key, value in completion.call_args.kwargs.items() if key == "timeout"} == expected+def test_older_bedrock_model_preserves_custom_temperature(self, monkeypatch: pytest.MonkeyPatch) -> None:monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "bedrock")monkeypatch.setenv("SKILL_EVAL_LLM_MODEL", "us.anthropic.claude-3-5-sonnet-20241022-v2:0")diff --git a/tests/test_base_install.py b/tests/test_base_install.pyindex 8656f598..469fe782 100644--- a/tests/test_base_install.py+++ b/tests/test_base_install.py@@ -97,6 +97,12 @@ direct_result = CliRunner().invoke()assert direct_result.exit_code == 0, direct_result.output+tier1_doctor = CliRunner().invoke(cli, ["doctor", "--tier", "1", "--checks", "schema,quality,lint"])+assert tier1_doctor.exit_code == 0, tier1_doctor.output+tier2_doctor = CliRunner().invoke(cli, ["doctor", "--tier", "2", "--verify-models"])+assert tier2_doctor.exit_code == 1, tier2_doctor.output+assert "Tier 2 extra" in tier2_doctor.output, tier2_doctor.output+leaked = sorted(m for m in BLOCKED if m in sys.modules)assert not leaked, f"base path imported extras-only modules: {leaked}"print("BASE_OK")diff --git a/tests/test_cli.py b/tests/test_cli.pyindex 5df605f7..9965abb9 100644--- a/tests/test_cli.py+++ b/tests/test_cli.py@@ -570,6 +570,170 @@ def test_doctor_verify_models_help_describes_live_catalog_probe(args: list[str])assert "authenticated" not in normalized_output+def doctor_calls(monkeypatch: pytest.MonkeyPatch) -> list[dict]:+ """Record each tier's doctor call instead of probing scanners, providers or backends."""+ import skillevaluator.doctor as readiness+ from skillevaluator.tier3 import commands as tier3_commands++ calls: list[dict] = []+ monkeypatch.setattr(readiness, "doctor", lambda **options: calls.append(options) or 0)+ monkeypatch.setattr(tier3_commands, "doctor", lambda **options: calls.append({"tier": "3", **options}) or 0)+ return calls++[email protected]("command", ["doctor", "health-check"])+def test_doctor_without_tier_keeps_checking_only_live_evaluation(doctor_calls, command) -> None:+ result = CliRunner().invoke(cli, [command, "-a", "codex", "--env-mode", "docker"])++ assert result.exit_code == 0, result.output+ assert doctor_calls == [+ {+ "tier": "3",+ "agents": "codex",+ "env_mode": "docker",+ "environment_kwarg": (),+ "verify_models": False,+ "agent_model": (),+ "environment_kwargs": {},+ }+ ]++[email protected]("command", ["doctor", "health-check"])+ ("args", "expected"),+ [+ (["--tier", "1"], {"tier": "1", "checks": None, "verify_models": False, "timeout": 60.0}),+ (+ ["--tier", "1", "--checks", "security,deps", "--timeout", "4"],+ {"tier": "1", "checks": "security,deps", "verify_models": False, "timeout": 4.0},+ ),+ (["--tier", "2"], {"tier": "2", "checks": None, "verify_models": False, "timeout": 60.0}),+ ],+)+def test_doctor_routes_lower_tiers_without_live_evaluation(doctor_calls, command, args, expected) -> None:+ result = CliRunner().invoke(cli, [command, *args])++ assert result.exit_code == 0, result.output+ assert doctor_calls == [expected]++[email protected](("args", "timeout"), [([], 60.0), (["--timeout", "5"], 5.0)])+def test_doctor_verify_models_reaches_tier2_live_probes(doctor_calls, args, timeout) -> None:+ result = CliRunner().invoke(cli, ["doctor", "--tier", "2", "--verify-models", *args])++ assert result.exit_code == 0, result.output+ assert doctor_calls == [{"tier": "2", "checks": None, "verify_models": True, "timeout": timeout}]+++ ("args", "message"),+ [+ (["doctor", "--tier", "2", "--timeout", "5"], "--timeout does not apply to --tier 2 without --verify-models."),+ (["health-check", "--tier", "2", "--timeout", "5"], "--timeout does not apply to --tier 2."),+ ],+)+def test_doctor_timeout_needs_a_live_tier2_request(doctor_calls, args, message) -> None:+ result = CliRunner().invoke(cli, args)++ assert result.exit_code == 2, result.output+ assert message in " ".join(result.output.split())+ assert doctor_calls == []++[email protected]("command", ["doctor", "health-check"])+def test_doctor_timeout_still_bounds_tier1_under_tier_all(doctor_calls, command) -> None:+ result = CliRunner().invoke(cli, [command, "--tier", "all", "--timeout", "5", "--env-mode", "local"])++ assert result.exit_code == 0, result.output+ assert [(call["tier"], call.get("timeout")) for call in doctor_calls] == [("1", 5.0), ("2", 5.0), ("3", None)]+++def test_health_check_timeout_help_covers_only_tier1() -> None:+ result = CliRunner().invoke(cli, ["health-check", "--help"])++ assert result.exit_code == 0, result.output+ normalized = " ".join(result.output.split())+ assert "Seconds allowed for each Tier 1 scanner probe." in normalized+ assert "live request" not in normalized++[email protected]("command", ["doctor", "health-check"])+def test_doctor_all_checks_every_tier_after_a_failure(monkeypatch, command) -> None:+ import skillevaluator.doctor as readiness+ from skillevaluator.tier3 import commands as tier3_commands++ tiers: list[str] = []+ monkeypatch.setattr(readiness, "doctor", lambda **options: tiers.append(options["tier"]) or 1)+ monkeypatch.setattr(tier3_commands, "doctor", lambda **_options: tiers.append("3") or 0)++ result = CliRunner().invoke(cli, [command, "--tier", "all", "--checks", "pii", "--env-mode", "local"])++ assert result.exit_code == 1, result.output+ assert tiers == ["1", "2", "3"]+++ "args",+ [+ ["doctor", "--tier", "4"],+ ["doctor", "--checks", "schema"],+ ["doctor", "--timeout", "5"],+ ["doctor", "--tier", "1", "--checks", "unknown"],+ ["doctor", "--tier", "all", "--checks", "unknown"],+ ["doctor", "--tier", "2", "--checks", "schema"],+ ["doctor", "--tier", "1", "--verify-models"],+ ["doctor", "--tier", "1", "--agents", "codex"],+ ["doctor", "--tier", "1", "--env-mode", "docker"],+ ["doctor", "--tier", "2", "--ek", "region=us"],+ ["doctor", "--tier", "2", "--agent-model", "codex=gpt"],+ ["doctor", "--tier", "1", "--timeout", "0"],+ ["doctor", "--tier", "1", "--timeout", "-1"],+ ["doctor", "--tier", "1", "--timeout", "nan"],+ ["doctor", "--tier", "1", "--timeout", "inf"],+ ["doctor", "--tier", "1", "--checks", "code", "--timeout", "1e300"],+ ["doctor", "--tier", "1", "--timeout", "3601"],+ ["doctor", "--tier", "2", "--timeout", "5"],+ ["doctor", "--tier", "3", "--timeout", "5"],+ ["health-check", "--tier", "2", "--timeout", "5"],+ ["doctor", "--tier", "all", "--ek", "malformed"],+ ["health-check", "--tier", "2", "--verify-models"],+ ["health-check", "--tier", "1", "-a", "codex"],+ ["tier3", "doctor", "--tier", "1"],+ ["tier3", "doctor", "--checks", "schema"],+ ["tier3", "doctor", "--timeout", "5"],+ ],+)+def test_doctor_rejects_options_for_unselected_tiers_before_any_probe(doctor_calls, args) -> None:+ result = CliRunner().invoke(cli, args)++ assert result.exit_code == 2, result.output+ assert doctor_calls == []++[email protected]("command", ["doctor", "health-check"])+def test_documented_readiness_examples_are_valid(doctor_calls, command) -> None:+ import shlex++ reference = (Path(__file__).parents[1] / "docs" / "cli-reference.mdx").read_text(encoding="utf-8")+ section = reference.split(f"\n## {command}\n", 1)[1].split("\n## ", 1)[0]+ examples = [line for line in section.splitlines() if line.startswith(f"skillevaluator {command}")]++ assert len(examples) >= 2+ for example in examples:+ result = CliRunner().invoke(cli, shlex.split(example, comments=True)[1:])+ assert result.exit_code == 0, f"{example}\n{result.output}"+++def test_tier3_doctor_stays_live_evaluation_only(doctor_calls) -> None:+ result = CliRunner().invoke(cli, ["tier3", "doctor", "-a", "codex", "--verify-models"])++ assert result.exit_code == 0, result.output+ assert [call["tier"] for call in doctor_calls] == ["3"]+ assert doctor_calls[0]["verify_models"] is True++def test_removed_benchmark_authoring_command_is_unavailable() -> None:removed_command = "convert" + "-benchmark"diff --git a/tests/test_doctor.py b/tests/test_doctor.pynew file mode 100644index 00000000..172899be--- /dev/null+++ b/tests/test_doctor.py@@ -0,0 +1,365 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Tier 1 scanner and Tier 2 provider readiness behind ``doctor --tier``."""++from __future__ import annotations++import json+import subprocess+import sys+from types import ModuleType, SimpleNamespace++import httpx+import pytest++from skillevaluator import doctor as readiness+from skillevaluator.utils import tool_runner+from skillevaluator.utils.tool_runner import ExternalTool, Tools++_SCANNERS = ("skillspector", "gitleaks", "bandit", "semgrep", "pip_audit")+_VERSION_OUTPUT = {+ "skillspector": "SkillSpector v2.11.2\n",+ "gitleaks": "8.21.2\n",+ "bandit": "bandit 1.9.4\n python version = 3.12.13\n",+ "semgrep": "1.166.0\n",+ "pip-audit": "pip-audit 2.10.1\n",+}+_PROVIDER_ENV = (+ "SKILL_EVAL_LLM_PROVIDER",+ "SKILL_EVAL_LLM_MODEL",+ "SKILL_EVAL_LLM_API_KEY",+ "SKILL_EVAL_LLM_BASE_URL",+ "SKILL_EVAL_EMBEDDING_PROVIDER",+ "SKILL_EVAL_EMBEDDING_MODEL",+ "SKILL_EVAL_EMBEDDING_API_KEY",+ "SKILL_EVAL_EMBEDDING_BASE_URL",+ "NVIDIA_API_KEY",+ "OPENAI_API_KEY",+ "OPENAI_BASE_URL",+ "ANTHROPIC_API_KEY",+ "ANTHROPIC_BASE_URL",+)+++def _rows(**options) -> dict[str, tuple[str, str]]:+ return {name: (status, detail) for name, status, detail in readiness.readiness_rows(**options)}+++def scanners(monkeypatch: pytest.MonkeyPatch) -> SimpleNamespace:+ """Resolve every scanner to a fake path and answer its version probe in-process."""+ for name in _SCANNERS:+ tool = getattr(Tools, name)+ monkeypatch.setattr(tool, "_path", f"/doctor-tools/{tool.command}")+ monkeypatch.setattr(tool, "_configuration_error", None)+ monkeypatch.setattr(tool, "resolution_source", "automatic")+ output = dict(_VERSION_OUTPUT)+ calls: list[tuple[list[str], dict]] = []++ def run(command, **kwargs):+ calls.append((list(command), kwargs))+ name = command[0].rsplit("/", 1)[-1]+ return subprocess.CompletedProcess(command, 0, stdout=output[name], stderr="")++ monkeypatch.setattr(tool_runner.subprocess, "run", run)+ # Tier 1 readiness is static: it passes with no provider configured.+ for name in _PROVIDER_ENV:+ monkeypatch.delenv(name, raising=False)+ return SimpleNamespace(calls=calls, output=output)+++ ("checks", "commands"),+ [+ (None, ["skillspector", "bandit", "semgrep", "gitleaks"]),+ ("security", ["skillspector"]),+ ("code", ["bandit", "semgrep", "gitleaks"]),+ ("deps,security", ["skillspector", "pip-audit"]),+ ],+)+def test_tier1_probes_each_scanner_the_selected_checks_run(scanners, checks, commands) -> None:+ assert readiness.doctor(tier="1", checks=checks, timeout=4) == 0++ assert [call[0] for call in scanners.calls] == [+ [f"/doctor-tools/{name}", "version" if name == "gitleaks" else "--version"] for name in commands+ ]+ assert all(call[1]["timeout"] == 4 for call in scanners.calls)+++def test_tier1_reports_each_scanner_version_and_path(scanners) -> None:+ rows = _rows(tier="1", checks="security,code-integrity,dependency")++ assert rows == {+ "CLI package": rows["CLI package"],+ "SkillSpector": ("pass", "2.11.2 at /doctor-tools/skillspector"),+ "Bandit": ("pass", "1.9.4 at /doctor-tools/bandit"),+ "Semgrep": ("pass", "1.166.0 at /doctor-tools/semgrep"),+ "Gitleaks": ("pass", "8.21.2 at /doctor-tools/gitleaks"),+ "pip-audit": ("pass", "2.10.1 at /doctor-tools/pip-audit"),+ }+++def test_tier1_checks_without_scanners_report_nothing_to_probe(scanners) -> None:+ rows = _rows(tier="1", checks="schema,pii,quality")++ assert rows["Tier 1 scanners"][0] == "pass"+ assert scanners.calls == []+++def test_tier1_version_probes_keep_credentials_out_of_skillspector(scanners, monkeypatch) -> None:+ monkeypatch.setenv("NVIDIA_API_KEY", "secret-canary")+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "nv_build")++ assert readiness.doctor(tier="1", checks="security,code") == 0++ environments = {call[0][0].rsplit("/", 1)[-1]: call[1]["env"] for call in scanners.calls}+ assert "secret-canary" not in environments["skillspector"].values()+ assert "NVIDIA_API_KEY" not in environments["skillspector"]+ # The semgrep probe skips Semgrep's network update check, as the scan does.+ assert environments["semgrep"]["SEMGREP_ENABLE_VERSION_CHECK"] == "0"+ assert environments["bandit"] is None+++def test_tier1_reports_an_explicit_executable_override(scanners, monkeypatch, tmp_path) -> None:+ executable = tmp_path / "bandit"+ executable.write_text("#!/bin/sh\n", encoding="utf-8")+ executable.chmod(0o755)+ monkeypatch.setenv("SKILLEVALUATOR_BANDIT_PATH", str(executable))+ monkeypatch.setattr(Tools, "bandit", ExternalTool("Bandit", "bandit", override_env="SKILLEVALUATOR_BANDIT_PATH"))++ rows = _rows(tier="1", checks="code")++ assert rows["Bandit"] == ("pass", f"1.9.4 at {executable.resolve()} (from SKILLEVALUATOR_BANDIT_PATH)")++[email protected]("failure", ["missing", "malformed override", "timeout", "exit status"])+def test_tier1_scanner_failure_does_not_hide_other_scanners(scanners, monkeypatch, failure) -> None:+ if failure == "missing":+ monkeypatch.setattr(Tools.bandit, "_path", None)+ elif failure == "malformed override":+ monkeypatch.setenv("SKILLEVALUATOR_BANDIT_PATH", "relative/bandit")+ monkeypatch.setattr(+ Tools, "bandit", ExternalTool("Bandit", "bandit", override_env="SKILLEVALUATOR_BANDIT_PATH")+ )+ else:+ answer = tool_runner.subprocess.run++ def failing_bandit(command, **kwargs):+ if command[0].endswith("/bandit"):+ if failure == "timeout":+ raise subprocess.TimeoutExpired(command, kwargs["timeout"], output="secret-canary")+ return subprocess.CompletedProcess(command, 2, stdout="secret-canary", stderr="secret-canary")+ return answer(command, **kwargs)++ monkeypatch.setattr(tool_runner.subprocess, "run", failing_bandit)++ rows = _rows(tier="1", checks="code", timeout=4)++ status, detail = rows["Bandit"]+ assert status == "fail"+ assert "secret-canary" not in detail+ expected = {+ "missing": "not installed. Bundled with the security extra",+ "malformed override": "SKILLEVALUATOR_BANDIT_PATH must name an absolute executable file",+ "timeout": "Bandit timed out after 4 seconds",+ "exit status": "version probe exited with status 2",+ }[failure]+ assert expected in detail+ assert rows["Semgrep"][0] == rows["Gitleaks"][0] == "pass"+ assert readiness.doctor(tier="1", checks="code", timeout=4) == 1+++ ("command", "output", "status"),+ [+ ("bandit", "bandit 1.9.3\n", "fail"),+ ("bandit", "bandit 1.9.4\n", "pass"),+ ("gitleaks", "v7.6.1\n", "fail"),+ ("gitleaks", "v8.0.0\n", "pass"),+ ],+)+def test_tier1_rejects_scanners_older_than_the_supported_release(scanners, command, output, status) -> None:+ scanners.output[command] = output++ rows = _rows(tier="1", checks="code")++ label = {"bandit": "Bandit", "gitleaks": "Gitleaks"}[command]+ assert rows[label][0] == status+ if status == "fail":+ assert "is older than the supported" in rows[label][1]+++def test_tier1_unreported_version_warns_without_failing(scanners) -> None:+ scanners.output["semgrep"] = "development build\n"++ rows = _rows(tier="1", checks="code")++ assert rows["Semgrep"] == ("warn", "version not reported; /doctor-tools/semgrep")+ assert readiness.doctor(tier="1", checks="code") == 0++[email protected]("timeout", [0, -1, float("nan"), float("inf"), 3600.5, 1e300])+def test_readiness_rejects_a_timeout_outside_the_supported_range(timeout) -> None:+ with pytest.raises(ValueError, match="finite, positive and at most 3600 seconds"):+ readiness.readiness_rows(tier="1", timeout=timeout)+++def test_readiness_only_covers_tiers_1_and_2() -> None:+ with pytest.raises(ValueError, match="tier 1 or 2"):+ readiness.readiness_rows(tier="3")+++def provider_api(monkeypatch: pytest.MonkeyPatch) -> SimpleNamespace:+ """Route the real provider SDKs through an in-process transport."""+ for name in _PROVIDER_ENV:+ monkeypatch.delenv(name, raising=False)+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "nv_build")+ monkeypatch.setenv("NVIDIA_API_KEY", "nvapi-secret-canary")+ requests: list[httpx.Request] = []+ responses: dict[str, httpx.Response] = {}++ def handler(request: httpx.Request) -> httpx.Response:+ requests.append(request)+ endpoint = request.url.path.rsplit("/", 1)[-1]+ if endpoint in responses:+ return responses[endpoint]+ if endpoint == "embeddings":+ return httpx.Response(+ 200,+ json={+ "object": "list",+ "model": "m",+ "data": [{"object": "embedding", "index": 0, "embedding": [0.6, 0.8]}],+ },+ )+ return httpx.Response(+ 200,+ json={+ "id": "probe",+ "object": "chat.completion",+ "created": 0,+ "model": "m",+ "choices": [{"index": 0, "message": {"role": "assistant", "content": "OK"}, "finish_reason": "stop"}],+ },+ )++ class _TransportClient(httpx.Client):+ def __init__(self, **kwargs):+ super().__init__(transport=httpx.MockTransport(handler), **kwargs)++ monkeypatch.setattr(httpx, "Client", _TransportClient)+ return SimpleNamespace(requests=requests, responses=responses)+++def test_tier2_resolves_providers_offline_by_default(provider_api) -> None:+ rows = _rows(tier="2")++ assert rows["Embedding provider"] == ("pass", "nv_build / nvidia/nemotron-3-embed-1b")+ assert rows["LLM provider"][0] == "pass"+ assert rows["LLM provider"][1].startswith("nv_build / ")+ assert "Embedding model" not in rows+ assert provider_api.requests == []+++def test_tier2_missing_provider_fails_without_network(provider_api, monkeypatch, capsys) -> None:+ monkeypatch.delenv("SKILL_EVAL_LLM_PROVIDER")+ monkeypatch.delenv("NVIDIA_API_KEY")++ assert readiness.doctor(tier="2", verify_models=True) == 1++ rows = _rows(tier="2", verify_models=True)+ assert rows["Embedding provider"][0] == rows["LLM provider"][0] == "fail"+ assert "No provider is configured" in rows["LLM provider"][1]+ assert "Embedding model" not in rows+ assert provider_api.requests == []+++def test_tier2_without_the_extra_reports_it_missing(monkeypatch) -> None:+ monkeypatch.setattr(readiness, "_module_available", lambda _module: False)++ rows = _rows(tier="2", verify_models=True)++ assert rows["Tier 2 extra"][0] == "fail"+ assert "tier2" in rows["Tier 2 extra"][1]+++def _request_timeouts(requests: list[httpx.Request]) -> list[dict[str, float]]:+ return [request.extensions["timeout"] for request in requests]+++# 5.0 equals httpx's default client timeout, which the provider SDKs would+# replace with their own 600-second default unless the bound is passed through.[email protected]("timeout", [4.0, 5.0])+def test_tier2_live_probe_sends_one_bounded_request_per_model(provider_api, timeout) -> None:+ rows = _rows(tier="2", verify_models=True, timeout=timeout)++ assert rows["Embedding model"] == ("pass", "returned a 2-dimension vector")+ assert rows["Deduplication model"] == ("pass", "returned a chat completion")+ assert [request.url.path for request in provider_api.requests] == ["/v1/embeddings", "/v1/chat/completions"]+ bound = dict.fromkeys(("connect", "read", "write", "pool"), timeout)+ assert _request_timeouts(provider_api.requests) == [bound, bound]+ chat = json.loads(provider_api.requests[1].content)+ assert chat["model"] == rows["LLM provider"][1].split(" / ", 1)[1]+++def test_tier2_anthropic_chat_probe_is_bounded_by_the_timeout(provider_api, monkeypatch) -> None:+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "anthropic")+ monkeypatch.setenv("SKILL_EVAL_EMBEDDING_PROVIDER", "nv_build")+ monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-secret-canary")+ provider_api.responses["messages"] = httpx.Response(+ 200,+ json={+ "id": "probe",+ "type": "message",+ "role": "assistant",+ "model": "m",+ "content": [{"type": "text", "text": "OK"}],+ "stop_reason": "end_turn",+ "stop_sequence": None,+ "usage": {"input_tokens": 1, "output_tokens": 1},+ },+ )++ rows = _rows(tier="2", verify_models=True, timeout=5.0)++ assert rows["Deduplication model"] == ("pass", "returned a chat completion")+ assert [request.url.path for request in provider_api.requests] == ["/v1/embeddings", "/v1/messages"]+ bound = dict.fromkeys(("connect", "read", "write", "pool"), 5.0)+ assert _request_timeouts(provider_api.requests) == [bound, bound]+++def test_tier2_bedrock_chat_probe_passes_the_timeout_to_litellm(provider_api, monkeypatch) -> None:+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "bedrock")+ monkeypatch.setenv("SKILL_EVAL_EMBEDDING_PROVIDER", "nv_build")+ calls: list[dict] = []++ def completion(**kwargs):+ calls.append(kwargs)+ return SimpleNamespace(choices=[SimpleNamespace(message=SimpleNamespace(content="OK"))])++ # Bedrock goes through litellm rather than the shared HTTP client.+ fake_litellm = ModuleType("litellm")+ fake_litellm.completion = completion+ monkeypatch.setitem(sys.modules, "litellm", fake_litellm)++ rows = _rows(tier="2", verify_models=True, timeout=5.0)++ assert rows["Deduplication model"] == ("pass", "returned a chat completion")+ assert [call["timeout"] for call in calls] == [5.0]++[email protected]("status", [401, 500])+def test_tier2_live_failures_are_reported_without_provider_bodies(provider_api, capsys, status) -> None:+ body = {"error": {"message": "secret-canary nvapi-secret-canary", "code": "secret-canary"}}+ provider_api.responses["embeddings"] = httpx.Response(status, json=body)+ provider_api.responses["completions"] = httpx.Response(status, json=body)++ assert readiness.doctor(tier="2", verify_models=True, timeout=4) == 1++ output = capsys.readouterr().out+ assert f"HTTP {status}" in output+ assert "secret-canary" not in output+ # The chat probe still runs after the embedding probe fails, and neither retries.+ assert [request.url.path for request in provider_api.requests] == ["/v1/embeddings", "/v1/chat/completions"]