Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

80.5 KB
Raw Blame
---
title: "CLI Reference"
description: "Run tiers directly, explore expert subcommands, and look up SkillEvaluator flags and defaults."
layout: reference
---

{/* Maintenance note: this page is maintained against the Click command tree and
    `skillevaluator --help` output. Re-verify every flag table each release;
    tests/golden/cli_surface.json also records compatibility aliases hidden in help. */}

This page lists every `skillevaluator` command with its full flag set and exact defaults. If you want task-oriented guidance instead, start from the tier guides: [Tier 1: Validation](/skills/skillevaluator/tier1-validation), [Tier 2: Deduplication](/skills/skillevaluator/tier2-deduplication), and [Tier 3: Live Agent Evaluation](/skills/skillevaluator/tier3-live-evaluation).

[Harbor](https://github.com/harbor-framework/harbor) is the open-source agent
evaluation framework behind the Tier 3 task, environment, trial, and trajectory
commands referenced below.

## Command map

| Command | Purpose | Needs |
| --- | --- | --- |
| [tier1 PATH](#direct-tier-workflows) | Run Tier 1 static and configured LLM checks | External scanners; provider for LLM stages |
| [tier2 PATH](#direct-tier-workflows) | Run intra-skill dedup; include inter-skill comparison with `--catalog` | Chat and embeddings provider |
| [tier3 PATH](#direct-tier-workflows) | Prepare a missing starter dataset and run live evaluation | Provider, supported agent, and sandbox |
| [validate](#validate) | Run all three tiers for skills, including Tier 3 autopilot, with explicit scope and gating controls | Provider access, scanners, supported agent, and sandbox; `--tiers 1` selects static checks |
| [quality-check](#quality-check) | Score skill quality across four weighted categories | Nothing |
| [rubric-eval](#rubric-eval) | LLM-as-judge rubric evaluation | LLM provider key |
| [security-scan](#security-scan) | Scan for security vulnerabilities | Nothing (key only with `--llm`/`--llm-verify`) |
| [pii-scan](#pii-scan) | Scan for PII and local identifiers | Nothing (key only with `--llm-verify`) |
| [lint-scripts](#lint-scripts) | Advisory lint checks on skill scripts | Nothing |
| [harness-check](#harness-check) | Show whether each agent harness loads, degrades or drops a skill | Nothing |
| [context-optimization-check](#context-optimization-check) | Detect redundant content within one skill | Embeddings + LLM provider key |
| [dedup-scan](#dedup-scan) | Alias of `context-optimization-check` | Embeddings + LLM provider key |
| [similarity-check](#similarity-check) | Detect duplicate content across a collection | Embeddings provider key |
| [tier3 evaluate](#tier3-evaluate) | Run Tier 3 live agent evaluation | Provider key plus a credential the selected agents can use (one `NVIDIA_API_KEY` covers both on NVIDIA Build), backend (Docker by default) |
| [tier3 select-decoys](#tier3-select-decoys) | Save a reproducible set of neighboring skills for group evaluation | Local description-embedding catalog, skill collection, embeddings provider key |
| [tier3 interact](#tier3-interact) | Measure departure from additive skill gains | At least two skills, one reviewed evaluation source, Tier 3 runtime and credentials |
| [tier3 compare-versions](#tier3-compare-versions) | Compare current and prior retained with-skill scores without writes | Two compatible retained runs |
| [create-eval-dataset](#create-eval-dataset) | Generate eval datasets | Substantive author guidance or usable matching retained traces; explicit `--grounding skill` permits skill-only generation. LLM provider key unless `--no-llm` |
| [init-custom-grader](#init-custom-grader) | Scaffold a bring-your-own-grader starter | Nothing |
| [init-harbor-task](#init-harbor-task) | Scaffold a bring-your-own-task Harbor starter | Nothing |
| [view](#view) | Open the latest HTML live-evaluation report | A previous `tier3 evaluate` run |
| [compare](#compare) | Compare live-evaluation results across agents | A previous `tier3 evaluate` run |
| [tier3 history](#tier3-history) | Inspect retained Skill Lift over skill revisions, grouped by compatible configuration | Retained Tier 3 summaries |
| [tier3 feedback](#tier3-feedback) | Chat about one completed live-evaluation run and possible skill changes | A completed `tier3 evaluate` run, LLM provider key |
| [dashboard](#dashboard) | Filter quantitative tables and paired usage differences | Saved JSON reports or retained runs; dashboard extra |
| [harbor-view](#harbor-view) | Browse retained Harbor job artifacts | A run made with `--harbor-keep-jobs` |
| [doctor](#doctor) | Live-evaluation readiness check, or Tier 1 scanners and Tier 2 providers with `--tier` | Nothing |
| [health-check](#health-check) | Quick CLI + backend readiness check, or another tier with `--tier` | Nothing |
| [models](#models) | List models visible through the selected provider's catalog | Provider configuration/key |
| [tier3 validate](#tier3-validate) | Validate `evals/` and the optional Harbor contract | Nothing |
| [tier3 import-traces](#tier3-import-traces) | Import redacted trace review candidates | Nothing |
| [tier3 accept-traces](#tier3-accept-traces) | Promote explicitly reviewed trace candidates | Nothing |
| [tier3 import-metrics](#tier3-import-metrics) | Report precomputed custom scores as external observations | A versioned metrics JSON file; no runtime or provider key |

## tier3 import-metrics

```bash
skillevaluator tier3 import-metrics INPUT.json --output-dir NEW_DIRECTORY
```

`INPUT.json` must exist. Required `-o, --output-dir` names a directory that must
not exist. The command validates the complete input before writing
`external_metrics.json` and `external_metrics.md`. It never executes tests or
assigns canonical scores; exit 0 means a valid import, not passing tests.
See the [input contract and simulator example](/skills/skillevaluator/custom-graders#import-precomputed-custom-metrics).

## Global conventions

**Reports.** Every Tier 1 and Tier 2 command accepts the same two report options.
Direct `tier1 PATH`, `tier1 validate PATH`, and `tier2 PATH` default to
`cli,json,html`; focused expert commands default to `cli`:

| Flag | Default | Effect |
| --- | --- | --- |
| `-r, --report [cli\|json\|html\|markdown\|sarif\|junit]` | `cli,json,html` for direct workflows; `cli` for focused commands | Report format(s). Accepts comma- or space-separated values (`-r cli,json,html` or `-r cli json html`) and may be repeated. The compact default view writes html+json unless `-r` is passed explicitly, which is honored exactly (including `cli`). |
| `-o, --output-dir DIRECTORY` | `reports` | Directory for generated reports. |

In practice: a default `validate` run (no `-r`) shows the compact pipeline view **and** writes `json` + `html` report files; passing `-r` takes over completely — `-r cli` really means terminal output only, no files. The focused expert commands take the `cli` default literally: they print the full terminal report and write files only for the formats you request with `-r`. A `validate` run writes `skillevaluator-output-<timestamp>.json` / `.html` (plus `.md` with `-r markdown`, `.sarif.json` with `-r sarif`, or `.junit.xml` with `-r junit`); standalone commands write a fixed `skillevaluator-<kind>` basename (for example `skillevaluator-quality.json`), so a re-run overwrites the previous report. Direct workflows write `skillevaluator-tier1` or `skillevaluator-tier2` reports in the selected formats and also overwrite previous files with that basename. Filename conventions and report anatomy live in [Reports & Results](/skills/skillevaluator/reports).

**Exit codes.** Commands communicate through a small exit-code contract — this is what your CI gates on (see [Gate Your CI](/skills/skillevaluator/ci-integration)):

| Code | Meaning |
| --- | --- |
| `0` | Success |
| `1` | Validation failed, the run could not complete, or an unexpected internal error occurred |
| `2` | Usage error — an unknown flag, a value outside an option's type or choices, or a target path that does not exist |

**Global options.** `skillevaluator --version` prints the installed version. `-v, --verbose`, given before the command (`skillevaluator -v validate ./my-skill`), enables debug logging; for `validate` it also replaces the compact pipeline view with the full detail stream. Every command accepts `-h` as a short form of `--help`.

## Direct tier workflows

```bash title="Run each tier independently"
skillevaluator tier1 ./my-skill
skillevaluator tier2 ./my-skill --catalog ./skill-catalog.json
skillevaluator tier3 ./my-skill
```

There is no required `run` subcommand. A bare `tier1`, `tier2`, or `tier3` shows
help and does not evaluate the current directory. Prefix a path with `./` when
its name matches an expert command: `tier1 ./validate` evaluates a directory
named `validate`; `tier1 validate ./my-skill` invokes the Tier 1 workflow.

| Workflow | Default scope | Scope controls |
| --- | --- | --- |
| `tier1 PATH` | All static checks including dependencies; rubric, LLM security, and finding verification when a provider is configured | `--no-llm` skips LLM stages; `--llm` requires provider access; `--checks` selects static checks |
| `tier2 PATH` | Intra-skill analysis | `--catalog FILE` adds inter-skill comparison against a saved JSON catalog; without it, inter-skill is explicitly not run |
| `tier3 PATH` | One missing starter case is generated, then the provider-native agent runs in Docker with and without the skill | Accepts the [tier3 evaluate](#tier3-evaluate) options; missing-source generation is implicit |

Direct Tier 1 can run keyless, but marks its LLM stages as skipped. Missing
required scanner evidence remains incomplete. Direct Tier 2 needs both chat and
embedding access and takes one skill, not a collection. Create its catalog with
`similarity-check ./skills --save-catalog ./skill-catalog.json`. Direct Tier 3
preserves valid existing sources and rejects invalid ones; it does not overwrite
a dataset to repair it.

`validate PATH` runs all three tiers for skills by default, including Tier 3
autopilot. It is separate from these tier-only workflows.

### Direct tiers compared with `validate --tiers`

`validate --tiers` narrows the combined pipeline; it does not run a tier on
its own. Tier 1 always runs and gates the exit code, so `--tiers 2`,
`--tiers 3`, and `--tiers 2,3` are rejected. Use `tier2 PATH` or
`tier3 PATH` to run Tier 2 or Tier 3 without Tier 1.

| | Direct workflows | `validate PATH --tiers ...` |
| --- | --- | --- |
| Tiers run | Only the named tier | Tier 1 plus any selected tiers |
| Tier 1 dependency audit | Included in `tier1 PATH` | Opt-in with `--checks`, which replaces the default list, so name every check you want plus `dependency` ([selection rules](/skills/skillevaluator/tier1-reference#narrow-to-specific-checks)) |
| Tier 1 LLM stages | Rubric, LLM security, and verification run when a provider is configured; `--no-llm` skips them | LLM security analysis runs only with `--llm`, and finding verification only with `--llm-verify`; no rubric stage |
| Reports | `tier1` and `tier2` default to `-r cli,json,html` and write `skillevaluator-tier1.*` or `skillevaluator-tier2.*` under `-o`. `tier3` writes its run under `--results-dir` and does not accept `-r` or `-o` | The compact view writes HTML and JSON as `skillevaluator-output-<timestamp>.*`; skill targets also get `BENCHMARK.md` |
| Exit code `1` | `tier1`: a check failed or required evidence is incomplete. `tier2`: HIGH or CRITICAL findings or a blocking analysis error. `tier3`: the run did not complete; a `FAIL` verdict alone exits `0` | Tier 1 failed; Tier 2 findings unless `--no-block-on-dedup`; Tier 3 only with `--block-on-agent-eval` |

For `tier2 PATH`, `tier3 PATH`, and `tier1 PATH --llm`, a missing provider is
a configuration error that exits `1` before any report is written. `tier1 PATH`
without `--llm` runs keyless. `validate` records Tier 2 and Tier 3 as skipped
and still writes its reports.

Direct Tier 3 checks provider, option, and runtime configuration before it
generates a starter case, so a configuration error leaves `evals/` unchanged and
no paid generation or agent run starts. The exception is `--env-mode` `ack`,
`cwsandbox`, `langsmith`, `openshift`, and `wandb`: their backend checks contact
a remote control plane, so they run after the starter case is generated. Usage
errors, such as an option the workflow does not accept, exit `2`.

### Direct Tier 1 options

| Flag | Default | Effect |
| --- | --- | --- |
| `--checks TEXT` | All applicable static checks, including dependency auditing | Comma-separated subset of static checks |
| `--skip CHECKS` | none | Comma-separated checks to leave out, e.g. `secrets` for the Gitleaks scan. Skipped checks are shown and reported as skipped, nothing is installed for them, and the run cannot certify Tier 1 for publication |
| `--llm / --no-llm` | Enabled when a provider is configured | Enable or skip rubric, LLM security, and finding verification; explicit `--llm` requires valid provider configuration |
| `--min-score INTEGER` | `70` | Minimum passing quality and rubric score, from 0 to 100; quality severity overrides cannot waive the minimum |
| `--quality-config PATH` | packaged `quality-default.yaml` | YAML file that adjusts the quality check's expected headings, error-handling rules, token budget, description length limits, and reserved name words; settings it omits keep their packaged values. See [Tier 1: Validation](/skills/skillevaluator/tier1-reference#the-checks). |
| `--pii-waivers PATH` | none | Reviewed, expiring PII waivers applied by the `pii` check. See [PII waivers](/skills/skillevaluator/tier1-reference#pii-waivers). |
| `--profile TEXT` | `SKILLEVALUATOR_PROFILE`, then `default` | Validation policy: `default` or `registry`. `external` is a compatibility alias for `default`. |
| `--policy FILE` | none | Explicit policy overlay, including reviewed ownership-contact severity. See [Tier 1 policies](/skills/skillevaluator/tier1-reference#validation-profiles-and-policies). |
| `--previous-version TEXT` | `SKILLEVALUATOR_PREVIOUS_VERSION`, else unset | Previous version for version checks |
| `--skillspector-baseline FILE` | unset | Audited SkillSpector suppression baseline; never discovered from the skill. See [Audited SkillSpector baselines](/skills/skillevaluator/tier1-reference#audited-skillspector-baselines). |

Also accepts `-r/--report` and `-o/--output-dir` described above.

### Direct Tier 2 options

| Flag | Default | Effect |
| --- | --- | --- |
| `--catalog FILE` | unset | Saved JSON catalog for inter-skill comparison |
| `--threshold FLOAT` | `0.80` | Intra-skill candidate similarity threshold |
| `--similarity-threshold FLOAT` | `0.75` | Inter-skill reporting threshold |
| `--full-body` | off | Compare the full `SKILL.md` against a catalog created in full-body mode |

Both thresholds accept values from 0 to 1. Also accepts `-r/--report` and
`-o/--output-dir`. Direct Tier 3 accepts the options in
[tier3 evaluate](#tier3-evaluate), with missing-source generation already enabled.

## Expert tier subcommands

The commands below remain available for focused operations.
`skillevaluator tier1 validate PATH` runs the same isolated Tier 1 workflow
as `tier1 PATH`. Use top-level `skillevaluator validate PATH` for the combined
pipeline.

| Group | Members |
| --- | --- |
| `tier1` | `validate`, `quality-check`, `rubric-eval`, `security-scan`, `pii-scan`, `lint-scripts` |
| `tier2` | `context-optimization-check`, `dedup-scan`, `similarity-check` |
| `tier3` | `evaluate`, `interact`, `select-decoys`, `create-eval-dataset`, `migrate-dataset`, `init-custom-grader`, `init-harbor-task`, `doctor`, `view`, `compare`, `feedback`, `validate`, `harbor-view`, `import-metrics`, `import-traces`, `accept-traces` |

`health-check` and [models](#models) are top-level only. The expert commands
[tier3 evaluate](#tier3-evaluate), [tier3 select-decoys](#tier3-select-decoys),
[tier3 feedback](#tier3-feedback), [tier3 import-metrics](#tier3-import-metrics),
[tier3 import-traces](#tier3-import-traces),
[tier3 accept-traces](#tier3-accept-traces), and `tier3 validate` are advertised
**only** inside the `tier3` group:

### tier3 import-traces

Import a production trace export into redacted candidates outside the skill.
See [Trace Datasets](/skills/skillevaluator/trace-datasets) for the v1 JSON/JSONL contract and redaction limits.

```bash
skillevaluator tier3 import-traces ./my-skill ./sessions.jsonl -o ./review
```

| Option | Default | Purpose |
| --- | --- | --- |
| `-o`, `--output DIR` | `./<skill-dir-name>-trace-review` | Review bundle directory, outside the skill. |
| `--force` | off | Overwrite an existing candidates file. |
| `--dry-run` | off | Print classifications, warnings, and the next-step hint; write nothing. |

### tier3 accept-traces

Promote only explicitly reviewed candidates into `evals/evals.json`. Requires
`SKILL_PATH` and `REVIEW_DIR`. `accept` requires a reviewer-written expectation;
`accept_observed` is allowed only for successful sessions with non-empty output.
Pending and rejected candidates are skipped. All decision/redaction errors abort
before writing; dataset validation errors after writing return a non-zero status.

```bash
skillevaluator tier3 accept-traces ./my-skill ./review --dry-run
skillevaluator tier3 accept-traces ./my-skill ./review
```

| Option | Default | Purpose |
| --- | --- | --- |
| `--dry-run` | off | Report additions and already-present ids without writing. |
| `--allow-native-harbor` | off | Explicitly accept traces when native Harbor tasks are selected. |

### tier3 validate

Validate the Tier 3 `evals/` directory and the optional Harbor bring-your-own-task contract before you spend money on a live run. The dataset and directory rules it enforces are documented in [Eval Datasets](/skills/skillevaluator/eval-datasets). Because the no-skill baseline is graded on the same text, it warns when a case's prompt points at `skills/<name>`, `<name>/SKILL.md` or reading or following `SKILL.md`, or when its expected output or an assertion names those files, calls `<name>` a skill, says the agent used, invoked or identified `<name>`, or runs one of the skill's scripts from `scripts/` or `tools/`. The check is a wording heuristic, and these warnings fail the command only with `--strict`.

```bash title="Check evals/ before a live run"
skillevaluator tier3 validate ./my-skill --strict
```

| Flag | Default | Effect |
| --- | --- | --- |
| `--json` | off | Emit JSON output. |
| `--strict` | off | Treat warnings as failures. |
| `--harbor-contract` | off | Validate the Harbor task and reward contract (see [Custom Graders & Tasks](/skills/skillevaluator/custom-graders)). |

## validate

Validate a skill, rule, workflow, or plugin. For skills, all three tiers run by default. Tier 3 uses autopilot to reuse an existing evaluation source or generate one starter case when none exists; invalid sources are reported rather than replaced. Tier 1 static, security, and quality checks always gate the exit code. Tier 2 deduplication runs and gates by default, but `--no-block-on-dedup` keeps the scan while making its findings advisory. Tier 3 is advisory unless `--block-on-agent-eval` is set. Rules, workflows, and plugins retain their applicable validation stages without automatically enabling live skill evaluation.

```bash title="Common invocations"
skillevaluator validate ./my-skill                        # Tier 1 + Tier 2 + Tier 3 autopilot
skillevaluator validate ./my-skill --llm                  # add LLM security analysis
skillevaluator validate ./my-skill -r cli,json,html       # multiple report formats
skillevaluator validate ./my-skill --no-dedup             # skip Tier 2 dedup
skillevaluator validate ./my-skill --no-block-on-dedup    # run Tier 2 as advisory
skillevaluator validate ./my-skill --profile default      # standard validation policy
skillevaluator validate ./my-skill -a codex               # choose the Tier 3 agent
skillevaluator validate ./my-skill -a codex \
    --block-on-agent-eval                                  # gate on Tier 3 too
skillevaluator validate ./my-skill --no-autopilot         # require an existing eval source
skillevaluator validate ./my-skill --full -a codex        # compatibility: same default scope
skillevaluator validate ./my-skill --tiers 1,2            # skip live evaluation
skillevaluator validate ./my-skill --tiers 1              # static checks only
skillevaluator validate ./my-skill --tiers 1,3            # explicit tier selection
skillevaluator validate ./skills-folder                   # whole catalog, serially
skillevaluator validate ./my-skill -a codex,claude-code \
    --env-mode docker --harbor-keep-jobs                  # Tier 3, retain Harbor jobs
```

Pointing `validate` at a directory of skills (one with no root `SKILL.md`) runs the whole pipeline for each skill inside it, writes each skill's reports under `<output-dir>/<skill>/`, and finishes with a per-skill summary — catalog validation in one command. Skills run serially by default; `--workers N` validates up to N skills at once in parallel processes. `--previous-version` is rejected for a directory of skills. By default the terminal shows a compact pipeline view; `--verbose` switches to the full per-check detail stream.

`--help` groups the flags into four panels, mirrored by the subsections below.

### Run & Reports

Applies to the whole run: target typing, policy profile, reports, tier selection.

| Flag | Default | Effect |
| --- | --- | --- |
| `--type [skill\|rules\|workflows\|plugin\|auto]` | `auto` | Force the content type instead of auto-detecting it from the target path. |
| `--tiers TEXT` | all three for skills | Explicit tier selection, e.g. `--tiers 1,2` skips live evaluation and `--tiers 1,3` skips dedup. Tier 1 always runs. |
| `--full` | unnecessary for skills | Compatibility flag for all three tiers with autopilot. Explicit tier selection and disable flags still apply. |
| `--verbose` | off | Print the full per-check detail stream instead of the compact pipeline view. |
| `--profile TEXT` | `$SKILLEVALUATOR_PROFILE`, then `default` | Bundled validation policy: `default` or `registry`. `external` is a compatibility alias for `default`. |
| `--external` | — | Compatibility alias for `--profile default`; does not change repository visibility. |
| `--policy FILE` | none | Custom policy YAML overlaid on top of `--profile`. |
| `--workers INTEGER RANGE` | `1` | Concurrent skill jobs when validating a folder of skills (`x>=1`). Values above 1 run skills in parallel processes and disable the per-skill pipeline view. |

The [evaluated source provenance](#evaluated-source-provenance) flags also appear in this panel.

Auto-detection recognizes: a `SKILL.md` for skills, `.mdc` files for rules, `workflow-rules.mdc` for workflows, and — for plugins — a bundle-reference `agent_plugin.yaml`/`.yml` manifest or a contained `.claude-plugin/plugin.json` manifest. Plugins are validated against their public contract; quality, lint, and version checks are skill-only and skipped for plugins.

When auto-detection finds none of these, the type is `unknown`. Tier 1 then records the `schema`, `version`, and `quality` checks as not applicable (N/A) instead of failing them with `No skills found`, and still runs the content scans. Tier 1 passes when those scans pass; Tier 2 still runs on the target unless `--no-dedup` or `--tiers 1`. The JSON report adds a root `tier1_applicability` object, whichever checks the run selects. Auto-detection also reads the directory names in the target's absolute path, so a repository whose path includes a directory named `skills`, `team-skills`, `team-rules`, `workflows`, or `team-workflows` is detected as that type, not `unknown`. A target folder whose own `SKILL.md`, `workflow-rules.mdc`, `agent_plugin.yaml`/`.yml`, or `.claude-plugin/plugin.json` is mis-cased (such as `Skill.md`) or a dangling symlink still runs the manifest checks, and fails them. A mis-cased manifest below the target is not discovered; validate that folder directly. Pass `--type` to validate the target as a given type anyway. See [Content types](/skills/skillevaluator/tier1-reference#content-types).

`validate` also takes the standard `-r/--report` and `-o/--output-dir` options described under [Global conventions](#global-conventions).

### Tier 1 · Static & Security

Static checks; LLM-free by default. Tier 1 gates the exit code and always runs.

| Flag | Default | Effect |
| --- | --- | --- |
| `--checks, --tier1-checks TEXT` | all applicable | Comma-separated subset of Tier 1 checks. Default choices: `schema`, `version`, `security`, `pii`, `license`, `code-integrity`, `unicode`, `quality`, `lint`; opt-in (not run by default): `dependency`. `quality`/`lint`/`version` are skill-only and skipped for rules and workflows. |
| `--skip CHECKS` | none | Comma-separated Tier 1 checks to leave out: any check above or `secrets` (the Gitleaks scan inside `code-integrity`). Skipped checks show as skipped in the terminal and reports, nothing is installed for them, and the run cannot certify Tier 1 for publication. |
| `--previous-version VERSION` | `SKILLEVALUATOR_PREVIOUS_VERSION`, else unset | Previous released version; the skill's version must be a strictly greater SemVer. Single-skill targets only. |
| `--fail-fast` | off | Stop on the first failing check instead of collecting all issues. |
| `-c, --continue-on-failure` | off | Run the full pipeline without stopping early; record all issues in the reports. Overrides `--fail-fast`, and for folder validation keeps scanning every skill past a CRITICAL finding. |
| `--llm, --tier1-llm / --no-llm, --no-tier1-llm` | `no-llm` | Enable LLM-backed security analysis (requires a configured public provider — see [Providers & Credentials](/skills/skillevaluator/configuration)). |
| `--llm-verify` | off | Run a second LLM pass to suppress false-positive findings. |
| `--skillspector-baseline FILE` | unset | Audited SkillSpector suppression baseline for the `security` check; never discovered from the skill. See [Audited SkillSpector baselines](/skills/skillevaluator/tier1-reference#audited-skillspector-baselines). |
| `--min-score INTEGER` | `70` | Minimum quality score (0–100) required per skill when the `quality` check runs; policy severity overrides cannot waive this gate. |
| `--quality-config PATH` | packaged `quality-default.yaml` | YAML file that adjusts the quality check's expected headings, error-handling rules, token budget, description length limits, and reserved name words; settings it omits keep their packaged values. See [Tier 1: Validation](/skills/skillevaluator/tier1-reference#the-checks). |
| `--pii-waivers PATH` | none | Reviewed, expiring PII waivers applied by the `pii` check; forwarded to each skill of a catalog run. See [PII waivers](/skills/skillevaluator/tier1-reference#pii-waivers). |

### Tier 2 · Deduplication

Embedding + LLM dedup; on by default, skips gracefully without a provider key.

| Flag | Default | Effect |
| --- | --- | --- |
| `--dedup, --tier2 / --no-dedup, --no-tier2` | `dedup` | Run Tier 2 intra-skill semantic-overlap checks. On by default; skipped gracefully without public embedding access. Use `--no-tier2` (or `--no-dedup`) to disable. |
| `--block-on-dedup / --no-block-on-dedup` | blocking | Choose whether Tier 2 findings participate in the `validate` exit gate. The scan and its report entries still run with `--no-block-on-dedup`. |

### Tier 3 · Live Agent Evaluation

<Note>
The following flags are forwarded to the live-eval engine when Tier 3 is enabled, which is the default for skills. They do not re-enable a stage excluded by `--tiers` or `--no-tier3`. The Dockerfile-, workspace-, resource-, and progress-tuning flags (`--custom-dockerfile-mode`, `--skill-workspace-mode`, the `--override-*` flags, `--progress`, and `--progress-interval`) are not forwardable — use the standalone [tier3 evaluate](#tier3-evaluate) command for those.
</Note>

| Flag | Default | Effect |
| --- | --- | --- |
| `--tier3 / --no-tier3` | enabled for skills | Run or skip Tier 3 live agent evaluation. Accepts a supported YAML/JSON dataset or native Harbor source and validates it before execution. `--agent-eval` / `--no-agent-eval` are compatibility aliases. Explicit `--tiers` selection controls scope; `--no-tier3` also disables autopilot. |
| `--block-on-agent-eval / --no-block-on-agent-eval` | advisory | Choose whether Tier 3 findings, including missing or invalid selected task-source evidence, participate in the `validate` exit gate. When blocking, only a completed `PASS` verdict with complete attempt coverage passes; `NEUTRAL` and `FAIL` exit 1. This flag does not enable Tier 3 by itself. |
| `--autopilot / --no-autopilot` | enabled for skills | Generate one starter case when no evaluation source exists, then run Tier 3. Existing sources are reused unchanged; invalid sources are reported. `--no-autopilot` requires an existing source. Explicit tier selection or `--no-tier3` can exclude Tier 3. |
| `-a, --agents TEXT` | provider-native | Comma-separated Harbor agents to evaluate. Defaults: NVIDIA Build=`opencode`, OpenAI=`codex`, Anthropic=`claude-code`. |
| `--env-mode` | `docker` | Harbor environment backend (full list under [tier3 evaluate](#tier3-evaluate)). |
| `--environment-kwarg`, `--ek` | none | Operator-only `KEY=VALUE` constructor option for Harbor-native non-Docker backends (repeatable). Forwarded only when Tier 3 is enabled; skill-owned `evals/config.yml` cannot set these values. Never pass secrets. |
| `--skip-baseline` | off | Skip the without-skill baseline (no lift analysis, faster). |
| `--n-concurrent INTEGER` | unset | Concurrent eval cases per agent. |
| `--max-agents INTEGER` | unset | Maximum agents to run in parallel. |
| `--max-cases INTEGER RANGE` | unset | Run only the first N eval cases in staged order (minimum `1`); without a limit every case runs. A run that skips cases is recorded in `run_config.json` and is never publication evidence. |
| `--n-attempts INTEGER` | unset | Attempts per eval case (pass@k). |
| `--pass-threshold FLOAT` | unset | Score threshold (0.0–1.0) for a case to count as passed. |
| `--max-agent-tokens INTEGER` | unset | Maximum measured agent-model tokens. |
| `--max-judge-calls INTEGER` | unset | Maximum recorded non-salvage judge attempts; excludes transport retries and RAGAS-internal requests. |
| `--max-attempts-total INTEGER` | unset | Maximum logical Harbor attempts admitted across agents and arms, including private comparison arms. |
| `--max-cost-usd FLOAT` | unset | Agent inference accounting admission limit, not billed spend or a hard currency ceiling; requires a price book. Judge, setup, container, backend, and infrastructure charges are excluded. |
| `--price-file FILE` | unset | User-supplied agent inference rates, not billed spend; keyed by exact resolved model identifier. File name/hash and applied rates are retained in `budget.pricing`; see [cost limits](/skills/skillevaluator/tier3-run-reference#cost-limits). |
| `--stop-on-pass / --no-stop-on-pass` | unset | Stop a case's remaining attempts once one passes. |
| `--model TEXT` | unset | Global agent model override. |
| `--agent-model TEXT` | unset | Per-agent model override, `AGENT=MODEL` (repeatable). |
| `--grading-mode [default\|default_plus_custom\|custom_only]` | unset | Reward/grading mode for live eval. |
| `--results-dir DIRECTORY` | unset | Directory for Harbor live-eval results. |
| `--include-skills PATH` | none | Additional skill(s) to mount into the eval environment (repeatable; paths must exist). Requires group workspace mode, set via `skill_workspace.mode: group` in `evals/config.yml`. |
| `--copy-repo` | off | Copy the surrounding repo into the eval environment. |
| `--timeout-multiplier FLOAT` | unset | Scale Harbor's agent, verifier, and environment-build timeouts (built-in default `1.0`). Must be a finite number greater than 0. |
| `--harbor-keep-jobs` | off | Retain Harbor job dirs and artifacts after the run for inspection. |
| `--agent-runtime-preflight / --no-agent-runtime-preflight` | disabled | Opt into an extra agent-only execution of the first staged task before the measured A/B matrix. The CLI value overrides `harbor.agent_runtime_preflight`; without either, no extra agent runs. |
| `--tolerate-baseline-timeouts / --no-tolerate-baseline-timeouts` | disabled | Keep an agent scored on its clean with-skill arm when only baseline attempts hit the agent execution timeout; its baseline and Skill Lift are not reported, and the run is not complete publication evidence. The CLI value overrides `harbor.tolerate_baseline_timeouts`. See [Baseline timeouts](/skills/skillevaluator/tier3-run-reference#baseline-timeouts). |

Flags marked "unset" fall back to their matching keys in `evals/config.yml` where one exists, then to the built-in defaults ([Eval Datasets](/skills/skillevaluator/eval-datasets) documents the config keys and precedence). The exception is `--results-dir`, which resolves independently of `evals/config.yml` (see [Reports & Results](/skills/skillevaluator/reports)).

### Evaluated source provenance

| Flag | Default | Effect |
| --- | --- | --- |
| `--evaluated-source-repository TEXT` | unset | Repository (`owner/name`) of the source tree being evaluated. |
| `--evaluated-source-revision TEXT` | unset | Immutable revision of the evaluated source: a full Git object id, or a `sha256`/`sha384`/`sha512` digest. A short prefix is refused. |
| `--evaluator-container-revision TEXT` | unset | Digest-pinned evaluator image reference recorded beside the evaluated source. |

The identity is recorded on `BENCHMARK.md`, forwarded to every child of a parallel catalog run so each card names the same source tree, and persisted into a Tier 3 run's `run_config.json` (see [Reports & Results](/skills/skillevaluator/reports)).

## quality-check

Score skill quality across correctness, discoverability, reliability, and efficiency. Runs entirely offline — no API key.

```bash title="Keyless quality score"
skillevaluator quality-check ./my-skill
```

| Flag | Default | Effect |
| --- | --- | --- |
| `--min-score INTEGER` | `70` | Minimum composite score required per skill; passing also requires no HIGH or CRITICAL quality findings. |
| `--quality-config PATH` | packaged `quality-default.yaml` | YAML file that adjusts the expected headings, error-handling rules, token budget, description length limits, and reserved name words used for grading; settings it omits keep their packaged values. |

Plus the standard report options. A collection's displayed average does not
replace the minimum required of each skill. Scoring weights, grade bands, the
independent minimum-score gate, and the `--quality-config` file format are
documented in [Tier 1: Validation](/skills/skillevaluator/tier1-validation).

## rubric-eval

Run LLM-as-judge rubric evaluation for a skill against nine weighted criteria. Requires a configured LLM provider.

```bash title="LLM rubric evaluation"
skillevaluator rubric-eval ./my-skill
```

| Flag | Default | Effect |
| --- | --- | --- |
| `--min-score INTEGER` | `70` | Minimum weighted rubric score required to pass; every criterion must also score at least 7/10. |

Plus the standard report options.

## security-scan

Scan for security vulnerabilities using the static scanner stack. Keyless by default; the LLM flags require a configured provider.

```bash title="Static security scan"
skillevaluator security-scan ./my-skill
```

| Flag | Default | Effect |
| --- | --- | --- |
| `--llm / --no-llm` | `no-llm` | Enable LLM security analysis. |
| `--llm-verify` | off | Use LLM verification to reduce false positives. |
| `--skillspector-baseline FILE` | unset | Audited SkillSpector suppression baseline; never discovered from the skill. See [Audited SkillSpector baselines](/skills/skillevaluator/tier1-reference#audited-skillspector-baselines). |

Plus the standard report options.

## pii-scan

Scan for PII and local identifiers (emails, credentials, home paths, and more).

```bash title="PII scan"
skillevaluator pii-scan ./my-skill
```

| Flag | Default | Effect |
| --- | --- | --- |
| `--llm-verify` | off | Use LLM verification to reduce false positives. |
| `--pii-waivers PATH` | none | Apply reviewed, expiring PII waivers from this YAML file. See [PII waivers](/skills/skillevaluator/tier1-reference#pii-waivers). |

Plus the standard report options.

## lint-scripts

Run advisory lint checks on skill scripts. Takes only the standard report options.

```bash title="Advisory script lint"
skillevaluator lint-scripts ./my-skill
```

## harness-check

Show whether each agent harness loads, degrades or drops a skill. This is the Tier 1 `harness` check on its own, and it is static and keyless: no agent or model runs. `TARGET_PATH` is a skill folder, a folder of skills, or a plugin folder, which also gets a plugin panel. See [Harness Check](/skills/skillevaluator/harness-check) for a walkthrough with example output, and [Tier 1: Validation](/skills/skillevaluator/tier1-validation) for every rule.

```bash title="Check one skill against two harnesses"
skillevaluator harness-check ./my-skill --harness codex,claude-code
```

```bash title="Describe your model setup"
skillevaluator harness-check ./my-skill --context-tokens 200000 --headless
```

| Flag | Default | Effect |
| --- | --- | --- |
| `--harness TEXT` | `all` | Comma-separated harnesses to check, for example `codex,claude-code`. |
| `--harness-profiles DIRECTORY` | packaged profiles | Directory of harness profile YAMLs to use instead of the packaged ones. |
| `-v, --verbose` | off | List every harness with its verified version and date. |
| `--json FILE` | unset | Also write the `harness_compatibility` block to this JSON file. |
| `--context-tokens INTEGER` | `0` | Context window of your model in tokens, for the loader limits that depend on it. `0` means unknown, so the smallest window is assumed, except that with `--custom-provider` OpenClaw assumes 128,000 tokens. A value you pass is always used as given. |
| `--custom-provider` | off | Your model is served through a custom or OpenAI-compatible provider, or is a model the harness ships no metadata for. With an unknown context window, OpenClaw then assumes 128,000 tokens instead of the smallest. |
| `--explicit-invocation` | off | Skills are activated by an explicit mention (for example Codex `$skill`), not opened by the model. |
| `--headless` | off | The harness runs non-interactively (for example `claude -p` in CI) with default permissions. |
| `--harness-version HARNESS=VERSION` | unset | The release you run, for example `codex=0.170.0`; repeat for each harness. A release other than the one a profile was verified on adds an advisory note. An unknown harness name is a usage error (`2`). |

Without the four model-setup flags (`--context-tokens`, `--custom-provider`, `--explicit-invocation`, `--headless`), the check assumes an unknown context window, a native provider, model-initiated activation and an interactive session.

This command does not take the standard report options; use `--json` for a file. It is advisory: findings never change the exit code. An unknown `--harness` name is a usage error (`2`), and a path with no skills and no plugin manifest exits `1`. A plugin folder with a recognized manifest still gets its plugin panel and exits `0`, even when it has no skills.

## context-optimization-check

Detect redundant content within one skill (intra-skill deduplication). Needs an embeddings-capable provider plus a chat LLM for the verification stage — see [Tier 2: Deduplication](/skills/skillevaluator/tier2-deduplication).

```bash title="Intra-skill redundancy check"
skillevaluator context-optimization-check ./my-skill
```

| Flag | Default | Effect |
| --- | --- | --- |
| `--threshold FLOAT` | `0.8` | Similarity threshold for clustering candidate duplicates. |
| `--model TEXT` | provider default | Embedding model override. |
| `--llm-model TEXT` | provider default | LLM model override for the verification stage. |

Plus the standard report options.

## dedup-scan

Alias of [context-optimization-check](#context-optimization-check) — same behavior, same flags (`--threshold` default `0.8`, `--model`, `--llm-model`, plus the standard report options). Only the report basename differs: `skillevaluator-dedup` instead of `skillevaluator-context`.

```bash title="Same check, alias name"
skillevaluator dedup-scan ./my-skill
```

## similarity-check

Detect duplicate content across a collection with embedding similarity (inter-skill deduplication). Takes a content path — a single item or a folder of them.

```bash title="Compare a collection"
skillevaluator similarity-check ./skills
```

| Flag | Default | Effect |
| --- | --- | --- |
| `--type [skill\|rules\|workflows\|auto]` | `auto` | Content type of the target. |
| `--threshold FLOAT` | `0.75` | Minimum similarity to report. |
| `--full-body` | off | Embed full file bodies instead of descriptions. |
| `--model TEXT` | provider default | Embedding model override. |
| `--catalog FILE` | none | Compare exactly one skill against a local catalog. |
| `--save-catalog FILE` | none | Build and save a versioned local catalog from this collection. |
| `--max-entries INTEGER` | `1024` | Maximum selected manifests for a fresh collection; range `1`–`5000`. Does not restrict loaded catalog queries. |
| `--max-scalar-comparisons INTEGER` | `128000000` | Maximum comparisons multiplied by embedding dimensions; must be positive. Applies to pairwise scans and catalog queries. |

Plus the standard report options. Classification tiers and catalog rules are in [Tier 2: Deduplication](/skills/skillevaluator/tier2-deduplication).

For older scripts, the hidden deprecated aliases `--cache` and `--save-cache`
still map to `--catalog` and `--save-catalog`, respectively. Use the catalog
spellings for new commands. A catalog query and a catalog build cannot be
combined in one invocation, and each alias cannot be supplied alongside its
replacement.

## tier3 select-decoys

Choose neighboring skills from a local description-embedding catalog and save
a frozen JSON manifest for group evaluation. This command makes an embedding
request for the target; it does not launch an agent or modify the run policy.

```bash title="Save a group-workspace selection"
skillevaluator tier3 select-decoys ./skills/my-skill \
  --catalog ./skill-catalog.json --catalog-root ./skills \
  --count 4 --output ./skills/my-skill/evals/decoys.json
```

| Flag | Default | Effect |
| --- | --- | --- |
| `--catalog FILE` | required | Skill-only catalog with at most 1,000 entries, saved by `similarity-check --save-catalog` without `--full-body`. |
| `--catalog-root DIRECTORY` | required | Local collection root corresponding to catalog-relative skill paths. |
| `--count INTEGER` | `4` | Number of eligible skills to select, from `1` to `1000`; fails if fewer are available. |
| `--exclude PATH` | none | Exclude a catalog-relative skill path; repeatable. |
| `--model TEXT` | configured embedding model | Embedding model override; must match the catalog. |
| `-o, --output FILE` | required | Selection manifest to write, outside the collection or inside the target's `evals/` directory. |

The manifest records the catalog hash and embedding identity, selected paths
and scores, target manifest hash, and selected runtime-content hashes. The
target's own collection uses a root relative to its parent directory, so an
identical checkout can reuse the manifest. The hashes cover everything staging
copies from a selected skill, including ignored or untracked files and empty
folders, so generate a manifest you plan to commit from a fresh clone.
Unrelated external collections
retain an absolute root. Evaluator-only files are excluded from decoy hashes;
changing a decoy's evaluation config does not require a new selection. Output
cannot overwrite evaluation inputs or an existing non-manifest file. Set
`skill_workspace.mode: group` and `skill_workspace.decoy_manifest` in
`evals/config.yml` to consume it. The manifest path is relative to the target
skill's parent directory. Evaluation validates and reuses the selection
without contacting the embedding provider. See
[Select a reproducible decoy set](/skills/skillevaluator/agents-and-sandboxes#select-a-reproducible-decoy-set)
for a complete example and the limits of similarity-based selection.

## tier3 interact

Run a fixed-source skill interaction experiment:

```bash
skillevaluator tier3 interact ./skills/a ./skills/b \
  --eval-source ./reviewed-suite --n-attempts 3 --agents codex --json
```

| Option | Default | Purpose |
| --- | --- | --- |
| `SKILLS...` | Required | At least two distinct skill directories. |
| `--eval-source` | Required | Skill directory containing the reviewed `evals/` source reused for every configuration. |
| `-a`, `--agents` | Provider-native agent | Comma-separated Harbor agents. |
| `--model` | Provider/config model | Global agent model override. |
| `--env-mode` | `docker` | Existing Harbor environment backend. |
| `--environment-kwarg`, `--ek` | None | Repeatable backend `KEY=VALUE`; never pass secrets. |
| `--n-attempts` | `1` | Positive fixed attempt budget for every case; disables early stopping. |
| `--n-concurrent` | Source config or `4` | Concurrent trials within a configuration. |
| `--metric` | `goal_accuracy` | Outcome metric: `goal_accuracy`, `accuracy`, or `overall` for `custom_only` grading. |
| `--results-dir` | `SKILLEVALUATOR_RESULTS_DIR` or `interaction-results` beside the source | Output root outside all inputs; home shortcuts and symlinked parents are resolved. |
| `--harbor-keep-jobs` | Off | Retain raw Harbor jobs alongside collected evidence. |
| `--json` | Off | Print the complete JSON report on stdout; progress uses stderr. |

Runs baseline, each individual skill, and the combined selection. Saves
`interaction.json`, fixed-source evidence, and separate Tier 3 results for every
configuration. Exit 0 means a comparable experiment completed; positive,
negative, and zero interaction all use exit 0. Exit 1 means incomplete or
incomparable evidence. Invalid command arguments use Click's usage-error exit.
See [Skill interaction experiments](/skills/skillevaluator/tier3-experiments#skill-interaction-experiments)
for the calculation, input restrictions, attempt budget, and interpretation.

## tier3 compare-versions

```bash
skillevaluator tier3 compare-versions ./my-skill --current CURRENT --prior PRIOR --results-dir ./results --json
```

Read-only comparison of two retained runs; no agent execution, credentials,
or artifact writes. Without `--json`, prints the prior/current score table.
Unavailable comparisons print reasons and exit nonzero.

| Flag | Default | Effect |
| --- | --- | --- |
| `--current RUN` | required | Current retained run id, or `latest` completed run. |
| `--prior RUN` | required | Prior retained run id, or `latest` eligible different skill revision. |
| `--results-dir DIRECTORY` | normal results-root resolution | External results root containing the per-skill directory. |
| `--json` | off | Print the complete comparison block as JSON. |

## tier3 evaluate

Run Tier 3 live agent evaluation: real agents attempt your eval cases with and without the skill, and the difference becomes Skill Lift. Requires the `tier3` extra, a provider key for grading, a credential the selected agents can use — with NVIDIA Build in `--env-mode docker`, `kata`, or `local`, the one `NVIDIA_API_KEY` covers both roles — and a running backend (Docker by default). The full walkthrough is [Tier 3: Live Agent Evaluation](/skills/skillevaluator/tier3-live-evaluation).

```bash title="Live evaluation with codex in Docker"
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker
```

<Note>
Use `skillevaluator tier3 PATH` for the complete workflow with automatic missing-source generation. `tier3 evaluate` retains explicit execution and opt-in `--autopilot`. The old top-level spelling `skillevaluator evaluate` still works for existing scripts but does not appear in `--help`.
</Note>

| Flag | Default | Effect |
| --- | --- | --- |
| `-a, --agents TEXT` | provider-native | Comma-separated Harbor agents. Defaults: NVIDIA Build=`opencode`, OpenAI=`codex`, Anthropic=`claude-code`. Supported: `claude-code`, `codex`, `opencode`; Experimental: `hermes` (Docker with NVIDIA Build only; see [Experimental Hermes](/skills/skillevaluator/agents-and-sandboxes#experimental-hermes)). The alias `claude` is accepted for `claude-code`. See [Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes). |
| `--env-mode` | `docker` | Where trials run. SkillEvaluator exposes 27 environment modes: 26 Harbor-native modes — `docker`, `daytona`, `e2b`, `modal`, `runloop`, `langsmith`, `ec2`, `gke`, `ack`, `openshift`, `novita`, `apple-container`, `singularity`, `islo`, `tensorlake`, `cwsandbox`, `wandb`, `use-computer`, `blaxel`, `beam`, `skypilot`, `hyperbrowser`, `vercel`, `kata`, `runta`, `mosaic` — plus SkillEvaluator `local` mode. `wandb` runs Harbor's `cwsandbox` backend with W&B authentication, so it needs the `harbor[cwsandbox]==0.25.0` extra; Harbor 0.25 has no `wandb` extra. Harbor's `cua-cloud`, `opensandbox`, `hf-sandbox`, `podman`, `prime`, `smol`, `sbx`, `sail`, and `kernel` backends are not selectable. See the complete extras and prerequisites matrix in [Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes#where-trials-run). Installed packages can add more modes; see [Custom execution backends](/skills/skillevaluator/agents-and-sandboxes#custom-execution-backends). |
| `--backend` | `harbor` | Execution backend: `harbor`, or the experimental `nemo-gym` task-folder adapter. Public Gym 0.7.0 is incompatible; see [NeMo Gym integration](/skills/skillevaluator/nemo-gym). |
| `--environment-kwarg`, `--ek` | none | Operator-only `KEY=VALUE` constructor option for Harbor-native non-Docker backends (repeatable). Values accept JSON-compatible types; skill-owned `evals/config.yml` cannot set them. Never pass secrets. Docker and local reject all kwargs, Kata accepts only `kata_runtime` and `kata_dns`, and Harbor runtime-policy fields for overrides, mounts, networks, extra Compose, lifecycle, and pod security are reserved, as are `stream`, `enable_environment_dir_upload`, `auto_mounts`, `auto_mounting`, and `auth`/`forward_compose_env` for `cwsandbox`/`wandb`. Names the selected backend does not accept in Harbor 0.25.0 are rejected. See [Native backend constructor options](/skills/skillevaluator/agents-and-sandboxes#native-backend-constructor-options). |
| `--autopilot` | off | Create one eval case when no dataset/task source exists, then evaluate. The case is LLM-generated with the configured provider, with a deterministic keyless template fallback; an existing source is never overwritten. |
| `--skip-baseline` | off | Skip the without-skill baseline (no lift analysis, faster). |
| `--skip-with-skill` | off | Run only the without-skill baseline, for example to calibrate a task set or measure a model change before running the skill. The report is marked baseline-only: the overall score is the baseline score, with-skill and lift are unavailable (shown as `skipped` or N/A, never 0.0), and no skill verdict is given. Mutually exclusive with `--skip-baseline`. Not available on `validate`. |
| `--reuse-baseline RUN` | off | Reuse a compatible fresh without-skill baseline from a retained run id in the same results root, or `latest`. An incompatible explicit id fails before execution; `latest` falls back to a fresh baseline. Mutually exclusive with `--skip-baseline` and `--skip-with-skill`. |
| `--compare-to RUN` | off | Compare with-skill scores to a compatible retained run. `latest` selects the newest eligible different skill revision, or records an unavailable comparison and continues. An incompatible explicit id fails before execution. Mutually exclusive with `--skip-with-skill`. Independent of no-skill Skill Lift. |
| `--n-attempts INTEGER` | unset | Attempts per eval case (pass@k). |
| `--pass-threshold FLOAT` | unset | Score threshold (0.0–1.0) for a case to count as passed. |
| `--max-agent-tokens INTEGER` | unset | Maximum measured agent-model tokens. |
| `--max-judge-calls INTEGER` | unset | Maximum recorded non-salvage judge attempts; excludes transport retries and RAGAS-internal requests. |
| `--max-attempts-total INTEGER` | unset | Maximum logical Harbor attempts admitted across agents and arms, including private comparison arms. |
| `--max-cost-usd FLOAT` | unset | Agent inference accounting admission limit, not billed spend or a hard currency ceiling; requires a price book. Judge, setup, container, backend, and infrastructure charges are excluded. |
| `--price-file FILE` | unset | User-supplied agent inference rates, not billed spend; keyed by exact resolved model identifier. File name/hash and applied rates are retained in `budget.pricing`; see [cost limits](/skills/skillevaluator/tier3-run-reference#cost-limits). |
| `--stop-on-pass / --no-stop-on-pass` | unset | Stop a case's remaining attempts once one passes. |
| `--n-concurrent INTEGER` | unset | Concurrent eval cases per agent. |
| `--max-agents INTEGER` | unset | Maximum agents to run in parallel. |
| `--max-cases INTEGER RANGE` | unset | Run only the first N eval cases in staged order (minimum `1`); without a limit every case runs. A run that skips cases is recorded in `run_config.json` and is never publication evidence. |
| `--model TEXT` | unset | Global agent model override. |
| `--agent-model TEXT` | unset | Per-agent model override, `AGENT=MODEL` (repeatable), e.g. `--agent-model codex=gpt-6.1-sol`. Accepts the `claude` alias for `claude-code`. |
| `--custom-dockerfile-mode [preserve\|rebase]` | unset | How a custom `evals/environment/Dockerfile` is applied: `rebase` (built-in default) swaps its `FROM` line onto the eval base image; `preserve` keeps your `FROM` and appends the eval dependencies. |
| `--skill-workspace-mode [isolated\|group]` | unset | `isolated` (built-in default) stages only the target skill in the agent workspace; `group` also stages the skills from `--include-skills` / `skill_workspace.include`. |
| `--include-skills PATH` | none | Additional skill(s) to mount into the eval environment (repeatable; paths must exist). Requires group workspace mode (`--skill-workspace-mode group` or `skill_workspace.mode: group` in `evals/config.yml`). |
| `--copy-repo` | off | Copy the surrounding repo into the eval environment. |
| `--grading-mode [default\|default_plus_custom\|custom_only]` | unset | Reward/grading mode — see [Custom Graders & Tasks](/skills/skillevaluator/custom-graders). The legacy aliases `aces_default` (→ `default`) and `aces_plus_custom` (→ `default_plus_custom`) are still accepted. |
| `--results-dir DIRECTORY` | unset | Directory for live-eval results (precedence rules in [Reports & Results](/skills/skillevaluator/reports)). |
| `--harbor-keep-jobs` | off | Retain Harbor job dirs and artifacts for inspection with [harbor-view](#harbor-view). |
| `--verbose` | off | Also print the per-evaluator and per-dimension tables and the metric findings after the case summary. |
| `--agent-runtime-preflight / --no-agent-runtime-preflight` | disabled | Opt into an extra agent-only execution of the first staged task before the full A/B evaluation matrix. |
| `--tolerate-baseline-timeouts / --no-tolerate-baseline-timeouts` | disabled | Keep an agent scored on its clean with-skill arm when only baseline attempts hit the agent execution timeout; its baseline and Skill Lift are not reported. The CLI value overrides `harbor.tolerate_baseline_timeouts`. See [Baseline timeouts](/skills/skillevaluator/tier3-run-reference#baseline-timeouts). |
| `--timeout-multiplier FLOAT` | unset | Scale Harbor's agent, verifier, and environment-build timeouts (built-in default `1.0`). Must be a finite number greater than 0. |
| `--override-cpus INTEGER` | unset | Override eval-environment CPU allocation (passed through to Harbor). |
| `--override-memory-mb INTEGER` | unset | Override eval-environment memory in MB (passed through to Harbor). |
| `--override-storage-mb INTEGER` | unset | Override eval-environment storage in MB (passed through to Harbor). |
| `--evaluated-source-repository TEXT` | unset | Repository (`owner/name`) of the source tree being evaluated, persisted into the run's `run_config.json`. |
| `--evaluated-source-revision TEXT` | unset | Immutable revision of the evaluated source: a full Git object id, or a `sha256`/`sha384`/`sha512` digest. A short prefix is refused. |
| `--evaluator-container-revision TEXT` | unset | Digest-pinned evaluator image reference recorded beside the evaluated source. |
| `--progress [auto\|rich\|plain\|off]` | `auto` | Progress display for the run: `rich` live view (a fixed-height table of the run plan and recent stages, with the full plan printed above it), `plain` line output, or `off`. |
| `--progress-interval SECONDS` | mode default | Seconds between progress heartbeats, from `0.05` to `86400`: how often the `rich` view refreshes elapsed times and how often `plain` output repeats a `still running` line for active stages. Defaults to 1 second for `rich` and 60 seconds for `plain`, including `auto` with redirected or CI output. |

Without a `--model`/`--agent-model` override, agent execution uses
`gpt-6.1-sol` for OpenAI (OpenCode: `openai/gpt-5.6-sol` unless
`SKILL_EVAL_LLM_MODEL` is set), `claude-opus-5-5` for Anthropic, and
`nvidia/nemotron-3.5-lightning-30b-a3b` for NVIDIA Build. OpenCode displays the
last value as `nvidia/nvidia/nemotron-3.5-lightning-30b-a3b` because its adapter
namespace precedes the NVIDIA catalog ID.

Flags marked "unset" fall back to their matching keys in `evals/config.yml` where one exists, then to the built-in defaults. The exception is `--results-dir`, which resolves independently of `evals/config.yml`.

## tier3 migrate-dataset

Preview a lossless conversion to the preferred dataset object. No provider or
Tier 3 runtime is required:

```bash
skillevaluator tier3 migrate-dataset ./my-skill/evals/evals.json --skill-name my-skill
```

| Option | Default | Purpose |
| --- | --- | --- |
| `--skill-name NAME` | inferred when unambiguous | Target skill identity. Required for all-negative or mixed-identity legacy datasets. |
| `--output DESTINATION` | preview only | Write a new `.json`, `.yaml`, or `.yml` file; existing paths and the source are refused. |

`SOURCE` accepts JSON, case-per-line JSONL, YAML, or YML. JSONL is input-only;
output is one JSON/YAML object. See [Migrate a legacy dataset](/skills/skillevaluator/eval-datasets#migrate-a-legacy-dataset)
for routing-label requirements, preserved fields, and how to promote the reviewed
conversion without changing fixture paths.

## create-eval-dataset

Create eval datasets in `evals/evals.json`. The default `auto` grounding prefers
substantive author guidance with content beyond headings, then usable matching
retained traces from a completed run scoped to the skill. If neither exists,
generation fails before provider access, collection, or writes. Write task
requests and observable outcomes in `evals/EVAL.md` or a `--prompt` file, or
explicitly choose `--grounding skill` for the previous skill-only workflow.
Uses the configured LLM provider unless you pass `--no-llm`. See
[Eval Datasets](/skills/skillevaluator/eval-datasets#choose-a-grounding-source)
for source selection and migration details.

```bash title="Generate after authoring ./my-skill/evals/EVAL.md"
skillevaluator create-eval-dataset ./my-skill --full --prompt ./my-skill/evals/EVAL.md
```

<Warning>
Auto leaves an existing `evals/evals.json` unchanged unless you pass `--force`
or `--refine`, even when matching traces exist. `--force` can overwrite hand-edited
cases. `--refine` regenerates intent- or skill-grounded datasets; auto trace
refinement preserves authored cases and IDs. Explicit `--grounding trace`
continues to opt into in-place refinement without either flag. Preview with
`--dry-run` before replacing a dataset, including an autopilot starter.
</Warning>

| Flag | Default | Effect |
| --- | --- | --- |
| `--full` | off | Generate the full four-bucket dataset (LLM). With `--no-llm`, template mode omits the negative bucket without an authored off-skill prompt in `EVAL.md`. |
| `--no-llm` | off | Use local templates only (no API key needed). |
| `--dry-run` | off | Preview without writing. |
| `--force` | off | Overwrite existing `evals/evals.json`. |
| `--prompt PATH` | none | Developer eval-guidance file for generation (default: auto-detects `evals/EVAL.md`). |
| `--grounding {auto,skill,intent,trace}` | `auto` | Prefer author guidance with content beyond headings, then usable matching retained skill-scoped traces. Explicit `skill` permits skill-only generation; `intent` requires guidance; `trace` may collect trajectories when none exists. |
| `--refine` | off | Refine generated cases with trajectories on top of the selected grounding source. With effective intent or skill grounding it regenerates the dataset; with trace grounding and an existing dataset, it refines those cases in place. Auto requires this flag or `--force` to modify an existing dataset; explicit `--grounding trace` already opts into refinement. Auto selection must find guidance or matching retained traces first; `--refine` does not collect evidence to bypass missing input. After selection, if no trajectory exists, collection runs `claude-code` through Harbor and needs the `tier3` extra, Docker, and a configured provider (see [Eval Datasets](/skills/skillevaluator/eval-datasets)). |
| `--from-results PATH` | none | With `auto`, `trace`, or `--refine`: results path containing agent trajectories (default: latest results). Auto requires a completed run scoped to the skill. |
| `--results-dir DIRECTORY` | unset | With `auto`, `trace`, or `--refine`: external results root to search for trajectories under the skill's directory. |

## init-custom-grader

Create a bring-your-own-grader (BYOG) starter under `evals/`. The grader contract it scaffolds is documented in [Custom Graders & Tasks](/skills/skillevaluator/custom-graders).

```bash title="Scaffold a Python grader"
skillevaluator init-custom-grader ./my-skill
```

| Flag | Default | Effect |
| --- | --- | --- |
| `--mode [default_plus_custom\|custom_only]` | `default_plus_custom` | How the custom grader combines with the default graders. |
| `--language [python\|shell]` | `python` | Grader language. |
| `--force` | off | Overwrite an existing top-level custom grader. |
| `--no-config` | off | Only create the grader file; do not create or update `evals/config.yml`. |

## init-harbor-task

Create a bring-your-own-task (BYOT) Harbor starter template under `evals/harbor/`. See [Custom Graders & Tasks](/skills/skillevaluator/custom-graders) for how case IDs map to eval entries.

```bash title="Scaffold a Harbor task"
skillevaluator init-harbor-task ./my-skill
```

| Flag | Default | Effect |
| --- | --- | --- |
| `--force` | off | Overwrite an existing starter case. |
| `--case-id TEXT` | `case-001` | Harbor case directory and eval entry id. |
| `--mode [default\|default_plus_custom\|custom_only]` | `custom_only` | Grading mode the starter configures. |
| `--language [python\|shell]` | `python` | Grader language. |
| `--with-config` | off | Create or update `evals/config.yml` for native Harbor mode. |

## view

Open the latest HTML live-evaluation report for a skill. If the results exist but `report.html` is missing, `view` regenerates it before opening. It reuses the feedback the run saved in `result.json`, and the run's `what_to_fix.json` when there is one, and makes no LLM call. A run that saved no feedback still has its feedback rebuilt, with the LLM when a provider is configured, and its `what_to_fix.json` rewritten.

```bash title="Open the latest report"
skillevaluator view ./my-skill
```

| Flag | Default | Effect |
| --- | --- | --- |
| `--results-dir DIRECTORY` | unset | Results directory to read from (default resolution in [Reports & Results](/skills/skillevaluator/reports)). |

## tier3 history

```bash
skillevaluator tier3 history ./my-skill
skillevaluator tier3 history ./my-skill --results-dir ./saved-runs --agent opencode --json
skillevaluator tier3 history ./my-skill --output ./lift-history.md
```

List every retained run chronologically, with one row per run and agent,
including failed, incomplete, and corrupt runs. An unavailable lift is JSON
`null` with `lift_status` and reasons, never an inferred zero. Segments and
visible break markers separate model, dataset, attempt, scoring-policy, and
other configuration changes; the change column never crosses a segment boundary.
Skill revisions alone do not break baseline compatibility.

| Flag | Default | Effect |
| --- | --- | --- |
| `--results-dir DIRECTORY` | normal results-root resolution | External root containing the per-skill directory; honors the configured location and legacy fallback like `compare`. |
| `--agent TEXT` | all | Include only this agent's rows. |
| `--json` | off | Print the complete history as JSON instead of a table. |
| `--output FILE` | none | Write exactly this `.json`, `.csv`, or `.md` export. The file must be outside all candidate results roots; no directories are created. Stdout still follows `--json`. |

The command reads summaries only: no agents, provider credentials, or raw
trajectories are needed. See [Lift history and retention](/skills/skillevaluator/reports#lift-history-and-retention).

## compare

Compare live-evaluation results across agents from previous runs.

```bash title="Compare agents"
skillevaluator compare ./my-skill
```

| Flag | Default | Effect |
| --- | --- | --- |
| `--results-dir DIRECTORY` | unset | Results directory to read from. |

## tier3 validate-grader

Qualify the shipped Tier 3 verifier against independently labelled evidence.
See [Grader Validation](/skills/skillevaluator/grader-validation) for the versioned suite format,
denominators, provenance and interpretation.

```bash
skillevaluator tier3 validate-grader --check-suite
skillevaluator tier3 validate-grader ./suite --repeats 3 -o qualification
```

| Option | Default | Meaning |
| --- | --- | --- |
| `SUITE_DIR` | bundled reference suite | Directory containing suite.json |
| `--repeats N` | 3 | Positive number of grades per case |
| `--split assessment\|tuning\|all` | assessment | Split selection; all keeps statistics separate |
| `--case ID` | all selected cases | Repeatable case filter |
| `-o, --output DIR` | grader-validation-output | Empty output directory for JSON and attempt artifacts |
| `--json` | off | Print full report as JSON |
| `--check-suite` | off | Validate suite and executable labels; with a policy also check expected coverage, no provider/output writes |
| `--qualification-policy FILE` | unset | Explicit assessment-only coverage and risk policy JSON; no default policy |
| `--concurrency N` | 1 | Maximum concurrent verifier subprocesses |
| `--timeout SECONDS` | 300 | Positive per-attempt timeout |
| `--max-false-pass-rate RATE` | unset | Gate any reported metric's false-pass rate |
| `--max-false-fail-rate RATE` | unset | Gate any reported metric's false-fail rate |
| `--max-unjudgeable-rate RATE` | unset | Gate any reported metric's error/insufficient rate |

Rates must be between 0 and 1. Exit 0 reports results (including disagreement),
exit 1 means an explicitly supplied gate was exceeded, and exit 2 means invalid
usage, suite or provider configuration. Gates use assessment only with split all.

With `--qualification-policy FILE`, exit 1 also means failed or insufficient policy
evidence. Missing required metrics/rates and unmet distinct-case class/category
minima cannot pass; repeats and behavior steps do not add cases. The policy only
accepts `--split assessment` and cannot be combined with standalone `--max-*-rate`
gates; put its criteria in the policy file. These combinations are usage exit 2
before provider access. Offline `ready` means expected-label coverage only,
with scored counts/rates unobserved. Live insufficient preflight retains a
`not_run` receipt with no attempts before provider access. Covered live runs retain the
evaluated decision in JSON stdout and `grader-validation.json`. Policy identity,
canonical hash, effective definition, coverage checks and reasons are retained;
see [policy details](/skills/skillevaluator/grader-validation#explicit-qualification-policy).

## tier3 feedback

Chat in the terminal about one completed live-evaluation run: which cases
explain a score, how the skill compared with the baseline, and what to change
in the skill. Answers are grounded in that run's saved evidence: scores,
with-skill and baseline comparisons, every case's attempts with per-metric
scores, measured agent tokens and seconds, execution errors, and the report's
saved feedback. The run is not evaluated again, and the report's LLM feedback
is not generated again.

```bash title="Discuss a completed run"
skillevaluator tier3 feedback ./my-skill/evals/results/20260709_141530_12345_a1b2c3d4e5f6

# Name the evaluated skill explicitly
skillevaluator tier3 feedback ./my-skill/evals/results/20260709_141530_12345_a1b2c3d4e5f6 --skill ./my-skill
```

Pass one completed run directory, also when results live under a custom
`--results-dir`. The command infers the evaluated skill from the run. An
explicit `--skill` must name the skill directory the run recorded; when that
skill has changed since the run, the chat says so, and its answers describe the
evaluated version.

`tier3 feedback` always opens this chat, also in a directory that contains a
skill folder named `feedback`. To evaluate that skill, run
`skillevaluator tier3 ./feedback` or `skillevaluator tier3 evaluate feedback`.

| Flag | Default | Effect |
| --- | --- | --- |
| `--skill DIRECTORY` | inferred from the run | Evaluated skill directory. It must match the skill the run recorded. |

Ask, for example, "Which cases explain the lower score?" or "What should I
change if agent time matters more than token cost?" Treat proposed changes as
hypotheses and check them with another evaluation; answer quality depends on
the configured model. Type `exit` or press Ctrl-D to end the chat. Ctrl-C
during an answer discards that question and returns to the prompt.

The chat uses the provider and model configured for LLM checks
(`SKILL_EVAL_LLM_PROVIDER` and `SKILL_EVAL_LLM_MODEL`; see [Environment
Variables](/skills/skillevaluator/environment-variables)). Each question sends the run's compact
evidence to that provider. The evidence names the run by its ID instead of its
local path. In every text field, such as errors, warnings, case questions and
saved feedback, an absolute path that ends in the run's directory is replaced
with that ID. This covers the path a copied run had on the machine where it ran,
also when it starts with `~user/`, `$HOME/`, `${HOME}/` or `%USERPROFILE%\`,
unless that path contains a space. A path joined to it, such as `/usr/bin` in
`PATH=/usr/bin:/home/ci/results/RUN_ID`, is kept. A root right after `,` `;` `:`
`=` `(` `[` or `{` starts a new path, so a directory name that ends in one of
those characters before a `/` is read as two paths and only the second is
replaced. So does a root right after a `\n`, `\r` or `\t` escape in JSON or repr
text, such as the next line of an env dump, also with ANSI codes after the
escape, as on a progress line (`\r\u001b[K`); a relative path after one is kept,
and a UNC server name that starts with `n`, `r` or `t`, such as `\\nas01\share`,
is not read as an escape. A path may also start right after an ANSI code, such
as the color code `\u001b[0m` or the erase-line code `\u001b[K`, and a path
colored in parts is replaced whole. The evaluated skill's directory is replaced
with its name and your home directory with `~`; the skill directory is known
when you pass `--skill` or the run is in the skill's `evals/results` directory.
Path masking leaves a URL unchanged, even a path in its query or fragment,
unless its scheme is `file:` or ends in `+file:`, such as `git+file:`. A URL
ends at whitespace; at a quote, `<`, `>`, a backtick, `^`, `{`, `|`, `}` or an
ANSI escape (raw, `\u001b` or `\x1b`); at a `\n`, `\r` or `\t` escape; and
before an absolute path or another scheme that follows a `,` or `;` (also with
an option or variable name and `=`, such as `LOG=` or `--find-links=`) or a
Markdown `](`. So a path after a URL in JSON, on the next line of an escaped env
dump or stderr, in a `,`- or `;`-separated list or in
`DOC=https://host/e;LOG=/path`, and a `file:` URL listed after one, are still
replaced. A path joined to a URL by any other character, such as `:`, `=`, `(`
or `)`, is not. The escape may have a doubled backslash, as in JSON inside JSON.
A backslash and `n`, `r` or `t` that start a Windows folder name made of
letters, digits, `_`, `.`, `~`, `$`, `%` or `-` and followed by as many
backslashes, such as `\runs\` or `\t\` (`\\runs\\` in JSON), are not an escape
unless an escaped character such as `\u001b` or `\xa0` comes next and is not
itself followed by a name and as many backslashes, so a Windows path with such
folders, such as `\t\x64\runs\`, in a URL is kept. Any other folder or file name
that starts with `n`, `r` or `t`, such as `\runs(old)\` or `\notes.txt`, ends
the URL there. Credential-shaped text is redacted before the text is shortened,
and a shortened field ends in `...<truncated>`. The command writes nothing to
disk.

No request is sent before the first question, so the command exits 0
immediately when standard input is empty. Piped input sends each line as a
question, and each one is a paid request. The command exits 1 when the run or
provider configuration cannot be used, or when questions were asked and every
one of them failed.

## dashboard

Open the optional local tables dashboard. Install the
[`dashboard` extra](/skills/skillevaluator/installation-options#choosing-extras) first, then pass JSON
reports, retained run directories, or directories containing reports. With no
paths, upload JSON files in the browser.

```bash
skillevaluator dashboard reports/run-a.json reports/run-b.json
skillevaluator dashboard --no-browser --port 8502
```

| Option | Default | Description |
| --- | --- | --- |
| `--port` | `8501`, or the next free port | Local port on 127.0.0.1, from 1 through 65535. Pass it to require a specific port; the command fails if that port is busy. |
| `--no-browser` | off | Start the server without opening a browser. |

The server listens on `127.0.0.1`; Ctrl+C stops it. See the
[dashboard guide](/skills/skillevaluator/performance-dashboard) for measurements and
comparison eligibility.

## harbor-view

Open retained Harbor job artifacts with Harbor's trajectory browser. Takes a jobs directory as its argument and has no options. Jobs are only retained when the original run passed `--harbor-keep-jobs`. `skillevaluator tier3 harbor-view` is an identical tier-prefixed spelling.

```bash title="Browse retained trajectories"
skillevaluator harbor-view ./path/to/jobs-dir
```

## doctor

Check live-evaluation runtime readiness: the CLI install, the configured provider, the requested agents and their credentials, and the selected backend. Run it before your first `tier3 evaluate` — see [Agents & Sandboxes](/skills/skillevaluator/agents-and-sandboxes) for what each check means.

```bash title="Deep readiness check"
skillevaluator doctor --env-mode docker
```

`--tier` selects other readiness checks. Without it, `doctor` checks Tier 3 only, as above.

```bash title="Tier 1 scanners and Tier 2 providers"
skillevaluator doctor --tier 1 --checks security,code-integrity,dependency
skillevaluator doctor --tier 2 --verify-models
skillevaluator doctor --tier all
```

- **Tier 1** reports each external scanner the selected checks run: SkillSpector for `security`, Bandit, Semgrep and Gitleaks for `code-integrity`, and pip-audit for `dependency`. Each row shows the resolved executable, the override variable when one selected it, and the version from a bounded version probe. The probe never starts a shell. A scanner fails when it is missing, its override is malformed, or the probe exits nonzero or times out. It also fails when it is older than the release the validators parse: Bandit 1.9.4 or Gitleaks 8. An unreported version is a warning.
- **Tier 2** checks that the embedding provider and the LLM provider used for duplicate analysis resolve, without a network request. With `--verify-models`, it also sends one embedding request and one chat request, each bounded by `--timeout` and sent without retries. Failures show only the HTTP status and a remedy, never the provider's response body.
- **all** runs Tiers 1, 2 and 3 in order. A failed tier does not stop the next, and the exit code is nonzero when any tier fails.

An option for a tier that is not selected exits with a usage error before any check runs. For example, `--checks` without `--tier 1` or `all` is an error, and so is `--timeout` with `--tier 2` but no `--verify-models`, because offline Tier 2 sends no request.

| Flag | Default | Effect |
| --- | --- | --- |
| `--tier [1\|2\|3\|all]` | `3` | Readiness to check: Tier 1 scanners, Tier 2 providers, Tier 3 live evaluation, or all three in order. |
| `--checks TEXT` | `validate`'s default checks | Tier 1 only. Comma-separated checks whose scanners to probe, with the same names as `validate --checks`; add `dependency` to include pip-audit. |
| `--timeout FLOAT` | `60` | Tier 1, and Tier 2 with `--verify-models`. Seconds allowed for each scanner version probe and each Tier 2 live request, including a Bedrock chat request; must be finite, greater than 0 and at most 3600. |
| `-a, --agents TEXT` | provider-native | Comma-separated agents to check. Defaults: NVIDIA Build=`opencode`, OpenAI=`codex`, Anthropic=`claude-code`. |
| `--env-mode` | `docker` | Backend to check (same 27 modes as [tier3 evaluate](#tier3-evaluate)). |
| `--backend [harbor\|nemo-gym]` | `harbor` | Execution backend to check, as in [tier3 evaluate](#tier3-evaluate). `nemo-gym` checks the Gym CLI, the Docker daemon, the `SKILL_EVAL_NEMO_GYM_*` settings and each agent's policy endpoint. |
| `--environment-kwarg`, `--ek` | none | `KEY=VALUE` non-secret constructor option for a Harbor-native non-Docker backend (repeatable). Pass the same values intended for evaluation. Docker and local reject all kwargs, and Kata accepts only `kata_runtime` and `kata_dns`; policy-owned fields are rejected for every eligible backend. |
| `--agent-model TEXT` | unset | Per-agent model override, `AGENT=MODEL` (repeatable) — check readiness with the model each agent will actually run. |
| `--verify-models` | off | Tier 3: live per-agent catalog probe. Prints pass for verified access, warn for an inconclusive or non-authoritative result (success or failure), and fail only for a definitive credential, model, or configuration rejection. Tier 2: one embedding and one chat request. |

`-a`, `--env-mode`, `--backend`, `--environment-kwarg` and `--agent-model` apply to Tier 3 only. `skillevaluator tier3 doctor` checks only Tier 3. It takes the Tier 3 flags, including `--agent-model` and `--verify-models`, and has no `--tier`, `--checks` or `--timeout`.

## health-check

Quick readiness check for the CLI and the selected live-eval backend. Use it as an install smoke test or a fast CI preflight. It differs from [doctor](#doctor) by two flags: `health-check` has no `--agent-model` and no `--verify-models`, so it never sends a live provider request. `--tier` and `--checks` work as in `doctor`, and `--timeout` bounds only the Tier 1 scanner probes.

```bash title="Quick readiness check"
skillevaluator health-check
skillevaluator health-check --tier 1    # CI preflight for the Tier 1 scanners
```

| Flag | Default | Effect |
| --- | --- | --- |
| `--tier [1\|2\|3\|all]` | `3` | Readiness to check, as in [doctor](#doctor). Tier 2 resolves providers offline. |
| `--checks TEXT` | `validate`'s default checks | Tier 1 only. Checks whose scanners to probe. |
| `--timeout FLOAT` | `60` | Tier 1 only. Seconds allowed for each scanner version probe; must be finite, greater than 0 and at most 3600. |
| `-a, --agents TEXT` | provider-native | Comma-separated agents to check. Defaults: NVIDIA Build=`opencode`, OpenAI=`codex`, Anthropic=`claude-code`. |
| `--env-mode` | `docker` | Backend to check (same 27 modes as [tier3 evaluate](#tier3-evaluate)). |
| `--environment-kwarg`, `--ek` | none | `KEY=VALUE` non-secret constructor option for a Harbor-native non-Docker backend (repeatable). Pass the same values intended for evaluation. Docker and local reject all kwargs, and Kata accepts only `kata_runtime` and `kata_dns`; policy-owned fields are rejected for every eligible backend. |

## models

List models visible through the selected provider's catalog. It uses the same provider and credential resolution as evaluation (see [Providers & Credentials](/skills/skillevaluator/configuration)), so it checks catalog reachability and model membership. NVIDIA Build and some compatible gateways expose public catalogs, so a listed model does **not** prove that the inference key works or that the model is compatible with the evaluation harness. `doctor --verify-models` reports such successful but non-authoritative checks as warnings. Top-level only; there is no tier-prefixed twin.

```bash title="List the provider's models"
skillevaluator models --limit 20
```

| Flag | Default | Effect |
| --- | --- | --- |
| `--limit INTEGER RANGE` | `10` | Maximum number of catalog entries to show (`1<=x<=100`). |
| `--json` | off | Emit machine-readable JSON. |

## See also

- [Environment Variables](/skills/skillevaluator/environment-variables) — every `SKILL_EVAL_*` and `SKILLEVALUATOR_*` variable
- [Gate Your CI](/skills/skillevaluator/ci-integration) — exit codes and JSON reports as a merge gate
- [Providers & Credentials](/skills/skillevaluator/configuration) — which commands need which key