Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
docs: bring the published docs in line with v0.5.0 (#186)
Narendran Raghavanee8ee4cb53f8
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex b56cf185..7b35b49e 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,6 +4,15 @@ All notable changes to SkillEvaluator are documented in this file.## Unreleased+### Fixed++- Brought the published documentation in line with 0.5.0. It now covers how+ standard grading builds judge evidence and rebuilds trajectories from+ OpenCode and Codex logs, structured judge output and retries, the evidence+ budget and judge time-budget variables, and the Harbor process environment.+ The CLI reference adds `validate --workers`, the full `--env-mode` list, and+ the exit codes the CLI actually returns (`0`, `1`, `2`).+## 0.5.0 - 2026-10-06### Addeddiff --git a/docs/AGENTS.md b/docs/AGENTS.mdindex 04dff302..a771c731 100644--- a/docs/AGENTS.md+++ b/docs/AGENTS.md@@ -11,10 +11,21 @@ folder holds MDX pages published to <https://docs.nvidia.com/skills/skillevaluat## What lives here- `index.mdx` — landing page and three-tier overview (Fern slug `/`)-- `installation.mdx`, `configuration.mdx` — setup and credential reference-- `tier1-validation.mdx`, `tier2-deduplication.mdx`, `tier3-live-evaluation.mdx`- — the three evaluation tiers-- `developer-guide.mdx` — contributor setup+- `quickstart.mdx` — install, one provider and key, and first runs+- `installation.mdx` — extras, uv, pip, Docker, requirements+- `configuration.mdx` — providers, embeddings, and Tier 3 credential roles+- `tier1-validation.mdx` — Tier 1 checks, profiles, scanner setup, exit codes+- `tier2-deduplication.mdx` — Tier 2 dedup commands, catalogs, thresholds, limits+- `tier3-live-evaluation.mdx` — Tier 3 live agent evaluation workflow+- `eval-datasets.mdx` — the `evals/` contract, dataset generation, run policy+- `agents-and-sandboxes.mdx` — agents, environment backends, local mode, readiness+- `custom-graders.mdx` — bring-your-own graders and native Harbor tasks+- `reports.mdx` — report formats, scores, and the results-on-disk contract+- `benchmark-rollout.mdx` — regenerating and promoting `BENCHMARK.md` cards+- `ci-integration.mdx` — exit codes, GitHub Actions recipe, progressive adoption+- `cli-reference.mdx` — every command and flag+- `environment-variables.mdx` — variable-by-variable inventory+- `developer-guide.mdx` — contributor setup, tests, and docs workflow- `assets/` — images referenced by the pages- `README.md` (human authoring guide) and this `AGENTS.md` are **not** publishedpages — do not add them to the navigationdiff --git a/docs/README.md b/docs/README.mdindex 4909e260..3482c500 100644--- a/docs/README.md+++ b/docs/README.md@@ -12,12 +12,21 @@ contributors making non-trivial edits.## Layout- `index.mdx` — landing page and three-tier overview-- `installation.mdx` — extras, pip, Docker, requirements-- `configuration.mdx` — credential map, providers, and embeddings-- `tier1-validation.mdx` — checks, flags, reports, CI recipe-- `tier2-deduplication.mdx` — dedup commands and thresholds+- `quickstart.mdx` — install, one provider and key, and first runs+- `installation.mdx` — extras, uv, pip, Docker, requirements+- `configuration.mdx` — providers, embeddings, and Tier 3 credential roles+- `tier1-validation.mdx` — checks, profiles, scanner setup, exit codes+- `tier2-deduplication.mdx` — dedup commands, catalogs, thresholds, limits- `tier3-live-evaluation.mdx` — skill evaluation with live agents-- `developer-guide.mdx` — contributor setup+- `eval-datasets.mdx` — the `evals/` contract, dataset generation, run policy+- `agents-and-sandboxes.mdx` — agents, environment backends, local mode, readiness+- `custom-graders.mdx` — bring-your-own graders and native Harbor tasks+- `reports.mdx` — report formats, scores, and the results-on-disk contract+- `benchmark-rollout.mdx` — regenerating and promoting `BENCHMARK.md` cards+- `ci-integration.mdx` — exit codes, GitHub Actions recipe, progressive adoption+- `cli-reference.mdx` — every command and flag+- `environment-variables.mdx` — variable-by-variable inventory+- `developer-guide.mdx` — contributor setup, tests, and docs workflowNavigation order and slugs are defined in [`../fern/docs.yml`](../fern/docs.yml).diff --git a/docs/agents-and-sandboxes.mdx b/docs/agents-and-sandboxes.mdxindex 618d197c..6e8aef3f 100644--- a/docs/agents-and-sandboxes.mdx+++ b/docs/agents-and-sandboxes.mdx@@ -23,7 +23,7 @@ SkillEvaluator selects the provider-native default: `opencode` for NVIDIA Build,| Agent | Notes || --- | --- || `claude-code` | Anthropic's Claude Code CLI. `claude` is an accepted alias. |-| `codex` | OpenAI's Codex CLI. The OpenAI-provider default. |+| `codex` | OpenAI's Codex CLI. The OpenAI-provider default. Runs with `reasoning_effort=high` so scores and cost stay comparable across Harbor releases. || `opencode` | The OpenCode CLI. |The `claude` alias is canonicalized to `claude-code` everywhere an agent@@ -209,9 +209,12 @@ use environment kwargs for resource overrides (`override_cpus`,mount injection (`mounts`, `mounts_json`), network policy (`network_policy`,`phase_network_policies`, `extra_allowed_hosts`), extra Compose definitions(`extra_docker_compose`), lifecycle controls (`delete`, `force_build`,-`keep_containers`), or Kubernetes pod/security overrides. Use SkillEvaluator's-first-class flags and config where available; advanced pod/security overrides-are intentionally not exposed through this CLI surface. Equivalent+`keep_containers`), Harbor's output streaming and environment upload controls+(`stream`, `enable_environment_dir_upload`), the per-trial wiring Harbor+supplies itself (such as `environment_dir`, `session_id`, and `trial_paths`),+or Kubernetes pod/security overrides. Use SkillEvaluator's first-class flags+and config where available; advanced pod/security overrides are intentionally+not exposed through this CLI surface. Equivalentbackend-specific aliases are also reserved when they directly bypass thoseguarantees, including provider volume/resource/network overrides, Kubernetes orcloud workload identities, disabled SSH host verification, and Singularity host@@ -252,16 +255,15 @@ custom config path by itself; configure one of Harbor's supported credentialforms before launching trials.Other eligible, backend-specific constructor options are forwarded after-validation. Harbor acts only on options implemented by the selected backend;-unknown kwargs are not a supported compatibility contract, so use the current-Harbor constructor documentation for that environment. ACK scheduling options-such as `node_selector` and-`tolerations` are accepted, and `image_pull_secret` accepts only the name of an-existing Kubernetes Secret object—not credential contents. Direct credential-forwarding controls such as HF Sandbox's `forward_hf_token` are rejected. Use-the same `--environment-kwarg` values and eligible non-Docker mode with-`doctor` or `health-check` to preflight the configuration you intend to-evaluate.+validation. A key that the selected backend's Harbor 0.24.0 constructor does not+accept is rejected before the run starts; for example, Harbor 0.24 removed+Modal's `modal_sandbox_v2` option, so drop it from existing `--ek` flags. ACK+scheduling options such as `node_selector` and `tolerations` are accepted, and+`image_pull_secret` accepts only the name of an existing Kubernetes Secret+object—not credential contents. Credential-bearing options such as LangSmith's+`api_key` and Vercel's `credential_injection` are rejected. Use the same+`--environment-kwarg` values and eligible non-Docker mode with `doctor` or+`health-check` to preflight the configuration you intend to evaluate.A few Harbor options contain provider-managed secret **references**, not secretvalues. These narrow shapes are allowed; every other credential-shaped kwarg,diff --git a/docs/benchmark-rollout.mdx b/docs/benchmark-rollout.mdxindex 193a9628..585f3c96 100644--- a/docs/benchmark-rollout.mdx+++ b/docs/benchmark-rollout.mdx@@ -56,7 +56,8 @@ skillevaluator validate ./skills/example-skill \--output-dir ./benchmark-backfill/example-skill```-Tier 3 is publication-required by default. Omitting `--agent-eval` therefore+Tier 3 runs by default for skills and is publication-required by default.+Excluding it, with `--no-tier3` or a `--tiers` selection without `3`, thereforeproduces an `INCOMPLETE` card with Tier 3 marked `NOT RUN`, and never apublication recommendation. If a catalog intentionally makes Tier 3 optional,its orchestration must persist `benchmark_policy.tier3_required = false`; the@@ -119,7 +120,7 @@ evaluated, or whose evaluator container is named by a mutable tag rather thanpinned by digest. That includes a card whose Tier 3 row is skipped under an`optional by policy` contract. The flag stays opt-in because cards generatedbefore the identity contract existed cannot satisfy it; without it the scan-behaves exactly as it did before.+skips these identity checks.Supply the identity when regenerating a card:diff --git a/docs/ci-integration.mdx b/docs/ci-integration.mdxindex 6174a752..513dac00 100644--- a/docs/ci-integration.mdx+++ b/docs/ci-integration.mdx@@ -12,16 +12,15 @@ Turn SkillEvaluator into a merge gate for your skill repository. The recommended| Exit code | Meaning | What to do || --- | --- | --- || `0` | Success — every gating check passed | Merge |-| `1` | Validation failed — one or more gating checks failed | Fix the skill; read the report |-| `2` | Configuration error — bad flags or an unreadable target | Fix the pipeline, not the skill |-| `3` | Runtime error — an unexpected internal failure | Re-run; the skill was never judged. File an issue if it persists |+| `1` | Validation failed — a gating check failed, the run could not complete, or an unexpected internal error occurred | Read the report or log; fix the skill, or re-run and file an issue if the run itself failed |+| `2` | Usage error — an unknown flag, a value outside an option's type or choices, or a target path that does not exist | Fix the pipeline, not the skill |Tier 1 checks always gate the exit code. Tier 2 deduplication findings gate when dedup runs unless you pass `--no-block-on-dedup`; the recommended keyless gate below selects `--tiers 1` to exclude Tier 2 and Tier 3. Tier 3 live-evaluation results are advisory inside `validate` unless you pass `--block-on-agent-eval`. Reports record the effective choice for each tier. Without an explicit scope, `validate` runs all three tiers for skills, including Tier 3 autopilot.## The recommended gate```bash title="Keyless CI gate"-skillevaluator validate ./skills --external --tiers 1 -r json,markdown -o reports --min-score 70+skillevaluator validate ./skills --external --tiers 1 -r json,markdown,sarif -o reports --min-score 70```Each flag earns its place:diff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdxindex f86ae0a3..10de1f8e 100644--- a/docs/cli-reference.mdx+++ b/docs/cli-reference.mdx@@ -59,11 +59,10 @@ In practice: a default `validate` run (no `-r`) shows the compact pipeline view| Code | Meaning || --- | --- || `0` | Success |-| `1` | Validation failed, or the run could not complete |-| `2` | Configuration error — bad flags or an unreadable target |-| `3` | Runtime error — an unexpected internal failure |+| `1` | Validation failed, the run could not complete, or an unexpected internal error occurred |+| `2` | Usage error — an unknown flag, a value outside an option's type or choices, or a target path that does not exist |-**Version.** `skillevaluator --version` prints the installed version.+**Global options.** `skillevaluator --version` prints the installed version. `-v, --verbose`, given before the command (`skillevaluator -v validate ./my-skill`), enables debug logging; for `validate` it also replaces the compact pipeline view with the full detail stream. Every command accepts `-h` as a short form of `--help`.## Direct tier workflows@@ -102,7 +101,7 @@ autopilot. It is separate from these tier-only workflows.| `--llm / --no-llm` | Enabled when a provider is configured | Enable or skip rubric, LLM security, and finding verification; explicit `--llm` requires valid provider configuration || `--min-score INTEGER` | `70` | Minimum passing quality and rubric score, from 0 to 100 || `--profile TEXT` | `SKILLEVALUATOR_PROFILE`, then `external` | Validation policy profile |-| `--previous-version TEXT` | unset | Previous version for version checks |+| `--previous-version TEXT` | `SKILLEVALUATOR_PREVIOUS_VERSION`, else unset | Previous version for version checks |Also accepts `-r/--report` and `-o/--output-dir` described above.@@ -122,7 +121,7 @@ Both thresholds accept values from 0 to 1. Also accepts `-r/--report` and## Expert tier subcommandsThe commands below remain available for focused operations.-`skillevaluator tier1 validate PATH` now runs the same isolated Tier 1 workflow+`skillevaluator tier1 validate PATH` runs the same isolated Tier 1 workflowas `tier1 PATH`. Use top-level `skillevaluator validate PATH` for the combinedpipeline.@@ -172,7 +171,7 @@ skillevaluator validate ./my-skill -a codex,claude-code \--env-mode docker --harbor-keep-jobs # Tier 3, retain Harbor jobs```-Pointing `validate` at a directory of skills (one with no root `SKILL.md`) runs the whole pipeline serially for each skill inside it and finishes with a per-skill summary — catalog validation in one command. By default the terminal shows a compact pipeline view; `--verbose` switches to the full per-check detail stream.+Pointing `validate` at a directory of skills (one with no root `SKILL.md`) runs the whole pipeline for each skill inside it, writes each skill's reports under `<output-dir>/<skill>/`, and finishes with a per-skill summary — catalog validation in one command. Skills run serially by default; `--workers N` validates up to N skills at once in parallel processes. `--previous-version` is rejected for a directory of skills. By default the terminal shows a compact pipeline view; `--verbose` switches to the full per-check detail stream.`--help` groups the flags into four panels, mirrored by the subsections below.@@ -189,6 +188,9 @@ Applies to the whole run: target typing, policy profile, reports, tier selection| `--profile TEXT` | `$SKILLEVALUATOR_PROFILE`, then `external` | Validation profile: `external` or a custom name. || `--external` | — | Shortcut for `--profile external` (validate for public publication). || `--policy FILE` | none | Custom policy YAML overlaid on top of `--profile`. |+| `--workers INTEGER RANGE` | `1` | Concurrent skill jobs when validating a folder of skills (`x>=1`). Values above 1 run skills in parallel processes and disable the per-skill pipeline view. |++The [evaluated source provenance](#evaluated-source-provenance) flags also appear in this panel.Auto-detection recognizes: a `SKILL.md` for skills, `.mdc` files for rules, `workflow-rules.mdc` for workflows, and — for plugins — a bundle-reference `agent_plugin.yaml`/`.yml` manifest or a contained `.claude-plugin/plugin.json` manifest. Plugins are validated against their public contract; quality, lint, and version checks are skill-only and skipped for plugins.@@ -201,6 +203,7 @@ Static checks; LLM-free by default. Tier 1 gates the exit code and always runs.| Flag | Default | Effect || --- | --- | --- || `--checks, --tier1-checks TEXT` | all applicable | Comma-separated subset of Tier 1 checks. Default choices: `schema`, `version`, `security`, `pii`, `license`, `code-integrity`, `unicode`, `quality`, `lint`; opt-in (not run by default): `dependency`. `quality`/`lint`/`version` are skill-only and skipped for rules and workflows. |+| `--previous-version VERSION` | `SKILLEVALUATOR_PREVIOUS_VERSION`, else unset | Previous released version; the skill's version must be a strictly greater SemVer. Single-skill targets only. || `--fail-fast` | off | Stop on the first failing check instead of collecting all issues. || `-c, --continue-on-failure` | off | Run the full pipeline without stopping early; record all issues in the reports. Overrides `--fail-fast`, and for folder validation keeps scanning every skill past a CRITICAL finding. || `--llm, --tier1-llm / --no-llm, --no-tier1-llm` | `no-llm` | Enable LLM-backed security analysis (requires a configured public provider — see [Providers & Credentials](configuration.mdx)). |@@ -242,7 +245,7 @@ The following flags are forwarded to the live-eval engine when Tier 3 is enabled| `--results-dir DIRECTORY` | unset | Directory for Harbor live-eval results. || `--include-skills PATH` | none | Additional skill(s) to mount into the eval environment (repeatable; paths must exist). Requires group workspace mode, set via `skill_workspace.mode: group` in `evals/config.yml`. || `--copy-repo` | off | Copy the surrounding repo into the eval environment. |-| `--timeout-multiplier FLOAT` | unset | Scale Harbor step timeouts. |+| `--timeout-multiplier FLOAT` | unset | Scale Harbor's agent, verifier, and environment-build timeouts (built-in default `1.0`). Must be a finite number greater than 0. || `--harbor-keep-jobs` | off | Retain Harbor job dirs and artifacts after the run for inspection. || `--agent-runtime-preflight / --no-agent-runtime-preflight` | disabled | Opt into an extra agent-only execution of the first staged task before the measured A/B matrix. The CLI value overrides `harbor.agent_runtime_preflight`; without either, no extra agent runs. |@@ -341,7 +344,7 @@ Plus the standard report options.## dedup-scan-Alias of [context-optimization-check](#context-optimization-check) — same behavior, same flags (`--threshold` default `0.8`, `--model`, `--llm-model`, plus the standard report options).+Alias of [context-optimization-check](#context-optimization-check) — same behavior, same flags (`--threshold` default `0.8`, `--model`, `--llm-model`, plus the standard report options). Only the report basename differs: `skillevaluator-dedup` instead of `skillevaluator-context`.```bash title="Same check, alias name"skillevaluator dedup-scan ./my-skill@@ -383,8 +386,8 @@ Use `skillevaluator tier3 PATH` for the complete workflow with automatic missing| Flag | Default | Effect || --- | --- | --- || `-a, --agents TEXT` | provider-native | Comma-separated Harbor agents. Defaults: NVIDIA Build=`opencode`, OpenAI=`codex`, Anthropic=`claude-code`. Supported: `claude-code`, `codex`, `opencode`; the alias `claude` is accepted for `claude-code`. See [Agents & Sandboxes](agents-and-sandboxes.mdx). |-| `--env-mode` | `docker` | Where trials run. SkillEvaluator exposes 24 environment modes: 23 Harbor-native backends, including Docker, plus SkillEvaluator `local` mode. See the complete extras and prerequisites matrix in [Agents & Sandboxes](agents-and-sandboxes.mdx). |-| `--environment-kwarg`, `--ek` | none | Operator-only `KEY=VALUE` constructor option for Harbor-native non-Docker backends (repeatable). Values accept JSON-compatible types; skill-owned `evals/config.yml` cannot set them. Never pass secrets. Docker and local reject all kwargs, and Harbor runtime-policy fields for overrides, mounts, networks, extra Compose, lifecycle, and pod security are reserved. |+| `--env-mode` | `docker` | Where trials run. SkillEvaluator exposes 24 environment modes: 23 Harbor-native modes — `docker`, `daytona`, `e2b`, `modal`, `runloop`, `langsmith`, `ec2`, `gke`, `ack`, `openshift`, `novita`, `apple-container`, `singularity`, `islo`, `tensorlake`, `cwsandbox`, `wandb`, `use-computer`, `blaxel`, `beam`, `skypilot`, `hyperbrowser`, `vercel` — plus SkillEvaluator `local` mode. `wandb` runs Harbor's `cwsandbox` backend with W&B authentication, so it needs the `harbor[cwsandbox]==0.24.0` extra; Harbor 0.24 has no `wandb` extra. Harbor's `cua-cloud`, `opensandbox`, `hf-sandbox`, `podman`, `kata`, `runta`, `prime`, `mosaic`, and `smol` backends are not selectable. See the complete extras and prerequisites matrix in [Agents & Sandboxes](agents-and-sandboxes.mdx#where-trials-run). |+| `--environment-kwarg`, `--ek` | none | Operator-only `KEY=VALUE` constructor option for Harbor-native non-Docker backends (repeatable). Values accept JSON-compatible types; skill-owned `evals/config.yml` cannot set them. Never pass secrets. Docker and local reject all kwargs. Harbor runtime-policy fields for overrides, mounts, networks, extra Compose, lifecycle, and pod security are reserved, as are `stream`, `enable_environment_dir_upload`, and `auth` for `cwsandbox`/`wandb`. Names the selected backend does not accept in Harbor 0.24.0 are rejected. See [Native backend constructor options](agents-and-sandboxes.mdx#native-backend-constructor-options). || `--autopilot` | off | Create one eval case when no dataset/task source exists, then evaluate. The case is LLM-generated with the configured provider, with a deterministic keyless template fallback; an existing source is never overwritten. || `--skip-baseline` | off | Skip the without-skill baseline (no lift analysis, faster). || `--n-attempts INTEGER` | unset | Attempts per eval case (pass@k). |@@ -402,7 +405,7 @@ Use `skillevaluator tier3 PATH` for the complete workflow with automatic missing| `--results-dir DIRECTORY` | unset | Directory for live-eval results (precedence rules in [Reports & Results](reports.mdx)). || `--harbor-keep-jobs` | off | Retain Harbor job dirs and artifacts for inspection with [harbor-view](#harbor-view). || `--agent-runtime-preflight / --no-agent-runtime-preflight` | disabled | Opt into an extra agent-only execution of the first staged task before the full A/B evaluation matrix. |-| `--timeout-multiplier FLOAT` | unset | Scale Harbor step timeouts. |+| `--timeout-multiplier FLOAT` | unset | Scale Harbor's agent, verifier, and environment-build timeouts (built-in default `1.0`). Must be a finite number greater than 0. || `--override-cpus INTEGER` | unset | Override eval-environment CPU allocation (passed through to Harbor). || `--override-memory-mb INTEGER` | unset | Override eval-environment memory in MB (passed through to Harbor). || `--override-storage-mb INTEGER` | unset | Override eval-environment storage in MB (passed through to Harbor). |@@ -428,7 +431,7 @@ skillevaluator create-eval-dataset ./my-skill --full```<Warning>-`--force` overwrites an existing `evals/evals.json`, including any hand-edited cases. Use `--dry-run` first if you are unsure.+`--force` overwrites an existing `evals/evals.json`, including any hand-edited cases, and `--refine` implies `--force`. Use `--dry-run` first if you are unsure.</Warning>| Flag | Default | Effect |@@ -438,7 +441,7 @@ skillevaluator create-eval-dataset ./my-skill --full| `--dry-run` | off | Preview without writing. || `--force` | off | Overwrite existing `evals/evals.json`. || `--prompt PATH` | none | Developer eval-guidance file for generation (default: auto-detects `evals/EVAL.md`). |-| `--refine` | off | Refine cases using existing or collected trajectories. |+| `--refine` | off | Refine cases using existing or collected trajectories. Reading trajectories from existing results does not require the `tier3` extra; when none exist, collecting one runs `claude-code` through Harbor and needs the `tier3` extra, Docker, and a configured provider (see [Eval Datasets](eval-datasets.mdx)). || `--from-results PATH` | none | With `--refine`: results path containing agent trajectories (default: latest results). || `--results-dir DIRECTORY` | unset | With `--refine`: external results root to search for trajectories. |diff --git a/docs/configuration.mdx b/docs/configuration.mdxindex db764ad2..631aaa7b 100644--- a/docs/configuration.mdx+++ b/docs/configuration.mdx@@ -27,8 +27,9 @@ No model override is needed. Defaults are pinned in the installed release,not automatically selected as the latest available model. Set`SKILL_EVAL_LLM_MODEL` only when you want a different evaluator model; thestandard judge inherits it unless a judge override is set. Embeddings use their-own model, and Tier 3 agent selection is separate. NVIDIA Build's Codex and-Claude Code agents use a Super default even when the evaluator uses Nano.+own model, and Tier 3 agent selection is separate. With NVIDIA Build, OpenCode+follows `SKILL_EVAL_LLM_MODEL`, while Codex and Claude Code keep the Super+default unless you override their models.Anthropic and Bedrock need a separate embedding provider for Tier 2; customOpenAI-compatible endpoints require an explicit URL and key, with optional model overrides.@@ -233,7 +234,7 @@ Live evaluation has two credential roles. They can share a key when the selected| Evaluator key | SkillEvaluator itself — dataset generation and verifier-side judging | The provider setup above (`SKILL_EVAL_LLM_PROVIDER` + its key) || Agent credential | The live agent under test (Codex, Claude Code, ...) inside its task environment | Your host environment only — export the agent's native variables (for example `OPENAI_API_KEY` and `OPENAI_BASE_URL` for Codex) before the run |-Both roles resolve from the operator's environment; a skill's `evals/config.yml` can never supply, rename, or redirect a credential. What each agent needs depends on the provider and `--env-mode`:+Both roles resolve from the operator's environment; a skill's `evals/config.yml` can never supply, rename, or redirect a credential. Export them in the shell or CI job that runs SkillEvaluator: Harbor starts in an empty, evaluator-owned working directory with `.env` loading disabled, so a `.env` or `.env.local` file near your project is not read. What each agent needs depends on the provider and `--env-mode`:- **NVIDIA Build, one key** (`--env-mode docker` or `local`): a single `NVIDIA_API_KEY` powers the evaluator and all three supported agents — `opencode` calls NVIDIA Build directly, while `claude-code` and `codex` route through SkillEvaluator's compatibility bridge (in-container in Docker mode; an authenticated in-process host bridge in `local` mode). Agent models must be full NVIDIA Build catalog IDs in `publisher/model` form.- **NVIDIA Build on Harbor cloud modes** (`daytona`, `e2b`, ...): the bridge is not available. `claude-code` needs its own native `ANTHROPIC_API_KEY`, `codex` needs a full OpenAI Responses API credential (`OPENAI_API_KEY` plus `OPENAI_BASE_URL`), and each needs an explicit model via `--agent-model` or `harbor.agents.<agent>.model`.@@ -302,8 +303,9 @@ KEY=VALUE` flags on `tier3 evaluate`, `doctor`, and `health-check`.Skill-owned `evals/config.yml` cannot set them. `docker` and `local` reject anyenvironment kwargs. Harbor runtime-policy fields for resource overrides,mounts, networks, extra Compose definitions, SkillEvaluator's own lifecycle-fields (`delete`, `force_build`, `keep_containers`), and Kubernetes pod security-are reserved. Other backend-specific timeout and retention knobs remain+fields (`delete`, `force_build`, `keep_containers`), Harbor's `stream` and+`enable_environment_dir_upload` controls, and Kubernetes pod security are+reserved. Other backend-specific timeout and retention knobs remainoperator-owned. Never pass secrets there; credentials remain host environmentvariables.diff --git a/docs/custom-graders.mdx b/docs/custom-graders.mdxindex d9d1e404..86c6654d 100644--- a/docs/custom-graders.mdx+++ b/docs/custom-graders.mdx@@ -133,6 +133,11 @@ non-finite, or outside 0.0–1.0 are omitted consistently from the persistedreward, aggregate, and report; they cannot contribute a score or consume the128-name publication budget.+SkillEvaluator assigns each trial's case ID from the staged Harbor task, not+from an `entry_id` your grader writes. A trial whose staged identity is+missing or ambiguous is kept only as an unscored diagnostic, and that+condition's case coverage fails.+Rather than hard-coding the paths, read them from the `HARBOR_*` environmentvariables and fall back to the defaults below when a variable is unset — thestarter grader already does. Harbor records the interaction history as the@@ -149,6 +154,12 @@ a standardized JSON trajectory:| `HARBOR_TESTS_DIR` | `/tests` | Root of the staged test files || `HARBOR_GRADER` | `/tests/grader.py` | Where your staged grader runs from |+Your grader reads Harbor's `trajectory.json` as written. Unlike the standard+grader, it does not [rebuild a missing or empty trajectory](tier3-live-evaluation.mdx#the-graded-trajectory)+from agent logs, so handle that case explicitly. A step's `message` and an+observation result's `content` can each be a string or a list of content parts+(`text`, `image`, or `audio`), so read both forms.+## Bring your own task (BYOT)BYOG replaces the scoring; BYOT replaces the whole task — instruction,@@ -210,7 +221,10 @@ Two rules keep BYOT predictable:[Eval Datasets](eval-datasets.mdx).- **Your source is never mutated in place.** SkillEvaluator stages with-skilland baseline copies of `evals/harbor/` under the results directory for each- run; the files you author stay untouched.+ run; the files you author stay untouched. In a paired run, the staged copies+ append `-with-skill` or `-without-skill` to each `[task] name` and record a+ missing `entry_id` under `[metadata]`, so Harbor keeps the arms apart;+ results still resolve to your case IDs.The staged task also has explicit compatibility boundaries:@@ -231,11 +245,15 @@ The staged task also has explicit compatibility boundaries:from a replaced `tests/skill_evaluator/` payload so authored helper modulescannot shadow its imports.- Use `custom_only` when the native task must own those surfaces. A shared- multi-step verifier can use a step-local test script or fall back to the- top-level test. A separate step verifier uses- `steps/<step-name>/tests/` as its image context whenever that directory- exists, so that context must contain its own test script; top-level fallback- works only when the step tests directory is absent. When every native+ verifier, or a separate verifier built from `environment/`, uses a step's+ `steps/<step-name>/tests/` script when one exists and otherwise falls back to+ the top-level test. A `Dockerfile` or `docker-compose.yaml` in a step or task+ `tests/` directory, or a `docker_image` in the step's or task's+ `[verifier.environment]`, instead defines a dedicated verifier image: that+ `tests/` directory must contain its own test script, and Harbor uploads no+ tests to it. A task with+ `tests/grader.py` or `tests/grader.sh` cannot use a dedicated verifier image,+ because SkillEvaluator's grader would never reach it. When every native`custom_only` verifier pass already resolves an authored test script and nonative `tests/grader.py` or `tests/grader.sh` selects SkillEvaluator's customrunner, staging preserves an authored `tests/skill_evaluator/` directory.@@ -306,9 +324,10 @@ layout and the machine-readable contract.test script for every verifier pass. Scaffold a grader with`skillevaluator init-custom-grader ./my-skill --mode custom_only`, or fora native Harbor task provide `tests/grader.py`, a top-level `tests/test.sh`,- or step-local test scripts. A separate step verifier whose own `tests/`- directory exists must put the script in that directory because Harbor uses- it as the verifier image context.+ or step-local test scripts. A dedicated verifier image, defined by a+ `Dockerfile` or `docker-compose.yaml` in a step or task `tests/` directory+ or by a verifier `docker_image`, must have its own test script in that+ directory because Harbor uploads no tests to it.</Accordion><Accordion title="custom_only requires a numeric overall score">In `custom_only` mode your grader must write a numeric `overall` betweendiff --git a/docs/developer-guide.mdx b/docs/developer-guide.mdxindex 4e386fec..f496a3f2 100644--- a/docs/developer-guide.mdx+++ b/docs/developer-guide.mdx@@ -136,16 +136,19 @@ Every commit needs a Developer Certificate of Origin sign-off — commit with `g| [CHANGELOG.md](https://github.com/NVIDIA/SkillEvaluator/blob/main/CHANGELOG.md) | User-visible changes per release || [CITATION.cff](https://github.com/NVIDIA/SkillEvaluator/blob/main/CITATION.cff) | How to cite SkillEvaluator |-If you use SkillEvaluator in academic or technical work, please cite it:+If you use SkillEvaluator in academic or technical work, cite the methodology+paper listed under `preferred-citation` in `CITATION.cff`, and record the+SkillEvaluator version you used for reproducibility:-```yaml title="CITATION.cff"+```yaml title="CITATION.cff (excerpt)"cff-version: 1.2.0-message: "If you use SkillEvaluator in academic or technical work, please cite this software."title: "SkillEvaluator"type: softwareversion: "0.5.0"-authors:- - name: "NVIDIA Corporation"-license: Apache-2.0-repository-code: "https://github.com/NVIDIA/SkillEvaluator"+preferred-citation:+ type: article+ title: "Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills"+ year: 2026+ doi: "10.48550/arXiv.2608.20614"+ url: "https://arxiv.org/abs/2608.20614"```diff --git a/docs/environment-variables.mdx b/docs/environment-variables.mdxindex 0ef64bf7..1bf49116 100644--- a/docs/environment-variables.mdx+++ b/docs/environment-variables.mdx@@ -6,7 +6,7 @@ layout: referenceEvery environment variable SkillEvaluator reads, grouped by feature area. Each group links to the guide that explains the workflow the variables belong to — this page is the inventory, not the tutorial.-Boolean variables accept `1`, `true`, `yes`, or `on` (case-insensitive) as true, and `0`, `false`, `no`, or `off` as false.+Boolean variables accept `1`, `true`, `yes`, or `on` (case-insensitive) as true, and `0`, `false`, `no`, or `off` as false. Any other value that is set, including an empty one, also counts as false.Two historical SkillEvaluator prefixes coexist. `SKILL_EVAL_*` covers providerand model configuration plus a small set of Tier 3 engine knobs;@@ -29,10 +29,6 @@ These variables select and configure the provider used for LLM-backed checks and| `SKILL_EVAL_LLM_MODEL` | OpenAI: `gpt-5.6-sol`; Anthropic: `claude-opus-5`; NVIDIA Build: `nvidia/nemotron-3-super-120b-a12b`; OpenAI-compatible: `nvidia/nvidia/nemotron-3-super-120b-long-ctx`; Bedrock: `us.anthropic.claude-opus-5` | Chat model override. For gateways, use the exact catalog ID when it differs from the default. Setting it to an empty string is a configuration error. For a lower-cost OpenAI option, use `gpt-5.4-mini`. || `SKILL_EVAL_LLM_BASE_URL` | provider default endpoint | Endpoint override; takes precedence over `OPENAI_BASE_URL` and `ANTHROPIC_BASE_URL`. Required for `openai-compatible`. Ignored by `nv_build`, whose endpoint is fixed to `https://integrate.api.nvidia.com/v1` — point a custom endpoint at the `openai-compatible` provider instead. || `SKILL_EVAL_LLM_API_KEY` | — | API key for the `openai-compatible` provider (local servers still require it to be set). |-| `SKILL_EVAL_LLM_MAX_RETRIES` | `3` | Extra attempts for transient LLM judge failures after the initial request. `0` disables retries; invalid or negative values use the default. |-| `SKILL_EVAL_LLM_RETRY_BASE_DELAY` | `1` second | Base for exponential jitter between direct LLM judge attempts. Values must be finite and nonnegative. |-| `SKILL_EVAL_LLM_RETRY_MAX_DELAY` | `30` seconds | Maximum sleep between direct LLM judge attempts. A `Retry-After` value above this limit fails fast; values must be finite and nonnegative. |-| `SKILL_EVAL_LLM_JUDGE_BUDGET_SEC` | `180` seconds | Wall-time budget per required LLM judge in the Harbor verifier. Values must be finite and positive; invalid or nonpositive values use the default. || `SKILL_EVAL_MODEL_CATALOG_ALLOW_HTTP_HOSTS` | unset | Comma-separated hosts whose model catalog may be read over plain HTTP. Catalog reads otherwise require HTTPS unless the host is loopback, because the request carries a bearer token. Use this only when the transport is already encrypted and authenticated below HTTP — a WireGuard or comparable tunnel peer, for example — where the tool cannot see that the link is protected. Each entry matches one whole host as written, with no resolution, no suffix matching, and no wildcards. A plain-HTTP request to an accepted host also bypasses any inherited HTTP proxy, so the bearer token is never offered to an intermediary. The transport rechecks authorization before dispatch and rejects hosts that are no longer allowed. HTTPS routing is unchanged. || `NVIDIA_API_KEY` | — | Credential for the `nv_build` provider (NVIDIA Build). || `OPENAI_API_KEY` | — | Credential for the `openai` provider. |@@ -42,11 +38,47 @@ These variables select and configure the provider used for LLM-backed checks and| `AWS_REGION` | `us-west-2` | Region for the `bedrock` provider. || `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`, `AWS_PROFILE`, ... | — | Standard AWS credential chain, used as-is by the `bedrock` provider. |-The retry settings apply to the shared host LLM client and direct public-provider-calls in the Harbor verifier. The canonical OpenAI RAGAS goal scorer uses its-own adapter and scoring path. Harbor's managed verifier limits each required-judge to `SKILL_EVAL_LLM_JUDGE_BUDGET_SEC` (180 seconds by default) so retries-leave time to write a failed-run report.+### LLM retries and judge time budget++SkillEvaluator retries transient LLM failures itself. The retry settings apply+on the host to every chat call made through the shared LLM client, including+Tier 1 `--llm` checks, Tier 2 LLM analysis, eval dataset generation, and Tier 3+report suggestions. They also apply in the Harbor verifier to the direct+provider calls made by the `accuracy`, `goal_accuracy`, and `behavior_check`+judges. Embedding requests and the canonical OpenAI RAGAS goal scorer use their+own client and scoring path, so these settings do not apply to them.++| Variable | Default | Effect |+| --- | --- | --- |+| `SKILL_EVAL_LLM_MAX_RETRIES` | `3` | Number of retries after the initial request. `0` disables retries. Non-integer or negative values use the default. |+| `SKILL_EVAL_LLM_RETRY_BASE_DELAY` | `1` second | Base for full-jitter exponential backoff. The wait before retry *n* is a random value from 0 up to `base × 2^(n-1)` seconds, capped at the maximum delay. Non-numeric, non-finite, or negative values use the default. |+| `SKILL_EVAL_LLM_RETRY_MAX_DELAY` | `30` seconds | Longest single wait between attempts. A value below the base delay is raised to the base delay. Non-numeric, non-finite, or negative values use the default. |+| `SKILL_EVAL_LLM_JUDGE_BUDGET_SEC` | `180` seconds | Wall-time budget for each required LLM judge in the Harbor verifier, covering all of its attempts and waits. Non-numeric, non-finite, zero, or negative values use the default. |++Retries cover HTTP 429, 500, 502, 503, and 504 responses, connection errors,+and timeouts. In the verifier, Bedrock throttling and service-unavailable+errors are also retried. Other HTTP errors, such as 400, 401, 403, or 404, are+not retried. When a retryable response carries a `Retry-After` header,+either in seconds or as an HTTP date, the next attempt waits that long plus 0.1+to 0.5 seconds of jitter instead of the computed backoff. An unparseable value+waits the base delay. If `Retry-After` asks for longer than+`SKILL_EVAL_LLM_RETRY_MAX_DELAY`, the call fails immediately with the provider's+error.++In the verifier, `SKILL_EVAL_LLM_JUDGE_BUDGET_SEC` also bounds retries. Each+request's timeout is cut to the judge's remaining time, and a retry whose wait+would pass the deadline ends the judge with a timeout. The RAGAS goal scorer+runs under the same budget. A judge that runs out of time is an infrastructure+failure, not a `0.0` score (see+[Tier 3 judge failure semantics](#tier-3-judge-failure-semantics)). The three+required judges run one after another inside the generated task's 600-second+verifier timeout, so the default leaves time to write a failed-run report.++Set these variables in the host environment that runs `skillevaluator`. When+one is set, Tier 3 stages it into the generated Harbor task's `[verifier.env]`,+so the verifier uses the same value. They are never added to the agent's+environment, and `evals/config.yml` `runtime_env` cannot set `SKILL_EVAL_*`+names.### Anthropic endpoint roots@@ -112,15 +144,65 @@ execution environments are covered in| Variable | Default | Effect || --- | --- | --- || `SKILLEVALUATOR_RESULTS_DIR` | `<skill>/evals/results` | External root for run results. Precedence for writes: the `--results-dir` flag, then this variable, then the legacy in-skill location. Read commands (`view`, `compare`) honor the same order and also fall back to the legacy location so older runs stay visible. |-| `SKILLEVALUATOR_LOCAL_SANDBOX` | `require` | Local-mode sandbox policy. `require` fails closed when no OS sandbox backend (Bubblewrap on Linux, Seatbelt on macOS) is usable; `prefer` degrades to advisory-only guardrails with a loud warning; `off` skips sandbox probing entirely — for skills you fully trust. No value enables native Windows: local mode fails closed there before anything runs — use WSL2 or `--env-mode docker`. |+| `SKILLEVALUATOR_OUTPUT_PROVENANCE_KEY_FILE` | macOS: `~/Library/Application Support/skillevaluator/output-provenance.key`; Linux: `$XDG_STATE_HOME/skillevaluator/output-provenance.key` (`~/.local/state` when `XDG_STATE_HOME` is unset); Windows: `%LOCALAPPDATA%\skillevaluator\output-provenance.key` | Local key that authenticates the markers SkillEvaluator writes into its generated output directories; a directory is replaced only when its marker verifies for that path. The key is created on first use. It must be outside every evaluated skill and generated output tree. On macOS and Linux, the file and its directory must be owned by the current user and not accessible to group or other users. |+| `SKILLEVALUATOR_LOCAL_SANDBOX` | `require` | Local-mode sandbox policy. `require` fails closed when no OS sandbox backend (Bubblewrap on Linux, Seatbelt on macOS) is usable; `prefer` degrades to advisory-only guardrails with a loud warning; `off` skips sandbox probing entirely — for skills you fully trust. Any other value is rejected. No value enables native Windows: local mode fails closed there before anything runs — use WSL2 or `--env-mode docker`. || `SKILLEVALUATOR_LOCAL_ALLOW_NET` | `true` | Network egress for local-mode trials. Set to `0` to airgap a skill that must not reach the network. Incompatible with the `nv_build` provider — NVIDIA Build local agents require network access, so airgapped runs are rejected up front. || `SKILLEVALUATOR_LOCAL_STRICT_READS` | `false` | Tightens the sandbox's read-only view of the host system to a stricter path set. || `SKILLEVALUATOR_LOCAL_INHERIT_AGENT_KEYS` | `false` | When true, local-mode trials inherit the host's agent credential variables into their ambient environment. Off by default so a hostile skill command cannot read credentials — agents and verifiers receive credentials per-exec instead. || `SKILLEVALUATOR_RUNTIME_DIR` | `~/.local/share/skillevaluator/runtimes` | Where local mode looks for managed agent CLIs. Must be a dedicated subdirectory — not the home directory or one of its parents. || `SKILL_EVAL_HARBOR_PREBUILD_TASK_ENVS` | unset | Set to exactly `1` to opt in to pre-building Harbor task environment images before trials start (other truthy spellings are not honored). Opt-in because concurrent trial cleanup can remove a shared prebuilt image that sibling trials still need. || `SKILL_EVAL_JUDGE_MODEL` | falls back to `SKILL_EVAL_LLM_MODEL` | Operator-controlled model override for the Tier 3 dimension judge in standard grading. `LLM_JUDGE_MODEL` is honored as a higher-precedence spelling; when neither is set, the judge uses `SKILL_EVAL_LLM_MODEL`, then the built-in judge default. Both spellings change only the model used with the configured `SKILL_EVAL_LLM_PROVIDER`; they do not select a separate judge provider, so the model must be supported by that provider. Standard grading makes both spellings resolve to the selected override in Harbor's parent process, then forwards them through its verifier-only job layer, which takes precedence over task and step verifier declarations. Neither name is injected into the evaluated agent's initial environment; attempts to set or reference one from skill-authored `runtime_env` or native `[environment.env]` tables are rejected. Native custom-only graders keep ownership of their verifier configuration. Shared agent/verifier containers are a process boundary, not a confidentiality boundary; use a separate verifier environment when the model name itself must be hidden from untrusted agent processes. `run_config.json` records the configured judge provider, model, source, `override_applied`, and catalog-verification status separately from agent models; an allowed provider fallback may use a different model for an individual judge call. When `override_applied` is false for a native Harbor task, `effective_model_source` is `native_harbor_runtime` and catalog verification is explicitly inconclusive because an authored task or step verifier declaration may supersede the recorded fallback. |+| `LLM_JUDGE_FALLBACK_MODELS` | unset | Comma-separated fallback models for the Tier 3 standard judges in the Harbor verifier. When the provider rejects the judge model as not found, invalid, or not accessible to the key, the verifier tries each fallback in order with the same provider. Fallback applies to the `openai`, `nv_build`, and `openai-compatible` providers. The value is forwarded only to standard verifier jobs, never to the agent environment or `custom_only` runs. || `SKILLEVALUATOR_HARBOR_DIAGNOSTIC_ARTIFACT_MAX_BYTES` | `5242880` (5 MiB) | Size cap per diagnostic artifact copied from a Harbor trial into the results directory. Values are bounded to 0–67108864 bytes (64 MiB); `0` skips non-empty artifacts. Oversized artifacts are skipped and recorded in the artifact manifest. |+### Harbor process environment++SkillEvaluator starts every Harbor process in an empty, evaluator-owned working+directory with `.env` loading and Harbor telemetry disabled:++- Harbor does not read `.env` or `.env.local` files from your project or+ current directory. Export variables in the shell or CI job that runs+ `skillevaluator` instead.+- An evaluation run passes Harbor only an allowlisted environment: basic+ process, locale, and temporary-directory variables such as `PATH`, `HOME`,+ `LANG`, and `TMPDIR`; proxy and CA-bundle variables; the selected backend's+ own variables (see [Agents & Sandboxes](agents-and-sandboxes.mdx)); the+ evaluator provider's settings and credential, plus the `AWS_*` chain for+ `bedrock`; each selected agent's credentials; validated `runtime_env`+ values; and the Tier 3 judge, retry, and evidence-budget variables on this+ page. Other variables in your shell do not reach Harbor.+- Relative paths in allowlisted path variables, such as `KUBECONFIG`,+ `DOCKER_CONFIG`, `SSL_CERT_FILE`, or `AWS_SHARED_CREDENTIALS_FILE`, resolve+ against the directory where you ran `skillevaluator`.+- Setting `HARBOR_TELEMETRY` or `PYTHON_DOTENV_DISABLED` yourself has no+ effect; SkillEvaluator always overrides both.+- Tier 3 needs python-dotenv 1.2.0 or newer, which the `tier3` extra installs.+ With an older version installed, the prerequisite check fails before Harbor+ starts.++### Judge evidence budgets++Before the Harbor verifier calls the `accuracy`, `goal_accuracy`, and+`behavior_check` judges, it condenses the agent's trajectory into a separate+evidence excerpt for each judge. These budgets cap each excerpt in characters.+When a run produces more evidence than fits, lower-priority material is+dropped or truncated first so the judge still sees the agent's final response.+Larger budgets give judges more context and cost more prompt tokens.++| Variable | Default | Effect |+| --- | --- | --- |+| `SKILL_EVAL_ACCURACY_BUDGET` | `8000` characters | Evidence budget for the `accuracy` judge: the agent's final response, files it wrote, and up to 6 recent tool results. |+| `SKILL_EVAL_GOAL_ACCURACY_BUDGET` | `12000` characters | Evidence budget for the `goal_accuracy` judge, including the RAGAS goal scorer: the final response, end-state file changes, and up to 12 recent tool results. |+| `SKILL_EVAL_BEHAVIOR_CHECK_BUDGET` | `8000` characters | Evidence budget for the `behavior_check` judge: file changes, the final response, the user request, and a compact tool history. The effective budget is never less than `SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT` plus 4000. |+| `SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT` | `800` characters | Maximum length of the agent's final response within the `behavior_check` evidence. Values above 4000 can also raise the effective behavior-check budget. |++Each value must be an integer. Values below `1` are treated as `1`, and+non-integer values such as `8k` or `1.5` use the default. Set these variables in+the host environment that runs `skillevaluator`. As with the+[retry settings](#llm-retries-and-judge-time-budget), Tier 3 stages any that+are set into the generated task's `[verifier.env]`, and never into the agent's+environment.+### Tier 3 judge failure semanticsIn standard grading, a required judge transport, authentication, timeout, HTTP, or exhausted response-parsing error is an evaluation infrastructure failure, not a numeric `0.0`. A condition containing a failed or incomplete default trial publishes none of its scores, dimensions, pass@k, or quality findings, and no cross-condition lift is published. Redacted diagnostics and coverage information remain available. A genuine model score of `0.0` remains valid and is published normally.@@ -144,7 +226,7 @@ The `HARBOR_*` variables are read by your custom grader *inside* the task contai## Set at runtime, not committed<Warning>-Keep credentials out of skill repositories and CI configuration files. Export provider keys as masked secrets at runtime. Agent credentials for Tier 3 come only from the host environment — `evals/config.yml` `runtime_env` carries non-credential values, and any entry that names or references an operator-owned credential (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `NVIDIA_API_KEY`, their base-URL variables, `CLAUDE_CODE_USE_BEDROCK`, and the `AWS_*` credential variables) is rejected at run time, as are names that control the host process (`PATH`, `DOCKER_*`, proxy variables, `SKILLEVALUATOR_*`, and similar). See the blocked-names table in [Providers & Credentials](configuration.mdx#tier-3-credentials-two-roles). With NVIDIA Build, one exported `NVIDIA_API_KEY` covers both the evaluator and the agents in the `docker` and `local` env-modes.+Keep credentials out of skill repositories and CI configuration files. Export provider keys as masked secrets at runtime. Agent credentials for Tier 3 come only from the host environment — `evals/config.yml` `runtime_env` carries non-credential values, and any entry that names or references an operator-owned credential (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `NVIDIA_API_KEY`, their base-URL variables, `CLAUDE_CODE_USE_BEDROCK`, and the `AWS_*` credential variables) is rejected at run time, as are names that control the host process (`PATH`, `DOCKER_*`, proxy variables, `SKILL_EVAL_*`, `SKILLEVALUATOR_*`, and similar). See the blocked-names table in [Providers & Credentials](configuration.mdx#tier-3-credentials-two-roles). With NVIDIA Build, one exported `NVIDIA_API_KEY` covers both the evaluator and the agents in the `docker` and `local` env-modes.</Warning>## See alsodiff --git a/docs/eval-datasets.mdx b/docs/eval-datasets.mdxindex 7bfdff38..a7b6d7f7 100644--- a/docs/eval-datasets.mdx+++ b/docs/eval-datasets.mdx@@ -200,8 +200,12 @@ skillevaluator create-eval-dataset ./my-skill --full --refine```Collecting a fresh trajectory needs Docker, the `tier3` extra, and a configured-provider. With `--no-llm`, refinement still works — it applies template-based-updates from the trajectory instead of an LLM rewrite.+provider. Refining from existing results does not need the `tier3` extra: case+IDs are read from each trial's `reward.json` or Harbor `result.json`, and+multimodal trajectory messages are read as text, with image and audio parts+shown as `[image]` and `[audio]` markers. With `--no-llm`, refinement still+works — it applies template-based updates from the trajectory instead of an LLM+rewrite.## Run policy: evals/config.ymldiff --git a/docs/installation.mdx b/docs/installation.mdxindex bae5b5f0..9dfaf572 100644--- a/docs/installation.mdx+++ b/docs/installation.mdx@@ -137,13 +137,13 @@ and environment backends.| --- | --- || `llm` | Shared LLM and embedding clients used by LLM-backed checks, Tier 2, and Tier 3 || `tier2` | Intra-skill deduplication and local-catalog inter-skill similarity; includes `llm` |-| `tier3` | Docker and cloud live-agent evaluation through Harbor; includes `llm` |+| `tier3` | Docker, local, and cloud live-agent evaluation through Harbor (pins `harbor==0.24.0` and requires `python-dotenv>=1.2.0`); includes `llm` || `security` | Bandit and pip-audit || `all` | `tier2`, `tier3`, and `security` in one bundle || `dev` | Build, test, coverage, and formatting tools for contributors |<Tip>-`[all]` installs every SkillEvaluator extra. External scanner tools, Docker, and agent CLIs stay separate no matter which extra you pick.+`[all]` installs every SkillEvaluator extra. External scanner tools, Docker, agent CLIs, and the Harbor extras for managed sandbox backends (such as `harbor[modal]==0.24.0`) stay separate no matter which extra you pick — see [Agents & Sandboxes](agents-and-sandboxes.mdx#where-trials-run).</Tip>## System toolsdiff --git a/docs/reports.mdx b/docs/reports.mdxindex 3b03aa89..0a353539 100644--- a/docs/reports.mdx+++ b/docs/reports.mdx@@ -214,7 +214,13 @@ completed run directly:(a `--skip-baseline` run has neither).- **`<agent>/{with-skill,without-skill}/`** — each arm's `summary.json`(scores, dimensions, pass@k, trial counts) and a `trials/` directory of- per-trial artifacts.+ per-trial artifacts: the trial's `reward.json`, Harbor's `trajectory.json`+ when available, and redacted, size-capped diagnostic copies such as+ `trial.log`, `exception.txt`, and agent logs (`codex.txt`, `opencode.txt`).+ With standard grading, `reward.json` records `trajectory_source` (the+ trajectory or agent log that was graded) and, under each LLM judge's+ `details`, an `omitted` record showing whether its evidence was truncated to+ fit the [evidence budget](tier3-live-evaluation.mdx#evidence-budgets).<Note>Read commands (`view`, `compare`, `create-eval-dataset --refine`) honor the@@ -305,6 +311,12 @@ baseline-only passes, both-pass, and neither-pass — plus a two-sided exactMcNemar diagnostic over the discordant pairs. Partial or unidentified pairingis labeled and does not receive that exact-test result.+Each logical attempt counts once: a multi-step native task's step rewards+combine into one attempt score for averages and pass@k. When one condition+mixes standard-metric and custom-only rewards, its overall score and lift use+that logical-attempt overall rather than the mean of the standard dimensions,+which would cover only the standard rows.+Case identifiers used as JSON keys must be printable, free of credentialmaterial, and no larger than 512 UTF-8 bytes. A reward with an unsafe identityis retained as a failed diagnostic but is not scored. For large evaluations,diff --git a/docs/tier1-validation.mdx b/docs/tier1-validation.mdxindex a7bd6827..bfccc50b 100644--- a/docs/tier1-validation.mdx+++ b/docs/tier1-validation.mdx@@ -151,6 +151,7 @@ For `validate`, the default set is `schema`, `version`, `security`, `pii`, `liceValidates `SKILL.md` frontmatter and repository placement:- YAML frontmatter must parse and satisfy the skill schema.+ - Frontmatter keys, including keys nested under `metadata`, must be strings. An unquoted key that YAML reads as a number, boolean, null, or date (`123`, `yes`, `null`, `2026-01-01`) is a HIGH finding that names the key in YAML form, such as `Field 'metadata.true': Keys should be strings`; quote the key (`"yes":`) to keep it. Rules, workflows, and plugin manifests report non-string keys as schema errors too.- `name` and `description` are required. Names are 1–64 characters; descriptions are 1–1024 characters.- Names use kebab-case, start with a letter, and cannot have trailing or consecutive hyphens. The directory name must match the frontmatter `name`. The reserved words `anthropic` and `claude` must not appear in a skill name.- `alwaysApply` and `globs` are forbidden skill fields. Optional fields include `license`, `compatibility`, `metadata`, and `allowed-tools`.@@ -315,8 +316,7 @@ Bundled Python scanners resolve next to the SkillEvaluator interpreter before `P| --- | --- || `0` | Success — every gate passed || `1` | Validation failed — a check failed or required scanner evidence is incomplete |-| `2` | Configuration error — invalid flags, arguments, or target path |-| `3` | Runtime error — an unexpected internal failure |+| `2` | Usage error — an unknown flag, a value outside an option's type or choices, or a target path that does not exist |Tier 1 checks always gate the exit code inside `validate`. Tier 2 gates when it runs unless `--no-block-on-dedup` is set. Tier 3 is advisory unless `--block-on-agent-eval` is set. The `quality` check passes when the composite score reaches `--min-score` (default `70`). Reports record the effective gate membership for every tier so saved evidence and process status agree.diff --git a/docs/tier2-deduplication.mdx b/docs/tier2-deduplication.mdxindex 13fadd8d..66c63d81 100644--- a/docs/tier2-deduplication.mdx+++ b/docs/tier2-deduplication.mdx@@ -332,7 +332,7 @@ handling terms before sending confidential skill content to a hosted endpoint.| Unsupported extension | Skips the file || Duplicate sections occur in one file | Reports the relevant headings and line ranges || Skill contains an exactly named, single-link regular `AGENTS.md` plus `CLAUDE.md -> AGENTS.md` | Validates and skips the alias, then scans `AGENTS.md` once |- | Skill carries large generated snapshots (`evals`, `.evals`, `results`, `.results`, `versions`, `.versions`) | Excluded directories are pruned during traversal, so their contents never count toward the discovered-path limit — a well-used skill with large evals or results snapshots no longer trips it, as it could in earlier releases |+ | Skill carries large generated snapshots (`evals`, `.evals`, `results`, `.results`, `versions`, `.versions`) | Excluded directories are pruned during traversal, so their contents never count toward the discovered-path limit || A directory is a Windows junction or other reparse point | Rejected during the walk, like symbolic links || Embedding or chat-LLM analysis is incomplete | Reports an error; does not claim a clean result |</Accordion>diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdxindex 434a98dc..0d6390b9 100644--- a/docs/tier3-live-evaluation.mdx+++ b/docs/tier3-live-evaluation.mdx@@ -73,7 +73,9 @@ using Harbor 0.24:with `.env` loading and Harbor telemetry disabled. Harbor ignores `.env.local`and `.env` files around your project, so it sees only SkillEvaluator'sallowlisted environment. Relative host paths such as `KUBECONFIG` keep their- meaning.+ meaning. Disabling `.env` loading needs python-dotenv 1.2.0 or newer, which+ the `tier3` extra requires; with an older release, `doctor` and+ `tier3 evaluate` fail the prerequisite check until you reinstall the extra.- Codex runs with `reasoning_effort=high`, the effort Harbor 0.22 used bydefault, so results stay comparable across Harbor upgrades.- Installed Codex adapters merge existing user and MCP configuration before the@@ -105,18 +107,22 @@ using Harbor 0.24:single-step `step_results: null`, authoritative multi-step results, andmultimodal tool output, which judges and reports see as text with`[image]`/`[audio]` markers.-- Native tasks follow Harbor 0.24's separate-verifier precedence: a verifier- image, or a `Dockerfile` or `docker-compose.yaml` in the step or task `tests/`- directory, defines a dedicated verifier that must ship its own test script.- Otherwise the verifier is built from `environment/` and receives the uploaded- tests. SkillEvaluator validates that Compose model against the verifier's- environment allowlist, and rejects `docker-compose.yml` verifier definitions,- which Harbor ignores.+- Native tasks follow Harbor 0.24's separate-verifier precedence. For a verify+ pass that runs in a separate verifier container, a `docker_image` in the step+ or task `[verifier.environment]`, or a `Dockerfile` or `docker-compose.yaml`+ in the step or task `tests/` directory, defines a dedicated verifier that+ must ship its own test script and never receives SkillEvaluator's grader.+ Otherwise the separate verifier is built from `environment/` and receives the+ uploaded tests. SkillEvaluator validates the verifier's Compose model against+ its environment allowlist, and rejects `docker-compose.yml` verifier+ definitions, which Harbor ignores. Standard grading modes reject separate+ verifiers entirely; see [Custom Graders & Tasks](custom-graders.mdx).- Harbor 0.24 rejects a `reward.json` whose top-level values are not all- numbers. SkillEvaluator's verifier and wrapped custom graders already write- numeric rewards; a native task's own test script must do the same. When- Harbor records such a failure on a step, the failed job names that step's- exception instead of reporting only a trial count.+ finite numbers. SkillEvaluator's verifier and wrapped custom graders already+ write numeric rewards; a native task's own test script must do the same. When+ Harbor's job statistics do not account for every trial, the failure names the+ first trial or step exception Harbor recorded, such as+ `VerifierOutputParseError`, instead of reporting only a trial count.## Start with one command@@ -322,7 +328,8 @@ so it can consume substantial tokens and exercise task actions; grading is theonly part disabled. It is off by default. Harbor then starts the selectedenvironment and runs the agent in both arms; with standard grading, each transcript isjudged against the case's expected output and assertions using the configured-evaluator provider. Finally, SkillEvaluator collects the artifacts and+evaluator provider (see+[How standard grading judges a trial](#how-standard-grading-judges-a-trial)). Finally, SkillEvaluator collects the artifacts andcomputes the dimension scores, pass rates, and Skill Lift. Harbor task and jobdirectories are transient: they are deleted after collection whether the runsucceeds or fails, unless you pass `--harbor-keep-jobs`.@@ -480,7 +487,7 @@ dataset iteration, `--skip-baseline` removes the without-skill arm and usuallycuts the live trial count roughly in half, but that run cannot produce SkillLift. Re-enable the baseline for any result you intend to compare or publish.For multi-attempt runs, `--stop-on-pass` can avoid remaining attempts after a-pass; it is not applicable with `--n-attempts 1`. By contrast,+pass; it is rejected with `--n-attempts 1`. By contrast,`--n-concurrent` and `--max-agents` control parallelism and elapsed time; theydo not reduce the total planned trials or model work.@@ -520,7 +527,7 @@ The high-signal flags, with defaults:| `--grading-mode` | `default` | `default`, `default_plus_custom`, or `custom_only` — see [Custom Graders & Tasks](custom-graders.mdx) || `--results-dir` | `evals/results`, or `SKILLEVALUATOR_RESULTS_DIR` when set | Write results under an external root instead of the skill directory || `--harbor-keep-jobs` | off | Retain Harbor job directories for inspection |-| `--timeout-multiplier` | `1.0` | Scale Harbor step timeouts |+| `--timeout-multiplier` | `1.0` | Scale Harbor's agent, verifier, and environment-build timeouts; must be a finite number greater than 0 whose scaled timeouts stay finite || `--copy-repo` | off | Copy the surrounding repository into the task environment || `--include-skills` | none | Stage additional skill directories; requires `--skill-workspace-mode group` |@@ -532,6 +539,109 @@ While iterating on a dataset, `--skip-baseline` roughly halves the runtime.Re-enable the baseline arm for any run whose Skill Lift you intend to trust.</Tip>+## How standard grading judges a trial++With `default` or `default_plus_custom` grading, SkillEvaluator's verifier+grades each trial after the agent finishes. `security`, `skill_execution`, and+`skill_efficiency` are deterministic trajectory checks. `accuracy`,+`goal_accuracy`, and `behavior_check` are LLM judges that call the configured+evaluator provider. `custom_only` runs none of them.++### The graded trajectory++The verifier grades Harbor's ATIF `trajectory.json`. When that file is missing+or has no steps, it rebuilds a trajectory from the agent's own log in the+trial's agent log directory, trying `claude-code.txt`, `cursor-cli.txt`,+`cline.txt`, `opencode.txt`, and `codex.txt` in that order:++- `opencode.txt` is OpenCode's JSON event stream. Text events become agent+ messages, and completed tool events become tool calls whose output keeps the+ tool's status, exit code, and error text.+- `codex.txt` is the Codex JSON event log. Shell commands, file changes, MCP+ tool calls, `web_search` actions (with the full search, open-page, or+ find-in-page payload), and `collab_tool_call` items become tool calls. Failed+ commands keep their `status=failed` and `exit_code=` evidence, so+ error-recovery checks still see them.++Each trial's reward records the artifact it graded in `trajectory_source`. If+neither `trajectory.json` nor a supported log yields any steps, the reward+records `No trajectory or reconstructible agent log`. Custom graders read+`trajectory.json` directly and do not get this rebuild.++### Evidence budgets++Each LLM judge sees a bounded evidence bundle drawn from the trajectory rather+than the full transcript. Sections are added in priority order until the+judge's character budget runs out, so lower-priority sections are dropped or+clipped first:++| Judge | Evidence, highest priority first | Budget variable |+| --- | --- | --- |+| `accuracy` | Final response, produced files and writes, the six most recent tool results | `SKILL_EVAL_ACCURACY_BUDGET` |+| `goal_accuracy` | Final response, end-state file changes, the twelve most recent tool results | `SKILL_EVAL_GOAL_ACCURACY_BUDGET` |+| `behavior_check` | File changes, final response (capped by `SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT`), the user request, and a compact tool history | `SKILL_EVAL_BEHAVIOR_CHECK_BUDGET` |++When the case's expected output or assertions contain a backtick-quoted+snippet of four or more characters, or an output path under+`/workspace/output`, `/output`, or `/logs/agent`, every bundle starts with a+short deterministic fact list marking each one observed or not observed in the+agent's tool calls.++A judge can score only what its bundle contains. Evidence that was dropped or+clipped cannot count, so on a long trajectory a behavior that appears only in+truncated history can be judged as not observed. In the results tree, each+judge's details in a trial's `reward.json` include an `omitted` record+(`count`, `truncated`, and `reason`); when `truncated` is `true` on cases that+matter, raise that judge's budget. Larger budgets send longer judge prompts and+use more evaluator tokens. The behavior budget never falls below the+final-response limit plus 4,000 characters, so raising the limit leaves room+for the tool history. Set these variables in the host environment;+SkillEvaluator forwards them to the verifier. Defaults and accepted values are+in [Judge evidence budgets](environment-variables.mdx#judge-evidence-budgets).++### Judge requests and retries++- **Structured output.** Each judge requests a fixed JSON schema. OpenAI and+ OpenAI-compatible chat-completions endpoints, including NVIDIA Build, receive+ a strict JSON Schema `response_format`; Anthropic `/v1/messages` receives+ `output_config`. Bedrock judges use prompt-only JSON.+- **Automatic downgrade.** If an endpoint rejects the schema option itself+ with HTTP 400 or 422 (for example, `response_format` is unsupported), the+ judge repeats that request without the schema and logs a warning. It then+ skips the schema for that provider, endpoint, and model for the remaining+ judge calls in that verifier run. Other 400 and 422 errors are not+ downgraded.+- **Reply validation.** Replies are checked against the expected fields either+ way. An empty, unparseable, or invalid reply, including an empty reply from a+ reasoning model that ran out of output tokens, gets one retry with a stricter+ format reminder. `behavior_check` can also salvage complete per-behavior+ results from a truncated reply and counts the missing behaviors as not+ passed.+- **RAGAS goal scorer.** With the `openai` provider on the canonical+ `https://api.openai.com/v1` endpoint, `goal_accuracy` first tries the RAGAS+ `AgentGoalAccuracyWithReference` scorer through its own client, without the+ schema or retry handling above but within the same time budget. If RAGAS is+ unavailable or fails, the prompt judge runs instead.+- **Transient failures.** HTTP 429, 500, 502, 503, and 504 responses,+ connection errors, timeouts, and Bedrock throttling or service-unavailable+ errors are retried transparently with jittered exponential backoff. A+ `Retry-After` header, in seconds or as an HTTP date, sets the wait; a value+ above the maximum retry delay fails immediately instead of stalling.+- **Time budget.** Each judge has its own wall-time budget (180 seconds by+ default) inside the generated task's 600-second verifier timeout, which+ leaves time to write failure artifacts. Requests are cut off at the+ deadline, and a retry whose wait would reach it is not attempted. If you+ raise the per-judge budget, scale the verifier timeout with+ `--timeout-multiplier` so the three judges still fit.++Retry counts, delays, and the per-judge budget come from the host environment+and are forwarded to the verifier; see+[LLM retries and judge time budget](environment-variables.mdx#llm-retries-and-judge-time-budget).+Retries and downgrades do not change scoring formulas. A judge that still fails+is an evaluation infrastructure failure, not a `0.0` score, and the affected+condition publishes no scores; see+[Tier 3 judge failure semantics](environment-variables.mdx#tier-3-judge-failure-semantics).+## Reading resultsIn the terminal, a finished run renders **Results by Evaluator** — each