Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
---
title: Agents & Sandboxes
description: Every agent Tier 3 can drive and every environment it can run them in — Docker tuning, local-mode sandbox policy, workspace modes, and readiness checks.
---
Tier 3 runs real agent CLIs against your eval cases, so two choices shape every
run: which agents attempt the tasks (`--agents`) and where the trials execute
(`--env-mode`). This page is the complete matrix for both, plus the resource,
safety, and workspace controls around them and the readiness checks that tell
you a run will work before you spend compute. The run workflow itself is
[Tier 3: Live Agent Evaluation](/skills/skillevaluator/tier3-live-evaluation).
[Harbor](https://github.com/harbor-framework/harbor) is the open-source agent
evaluation framework that executes each Tier 3 trial and provides its Docker,
local, and cloud environment backends.
## Supported agents
Pass agents as a comma-separated list with `-a/--agents`. Without that option,
SkillEvaluator selects the provider-native default: `opencode` for NVIDIA Build,
`codex` for OpenAI, and `claude-code` for Anthropic. Three agents are supported,
with Hermes available as an explicit experimental option:
| Agent | Notes |
| --- | --- |
| `claude-code` | Anthropic's Claude Code CLI. `claude` is an accepted alias. Harbor installs Claude Code 2.1.292. Claude 5.5 and later models run with `reasoning_effort=medium`; older models keep Claude Code's default effort. |
| `codex` | OpenAI's Codex CLI. The OpenAI-provider default. Harbor installs Codex 0.161.0. Runs with `reasoning_effort=high` so scores and cost stay comparable across Harbor releases. |
| `opencode` | The OpenCode CLI. Harbor installs OpenCode 1.19.5. |
| `hermes` | Experimental. Hermes v2026.9.24, Docker with NVIDIA Build only; no OpenRouter fallback. |
### Experimental Hermes
Hermes uses NVIDIA Build directly, not a protocol bridge. The initial matrix is
`--agents hermes --env-mode docker` with `SKILL_EVAL_LLM_PROVIDER=nv_build` and
`NVIDIA_API_KEY`. Local mode, Kata, cloud backends, and other evaluator providers
are rejected before execution. OpenCode remains the NVIDIA Build default.
The compatibility smoke uses Linux amd64, Docker Engine 29.9.3, Harbor 0.24.0,
and the Hermes/model versions below.
```bash title="Run Hermes with NVIDIA Build"
export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
skillevaluator doctor --agents hermes --env-mode docker
skillevaluator tier3 evaluate ./my-skill --agents hermes --env-mode docker \
--harbor-keep-jobs
```
The Hermes default is `nvidia/nemotron-3.5-super-vl-120b-a12b`. Override it with
`--agent-model hermes=publisher/model` or `harbor.agents.hermes.model` in
`evals/config.yml`; model IDs must be full NVIDIA Build catalog IDs, without an
extra agent-provider prefix. Account access and tool support still depend on
the selected model. The grader model is independent; set
`SKILL_EVAL_LLM_MODEL` to a model your Build account can invoke for grading.
Harbor 0.24.0 supplies session export and ATIF conversion. SkillEvaluator pins
both the installer and checkout to Hermes commit
`f97608f178d1ffeca59860195ab7da295f7c8e5f` (v2026.9.24). The installer needs
GitHub and Python/Node package download access; first installs may take several
minutes. Browser and computer-use dependencies are not installed. Only the
selected NVIDIA credential reaches the agent; an ambient `OPENROUTER_API_KEY`
does not enable fallback.
Both arms use the same cases, inputs, model and grader. Hermes starts without
bundled skills or persistent memories; only evaluator-staged skills are copied
to `/tmp/hermes/skills`. Successful `skill_view` manifest reads and terminal
calls are normalized for the existing activation and grading checks. Failed
reads and support-file reads do not establish activation. Session export is
attempted on errors and timeouts too; a missing/malformed session or a session
without an agent response is an execution failure, not a scored skill defect.
Retained jobs include `trajectory.json` and `hermes.txt`; normal bounded,
redacted diagnostic retention applies. Token counts come from Hermes session
totals (or per-message usage when present); cost, cached tokens and other unreported fields remain
unknown rather than estimated as zero. Tool approval is noninteractive inside
the Docker sandbox, as for the other installed harnesses.
The repeatable smoke dataset is
[`tests/fixtures/skills/lookup-code`](https://github.com/NVIDIA/SkillEvaluator/tree/main/tests/fixtures/skills/lookup-code):
one script-backed skill lookup and one arithmetic non-trigger case. To validate
your own installation, run both arms without early stopping:
```bash title="Hermes compatibility smoke from a source checkout"
SKILL_EVAL_LLM_MODEL=nvidia/nemotron-3.5-super-vl-120b-a12b \
skillevaluator tier3 evaluate tests/fixtures/skills/lookup-code \
--agents hermes --env-mode docker --n-attempts 1 --n-concurrent 1 \
--no-stop-on-pass --harbor-keep-jobs
```
An opt-in integration test checks the four trials, native activation, terminal
conversion, diagnostics and nonzero token usage. It installs the harness in
Docker and makes real API calls (including grading):
```bash
SKILL_EVAL_HERMES_INTEGRATION=1 python -m pytest -m integration \
tests/test_hermes_agent.py -k real_hermes
```
The pinned CLI releases apply wherever Harbor installs the agent: Docker and
the remote environments. [Local mode](#local-mode) runs the CLI already on your
machine instead.
In Docker and Kata, SkillEvaluator reruns a Claude Code or Codex install that
fails, for example on a dropped download or a package mirror error. It makes up
to three attempts, waiting 2 and then 4 seconds, all within Harbor's agent setup
timeout, so an install that keeps failing still fails the trial. A slow failure
is not retried when the attempt took over a third of that timeout, or when
another attempt as long would not finish within it, so the trial reports the
install error rather than a setup timeout. Each retry logs a warning that ends
with the failed command's output. Claude Code on GKE, and Codex behind an
OpenAI-compatible gateway, are retried the same way.
Claude Code runs with background tasks turned off. Every staged task sets
`CLAUDE_CODE_DISABLE_BACKGROUND_TASKS=1` in its environment, in every env mode,
so a shell command finishes in the agent's turn or is stopped at Claude Code's
Bash timeout (2 minutes unless the agent requests a longer one) instead of
moving to the background. A headless Claude Code run waits for its background
commands before it exits, so a command that never finishes would otherwise hold
a trial that has already answered until the agent timeout. Neither
`harbor.runtime_env` nor a native task's `[environment.env]` can turn
background tasks back on. If a skill needs a single command to run
longer, raise Claude Code's default in `evals/config.yml` with
`harbor.runtime_env: {BASH_DEFAULT_TIMEOUT_MS: "240000"}` (milliseconds), and
keep it below the trial's agent timeout: 300 seconds times
`harbor.timeout_multiplier` for generated tasks.
Claude Code also runs with its refusal model fallback turned off. When the
model provider's safety classifier stops the selected model, Claude Code
would otherwise retry the turn on another model from its own fixed table (for
example `claude-opus-4-8` after Claude Opus 5.5) under that bare Anthropic
model ID. A route that does not serve that ID, such as a gateway that serves
only the selected model, rejects the retry with HTTP 403, and Harbor records
an unknown API error. Where the retry succeeds, one trial mixes two models but
is scored as the selected one. Every staged task therefore sets
`CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK=1` in its environment, in every env mode
and for every provider route, including the Anthropic API, Bedrock, Vertex AI
and gateways. With the pinned Claude Code 2.1.292, a refused turn is not
retried on another model, and the refusal ends the attempt with Harbor's
`AgentSafetyRefusalError`. The variable was checked with that release only.
[Local mode](#local-mode) runs the Claude Code already on your machine, so it
keeps the fallback off only if that release honors the variable. Neither
`harbor.runtime_env` nor a native task's `[environment.env]` can turn the
fallback back on, and other agents ignore the variable.
Like any other failed trial, a refused attempt is not scored, so the arm no
longer has complete attempt coverage and its execution fails: after one
refusal in 20 cases with 3 attempts each, the run's `result.json` records
`Scored attempt coverage is 59/60`. `tier3 evaluate` then exits 1, and so
does `validate --block-on-agent-eval`. On a route that serves the fallback
model, such as the Anthropic API, the retry used to finish such an attempt,
and the attempt was scored, so a run that completed before can now fail. See
[Troubleshooting](/skills/skillevaluator/tier3-live-evaluation#troubleshooting) for what to do.
The `claude` alias is canonicalized to `claude-code` everywhere an agent
name can appear — `-a/--agents`, `--agent-model`, and `harbor.agents` in
`evals/config.yml` — so both spellings always name the same agent.
Naming both `claude` and `claude-code` in `--agent-model` or in
`harbor.agents` is rejected as a duplicate; in `-a/--agents` the repeated
agent is simply deduplicated.
Each agent runs on a model, and the evaluator provider's default model is not
automatically valid for every agent. Override the model per agent with
`--agent-model AGENT=MODEL` (repeatable), or globally with `--model`:
```bash title="Per-agent model override"
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker \
--agent-model codex=nvidia/nemotron-3.5-lightning-30b-a3b
```
To make an override persistent, set it in `evals/config.yml` under
`harbor.agents.<name>.model` — see [Eval Datasets](/skills/skillevaluator/eval-datasets).
<Note>
Harbor installs agent CLIs in Docker and remote task environments. Local mode
requires an agent CLI already installed on your machine.
Operator credentials come only from the host environment — a
`harbor.runtime_env` entry in `evals/config.yml` that sets an operator-owned
credential name (`ANTHROPIC_API_KEY`, `ANTHROPIC_BASE_URL`,
`CLAUDE_CODE_USE_BEDROCK`, `NVIDIA_API_KEY`, `OPENAI_API_KEY`,
`OPENAI_BASE_URL`) or references one with `${...}` is rejected at run time.
See the two credential roles in [Providers & Credentials](/skills/skillevaluator/configuration).
</Note>
## One NVIDIA Build key for everything
When the evaluator provider is NVIDIA Build (`nv_build`) and trials run in
`--env-mode docker`, `kata`, or `local`, a single `NVIDIA_API_KEY` powers the
evaluator and all three agents:
- **OpenCode** calls NVIDIA Build directly through its `nvidia` provider
adapter.
- **Codex** runs against a SkillEvaluator-owned loopback compatibility
bridge that translates the OpenAI Responses API into NVIDIA Build Chat
Completions calls.
- **Claude Code** (experimental on this path) runs against an equivalent
Messages-to-Chat-Completions bridge.
Both bridges support multi-turn tool-call continuation, so agents complete
real tool loops against Build models. The Claude Code bridge drops
orchestration and server tools Build cannot execute — `WebFetch`,
`WebSearch`, `Task`, and similar — before forwarding a request.
### Models on the one-key path
With `nv_build` and no explicit model override, every supported agent defaults
to `nvidia/nemotron-3.5-lightning-30b-a3b`. The provider-native agent selection is
OpenCode. Its model is automatically qualified with the `nvidia/` adapter
namespace, so it renders as `nvidia/nvidia/nemotron-3.5-lightning-30b-a3b` in run
output — the doubled prefix is the adapter namespace plus the catalog publisher,
not a bug.
An explicit `--agent-model` value with NVIDIA Build must be a full
`publisher/model` catalog ID; an explicit OpenCode override also carries the
adapter namespace (`nvidia/publisher/model`). Overrides are never rewritten
silently, and an unavailable model fails the run rather than falling back.
### Key protection
The bridged CLIs — Codex and Claude Code — never see the real Build key:
- **Docker mode** passes the key to Harbor's trusted parent process over stdin,
then uses a short-lived, container-only stdin handoff for each main-container
exec. The handoff is removed before the agent command runs, and values never
appear on Docker Compose argv.
- **Local mode** keeps the real key in Harbor's trusted parent process; each
bridged CLI receives a unique per-trial capability token that only the
loopback bridge accepts.
OpenCode is the exception: it calls NVIDIA Build directly, so it necessarily
receives the key itself.
NVIDIA Build local agents require network access. When
`SKILLEVALUATOR_LOCAL_ALLOW_NET` is disabled, the run is rejected with
`NVIDIA Build local agents require network access; unset
SKILLEVALUATOR_LOCAL_ALLOW_NET or set it to 1`.
### Other backends still need native credentials
The bridges exist only in `docker`, `kata`, and `local` mode. On any other Harbor
backend, each agent calls its native provider:
| Agent | Requirement with `nv_build` on other backends |
| --- | --- |
| `claude-code` | Its own `ANTHROPIC_API_KEY` in the host environment, plus an explicit model (`--agent-model claude-code=MODEL`). |
| `codex` | A full OpenAI Responses credential — `OPENAI_API_KEY` + `OPENAI_BASE_URL`, distinct from Build's — plus an explicit model. NVIDIA Build's `/responses` endpoint does not support Codex's tool schema. |
| `opencode` | Works with `NVIDIA_API_KEY` directly. |
Two provider-agent pairings are restricted regardless of NVIDIA Build: a
Bedrock evaluator with `claude-code` requires an explicit
`AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` pair or
`AWS_BEARER_TOKEN_BEDROCK` and does not support local mode at all, and an
Anthropic evaluator with `opencode` also does not support local mode.
## Where trials run
SkillEvaluator exposes 27 environment modes: 26 Harbor-native backends,
including Docker, plus SkillEvaluator's own `local` host-execution mode.
Docker is the default. The `tier3` extra installs `harbor[gke]==0.25.0`, including
GKE/ACK dependencies. Install the environment extra shown below when another
managed backend needs one.
| Mode | Type | Harbor extra or system prerequisite |
| --- | --- | --- |
| `docker` | Harbor-native | No additional Python extra. Requires Docker Engine and Docker Compose v2 — see [Docker mode](#docker-mode). |
| `kata` | Harbor-native | No additional Python extra. Requires a Linux host with `/dev/kvm` and a Kata Containers runtime registered with Docker — see [Kata mode](#kata-mode). |
| `daytona` | Harbor-native | `harbor[daytona]==0.25.0` |
| `e2b` | Harbor-native | `harbor[e2b]==0.25.0` |
| `modal` | Harbor-native | `harbor[modal]==0.25.0` |
| `runloop` | Harbor-native | `harbor[runloop]==0.25.0` |
| `langsmith` | Harbor-native | `harbor[langsmith]==0.25.0` |
| `ec2` | Harbor-native | `harbor[ec2]==0.25.0` |
| `gke` | Harbor-native | `harbor[gke]==0.25.0` |
| `ack` | Harbor-native | `harbor[gke]==0.25.0` |
| `openshift` | Harbor-native | No Harbor Python extra; install and authenticate the OpenShift `oc` CLI. |
| `novita` | Harbor-native | `harbor[novita]==0.25.0` |
| `apple-container` | Harbor-native | No Harbor Python extra; install the Apple `container` CLI. |
| `singularity` | Harbor-native | No Harbor Python extra; install the `singularity` CLI Harbor invokes. |
| `islo` | Harbor-native | `harbor[islo]==0.25.0` |
| `tensorlake` | Harbor-native | `harbor[tensorlake]==0.25.0` |
| `cwsandbox` | Harbor-native | `harbor[cwsandbox]==0.25.0` |
| `wandb` | Harbor-native | `harbor[cwsandbox]==0.25.0` |
| `use-computer` | Harbor-native | `harbor[use-computer]==0.25.0` |
| `blaxel` | Harbor-native | `harbor[blaxel]==0.25.0` |
| `beam` | Harbor-native | `harbor[beam]==0.25.0` |
| `skypilot` | Harbor-native | `harbor[skypilot]==0.25.0` |
| `hyperbrowser` | Harbor-native | `harbor[hyperbrowser]==0.25.0` |
| `vercel` | Harbor-native | `harbor[vercel]==0.25.0` |
| `runta` | Harbor-native | `harbor[runta]==0.25.0` |
| `mosaic` | Harbor-native | `harbor[mosaic]==0.25.0` |
| `local` | SkillEvaluator | Your machine, under an OS sandbox policy — see [Local mode](#local-mode). |
Installed packages can add more modes, for example an organization's internal
sandbox service. See [Custom execution backends](#custom-execution-backends).
<Tip>
SkillEvaluator documents setup for `docker`, `kata`, and `local`. For any other
backend, install the mapped Harbor extra or system CLI shown above, set the
credentials that backend requires, and confirm readiness with
`skillevaluator doctor --env-mode <mode>` before a full run.
</Tip>
Harbor 0.24 merged its W&B sandbox into `cwsandbox`. SkillEvaluator keeps
`--env-mode wandb` and runs it as Harbor's `cwsandbox` backend with W&B
authentication (`auth=wandb`), using `WANDB_API_KEY` or the netrc file that
`wandb login` writes. Because SkillEvaluator sets `auth` itself, `--ek auth=...`
is rejected for both `cwsandbox` and `wandb`.
Harbor 0.25 also defines `cua-cloud`, `opensandbox`, `hf-sandbox`, `podman`,
`prime`, `smol`, `sbx`, `sail`, and `kernel`, but those backends are not exposed
until SkillEvaluator can project the complete task bundle — skill, verifier, dependencies, and
inputs — through a trusted image or backend-native provisioning path, with
credentials and constructor options reviewed per backend. Selecting one of those
names is rejected before a run starts. Prime cannot build a Dockerfile-only
task, which is how SkillEvaluator stages most tasks, and Smol runs only a
published `[environment].docker_image`, which SkillEvaluator rejects because it
bypasses the evaluator-staged image. Exposing either one needs Compose-only
staging or a trusted image-publishing step, validated live. Podman also needs a
runtime-aware hardened Compose runner: Harbor runs Podman without Compose's
`up --wait` readiness gate and, for some frontends, without `compose cp`, which
SkillEvaluator's Docker isolation relies on.
### Native backend constructor options
Environment kwargs are operator-only and apply only to Harbor-native
non-Docker backends. `docker` and `local` reject any environment kwargs, and
`kata` accepts only its two runtime options (see [Kata mode](#kata-mode)). For an
eligible backend that needs non-secret constructor options in addition to host
credentials, pass each option with the repeatable
`--environment-kwarg KEY=VALUE` / `--ek KEY=VALUE` flag:
```bash title="GKE constructor options"
skillevaluator doctor --env-mode gke \
--ek cluster_name=my-cluster \
--ek region=us-central1 \
--ek namespace=skill-evals \
--ek registry_location=us-central1 \
--ek registry_name=skill-evals
```
Values accept Harbor's JSON-compatible types. Skill-owned `evals/config.yml`
cannot set constructor kwargs because backend endpoints and sandbox controls
belong to the operator trust boundary. Never pass secrets through the CLI;
backend credentials belong in the operator's host environment and are kept
out of Harbor argv and stored run configuration.
SkillEvaluator also rejects fields reserved for Harbor runtime policy. Do not
use environment kwargs for resource overrides (`override_cpus`,
`override_gpus`, `override_memory_mb`, `override_storage_mb`, `override_tpu`),
mount injection (`mounts`, `mounts_json`), network policy (`network_policy`,
`phase_network_policies`, `extra_allowed_hosts`), extra Compose definitions
(`extra_docker_compose`), lifecycle controls (`delete`, `force_build`,
`keep_containers`), Harbor's output streaming and environment upload controls
(`stream`, `enable_environment_dir_upload`), the per-trial wiring Harbor
supplies itself (such as `environment_dir`, `session_id`, and `trial_paths`),
or Kubernetes pod/security overrides. Use SkillEvaluator's first-class flags
and config where available; advanced pod/security overrides are intentionally
not exposed through this CLI surface. Equivalent
backend-specific aliases are also reserved when they directly bypass those
guarantees, including provider volume/resource/network overrides, Kubernetes or
cloud workload identities, disabled SSH host verification, and Singularity host
mount policy.
The backends with required constructor keys are:
| Mode | Required keys |
| --- | --- |
| `gke` | `cluster_name`, `region`, `namespace`, `registry_location`, `registry_name` |
| `ack` | `namespace` |
| `ec2` | `region`, plus `ami_id` for the default `launch_mode=ephemeral`; use `launch_mode=attach` with `instance_id` to attach to an existing instance. |
EC2 accepts either Harbor's explicit `ssh_key_path` option or the host's
authenticated SSH agent. In the latter case, keep `SSH_AUTH_SOCK` set when
running SkillEvaluator; it is forwarded only to the EC2 backend process. A
provided `ssh_key_path` must resolve to an existing regular file. Ephemeral EC2
with `use_public_ip=false` also requires a non-empty `subnet_id`.
Managed-backend readiness is not a connectivity test. EC2 preflight checks the
Python extra, the local `ssh` executable, and the constructor invariants above;
it does not contact AWS or prove credentials, AMI/instance access, networking,
or quota. GKE preflight checks the required constructor options, local `gcloud`
availability, and that a kubeconfig path exists; it does not contact the GKE
cluster or prove authentication, permissions, registry access, or capacity.
The first real trial remains the end-to-end infrastructure check.
For ACK, optional `context` and `kubeconfig` kwargs select the Kubernetes
configuration. Without an available kubeconfig, readiness falls back to the
in-cluster service account and performs a bounded, namespaced pod-list probe
before launching trials. `KUBERNETES_SERVICE_HOST` and
`KUBERNETES_SERVICE_PORT` are forwarded only for this ACK client path.
Harbor 0.25 Modal preflight requires `~/.modal.toml` or `MODAL_TOKEN_ID` plus
`MODAL_TOKEN_SECRET`. SkillEvaluator validates and forwards an explicitly set
`MODAL_CONFIG_PATH`, but Harbor's child-process preflight does not recognize a
custom config path by itself; configure one of Harbor's supported credential
forms before launching trials.
Runta authenticates with `RUNTA_TOKEN` or the configuration file named by
`RUNTA_CONFIG` (default `~/.config/runta/config.toml`), and Harbor's preflight
requires one of the two variables. `RUNTA_ENDPOINT` selects a non-default API
endpoint. A relative `RUNTA_CONFIG` path is resolved against the directory you
run SkillEvaluator from, and a configured file must exist. Runta's `mode`
kwarg is reserved: `mode=direct` runs commands in Runta's base runtime and
ignores the task image, while Harbor's automatic selection builds the task's
Dockerfile or Compose definition inside the runtime.
Mosaic authenticates with `MOSAIC_API_TOKEN` or the token `mos login` saves in
`~/.config/mosaic-sandbox/config.json`; `MOSAIC_CONFIG` names a different file,
which must exist and is resolved against the directory you run SkillEvaluator
from when relative.
`MOSAIC_API_URL` and `MOSAIC_RETRIES` tune the API client, the SDK's older
`MAR_API_TOKEN`, `MAR_ENDPOINT`, and `MAR_CONFIG` names are forwarded too, and
`MOSAIC_REGISTRY_USERNAME` and `MOSAIC_REGISTRY_PASSWORD` cover private base
images. Mosaic's `volume`, `persist`, and `enable_ssh` kwargs are
reserved because they let state or access outlive one isolated trial, and
`build_args` and `build_target` are reserved because they can skip or alter
evaluator-staged Dockerfile layers. Mosaic's `secrets` kwarg is rejected because
it injects provider secrets into the agent's sandbox, and a Mosaic token stored
only in `E2B_API_KEY` is rejected because that variable is never forwarded to
Mosaic. Mosaic runs one VM per trial, so tasks with an
`environment/docker-compose.yaml` are rejected before the run starts.
Harbor builds each Mosaic task image into a snapshot named from the task and its
build context, reuses an existing snapshot with that name in later runs, and
never deletes snapshots. Run evaluations in a Mosaic account that only trusted
operators can write to, and delete old `harbor-*` snapshots when you no longer
need the staged skills and inputs they contain.
Harbor's `auto_mounts` and `auto_mounting` output controls are reserved for
SkillEvaluator's runtime policy. The `cwsandbox` and `wandb` modes also reserve
`forward_compose_env`, which could copy operator environment values into the
task's Compose configuration. Credentials continue to use the evaluator's
controlled handoff.
Other eligible, backend-specific constructor options are forwarded after
validation. A key that the selected backend's Harbor 0.25.0 constructor does not
accept is rejected before the run starts; for example, Harbor 0.24 removed
Modal's `modal_sandbox_v2` option, so drop it from existing `--ek` flags. ACK
scheduling options such as `node_selector` and `tolerations` are accepted, and
`image_pull_secret` accepts only the name of an existing Kubernetes Secret
object—not credential contents. Credential-bearing options such as LangSmith's
`api_key` and Vercel's `credential_injection` are rejected. Use the same
`--environment-kwarg` values and eligible non-Docker mode with `doctor` or
`health-check` to preflight the configuration you intend to evaluate.
Harbor 0.25 adds eligible `dockerd_wait_sec` for Daytona,
`sandbox_timeout_secs` for E2B, `connection_pool_maxsize` for Runta, and
`dind_image` for `cwsandbox` and `wandb`.
A few Harbor options contain provider-managed secret **references**, not secret
values. These narrow shapes are allowed; every other credential-shaped kwarg,
embedded credential URI, or raw secret value is rejected:
| Mode | Allowed reference shape |
| --- | --- |
| `ack` | `image_pull_secret="name"`, where `name` is a Kubernetes DNS-subdomain Secret name |
| `modal` | `registry_secret="name"`; `secrets=["name", ...]` |
| `skypilot` | `secrets=["name", ...]` |
| `daytona` | `secrets={"TARGET_ENV": "organization-secret-name"}` |
| `cwsandbox`, `wandb` | `secrets=[{...}]` provider reference mappings using only `env_var`, `field`, `name`, and `store` |
Complex values use JSON syntax in the `KEY=VALUE` argument. Reference names
must be trimmed and bounded; they cannot contain credential-bearing URIs.
## Custom execution backends
A Python package can add an `--env-mode` that runs trials on infrastructure
Harbor does not ship, without changing SkillEvaluator. The package registers
an entry point in the `skillevaluator.env_modes` group. The entry-point name
becomes the mode name, and its value names an `EnvironmentBackendSpec` (or a
function that returns one):
```toml
# pyproject.toml of the backend package
[project.entry-points."skillevaluator.env_modes"]
acme = "skillevaluator_acme.backend:SPEC"
```
```python
# skillevaluator_acme/backend.py
from skillevaluator.tier3_environments import EnvironmentBackendSpec
SPEC = EnvironmentBackendSpec(
import_path="skillevaluator_acme.environment:AcmeEnvironment",
environment_kwargs=frozenset({"profile", "lease_seconds"}),
required_environment_kwargs=("profile",),
host_env_vars=frozenset({"ACME_SANDBOX_URL", "ACME_SANDBOX_TOKEN"}),
prerequisites="skillevaluator_acme.checks:prerequisites",
install_hint="Install 'skillevaluator-acme' from your package index.",
)
```
`import_path` names a Harbor `BaseEnvironment` subclass. SkillEvaluator passes
it to `harbor run --env`, so the class receives the same evaluator-staged task
directory, verifier, and trial paths as a Harbor-native backend.
### What a backend must provide
- Support the Harbor release SkillEvaluator pins (`harbor==0.25.0`), including
its environment constructor options.
- Run the evaluator-staged image: build each task's `environment/Dockerfile` as
written, or an equivalent image with the same layers. Do not substitute a
generic image.
- Preserve the task workspace: `/workspace` with the staged skills under the
task's `skills_dir`, task input, and the agent working directory.
- Preserve the verifier contract: run `tests/` with the verifier environment
SkillEvaluator stages, and return `/logs/verifier` and `/logs/agent`
(including the trajectory) to the host.
- Clean up every trial resource, including after cancellation.
The acceptance test for a backend is a full SkillEvaluator Tier 3 run of a
skill with both the baseline and with-skill conditions on that backend. The
run must produce the expected scores, retain trajectories and verifier
artifacts, and leave no trial resources behind. Compare the results with the
same skill on `--env-mode docker`.
### Policy SkillEvaluator applies
A custom execution backend follows the same policy as the Harbor-native
backends:
- `--ek` accepts only the names in `environment_kwargs`. Names in
`reserved_environment_kwargs`, SkillEvaluator's runtime-policy kwargs such as
`network_policy` and `session_id`, and secret-bearing names or values are
rejected.
- Only the upper-case variables in `host_env_vars` reach the Harbor process,
in addition to the usual proxy, CA bundle, and locale variables. A skill's
`harbor.runtime_env` cannot set them. A backend cannot forward agent or
provider credentials (for example `ANTHROPIC_*`, `CLAUDE_*`, `CODEX_*`,
`OPENAI_*`, `OPENCODE_*`, `NVIDIA_*`, `AWS_*`, `GITHUB_*`, `GH_*`, `HF_*`),
another backend's credentials, judge settings, or process-control variables,
because Harbor's stock agents read some of them and can copy them into the
trial sandbox. `KUBECONFIG` is allowed. The runner supplies agent credentials
per agent as it does for every backend.
- Harbor runs in a private working directory, so list host variables and `--ek`
kwargs that hold file paths in `host_path_env_vars` and
`host_path_environment_kwargs`; relative values are made absolute first.
- `skillevaluator doctor --env-mode <mode>` checks
`required_environment_kwargs`, then runs the optional `prerequisites`
function (`fn(environment_kwargs) -> list[str]`), then the environment
class's Harbor `preflight()`. Diagnostics are redacted like built-in ones.
- Tasks with a `docker-compose.yaml` are rejected unless the backend sets
`supports_docker_compose=True`.
- `agent_import_paths` may map an agent name to a Harbor agent class, for
example one that uses a CLI already installed in the backend's image.
Agents without an entry use Harbor's stock agent. When the evaluator
provider needs its own wrapper (for example `SkillEvaluatorGatewayCodex` for
Codex with an OpenAI-compatible gateway), the backend's class must subclass
that wrapper; otherwise the run stops before any trial starts.
A custom execution backend cannot replace a built-in mode name, and a name
registered by more than one installed package is refused with every package
name until all but one is uninstalled. Listing the available modes reads
package metadata only; the backend module is imported when its mode is
selected. Published benchmark cards show a custom execution backend as
`Isolated sandbox` and do not name it. Set
`SKILLEVALUATOR_DISABLE_CUSTOM_BACKENDS=1` to ignore every installed custom
execution backend. Backend packages run with your privileges, so install only
ones you trust.
## Docker mode
The default path needs a running Docker daemon and Docker Compose v2 —
`skillevaluator doctor --env-mode docker` checks both. Each trial runs in its
own container, and both arms of the comparison are staged from the same
environment definition — the baseline image differs only by not containing
the skill — so the skill stays the only variable.
### Windows hosts
On native Windows, use Docker Desktop with Linux containers. Keep the Windows
system variables available in the shell that runs SkillEvaluator: Harbor
inherits `PROGRAMFILES`, `PROGRAMW6432`, and `PROGRAMDATA` so Docker Desktop
can discover its Compose plugin.
For a non-UTF-8 Windows locale, enable Python UTF-8 mode in PowerShell before
running readiness checks or evaluations:
```powershell title="Windows Docker readiness"
$env:PYTHONUTF8 = "1"
$env:PYTHONIOENCODING = "utf-8"
skillevaluator doctor --env-mode docker
```
SkillEvaluator forwards both Python settings to Harbor when they are set;
it does not enable them automatically. Native Windows local mode remains
unsupported; use Docker or run local mode inside WSL2.
Verifier scripts run inside the Linux container, so they need LF line
endings. SkillEvaluator writes generated `test.sh` files and the
`init-custom-grader` and `init-harbor-task` grader starters with LF on every
host. Keep LF when you edit them: a CRLF `grader.sh` or `test.sh` fails in the
container before grading starts.
### Resource tuning
If an agent's work is heavier than the defaults allow — large builds,
memory-hungry test suites — override the per-container allocation. The flags
pass straight through to Harbor:
```bash title="Give each trial more headroom"
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker \
--override-cpus 4 --override-memory-mb 8192 --override-storage-mb 16384
```
To make the sizing part of the skill's run policy instead of a flag, set
`harbor.resources` (`cpus`, `memory_mb`, `storage_mb`) in `evals/config.yml`
— see [Eval Datasets](/skills/skillevaluator/eval-datasets). A trial that dies mid-run with the
agent process killed is the classic out-of-memory symptom; raise
`--override-memory-mb` first.
### Custom Dockerfiles
Declare system tools in `scripts/apt-packages.txt` and Python packages in
`scripts/requirements.txt` anywhere under the skill directory. These files are
auto-installed only when there is no custom `evals/environment/Dockerfile`.
With a custom Dockerfile, install those dependencies there yourself. A required
tool or module that is unavailable makes the case `environment_invalid` and
UNSCORED, not a skill defect; the report names the missing dependency and fix.
When an import or executable name differs from its package name, the report
asks you to install the package that provides it or ship it with the skill.
When a skill ships its own `evals/environment/Dockerfile`,
`--custom-dockerfile-mode` decides how it combines with the eval image:
| Value | Effect |
| --- | --- |
| `rebase` (built-in default) | Swaps your `FROM` line onto the eval base image, keeping the rest of your Dockerfile. |
| `preserve` | Keeps your `FROM` exactly as written and appends the eval dependencies on top. |
With either value, SkillEvaluator appends a `python -m pip install` of the eval
dependencies after your instructions. It then checks that `python3 -I` can
import the packages the verifier uses, because the verifier runs as
`python3 -I`. `-I` ignores `PYTHONPATH` and the user site, so make sure those
packages land in the system or virtual environment site-packages that
`python3` uses. In a custom image, `python` and `python3` must use the same
interpreter so the install and verifier see the same packages. Do not use
`ENV PIP_USER=1`, `PIP_TARGET` or a `PYTHONPATH` entry for them. If the check fails, the image build stops with
`ModuleNotFoundError`. `custom_only` grading never runs that verifier, so it
skips the check.
Environment rules — size limits, auto-generated fallbacks, and sidecar
containers — are part of the `evals/` contract in
[Eval Datasets](/skills/skillevaluator/eval-datasets).
Docker Compose project `.env` files are rejected before startup. Protected
main-container and sidecar operations disable Compose's implicit env-file
loading so the same definition cannot start with one interpolation value and
later run with another. Put non-secret interpolation values in
`harbor.runtime_env` in `evals/config.yml`; keep operator credentials in the
host environment as described above.
## GKE mode
When running with `--env-mode gke`, SkillEvaluator enforces least-privilege pod
identity by default unless you explicitly opt in for trusted skills via
`SKILLEVALUATOR_GKE_ALLOW_WORKLOAD_IDENTITY=1` or `--ek allow_workload_identity=1`:
- **ServiceAccount and metadata isolation**: Pods run with
`automountServiceAccountToken: false`, `hostNetwork: false`,
`GCE_METADATA_HOST=127.0.0.1:1`, and a pod-scoped egress `NetworkPolicy`
(`skillevaluator-block-gce-metadata`) that blocks HTTP egress to
`169.254.169.254/32` and `169.254.169.252/32` while preserving UDP/TCP port 53 DNS.
Kubernetes ServiceAccounts annotated with `iam.gke.io/gcp-service-account` or
`iam.gke.io/return-principal-id-as-email=true` are rejected.
- **Trusted init and companion metadata probes**: Single-container evaluation
pods include a trusted init container (`skillevaluator-metadata-probe-init`)
that verifies metadata endpoints are unreachable *before* the `main` task
container starts, plus a companion container (`skillevaluator-metadata-probe`)
that re-verifies isolation before task upload. Set
`SKILLEVALUATOR_GKE_METADATA_PROBE_IMAGE` when your cluster requires pulling
the probe image (`python:3.12-slim` by default) from a private Artifact Registry mirror.
- **GKE Autopilot ephemeral storage**: On GKE Autopilot (auto-detected via node
inspection or `gcloud`, or set explicitly with `SKILLEVALUATOR_GKE_AUTOPILOT`
/ `--ek autopilot=<bool>`), general-purpose pods have a `10Gi` (`10240Mi`)
cumulative ephemeral-storage ceiling. When a non-GPU/TPU `main` container
requests `10Gi`, SkillEvaluator caps `main` to `10176Mi` so `main` (`10176Mi`) +
`skillevaluator-metadata-probe` (`64Mi`) equals `10240Mi`. Standard GKE
clusters, GPU/TPU pods, and explicit storage requests above `10Gi` are not
lowered.
## Kata mode
`--env-mode kata` uses the same hardened Docker Compose backend as Docker mode,
including its credential handoff and container containment, but Harbor adds a
final Compose overlay that runs every task service under a Kata Containers
runtime, so agent execution and verification happen inside KVM microVMs instead
of on the host kernel. It needs a Linux host with `/dev/kvm`, Docker Compose v2,
and a Kata runtime handler registered with Docker (for example with Kata's
`kata-manager.sh`). `skillevaluator doctor --env-mode kata` checks all three,
including that Docker registers the selected handler.
Two environment kwargs are accepted:
| Kwarg | Default | Rule |
| --- | --- | --- |
| `kata_runtime` | `kata` | A registered Docker runtime handler whose name contains `kata`, such as `kata-clh` or `kata-qemu`. Other handlers, such as `runc`, are rejected. |
| `kata_dns` | `1.1.1.1,8.8.8.8` | At least one IP address, written to each microVM's `resolv.conf`, as a list or comma-separated string. The default replaces the resolvers Docker's embedded DNS would forward to, so set your network's resolvers when public DNS is blocked or when a gateway, judge, or package endpoint uses an internal hostname. |
Harbor's Kata backend has limits that SkillEvaluator keeps:
- Image builds are not VM-isolated. Dockerfile `RUN` steps execute on the
host's BuildKit builder; the microVM boundary covers the agent and verifier.
- Non-public network policies are rejected before start, because Kata VMs
cannot join the egress-control sidecar's network namespace. Generated tasks
use public networking.
- Compose service-name DNS does not work inside the microVM, so tasks whose
Compose model declares services other than `main`, such as a database
sidecar, are rejected before the run starts.
## Local mode
<Warning>
Local mode is experimental and executes the agent directly on your host
rather than in a container. Its OS sandbox is designed for trusted skills
and workspaces on a developer machine — use Docker or a remote Harbor backend for
untrusted code. Every local run prints a "Local mode · Experimental"
warning panel.
</Warning>
`--env-mode local` skips containers entirely: the agent CLI already installed
on your machine runs the trials, wrapped in an OS-level sandbox. On Linux that
is Bubblewrap — kernel namespace isolation, a read-only view of the system,
writes confined to the run directories, and a network namespace that is
isolated when the run is airgapped and shared for model egress when network
is allowed. SkillEvaluator resolves `bwrap` on `PATH` to an absolute path once,
smoke-tests that binary, and launches the same binary for every command. On
macOS it is Seatbelt, which confines reads and writes and —
even when network is allowed — denies inbound connections, socket binding,
and Unix sockets (except the `mDNSResponder` DNS socket); it cannot provide
full process isolation, so it is the weaker of the two. Native Windows is
hard-unsupported and fails closed before agent-runtime discovery —
`SKILLEVALUATOR_LOCAL_SANDBOX=prefer` or `off` is not a bypass; the error
directs you to WSL2 or `--env-mode docker`. Only `claude-code`, `codex`,
and `opencode` can run in local mode.
When the selected agent's CLI is missing, a local-mode run offers to install it
under `SKILLEVALUATOR_RUNTIME_DIR`, at the release Harbor installs; without a
terminal it prints the `npm install` command instead. Each trial checks the
installed version against the selected model and fails before the agent starts
when it is too old: Claude 5.5 and later models need Claude Code 2.1.280 or
newer, and GPT-6 models need Codex 0.159.1 or newer.
The sandbox policy is controlled by environment variables:
| Variable | Default | Effect |
| --- | --- | --- |
| `SKILLEVALUATOR_LOCAL_SANDBOX` | `require` | `require` fails closed when no OS sandbox backend is usable; `prefer` degrades to advisory-only guardrails with a loud warning; `off` skips the sandbox entirely — only for skills you fully trust. No value enables native Windows: the platform check rejects the run first. |
| `SKILLEVALUATOR_LOCAL_ALLOW_NET` | `true` | Network egress for trials. Set to `0` to airgap a skill that must not reach the network — this also blocks NVIDIA Build local agents entirely, since they require network access. |
| `SKILLEVALUATOR_LOCAL_STRICT_READS` | `false` | Tightens the sandbox's read-only view of the host system to a stricter path set. |
| `SKILLEVALUATOR_LOCAL_INHERIT_AGENT_KEYS` | `false` | When true, trials inherit the host's agent credential variables into their ambient environment. Off by default so a hostile skill command cannot read them — agents receive credentials per-exec instead. |
| `SKILLEVALUATOR_RUNTIME_DIR` | `~/.local/share/skillevaluator/runtimes` | Where local mode looks for managed agent CLIs. Must be a dedicated subdirectory — not the home directory or one of its parents. |
Full semantics for each variable are in
[Environment Variables](/skills/skillevaluator/environment-variables). `doctor --env-mode local`
verifies the sandbox backend and the installed agent CLIs before you commit to
a run.
When Bubblewrap is installed but fails its startup probe, the error includes
Bubblewrap's own failure line. Inside a container,
`pivot_root: Operation not permitted` can mean the container's seccomp profile
blocks Bubblewrap; only the options the container was launched with can change
that. On such a host, use `--env-mode docker` where a Docker daemon is
available, or set `SKILLEVALUATOR_LOCAL_SANDBOX=prefer` (advisory-only
guardrails) or `off` for skills you fully trust.
## Workspace modes
SkillEvaluator stages the agent's workspace itself, in both arms, so the
skill under test never leaks into the without-skill baseline — that isolation
is what keeps Skill Lift trustworthy. Three flags control what else the agent
can see:
| Flag | Default | Effect |
| --- | --- | --- |
| `--skill-workspace-mode [isolated\|group]` | `isolated` | `isolated` stages only the target skill; `group` also stages the sibling skills you name. |
| `--include-skills PATH` | none | Additional skill directories to stage (repeatable). Requires group mode, via the flag or `skill_workspace.mode: group` in `evals/config.yml`. |
| `--copy-repo` | off | Copy the surrounding repository into the task environment, for skills that operate on their own repo's files. |
Use group mode when the skill delegates to companion skills and evaluating it
alone would be unfair; keep the default otherwise — a smaller workspace is a
cleaner experiment.
### Select a reproducible decoy set
`tier3 select-decoys` ranks real skills from a local description-embedding
catalog and saves a selection manifest. Selection accepts skill-only catalogs
with at most 1,000 entries. Build the catalog without `--full-body` and use the
same embedding provider and model for selection:
```bash title="Select four neighboring skills"
skillevaluator similarity-check ./skills --save-catalog ./skill-catalog.json
skillevaluator tier3 select-decoys ./skills/my-skill \
--catalog ./skill-catalog.json --catalog-root ./skills \
--count 4 --exclude required-helper \
--output ./skills/my-skill/evals/decoys.json
```
Opt in through the target skill's run policy. Manifest paths, like included
skill paths, are relative to the target skill's parent directory:
```yaml title="skills/my-skill/evals/config.yml"
schema_version: 1
skill_workspace:
mode: group
decoy_manifest: my-skill/evals/decoys.json
include:
- required-helper
```
Selection excludes the target, skills bundled inside it, and duplicate
candidates, orders equal scores by catalog-relative path, and fails if too few
eligible skills remain. Required
helpers stay explicit in `include`; the selector does not infer prerequisites.
A collision between a selected skill and an explicitly included skill is an
error.
Selection and group-mode evaluation read the `name` and `description` of the
target and of each selected skill as `similarity-check` does, within the same
[frontmatter structure limits](/skills/skillevaluator/tier2-reference#defaults-and-safety-limits).
The command fails, and says the skill needs a root `SKILL.md` whose frontmatter
is a valid YAML mapping with a non-empty string name and description, when that
file or its frontmatter is missing, the frontmatter is invalid YAML or not a
mapping, or either value is empty or not a string. A value that is not a string,
such as `name: yes` or `name: 2048`, is never converted to text. Quote such a
value, as in `name: "yes"`, and quote a value that contains a colon followed by
a space, as in `description: "Use when: debugging logs"`. Frontmatter over a
structure limit, or a value longer than 16,384 characters, fails with an error
that names the manifest, such as `my-skill/SKILL.md`.
The manifest records catalog and model identity, scores, the target manifest
hash, and selected skill runtime hashes. Evaluation verifies the hashes and
uses one snapshot of the selected skills in both arms; it makes no embedding
request and does not select again. The runtime hashes exclude evaluator-only
directories, including each nested skill's `evals/`, exactly as staging does.
Editing those configs, datasets, or selection manifests does not invalidate
other skills' selections. Agent-visible references still participate in the
hash, including an ordinary `references/evals/` folder.
For a collection that contains the target, or lies inside the target's parent
directory, `catalog_root` is stored relative to that parent (`.` in the example
above). Identical clones and relocated collections can reuse the same manifest;
evaluation checks the sources in the current checkout. An unrelated external
collection retains an absolute root and must be available at that location.
The runtime hashes cover everything staging copies from a selected skill,
including files that git ignores or does not track (such as `.DS_Store` or
`.pytest_cache/`) and empty folders, so generate a manifest you plan to commit
from a fresh clone. Of the permission bits, only whether a file is executable
by its owner counts, so clones made under different umasks agree. Windows
checkouts can differ: Git may convert line endings there, and Python reports
`.exe`, `.bat`, `.cmd` and `.com` files as executable and other files as not
executable. Generate a separate selection on Windows.
Regenerate after editing the target's `SKILL.md`, changing selected runtime
content, or moving an external collection. Manifests using an earlier digest
algorithm must also be regenerated. External workspace skills retain the
existing publication-evidence limitations.
Use a separate manifest filename such as `decoys.json`. Selection output cannot
replace an evaluation dataset, run policy, embedding catalog, or existing
non-manifest file.
Similarity is a heuristic for plausible alternatives, not evidence of actual
routing confusion. Skill Lift remains conditional on the chosen workspace;
a neighboring skill may legitimately substitute for the target. Review the
selection before running the evaluation, and keep the manifest with the run
policy when comparing results.
## Readiness checks
`doctor` and `health-check` run the same readiness checks and exit nonzero when
any check fails, so either can gate a script or CI job. Warnings from
`doctor --verify-models` are nonblocking because the catalog result is
inconclusive rather than a definitive rejection. Both commands check Tier 3 by
default. `--tier 1` checks the Tier 1 scanners, `--tier 2` the embedding and LLM
providers, and `--tier all` every tier; see the
[CLI Reference](/skills/skillevaluator/cli-reference#doctor).
### doctor
```bash title="Readiness check before the first run"
skillevaluator doctor --agents codex --env-mode docker
```
`doctor` prints a check-by-check table and names exactly what is missing:
| Check | What it verifies |
| --- | --- |
| CLI package | SkillEvaluator is installed, with its version. |
| Public LLM provider | The evaluator provider resolves — the key that powers dataset generation and grading. |
| Agent runtime credential | The full credential and model plan for each selected agent resolves — the same plan a real run would use, including the host credentials the agents need. |
| Harbor agents | Every requested agent name is known. |
| `<env-mode>` prerequisite | The selected `--env-mode` is usable — Docker Compose responds, the local sandbox and agent CLIs are ready, or the selected Harbor backend's prerequisites are met. |
`doctor` also accepts `--agent-model AGENT=MODEL` (repeatable), so you can
check the exact model plan a run would use. Add `--verify-models` to probe
each agent's resolved model live against the provider's catalog — one extra
pass, warn, or fail row per agent. A warning means the catalog result is
non-authoritative, such as a public NVIDIA Build listing or an OpenAI catalog
permission failure that does not decide inference access. Browse catalog
membership with `skillevaluator models` (top 10 by default; `--limit` and
`--json` adjust the result). To prove that the agent can actually invoke the
model before the measured matrix, opt into the extra Tier 3 execution with
`--agent-runtime-preflight`. A different route used only by the
standard grader is not exercised by that smoke and remains unproven until
grading begins.
### health-check
```bash title="Quick readiness check"
skillevaluator health-check
```
`health-check` runs the same checks and prints the same table as a plain
`doctor` run — the only difference is that it lacks two of doctor's flags:
`--verify-models` and `--agent-model`. Use it as an install smoke test or a
CI preflight.
## Debug a run
Staged-task and raw Harbor job directories are transient: they are deleted
by default when a run finishes, on success and on failure alike. Retain them
with `--harbor-keep-jobs`, then open them with Harbor's own trajectory
browser — the final Artifacts panel prints the exact `harbor-view` command
whenever a retained jobs directory exists:
```bash title="Retain job artifacts, then browse them"
skillevaluator tier3 evaluate ./my-skill --agents codex --harbor-keep-jobs
skillevaluator harbor-view <jobs-dir>
```
The retained tree holds the full agent trajectory and grader output for every
trial — the fastest way to see what an agent actually did. The results layout
and where `<jobs-dir>` lands on disk are in
[Reports & Results](/skills/skillevaluator/reports); debugging custom graders this way is
covered in [Custom Graders & Tasks](/skills/skillevaluator/custom-graders).
## Next steps
<CardGroup cols={2}>
<Card title="Tier 3: Live Agent Evaluation" icon="flask" href="/skills/skillevaluator/tier3-live-evaluation">
The end-to-end run workflow — readiness, dataset, evaluation, and results.
</Card>
<Card title="Custom Graders & Tasks" icon="terminal" href="/skills/skillevaluator/custom-graders">
Bring your own grader or Harbor task when the defaults can't express
success.
</Card>
</CardGroup>