Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

feat(tier3): run hardened Docker trials under Kata Containers (#182)

Narendran Raghavan004a49d29668
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 386b4669..0a656c5f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,6 +12,11 @@ All notable changes to SkillEvaluator are documented in this file.
scanner release, so reviewers can tell which SkillSpector produced them
without rerunning the scan. The version is shown even when the report omits
its timestamp.
+- `--env-mode kata` runs Tier 3 trials through the hardened Docker backend with
+ every task service in a Kata Containers microVM. It requires a Linux host with
+ `/dev/kvm` and a Kata runtime registered with Docker, and accepts only the
+ `kata_runtime` and `kata_dns` environment kwargs. Image builds still run on
+ the host, and tasks with Compose sidecars are rejected.
### Changed
@@ -82,6 +87,9 @@ All notable changes to SkillEvaluator are documented in this file.
only for the Codex process. Skills whose scripts, tasks, or stdio MCP servers
need these tools must install them through `scripts/apt-packages.txt` or a
custom `evals/environment/Dockerfile`.
+- Exposed `kata` as the 24th Harbor 0.24 backend alongside local mode.
+ `cua-cloud`, `opensandbox`, `hf-sandbox`, `podman`, `runta`, `prime`,
+ `mosaic`, and `smol` remain disabled.
### Fixed
diff --git a/docs/agents-and-sandboxes.mdx b/docs/agents-and-sandboxes.mdx
index cff01733..99a60aa2 100644
--- a/docs/agents-and-sandboxes.mdx
+++ b/docs/agents-and-sandboxes.mdx
@@ -62,7 +62,7 @@ See the two credential roles in [Providers & Credentials](configuration.mdx).
## One NVIDIA Build key for everything
When the evaluator provider is NVIDIA Build (`nv_build`) and trials run in
-`--env-mode docker` or `local`, a single `NVIDIA_API_KEY` powers the
+`--env-mode docker`, `kata`, or `local`, a single `NVIDIA_API_KEY` powers the
evaluator and all three agents:
- **OpenCode** calls NVIDIA Build directly through its `nvidia` provider
@@ -114,7 +114,7 @@ SKILLEVALUATOR_LOCAL_ALLOW_NET or set it to 1`.
### Other backends still need native credentials
-The bridges exist only in `docker` and `local` mode. On any other Harbor
+The bridges exist only in `docker`, `kata`, and `local` mode. On any other Harbor
backend, each agent calls its native provider:
| Agent | Requirement with `nv_build` on other backends |
@@ -131,7 +131,7 @@ Anthropic evaluator with `opencode` also does not support local mode.
## Where trials run
-SkillEvaluator exposes 24 environment modes: 23 Harbor-native backends,
+SkillEvaluator exposes 25 environment modes: 24 Harbor-native backends,
including Docker, plus SkillEvaluator's own `local` host-execution mode.
Docker is the default. The `tier3` extra installs base `harbor==0.24.0`; install
the environment extra shown below when a managed backend needs one.
@@ -139,6 +139,7 @@ the environment extra shown below when a managed backend needs one.
| Mode | Type | Harbor extra or system prerequisite |
| --- | --- | --- |
| `docker` | Harbor-native | No additional Python extra. Requires Docker Engine and Docker Compose v2 — see [Docker mode](#docker-mode). |
+| `kata` | Harbor-native | No additional Python extra. Requires a Linux host with `/dev/kvm` and a Kata Containers runtime registered with Docker — see [Kata mode](#kata-mode). |
| `daytona` | Harbor-native | `harbor[daytona]==0.24.0` |
| `e2b` | Harbor-native | `harbor[e2b]==0.24.0` |
| `modal` | Harbor-native | `harbor[modal]==0.24.0` |
@@ -164,7 +165,7 @@ the environment extra shown below when a managed backend needs one.
| `local` | SkillEvaluator | Your machine, under an OS sandbox policy — see [Local mode](#local-mode). |
<Tip>
-SkillEvaluator documents setup for `docker` and `local`. For any other
+SkillEvaluator documents setup for `docker`, `kata`, and `local`. For any other
backend, install the mapped Harbor extra or system CLI shown above, set the
credentials that backend requires, and confirm readiness with
`skillevaluator doctor --env-mode <mode>` before a full run.
@@ -177,17 +178,20 @@ authentication (`auth=wandb`), using `WANDB_API_KEY` or the netrc file that
is rejected for both `cwsandbox` and `wandb`.
Harbor 0.24 also defines `cua-cloud`, `opensandbox`, `hf-sandbox`, `podman`,
-`kata`, `runta`, `prime`, `mosaic`, and `smol`, but those backends are not
-exposed until SkillEvaluator can project the complete task bundle — skill,
-verifier, dependencies, and inputs — through a trusted image or
-backend-native provisioning path, with credentials and constructor options
-reviewed per backend. Selecting one of those names is rejected before a run
-starts.
+`runta`, `prime`, `mosaic`, and `smol`, but those backends are not exposed
+until SkillEvaluator can project the complete task bundle — skill, verifier,
+dependencies, and inputs — through a trusted image or backend-native
+provisioning path, with credentials and constructor options reviewed per
+backend. Selecting one of those names is rejected before a run starts. Podman
+also needs a runtime-aware hardened Compose runner: Harbor runs Podman without
+Compose's `up --wait` readiness gate and, for some frontends, without
+`compose cp`, which SkillEvaluator's Docker isolation relies on.
### Native backend constructor options
Environment kwargs are operator-only and apply only to Harbor-native
-non-Docker backends. `docker` and `local` reject any environment kwargs. For an
+non-Docker backends. `docker` and `local` reject any environment kwargs, and
+`kata` accepts only its two runtime options (see [Kata mode](#kata-mode)). For an
eligible backend that needs non-secret constructor options in addition to host
credentials, pass each option with the repeatable
`--environment-kwarg KEY=VALUE` / `--ek KEY=VALUE` flag:
@@ -330,6 +334,35 @@ later run with another. Put non-secret interpolation values in
`harbor.runtime_env` in `evals/config.yml`; keep operator credentials in the
host environment as described above.
+## Kata mode
+
+`--env-mode kata` uses the same hardened Docker Compose backend as Docker mode,
+including its credential handoff and container containment, but Harbor adds a
+final Compose overlay that runs every task service under a Kata Containers
+runtime, so agent execution and verification happen inside KVM microVMs instead
+of on the host kernel. It needs a Linux host with `/dev/kvm`, Docker Compose v2,
+and a Kata runtime handler registered with Docker (for example with Kata's
+`kata-manager.sh`). `skillevaluator doctor --env-mode kata` checks all three,
+including that Docker registers the selected handler.
+
+Two environment kwargs are accepted:
+
+| Kwarg | Default | Rule |
+| --- | --- | --- |
+| `kata_runtime` | `kata` | A registered Docker runtime handler whose name contains `kata`, such as `kata-clh` or `kata-qemu`. Other handlers, such as `runc`, are rejected. |
+| `kata_dns` | `1.1.1.1,8.8.8.8` | At least one IP address, written to each microVM's `resolv.conf`, as a list or comma-separated string. The default replaces the resolvers Docker's embedded DNS would forward to, so set your network's resolvers when public DNS is blocked or when a gateway, judge, or package endpoint uses an internal hostname. |
+
+Harbor's Kata backend has limits that SkillEvaluator keeps:
+
+- Image builds are not VM-isolated. Dockerfile `RUN` steps execute on the
+ host's BuildKit builder; the microVM boundary covers the agent and verifier.
+- Non-public network policies are rejected before start, because Kata VMs
+ cannot join the egress-control sidecar's network namespace. Generated tasks
+ use public networking.
+- Compose service-name DNS does not work inside the microVM, so tasks whose
+ Compose model declares services other than `main`, such as a database
+ sidecar, are rejected before the run starts.
+
## Local mode
<Warning>
diff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdx
index bc224cb6..1f7ae661 100644
--- a/docs/cli-reference.mdx
+++ b/docs/cli-reference.mdx
@@ -373,7 +373,7 @@ Plus the standard report options. Classification tiers and catalog rules are in
## tier3 evaluate
-Run Tier 3 live agent evaluation: real agents attempt your eval cases with and without the skill, and the difference becomes Skill Lift. Requires the `tier3` extra, a provider key for grading, a credential the selected agents can use — with NVIDIA Build in `--env-mode docker` or `local`, the one `NVIDIA_API_KEY` covers both roles — and a running backend (Docker by default). The full walkthrough is [Tier 3: Live Evaluation](tier3-live-evaluation.mdx).
+Run Tier 3 live agent evaluation: real agents attempt your eval cases with and without the skill, and the difference becomes Skill Lift. Requires the `tier3` extra, a provider key for grading, a credential the selected agents can use — with NVIDIA Build in `--env-mode docker`, `kata`, or `local`, the one `NVIDIA_API_KEY` covers both roles — and a running backend (Docker by default). The full walkthrough is [Tier 3: Live Evaluation](tier3-live-evaluation.mdx).
```bash title="Live evaluation with codex in Docker"
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker
@@ -386,8 +386,8 @@ Use `skillevaluator tier3 PATH` for the complete workflow with automatic missing
| Flag | Default | Effect |
| --- | --- | --- |
| `-a, --agents TEXT` | provider-native | Comma-separated Harbor agents. Defaults: NVIDIA Build=`opencode`, OpenAI=`codex`, Anthropic=`claude-code`. Supported: `claude-code`, `codex`, `opencode`; the alias `claude` is accepted for `claude-code`. See [Agents & Sandboxes](agents-and-sandboxes.mdx). |
-| `--env-mode` | `docker` | Where trials run. SkillEvaluator exposes 24 environment modes: 23 Harbor-native modes — `docker`, `daytona`, `e2b`, `modal`, `runloop`, `langsmith`, `ec2`, `gke`, `ack`, `openshift`, `novita`, `apple-container`, `singularity`, `islo`, `tensorlake`, `cwsandbox`, `wandb`, `use-computer`, `blaxel`, `beam`, `skypilot`, `hyperbrowser`, `vercel` — plus SkillEvaluator `local` mode. `wandb` runs Harbor's `cwsandbox` backend with W&B authentication, so it needs the `harbor[cwsandbox]==0.24.0` extra; Harbor 0.24 has no `wandb` extra. Harbor's `cua-cloud`, `opensandbox`, `hf-sandbox`, `podman`, `kata`, `runta`, `prime`, `mosaic`, and `smol` backends are not selectable. See the complete extras and prerequisites matrix in [Agents & Sandboxes](agents-and-sandboxes.mdx#where-trials-run). |
-| `--environment-kwarg`, `--ek` | none | Operator-only `KEY=VALUE` constructor option for Harbor-native non-Docker backends (repeatable). Values accept JSON-compatible types; skill-owned `evals/config.yml` cannot set them. Never pass secrets. Docker and local reject all kwargs. Harbor runtime-policy fields for overrides, mounts, networks, extra Compose, lifecycle, and pod security are reserved, as are `stream`, `enable_environment_dir_upload`, and `auth` for `cwsandbox`/`wandb`. Names the selected backend does not accept in Harbor 0.24.0 are rejected. See [Native backend constructor options](agents-and-sandboxes.mdx#native-backend-constructor-options). |
+| `--env-mode` | `docker` | Where trials run. SkillEvaluator exposes 25 environment modes: 24 Harbor-native modes — `docker`, `daytona`, `e2b`, `modal`, `runloop`, `langsmith`, `ec2`, `gke`, `ack`, `openshift`, `novita`, `apple-container`, `singularity`, `islo`, `tensorlake`, `cwsandbox`, `wandb`, `use-computer`, `blaxel`, `beam`, `skypilot`, `hyperbrowser`, `vercel`, `kata` — plus SkillEvaluator `local` mode. `wandb` runs Harbor's `cwsandbox` backend with W&B authentication, so it needs the `harbor[cwsandbox]==0.24.0` extra; Harbor 0.24 has no `wandb` extra. Harbor's `cua-cloud`, `opensandbox`, `hf-sandbox`, `podman`, `runta`, `prime`, `mosaic`, and `smol` backends are not selectable. See the complete extras and prerequisites matrix in [Agents & Sandboxes](agents-and-sandboxes.mdx#where-trials-run). |
+| `--environment-kwarg`, `--ek` | none | Operator-only `KEY=VALUE` constructor option for Harbor-native non-Docker backends (repeatable). Values accept JSON-compatible types; skill-owned `evals/config.yml` cannot set them. Never pass secrets. Docker and local reject all kwargs, Kata accepts only `kata_runtime` and `kata_dns`, and Harbor runtime-policy fields for overrides, mounts, networks, extra Compose, lifecycle, and pod security are reserved, as are `stream`, `enable_environment_dir_upload`, and `auth` for `cwsandbox`/`wandb`. Names the selected backend does not accept in Harbor 0.24.0 are rejected. See [Native backend constructor options](agents-and-sandboxes.mdx#native-backend-constructor-options). |
| `--autopilot` | off | Create one eval case when no dataset/task source exists, then evaluate. The case is LLM-generated with the configured provider, with a deterministic keyless template fallback; an existing source is never overwritten. |
| `--skip-baseline` | off | Skip the without-skill baseline (no lift analysis, faster). |
| `--n-attempts INTEGER` | unset | Attempts per eval case (pass@k). |
@@ -520,8 +520,8 @@ skillevaluator doctor --env-mode docker
| Flag | Default | Effect |
| --- | --- | --- |
| `-a, --agents TEXT` | provider-native | Comma-separated agents to check. Defaults: NVIDIA Build=`opencode`, OpenAI=`codex`, Anthropic=`claude-code`. |
-| `--env-mode` | `docker` | Backend to check (same 24 modes as [tier3 evaluate](#tier3-evaluate)). |
-| `--environment-kwarg`, `--ek` | none | `KEY=VALUE` non-secret constructor option for a Harbor-native non-Docker backend (repeatable). Pass the same values intended for evaluation. Docker and local reject all kwargs; policy-owned fields are rejected for every eligible backend. |
+| `--env-mode` | `docker` | Backend to check (same 25 modes as [tier3 evaluate](#tier3-evaluate)). |
+| `--environment-kwarg`, `--ek` | none | `KEY=VALUE` non-secret constructor option for a Harbor-native non-Docker backend (repeatable). Pass the same values intended for evaluation. Docker and local reject all kwargs, and Kata accepts only `kata_runtime` and `kata_dns`; policy-owned fields are rejected for every eligible backend. |
| `--agent-model TEXT` | unset | Per-agent model override, `AGENT=MODEL` (repeatable) — check readiness with the model each agent will actually run. |
| `--verify-models` | off | Live per-agent catalog probe. Prints pass for verified access, warn for an inconclusive or non-authoritative result (success or failure), and fail only for a definitive credential, model, or configuration rejection. |
@@ -538,8 +538,8 @@ skillevaluator health-check
| Flag | Default | Effect |
| --- | --- | --- |
| `-a, --agents TEXT` | provider-native | Comma-separated agents to check. Defaults: NVIDIA Build=`opencode`, OpenAI=`codex`, Anthropic=`claude-code`. |
-| `--env-mode` | `docker` | Backend to check (same 24 modes as [tier3 evaluate](#tier3-evaluate)). |
-| `--environment-kwarg`, `--ek` | none | `KEY=VALUE` non-secret constructor option for a Harbor-native non-Docker backend (repeatable). Pass the same values intended for evaluation. Docker and local reject all kwargs; policy-owned fields are rejected for every eligible backend. |
+| `--env-mode` | `docker` | Backend to check (same 25 modes as [tier3 evaluate](#tier3-evaluate)). |
+| `--environment-kwarg`, `--ek` | none | `KEY=VALUE` non-secret constructor option for a Harbor-native non-Docker backend (repeatable). Pass the same values intended for evaluation. Docker and local reject all kwargs, and Kata accepts only `kata_runtime` and `kata_dns`; policy-owned fields are rejected for every eligible backend. |
## models
diff --git a/docs/configuration.mdx b/docs/configuration.mdx
index 8b3c7db5..fddbf68d 100644
--- a/docs/configuration.mdx
+++ b/docs/configuration.mdx
@@ -12,7 +12,7 @@ export NVIDIA_API_KEY='nvapi-...'
```
The same key covers Tier 1 LLM checks, Tier 2 embeddings and chat, and supported
-Tier 3 agents in Docker or local mode. OpenAI likewise starts with
+Tier 3 agents in Docker, Kata, or local mode. OpenAI likewise starts with
`SKILL_EVAL_LLM_PROVIDER=openai` and `OPENAI_API_KEY`, using Codex for Tier 3.
External scanners, agent runtimes, and sandboxes remain prerequisites; model
credentials do not install them.
@@ -236,7 +236,7 @@ Live evaluation has two credential roles. They can share a key when the selected
Both roles resolve from the operator's environment; a skill's `evals/config.yml` can never supply, rename, or redirect a credential. Export them in the shell or CI job that runs SkillEvaluator: Harbor starts in an empty, evaluator-owned working directory with `.env` loading disabled, so a `.env` or `.env.local` file near your project is not read. What each agent needs depends on the provider and `--env-mode`:
-- **NVIDIA Build, one key** (`--env-mode docker` or `local`): a single `NVIDIA_API_KEY` powers the evaluator and all three supported agents — `opencode` calls NVIDIA Build directly, while `claude-code` and `codex` route through SkillEvaluator's compatibility bridge (in-container in Docker mode; an authenticated in-process host bridge in `local` mode). Agent models must be full NVIDIA Build catalog IDs in `publisher/model` form.
+- **NVIDIA Build, one key** (`--env-mode docker`, `kata`, or `local`): a single `NVIDIA_API_KEY` powers the evaluator and all three supported agents — `opencode` calls NVIDIA Build directly, while `claude-code` and `codex` route through SkillEvaluator's compatibility bridge (in-container in Docker and Kata modes; an authenticated in-process host bridge in `local` mode). Agent models must be full NVIDIA Build catalog IDs in `publisher/model` form.
- **NVIDIA Build on Harbor cloud modes** (`daytona`, `e2b`, ...): the bridge is not available. `claude-code` needs its own native `ANTHROPIC_API_KEY`, `codex` needs a full OpenAI Responses API credential (`OPENAI_API_KEY` plus `OPENAI_BASE_URL`), and each needs an explicit model via `--agent-model` or `harbor.agents.<agent>.model`.
- **Matching native provider**: when the evaluator provider is the agent's own (for example `anthropic` with `claude-code`, or `openai` with `codex`), the same key serves both roles automatically.
- **OpenAI-compatible gateway**: Codex, OpenCode, and Claude Code inherit the shared URL and key. The gateway must support the selected harness's API and model. Explicit Claude credentials and endpoint settings take precedence as described above.
@@ -301,7 +301,7 @@ Non-secret constructor kwargs are only for Harbor-native non-Docker backends.
They are operator-only: use repeatable `--environment-kwarg KEY=VALUE` / `--ek
KEY=VALUE` flags on `tier3 evaluate`, `doctor`, and `health-check`.
Skill-owned `evals/config.yml` cannot set them. `docker` and `local` reject any
-environment kwargs. Harbor runtime-policy fields for resource overrides,
+environment kwargs, and `kata` accepts only `kata_runtime` and `kata_dns`. Harbor runtime-policy fields for resource overrides,
mounts, networks, extra Compose definitions, SkillEvaluator's own lifecycle
fields (`delete`, `force_build`, `keep_containers`), Harbor's `stream` and
`enable_environment_dir_upload` controls, and Kubernetes pod security are
diff --git a/docs/environment-variables.mdx b/docs/environment-variables.mdx
index 7cdcc5b2..6e84b70e 100644
--- a/docs/environment-variables.mdx
+++ b/docs/environment-variables.mdx
@@ -229,7 +229,7 @@ The `HARBOR_*` variables are read by your custom grader *inside* the task contai
## Set at runtime, not committed
<Warning>
-Keep credentials out of skill repositories and CI configuration files. Export provider keys as masked secrets at runtime. Agent credentials for Tier 3 come only from the host environment — `evals/config.yml` `runtime_env` carries non-credential values, and any entry that names or references an operator-owned credential (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `NVIDIA_API_KEY`, their base-URL variables, `CLAUDE_CODE_USE_BEDROCK`, and the `AWS_*` credential variables) is rejected at run time, as are names that control the host process (`PATH`, `DOCKER_*`, proxy variables, `SKILL_EVAL_*`, `SKILLEVALUATOR_*`, and similar). See the blocked-names table in [Providers & Credentials](configuration.mdx#tier-3-credentials-two-roles). With NVIDIA Build, one exported `NVIDIA_API_KEY` covers both the evaluator and the agents in the `docker` and `local` env-modes.
+Keep credentials out of skill repositories and CI configuration files. Export provider keys as masked secrets at runtime. Agent credentials for Tier 3 come only from the host environment — `evals/config.yml` `runtime_env` carries non-credential values, and any entry that names or references an operator-owned credential (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `NVIDIA_API_KEY`, their base-URL variables, `CLAUDE_CODE_USE_BEDROCK`, and the `AWS_*` credential variables) is rejected at run time, as are names that control the host process (`PATH`, `DOCKER_*`, proxy variables, `SKILL_EVAL_*`, `SKILLEVALUATOR_*`, and similar). See the blocked-names table in [Providers & Credentials](configuration.mdx#tier-3-credentials-two-roles). With NVIDIA Build, one exported `NVIDIA_API_KEY` covers both the evaluator and the agents in the `docker`, `kata`, and `local` env-modes.
</Warning>
## See also
diff --git a/docs/eval-datasets.mdx b/docs/eval-datasets.mdx
index ec1a04d0..078806de 100644
--- a/docs/eval-datasets.mdx
+++ b/docs/eval-datasets.mdx
@@ -294,7 +294,7 @@ documents. An eval entry's `files` string or list selects only those paths
| File | Rules |
| --- | --- |
| `Dockerfile` | At most 20 KB; the first non-comment line must be `FROM`. The build context is `environment/` itself, so `COPY` paths are relative to it (`COPY skills/`, not `COPY environment/skills/`). If absent, an image is auto-generated from `scripts/requirements.txt` and `scripts/apt-packages.txt` on a `python:3.12-slim` base. |
-| `docker-compose.yaml` | Sidecar services only (databases, APIs, mock servers) launched alongside the agent container. A strict allowlist applies: host port mappings are removed, and host mounts, privileged access, and unrecognized keys are rejected. Also accepted as `docker-compose.yml`. |
+| `docker-compose.yaml` | Sidecar services only (databases, APIs, mock servers) launched alongside the agent container. Not supported with `--env-mode kata`, where services cannot resolve each other by name. A strict allowlist applies: host port mappings are removed, and host mounts, privileged access, and unrecognized keys are rejected. Also accepted as `docker-compose.yml`. |
| `mcp_servers.toml` | MCP servers injected into the container. TOML array-of-tables `[[mcp_servers]]`; each entry needs `name` plus `url` or `command`. Command-based servers use `transport = "stdio"` (auto-inferred); URL-based servers default to `sse`. |
| Subdirectories | Copied into the container environment (config files, seed data for sidecars). Any name except `__pycache__` and `.git`. |
diff --git a/docs/quickstart.mdx b/docs/quickstart.mdx
index 8233972c..e474ec03 100644
--- a/docs/quickstart.mdx
+++ b/docs/quickstart.mdx
@@ -35,7 +35,7 @@ Default models are already configured for NVIDIA Build and OpenAI.
Get a key from [build.nvidia.com](https://build.nvidia.com/). That same key
covers evaluator chat, Tier 2 embeddings, and supported Tier 3 agents in
- Docker or local mode. No model or embedding settings are required.
+ Docker, Kata, or local mode. No model or embedding settings are required.
For OpenAI, use `SKILL_EVAL_LLM_PROVIDER=openai` and `OPENAI_API_KEY` instead;
the default Codex agent uses the same OpenAI key. Other providers and optional
diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdx
index d9acd954..64486c6b 100644
--- a/docs/tier3-live-evaluation.mdx
+++ b/docs/tier3-live-evaluation.mdx
@@ -7,7 +7,7 @@ description: "Run a real agent with and without your skill in a sandboxed enviro
Tier 3 answers the question the other tiers can't: **does your skill actually
make an agent better at the task?** It is the one tier that needs real
infrastructure — model access for the evaluator and selected agent, and
-Docker or another sandbox. NVIDIA Build in Docker or local mode uses one key
+Docker or another sandbox. NVIDIA Build in Docker, Kata, or local mode uses one key
for both roles; OpenAI with Codex can also share its key. This guide takes you from a skill directory to a
finished with-skill vs. without-skill comparison.
@@ -85,7 +85,7 @@ using Harbor 0.24:
- Installed Codex adapters merge existing user and MCP configuration before the
selected runtime provider and endpoint take precedence over conflicting route
keys.
-- Docker and local execution stream stdout and stderr through secret redaction
+- Docker, Kata, and local execution stream stdout and stderr through secret redaction
before Harbor callbacks receive them. Each command has a 16 MiB combined raw
stdout/stderr limit; exceeding it terminates the command process tree and
fails the trial instead of returning partial output.
@@ -367,9 +367,9 @@ Compose, proxy, or tracing controls. Configure the selected backend in the
host environment instead.
One deliberate exception collapses the two roles: with the `nv_build` provider
-in `--env-mode docker` or `local`, a single `NVIDIA_API_KEY` powers the
+in `--env-mode docker`, `kata`, or `local`, a single `NVIDIA_API_KEY` powers the
evaluator and all three agents. Direct OpenCode talks to NVIDIA Build directly;
-Codex and experimental Claude Code use a Docker or local compatibility bridge.
+Codex and experimental Claude Code use an in-container (Docker and Kata) or local compatibility bridge.
Explicit model overrides are preserved exactly. Without an override, all three
agents default to Nemotron Super (`nvidia/nemotron-3-super-120b-a12b`); direct
OpenCode renders it as `nvidia/nvidia/nemotron-3-super-120b-a12b` because its
@@ -394,7 +394,7 @@ explicit models — see [Agents & Sandboxes](agents-and-sandboxes.mdx).
## Where agents run
-`--env-mode` selects one of 24 environment modes: 23 Harbor-native backends,
+`--env-mode` selects one of 25 environment modes: 24 Harbor-native backends,
including the default Docker backend, plus SkillEvaluator's **local mode**,
which runs the agent CLI directly on the host under an OS-sandbox policy.
Managed backends such as `daytona`, `e2b`, and `modal` require their mapped
@@ -402,12 +402,13 @@ Harbor 0.24 extra; OpenShift, Apple Container, and Singularity instead use
system CLIs and have no Harbor Python extra. These are selectable integration
modes, not evidence that every managed backend was exercised end to end for
this release. Harbor 0.24 transport verification covers Docker and local;
-validate any managed backend's extra, credentials, infrastructure, and
+Kata mode reuses the Docker transport but has not been exercised on a KVM host.
+Validate any managed backend's extra, credentials, infrastructure, and
constructor settings with `doctor` and its Harbor preflight before relying on
it.
-Harbor's `cua-cloud`, `opensandbox`, `hf-sandbox`, `podman`, `kata`, `runta`,
-`prime`, `mosaic`, and `smol` backends are not selectable because they cannot
+Harbor's `cua-cloud`, `opensandbox`, `hf-sandbox`, `podman`, `runta`, `prime`,
+`mosaic`, and `smol` backends are not selectable because they cannot
yet receive SkillEvaluator's complete generated task bundle through a trusted
image or backend-native provisioning path.
@@ -415,7 +416,7 @@ Environment kwargs are only for Harbor-native non-Docker backends. Pass
non-secret constructor settings with the repeatable, operator-only
`--environment-kwarg KEY=VALUE` / `--ek KEY=VALUE` option. Skill-owned
`evals/config.yml` cannot set them. `docker` and `local` reject any environment
-kwargs. Fields owned by Harbor runtime policy are also reserved: resource
+kwargs, and `kata` accepts only `kata_runtime` and `kata_dns`. Fields owned by Harbor runtime policy are also reserved: resource
overrides, mounts, network policy, extra Compose definitions, SkillEvaluator's
own lifecycle fields (`delete`, `force_build`, `keep_containers`), and
Kubernetes pod security are not exposed through this surface. Other
@@ -750,7 +751,7 @@ machine-readable JSON contract, and how to judge whether a given lift is good.
Export the agent's credential in the host environment where you run
`skillevaluator` — agent credentials are read only from the host, never
from `evals/config.yml`. It is a separate role from the evaluator key,
- except with `nv_build` in Docker or local mode, where one
+ except with `nv_build` in Docker, Kata, or local mode, where one
`NVIDIA_API_KEY` covers both.
</Accordion>
<Accordion title="evaluate cannot start the environment">
diff --git a/src/skillevaluator/cli.py b/src/skillevaluator/cli.py
index 2b571e3e..a1890582 100644
--- a/src/skillevaluator/cli.py
+++ b/src/skillevaluator/cli.py
@@ -1877,6 +1877,7 @@ def validate(
view.tier_start(tier3_index)
env_note = {
"docker": "isolated containers per trial",
+ "kata": "isolated Kata microVMs per trial",
"local": "experimental host sandbox — trusted skills and workspaces only",
}.get(env_mode, "")
model_display = ", ".join(agent_model) if agent_model else (model or "agent defaults")
diff --git a/src/skillevaluator/reporting/benchmark.py b/src/skillevaluator/reporting/benchmark.py
index 57e14112..10f64f86 100644
--- a/src/skillevaluator/reporting/benchmark.py
+++ b/src/skillevaluator/reporting/benchmark.py
@@ -877,6 +877,8 @@ def _environment_note(environment: str | None) -> str | None:
return "Each task attempt ran in its own isolated sandbox."
if "k8s" in lowered or "sandbox" in lowered:
return "Each task attempt ran in its own isolated sandbox pod."
+ if lowered == "kata":
+ return "Each task attempt ran in its own Kata Containers microVM."
if "docker" in lowered:
return "Each task attempt ran in its own isolated Docker container."
if "local" in lowered:
diff --git a/src/skillevaluator/tier3/harbor/runner.py b/src/skillevaluator/tier3/harbor/runner.py
index 13d58a01..e4b1ad94 100644
--- a/src/skillevaluator/tier3/harbor/runner.py
+++ b/src/skillevaluator/tier3/harbor/runner.py
@@ -8,6 +8,7 @@ from __future__ import annotations
import codecs
import hashlib
import importlib.util
+import ipaddress
import json
import logging
import math
@@ -23,7 +24,7 @@ import tempfile
import threading
import time
import tomllib
-from collections.abc import Iterator, Mapping
+from collections.abc import Iterable, Iterator, Mapping
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, as_completed, wait
from contextlib import ExitStack, contextmanager, suppress
from dataclasses import dataclass, field
@@ -103,7 +104,10 @@ from skillevaluator.tier3.harbor.report_data import (
load_staged_harbor_dataset,
)
from skillevaluator.tier3.harbor.secure_copy import copytree_secure
-from skillevaluator.tier3.harbor.secure_docker_environment import SECURE_DOCKER_ENV_IMPORT_PATH
+from skillevaluator.tier3.harbor.secure_docker_environment import (
+ SECURE_DOCKER_ENV_IMPORT_PATH,
+ SECURE_KATA_ENV_IMPORT_PATH,
+)
from skillevaluator.tier3.harbor.sensitive_stdin import (
NVIDIA_BUILD_KEY_STDIN_ENV as _NVIDIA_BUILD_KEY_STDIN_ENV,
)
@@ -126,6 +130,7 @@ from skillevaluator.tier3.results_location import publish_latest_results
from skillevaluator.tier3_environments import (
DEFAULT_ENV_MODE,
ENV_MODE_LOCAL,
+ HARBOR_DOCKER_ENGINE_ENV_MODES,
HARBOR_ENV_MODES,
HARBOR_ENVIRONMENT_ALIAS_KWARGS,
HARBOR_ENVIRONMENT_EXTRAS,
@@ -761,6 +766,7 @@ _DOCKER_HOST_ENV_VARS = frozenset(
)
_HARBOR_ENV_MODE_VARS = {
"docker": _DOCKER_HOST_ENV_VARS,
+ "kata": _DOCKER_HOST_ENV_VARS,
"daytona": frozenset(
{
"DAYTONA_API_KEY",
@@ -1068,7 +1074,7 @@ def _nvidia_build_key_handoff(
subprocess_env.pop(_NVIDIA_BUILD_KEY_STDIN_ENV, None)
api_key = subprocess_env.get("NVIDIA_API_KEY", "")
if (
- env_mode == "docker"
+ env_mode in HARBOR_DOCKER_ENGINE_ENV_MODES
and subprocess_env.get("SKILL_EVAL_LLM_PROVIDER") == "nv_build"
and api_key
and api_key not in {_NVIDIA_BUILD_FILE_SENTINEL, _NVIDIA_BUILD_STDIN_SENTINEL}
@@ -1150,6 +1156,10 @@ _HARBOR_ENVIRONMENT_RUNTIME_POLICY_KWARGS: dict[str, frozenset[str]] = {
}
+_KATA_RUNTIME_NAME_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}")
+_KATA_DNS_SERVER_RE = re.compile(r"[0-9A-Fa-f:.]{2,45}")
+
+
def _environment_kwarg_policy_error(env_mode: str, environment_kwargs: Mapping[str, Any]) -> str | None:
if not environment_kwargs:
return None
@@ -1164,6 +1174,64 @@ def _environment_kwarg_policy_error(env_mode: str, environment_kwargs: Mapping[s
return f"Harbor {HARBOR_VERSION} environment '{env_mode}' does not accept environment kwarg(s): " + ", ".join(
unknown
)
+ if env_mode == "kata":
+ return _kata_environment_kwarg_error(environment_kwargs)
+ return None
+
+
+def _kata_environment_kwarg_error(environment_kwargs: Mapping[str, Any]) -> str | None:
+ """Keep Kata's runtime overlay pointed at a Kata handler and literal DNS servers."""
+ runtime = environment_kwargs.get("kata_runtime", "kata")
+ # Harbor treats Docker runtimes whose names contain "kata" as Kata handlers;
+ # any other handler, such as runc, would silently drop the microVM boundary.
+ if not isinstance(runtime, str) or "kata" not in runtime or not _KATA_RUNTIME_NAME_RE.fullmatch(runtime):
+ return "Harbor environment 'kata' requires kata_runtime to name a registered Kata runtime handler"
+ # Harbor writes each server into the microVM's resolv.conf. Parse them the
+ # way Harbor does (comma strings are split and trimmed) and allow only IP
+ # literals: an IPv6 zone ID (%...) could otherwise carry resolver directives.
+ if "kata_dns" not in environment_kwargs:
+ return None
+ dns = environment_kwargs["kata_dns"]
+ if isinstance(dns, str):
+ servers = [server.strip() for server in dns.split(",") if server.strip()]
+ elif isinstance(dns, list) and all(isinstance(server, str) for server in dns):
+ servers = dns
+ else:
+ return "Harbor environment 'kata' requires kata_dns to be a list of IP addresses"
+ # Without a server Harbor keeps Docker's embedded resolver, which a Kata VM cannot reach.
+ if not servers:
+ return "Harbor environment 'kata' requires kata_dns to list at least one IP address"
+ for server in servers:
+ if not _KATA_DNS_SERVER_RE.fullmatch(server):
+ return "Harbor environment 'kata' requires kata_dns to be a list of IP addresses"
+ try:
+ ipaddress.ip_address(server)
+ except ValueError:
+ return "Harbor environment 'kata' requires kata_dns to be a list of IP addresses"
+ return None
+
+
+def _staged_task_environment_error(env_mode: str, task_dirs: Iterable[Path]) -> str | None:
+ """Reject staged task environments the selected backend cannot run."""
+ if env_mode != "kata":
+ return None
+ import yaml
+
+ for task_dir in task_dirs:
+ compose_path = task_dir / "environment" / "docker-compose.yaml"
+ if not compose_path.is_file():
+ continue
+ try:
+ compose = yaml.safe_load(compose_path.read_text(encoding="utf-8"))
+ except (OSError, UnicodeDecodeError, yaml.YAMLError):
+ return f"Could not read the Docker Compose model of task '{task_dir.name}'"
+ services = compose.get("services") if isinstance(compose, dict) else None
+ if sidecars := sorted(str(name) for name in services or {} if name != "main"):
+ # Kata replaces each service's resolv.conf, so Compose service-name DNS stops working.
+ return (
+ f"Harbor environment 'kata' cannot run task '{task_dir.name}' with Compose sidecar services "
+ f"({', '.join(sidecars)}): services in Kata microVMs cannot resolve each other by name"
+ )
return None
@@ -1231,10 +1299,10 @@ def build_harbor_run_command(
timeout_multiplier = _validated_timeout_multiplier(timeout_multiplier)
if (
agent_import_path
- and env_mode not in {"docker", ENV_MODE_LOCAL}
+ and env_mode not in HARBOR_DOCKER_ENGINE_ENV_MODES | {ENV_MODE_LOCAL}
and agent_import_path not in _NATIVE_BACKEND_AGENT_IMPORT_PATHS
):
- raise ValueError("agent_import_path is supported only with --env docker or local")
+ raise ValueError("agent_import_path is supported only with --env docker, kata, or local")
validated_environment_kwargs = validate_environment_kwargs(
dict(environment_kwargs or {}),
env_mode=env_mode,
@@ -1290,11 +1358,11 @@ def build_harbor_run_command(
f"inherit_agent_keys={str(local_sandbox.coerce_flag(None, env_var=local_sandbox.INHERIT_AGENT_KEYS_ENV)).lower()}",
]
)
- elif env_mode == "docker":
+ elif env_mode in HARBOR_DOCKER_ENGINE_ENV_MODES:
from skillevaluator.tier3.harbor import DOCKER_AGENT_IMPORT_PATHS
command.extend(["--agent", agent_import_path or DOCKER_AGENT_IMPORT_PATHS.get(agent, agent)])
- command.extend(["--env", SECURE_DOCKER_ENV_IMPORT_PATH])
+ command.extend(["--env", SECURE_KATA_ENV_IMPORT_PATH if env_mode == "kata" else SECURE_DOCKER_ENV_IMPORT_PATH])
else:
command.extend(["--agent", agent_import_path or agent, "--env", harbor_environment_type(env_mode)])
for name, value in sorted(_harbor_agent_kwargs(agent, model, env_mode).items()):
@@ -1483,7 +1551,7 @@ def _validate_agent_provider_credentials(
"NVIDIA Build local agents require network access; unset SKILLEVALUATOR_LOCAL_ALLOW_NET or set it to 1."
]
- if env_mode in {"docker", ENV_MODE_LOCAL}:
+ if env_mode in HARBOR_DOCKER_ENGINE_ENV_MODES | {ENV_MODE_LOCAL}:
for agent in agents:
model = models.get(agent, "")
raw_model = model.removeprefix("nvidia/") if agent == "opencode" else model
@@ -1835,7 +1903,8 @@ def _check_prerequisites(
except ValueError as exc:
return [f"Invalid local runtime configuration: {exc}"]
- if env_mode == "docker":
+ if env_mode in HARBOR_DOCKER_ENGINE_ENV_MODES:
+ mode_label = "Kata" if env_mode == "kata" else "Docker"
try:
compose = subprocess.run(
["docker", "compose", "version"],
@@ -1849,7 +1918,7 @@ def _check_prerequisites(
exc,
secret_values=secret_values_from_environment(os.environ),
)
- return [f"Docker Compose v2 is required for Tier 3 Docker mode: {detail}"]
+ return [f"Docker Compose v2 is required for Tier 3 {mode_label} mode: {detail}"]
if compose.returncode != 0:
detail = (compose.stderr or compose.stdout).strip()
safe_detail = redact_progress_detail(
@@ -1857,7 +1926,7 @@ def _check_prerequisites(
secret_values=secret_values_from_environment(os.environ),
)
suffix = f": {safe_detail}" if safe_detail else ""
- return [f"Docker Compose v2 is required for Tier 3 Docker mode{suffix}"]
+ return [f"Docker Compose v2 is required for Tier 3 {mode_label} mode{suffix}"]
try:
from harbor.environments.factory import EnvironmentFactory
@@ -1866,6 +1935,13 @@ def _check_prerequisites(
if cwsandbox_errors := _cwsandbox_prerequisite_errors(env_mode):
return cwsandbox_errors
EnvironmentFactory.run_preflight(EnvironmentType(harbor_environment_type(env_mode)))
+ if env_mode == "kata":
+ from harbor.environments.kata import DEFAULT_KATA_RUNTIME, KataEnvironment
+
+ # Harbor's preflight only requires some Kata handler; require the selected one.
+ kata_runtime = (environment_kwargs or {}).get("kata_runtime", DEFAULT_KATA_RUNTIME)
+ if kata_runtime not in KataEnvironment._registered_kata_runtimes():
+ return [f"Harbor environment 'kata' requires Docker to register the Kata runtime {kata_runtime!r}."]
if env_mode == "ack":
ack_subprocess_env = (
dict(subprocess_env)
@@ -2133,7 +2209,7 @@ def _agent_credentials(
if env_mode == ENV_MODE_LOCAL:
return _local_agent_credentials(provider)
return {"NVIDIA_API_KEY": provider.api_key or ""}
- if env_mode in {"docker", ENV_MODE_LOCAL} and agent in {"claude-code", "codex"}:
+ if env_mode in HARBOR_DOCKER_ENGINE_ENV_MODES | {ENV_MODE_LOCAL} and agent in {"claude-code", "codex"}:
# The Docker bridge wrapper reads the evaluator credential from
# the Harbor parent handoff; the vendor CLI receives only a local
# sentinel and must not inherit NVIDIA_API_KEY in task env.
@@ -2212,11 +2288,7 @@ def _agent_provider_config(
if (
evaluator_provider.provider == "nv_build"
and agent == "claude-code"
- and env_mode
- not in {
- "docker",
- ENV_MODE_LOCAL,
- }
+ and env_mode not in HARBOR_DOCKER_ENGINE_ENV_MODES | {ENV_MODE_LOCAL}
):
resolved_model = model.removeprefix("anthropic/")
return ProviderConfig(
@@ -2229,11 +2301,7 @@ def _agent_provider_config(
if (
evaluator_provider.provider == "nv_build"
and agent == "codex"
- and env_mode
- not in {
- "docker",
- ENV_MODE_LOCAL,
- }
+ and env_mode not in HARBOR_DOCKER_ENGINE_ENV_MODES | {ENV_MODE_LOCAL}
):
resolved_model = model.removeprefix("openai/")
return ProviderConfig(
@@ -2487,7 +2555,7 @@ def _agent_import_path(provider: ProviderConfig, agent: str, env_mode: str) -> s
"""Select only the provider-specific wrappers required for this environment."""
if provider.provider == "openai-compatible" and agent == "codex" and env_mode != ENV_MODE_LOCAL:
return _GATEWAY_CODEX_IMPORT_PATH
- if provider.provider == "openai-compatible" and agent == "opencode" and env_mode == "docker":
+ if provider.provider == "openai-compatible" and agent == "opencode" and env_mode in HARBOR_DOCKER_ENGINE_ENV_MODES:
return "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorGatewayOpenCode"
return _nvidia_build_agent_import_path(provider, agent, env_mode)
@@ -2501,7 +2569,7 @@ def _nvidia_build_agent_import_path(provider: ProviderConfig, agent: str, env_mo
NVIDIA_BUILD_LOCAL_AGENT_IMPORT_PATHS,
)
- if env_mode == "docker":
+ if env_mode in HARBOR_DOCKER_ENGINE_ENV_MODES:
return NVIDIA_BUILD_AGENT_IMPORT_PATHS.get(agent)
if env_mode == ENV_MODE_LOCAL:
return NVIDIA_BUILD_LOCAL_AGENT_IMPORT_PATHS.get(agent)
@@ -4140,7 +4208,7 @@ def _run_harbor_eval_impl(
emitter = stage_native_harbor_tasks if task_source == "native_harbor" else generate_harbor_tasks
resource_config = harbor_config.get("resources", {})
- use_base_image = env_mode == "docker" and base_image_mode != "disabled"
+ use_base_image = env_mode in HARBOR_DOCKER_ENGINE_ENV_MODES and base_image_mode != "disabled"
base_image = ""
if use_base_image:
reporter.emit(
@@ -4209,6 +4277,8 @@ def _run_harbor_eval_impl(
evaluator_skill_path=evaluator_skill_path,
arm_suffix=with_arm_suffix,
)
+ if staged_environment_error := _staged_task_environment_error(env_mode, task_paths):
+ raise ValueError(staged_environment_error)
task_selectors = validate_case_ids(task.name for task in task_paths)
logical_case_ids = validate_case_ids(_native_entry_id(task) for task in task_paths)
case_id_by_task_selector = dict(zip(task_selectors, logical_case_ids, strict=True))
diff --git a/src/skillevaluator/tier3/harbor/runtime_preflight.py b/src/skillevaluator/tier3/harbor/runtime_preflight.py
index b402091e..2ab2be05 100644
--- a/src/skillevaluator/tier3/harbor/runtime_preflight.py
+++ b/src/skillevaluator/tier3/harbor/runtime_preflight.py
@@ -79,6 +79,7 @@ from skillevaluator.tier3.harbor.runner import (
_nvidia_build_key_handoff,
build_harbor_run_command,
)
+from skillevaluator.tier3_environments import HARBOR_DOCKER_ENGINE_ENV_MODES
if TYPE_CHECKING:
from skillevaluator.provider_config import ProviderConfig
@@ -1084,7 +1085,7 @@ def validate_harbor_agent_only_job_result(
f"Harbor agent-only trial {trial_result_path.parent.name} step {step_name!r} has no agent result"
)
- if env_mode == "docker" and not any(
+ if env_mode in HARBOR_DOCKER_ENGINE_ENV_MODES and not any(
_has_visible_artifact(agent_dir, root=job_dir) for agent_dir in agent_artifact_dirs
):
return False, (
diff --git a/src/skillevaluator/tier3/harbor/secure_docker_environment.py b/src/skillevaluator/tier3/harbor/secure_docker_environment.py
index f34bc03b..6bf23fdb 100644
--- a/src/skillevaluator/tier3/harbor/secure_docker_environment.py
+++ b/src/skillevaluator/tier3/harbor/secure_docker_environment.py
@@ -44,6 +44,7 @@ from harbor.environments.base import (
)
from harbor.environments.docker.docker import DockerEnvironment, _sanitize_docker_compose_project_name
from harbor.environments.docker.runtime import DOCKER_RUNTIME, ContainerRuntime
+from harbor.environments.kata import KataEnvironment
from skillevaluator.tier3.harbor.progress import secret_values_from_environment
from skillevaluator.tier3.harbor.secure_copy import (
@@ -65,6 +66,9 @@ if TYPE_CHECKING:
SECURE_DOCKER_ENV_IMPORT_PATH = (
"skillevaluator.tier3.harbor.secure_docker_environment:SkillEvaluatorSecureDockerEnvironment"
)
+SECURE_KATA_ENV_IMPORT_PATH = (
+ "skillevaluator.tier3.harbor.secure_docker_environment:SkillEvaluatorSecureKataEnvironment"
+)
_ENV_NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
_ENV_NAME_PREFIX_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*")
@@ -3527,3 +3531,15 @@ class SkillEvaluatorSecureDockerEnvironment(SkillEvaluatorDockerEnvironment):
timeout_sec=timeout_sec,
user=user,
)
+
+
+class SkillEvaluatorSecureKataEnvironment(SkillEvaluatorSecureDockerEnvironment, KataEnvironment):
+ """Hardened Docker orchestration with every task service in a Kata microVM.
+
+ Harbor's Kata backend changes only the Compose model: a final overlay sets
+ ``runtime:`` on every task service and binds a literal ``resolv.conf``.
+ The hardened Docker runner already passes every ``_docker_compose_paths``
+ entry to Compose and keeps its argv, credential handoff, and containment
+ guarantees, so this class only combines the two. Harbor's Kata preflight
+ and its rejection of non-public network policies stay in force.
+ """
diff --git a/src/skillevaluator/tier3_environments.py b/src/skillevaluator/tier3_environments.py
index 60f1a300..af0bf32f 100644
--- a/src/skillevaluator/tier3_environments.py
+++ b/src/skillevaluator/tier3_environments.py
@@ -15,6 +15,7 @@ HARBOR_VERSION = "0.24.0"
HARBOR_ENVIRONMENTS = (
"docker",
+ "kata",
"daytona",
"e2b",
"modal",
@@ -45,6 +46,9 @@ HARBOR_ENVIRONMENTS = (
HARBOR_ENV_MODES = frozenset(HARBOR_ENVIRONMENTS)
#: Public env modes that Harbor accepts natively via ``--env`` (everything except ``local``).
HARBOR_NATIVE_ENV_MODES = frozenset(m for m in HARBOR_ENVIRONMENTS if m != "local")
+#: Modes that run through SkillEvaluator's hardened Docker Compose backend on the
+#: host's Docker engine. ``kata`` adds Harbor's Kata Containers runtime overlay.
+HARBOR_DOCKER_ENGINE_ENV_MODES = frozenset({"docker", "kata"})
#: SkillEvaluator modes that run as a different Harbor environment type. Harbor
#: 0.24 merged its W&B sandbox into ``cwsandbox``, selected with ``auth=wandb``.
HARBOR_ENVIRONMENT_TYPE_ALIASES: dict[str, str] = {"wandb": "cwsandbox"}
@@ -59,10 +63,11 @@ def harbor_environment_type(env_mode: str) -> str:
# Exact Harbor ``Provides-Extra`` names. ``ack`` reuses the Kubernetes
# dependencies supplied by ``gke``, and ``wandb`` the W&B-capable ``cwsandbox``
-# extra. Of the four ``None`` entries, Docker needs no additional Python extra
-# and the other three are system-CLI backends.
+# extra. Of the five ``None`` entries, Docker and Kata need no additional Python
+# extra and the other three are system-CLI backends.
HARBOR_ENVIRONMENT_EXTRAS: dict[str, str | None] = {
"docker": None,
+ "kata": None,
"daytona": "daytona",
"e2b": "e2b",
"modal": "modal",
@@ -91,8 +96,8 @@ HARBOR_ENVIRONMENT_EXTRAS: dict[str, str | None] = {
# importing the base SkillEvaluator CLI never imports Harbor or optional provider
# SDKs. The packaging parity test AST-reads the pinned Harbor sources and catches
# additions, removals, and provider kwargs that are consumed through **kwargs.
-# Registry-only entries (cua-cloud, opensandbox, hf-sandbox, podman, kata, runta,
-# prime, mosaic, smol) keep that parity exact; those modes are not exposed.
+# Registry-only entries (cua-cloud, opensandbox, hf-sandbox, podman, runta, prime,
+# mosaic, smol) keep that parity exact; those modes are not exposed.
_HARBOR_BASE_ENVIRONMENT_KWARGS = frozenset(
{
"cpu_enforcement_policy",
diff --git a/tests/golden/cli_surface.json b/tests/golden/cli_surface.json
index d51f092a..31945d8b 100644
--- a/tests/golden/cli_surface.json
+++ b/tests/golden/cli_surface.json
@@ -252,6 +252,7 @@
{
"choices": [
"docker",
+ "kata",
"daytona",
"e2b",
"modal",
@@ -337,6 +338,7 @@
{
"choices": [
"docker",
+ "kata",
"daytona",
"e2b",
"modal",
@@ -641,6 +643,7 @@
{
"choices": [
"docker",
+ "kata",
"daytona",
"e2b",
"modal",
@@ -2105,6 +2108,7 @@
{
"choices": [
"docker",
+ "kata",
"daytona",
"e2b",
"modal",
@@ -2190,6 +2194,7 @@
{
"choices": [
"docker",
+ "kata",
"daytona",
"e2b",
"modal",
@@ -2693,6 +2698,7 @@
{
"choices": [
"docker",
+ "kata",
"daytona",
"e2b",
"modal",
@@ -3209,6 +3215,7 @@
{
"choices": [
"docker",
+ "kata",
"daytona",
"e2b",
"modal",
diff --git a/tests/test_harbor_environment_contract.py b/tests/test_harbor_environment_contract.py
index c85293cb..307f0259 100644
--- a/tests/test_harbor_environment_contract.py
+++ b/tests/test_harbor_environment_contract.py
@@ -11,8 +11,9 @@ from pathlib import Path
import pytest
-from skillevaluator.tier3.harbor import runner
+from skillevaluator.tier3.harbor import DOCKER_AGENT_IMPORT_PATHS, runner
from skillevaluator.tier3.harbor.runner import build_harbor_run_command
+from skillevaluator.tier3.harbor.secure_docker_environment import NVIDIA_BUILD_STDIN_SENTINEL
def _environment_kwargs(command: list[str]) -> list[str]:
@@ -85,6 +86,170 @@ def test_backend_kwargs_track_the_pinned_harbor_release() -> None:
assert f"dind_image={json.dumps('docker:dind')}" in _environment_kwargs(tensorlake)
+def test_kata_runs_the_hardened_kata_backend() -> None:
+ from skillevaluator.tier3.harbor.secure_docker_environment import SECURE_KATA_ENV_IMPORT_PATH
+
+ command = build_harbor_run_command(
+ dataset_path="/tmp/dataset",
+ agent="codex",
+ job_name="kata",
+ env_mode="kata",
+ environment_kwargs={"kata_runtime": "kata-clh"},
+ )
+
+ assert command[command.index("--env") + 1] == SECURE_KATA_ENV_IMPORT_PATH
+ # Kata runs the Docker agent wrappers, which skip the redundant Node/npm bootstrap.
+ assert command[command.index("--agent") + 1] == DOCKER_AGENT_IMPORT_PATHS["codex"]
+ assert command[command.index("--ak") + 1] == f"reasoning_effort={json.dumps('high')}"
+ assert _environment_kwargs(command) == [f"kata_runtime={json.dumps('kata-clh')}"]
+
+
+ ("environment_kwargs", "message"),
+ [
+ ({"keep_containers": True}, "reserved for Harbor runtime policy"),
+ ({"privileged": True}, "does not accept environment kwarg"),
+ ({"kata_runtime": "runc"}, "kata_runtime"),
+ ({"kata_runtime": "kata --privileged"}, "kata_runtime"),
+ ({"kata_runtime": 7}, "kata_runtime"),
+ ({"kata_dns": "1.1.1.1\noptions ndots:15"}, "kata_dns"),
+ ({"kata_dns": ["dns.example"]}, "kata_dns"),
+ ({"kata_dns": [" 1.1.1.1"]}, "kata_dns"),
+ ({"kata_dns": {"server": "1.1.1.1"}}, "kata_dns"),
+ ({"kata_dns": "fe80::1%eth0\nsearch attacker.example"}, "kata_dns"),
+ ({"kata_dns": ["fe80::1%eth0\noptions ndots:15"]}, "kata_dns"),
+ ({"kata_dns": ["fe80::1%eth0"]}, "kata_dns"),
+ ({"kata_dns": ""}, "at least one IP address"),
+ ({"kata_dns": ","}, "at least one IP address"),
+ ({"kata_dns": []}, "at least one IP address"),
+ ({"kata_dns": None}, "kata_dns"),
+ ],
+)
+def test_kata_kwargs_keep_the_microvm_boundary(environment_kwargs: dict[str, object], message: str) -> None:
+ with pytest.raises(ValueError, match=message):
+ build_harbor_run_command(
+ dataset_path="/tmp/dataset",
+ agent="opencode",
+ job_name="kata",
+ env_mode="kata",
+ environment_kwargs=environment_kwargs,
+ )
+
+
[email protected]("dns", ["1.1.1.1, 2606:4700:4700::1111", ["9.9.9.9"], ["10.0.0.2", "::1"]])
+def test_kata_accepts_literal_dns_servers(dns: object) -> None:
+ command = build_harbor_run_command(
+ dataset_path="/tmp/dataset",
+ agent="opencode",
+ job_name="kata",
+ env_mode="kata",
+ environment_kwargs={"kata_dns": dns},
+ )
+
+ assert _environment_kwargs(command) == [f"kata_dns={json.dumps(dns, separators=(',', ':'))}"]
+
+
+def test_docker_still_rejects_environment_kwargs() -> None:
+ with pytest.raises(ValueError, match="not supported for SkillEvaluator Docker mode"):
+ build_harbor_run_command(
+ dataset_path="/tmp/dataset",
+ agent="opencode",
+ job_name="docker",
+ env_mode="docker",
+ environment_kwargs={"kata_runtime": "kata"},
+ )
+
+
[email protected](("registered", "ready"), [(["kata", "kata-clh"], True), (["kata-qemu"], False)])
+def test_kata_prerequisites_require_the_selected_runtime(
+ monkeypatch: pytest.MonkeyPatch,
+ registered: list[str],
+ ready: bool,
+) -> None:
+ from harbor.environments.factory import EnvironmentFactory
+ from harbor.environments.kata import KataEnvironment
+ from harbor.models.environment_type import EnvironmentType
+
+ preflights: list[object] = []
+ monkeypatch.setattr(
+ EnvironmentFactory, "run_preflight", lambda environment_type, **_kwargs: preflights.append(environment_type)
+ )
+ monkeypatch.setattr(KataEnvironment, "_registered_kata_runtimes", classmethod(lambda _cls: registered))
+ monkeypatch.setattr(
+ runner.subprocess,
+ "run",
+ lambda *_args, **_kwargs: runner.subprocess.CompletedProcess(
+ [], 0, stdout="Docker Compose version v2", stderr=""
+ ),
+ )
+
+ errors = runner._check_prerequisites(
+ env_mode="kata", agents=["opencode"], environment_kwargs={"kata_runtime": "kata-clh"}
+ )
+
+ assert preflights == [EnvironmentType.KATA]
+ if ready:
+ assert errors == []
+ else:
+ assert errors == ["Harbor environment 'kata' requires Docker to register the Kata runtime 'kata-clh'."]
+
+
+def test_kata_default_runtime_must_be_registered(monkeypatch: pytest.MonkeyPatch) -> None:
+ from harbor.environments.factory import EnvironmentFactory
+ from harbor.environments.kata import KataEnvironment
+
+ monkeypatch.setattr(EnvironmentFactory, "run_preflight", lambda _environment_type, **_kwargs: None)
+ monkeypatch.setattr(KataEnvironment, "_registered_kata_runtimes", classmethod(lambda _cls: ["kata-clh"]))
+ monkeypatch.setattr(
+ runner.subprocess,
+ "run",
+ lambda *_args, **_kwargs: runner.subprocess.CompletedProcess(
+ [], 0, stdout="Docker Compose version v2", stderr=""
+ ),
+ )
+
+ assert runner._check_prerequisites(env_mode="kata", agents=["opencode"]) == [
+ "Harbor environment 'kata' requires Docker to register the Kata runtime 'kata'."
+ ]
+
+
+def test_kata_rejects_compose_sidecars_before_harbor_starts(tmp_path: Path) -> None:
+ def task(name: str, compose: str | None) -> Path:
+ environment = tmp_path / name / "environment"
+ environment.mkdir(parents=True)
+ if compose is not None:
+ (environment / "docker-compose.yaml").write_text(compose, encoding="utf-8")
+ return tmp_path / name
+
+ dockerfile_only = task("dockerfile-only", None)
+ main_only = task("main-only", "services:\n main:\n environment:\n MODE: test\n")
+ sidecar = task("with-db", "services:\n main: {}\n db:\n image: postgres:16\n")
+
+ assert runner._staged_task_environment_error("kata", [dockerfile_only, main_only]) is None
+ error = runner._staged_task_environment_error("kata", [dockerfile_only, sidecar])
+ assert error is not None
+ assert "task 'with-db'" in error
+ assert "(db)" in error
+ assert runner._staged_task_environment_error("docker", [sidecar]) is None
+
+
+def test_kata_runs_are_described_as_microvm_attempts() -> None:
+ from skillevaluator.reporting.benchmark import _environment_note
+
+ assert _environment_note("kata") == "Each task attempt ran in its own Kata Containers microVM."
+ assert _environment_note("docker") == "Each task attempt ran in its own isolated Docker container."
+
+
+def test_kata_uses_the_docker_nvidia_build_handoff_and_host_environment() -> None:
+ assert runner._HARBOR_ENV_MODE_VARS["kata"] == runner._HARBOR_ENV_MODE_VARS["docker"]
+ handoff = runner._nvidia_build_key_handoff(
+ {"NVIDIA_API_KEY": "nvapi-kata-handoff", "SKILL_EVAL_LLM_PROVIDER": "nv_build"},
+ env_mode="kata",
+ )
+ assert handoff.subprocess_env["NVIDIA_API_KEY"] == NVIDIA_BUILD_STDIN_SENTINEL
+ assert handoff.stdin_text == "nvapi-kata-handoff"
+
+
def test_cwsandbox_prerequisites_require_sdk_and_api_key(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(runner.importlib.util, "find_spec", lambda name: None if name == "cwsandbox" else object())
assert "harbor[cwsandbox]==0.24.0" in runner._cwsandbox_prerequisite_errors("cwsandbox")[0]
diff --git a/tests/test_harbor_local_mode.py b/tests/test_harbor_local_mode.py
index 494fb80d..7afc2297 100644
--- a/tests/test_harbor_local_mode.py
+++ b/tests/test_harbor_local_mode.py
@@ -465,7 +465,6 @@ def test_registered_native_env_modes_are_supported_subset_of_pinned_harbor_relea
"opensandbox",
"hf-sandbox",
"podman",
- "kata",
"runta",
"prime",
"mosaic",
diff --git a/tests/test_harbor_runner_environment.py b/tests/test_harbor_runner_environment.py
index e1382205..a05a0e30 100644
--- a/tests/test_harbor_runner_environment.py
+++ b/tests/test_harbor_runner_environment.py
@@ -1863,6 +1863,9 @@ def test_harbor_backend_environment_allowlist_covers_every_native_022_mode() ->
"vercel": {"VERCEL_OIDC_TOKEN", "VERCEL_PROJECT_ID", "VERCEL_TEAM_ID", "VERCEL_TOKEN"},
}
+ # Kata runs on the host's Docker engine, so it selects exactly Docker's variables.
+ expected["kata"] = expected["docker"]
+
assert set(expected) == HARBOR_NATIVE_ENV_MODES
assert {mode: set(names) for mode, names in runner._HARBOR_ENV_MODE_VARS.items()} == expected
diff --git a/tests/test_harbor_secure_docker_environment.py b/tests/test_harbor_secure_docker_environment.py
index 87d588a1..15cda13e 100644
--- a/tests/test_harbor_secure_docker_environment.py
+++ b/tests/test_harbor_secure_docker_environment.py
@@ -38,6 +38,7 @@ from skillevaluator.tier3.harbor.secure_docker_environment import (
SECURE_DOCKER_ENV_IMPORT_PATH,
SkillEvaluatorDockerEnvironment,
SkillEvaluatorSecureDockerEnvironment,
+ SkillEvaluatorSecureKataEnvironment,
_collision_safe_redaction_marker,
_compose_client_credential_values,
_compose_interpolation_names,
@@ -8501,6 +8502,102 @@ def test_secure_docker_disables_ssh_streaming_and_gpu_reservations(tmp_path: Pat
)
+def _initialized_secure_kata_environment(tmp_path: Path, **kwargs: object) -> SkillEvaluatorSecureKataEnvironment:
+ environment_dir = tmp_path / "kata-environment"
+ environment_dir.mkdir(exist_ok=True)
+ (environment_dir / "Dockerfile").write_text("FROM scratch\n", encoding="utf-8")
+ return SkillEvaluatorSecureKataEnvironment(
+ environment_dir=environment_dir,
+ environment_name="secure-kata-test",
+ session_id="secure-kata-test",
+ trial_paths=TrialPaths(tmp_path / "kata-trial"),
+ task_env_config=EnvironmentConfig(),
+ **kwargs,
+ )
+
+
+def test_secure_kata_keeps_docker_hardening_under_the_kata_runtime(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ from harbor.environments.docker.runtime import DOCKER_RUNTIME
+ from harbor.environments.podman import PodmanEnvironment
+ from harbor.models.environment_type import EnvironmentType
+
+ monkeypatch.setattr(DockerEnvironment, "runtime", classmethod(lambda _cls: PodmanEnvironment.runtime()))
+ environment = _initialized_secure_kata_environment(tmp_path, kata_runtime="kata-clh", kata_dns=["9.9.9.9"])
+
+ assert environment.type() == EnvironmentType.KATA
+ assert SkillEvaluatorSecureKataEnvironment.runtime() is DOCKER_RUNTIME
+ # Plain Harbor KataEnvironment also reports stream/gpus/windows as False, so
+ # prove the hardened methods are the ones that run.
+ hardened_owners = {
+ name: next(cls for cls in SkillEvaluatorSecureKataEnvironment.__mro__ if name in cls.__dict__)
+ for name in (
+ "preflight",
+ "start",
+ "exec",
+ "download_file",
+ "download_dir",
+ "capabilities",
+ "runtime",
+ "_run_docker_compose_command",
+ )
+ }
+ assert hardened_owners["exec"] is SkillEvaluatorSecureDockerEnvironment
+ assert all(
+ owner in {SkillEvaluatorSecureDockerEnvironment, SkillEvaluatorDockerEnvironment}
+ for owner in hardened_owners.values()
+ ), hardened_owners
+ assert environment.capabilities.stream is False
+ assert environment.capabilities.gpus is False
+ assert environment.capabilities.windows is False
+ # Kata VMs cannot share the egress sidecar's network namespace, so Harbor
+ # rejects non-public network policies instead of running them unenforced.
+ assert environment._requires_egress_control() is False
+ overlay = json.loads(environment._docker_compose_paths[-1].read_text(encoding="utf-8"))
+ assert overlay["services"]["main"]["runtime"] == "kata-clh"
+ assert overlay["services"]["main"]["volumes"][0]["target"] == "/etc/resolv.conf"
+ assert "main" in environment._compose_model_metadata()[1]
+
+
+def test_secure_kata_preflight_consumes_the_handoff_before_kata_checks(monkeypatch: pytest.MonkeyPatch) -> None:
+ from harbor.environments.kata import KataEnvironment
+
+ calls: list[str] = []
+ monkeypatch.setenv("NVIDIA_API_KEY", NVIDIA_BUILD_STDIN_SENTINEL)
+ monkeypatch.setattr(
+ "skillevaluator.tier3.harbor.secure_docker_environment.read_nvidia_build_key_from_stdin",
+ lambda: calls.append("handoff"),
+ )
+ monkeypatch.setattr(DockerEnvironment, "preflight", classmethod(lambda _cls: calls.append("docker")))
+ monkeypatch.setattr(KataEnvironment, "_host_is_linux", staticmethod(lambda: False))
+
+ with pytest.raises(SystemExit, match="Linux host with KVM"):
+ SkillEvaluatorSecureKataEnvironment.preflight()
+ assert calls == ["handoff", "docker"]
+
+
+def test_secure_kata_compose_commands_force_the_runtime_overlay(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ environment = _initialized_secure_kata_environment(tmp_path)
+ commands: list[tuple[str, ...]] = []
+
+ async def create_subprocess(*args: object, **_kwargs: object) -> _BufferedComposeProcess:
+ commands.append(tuple(str(argument) for argument in args))
+ return _BufferedComposeProcess(stdout=b"")
+
+ monkeypatch.setattr(asyncio, "create_subprocess_exec", create_subprocess)
+
+ asyncio.run(environment.download_file("/app/result.txt", tmp_path / "result.txt"))
+
+ compose_files = [command[index + 1] for command in commands for index, value in enumerate(command) if value == "-f"]
+ assert compose_files[-1] == str(environment._docker_compose_paths[-1].resolve())
+ assert compose_files[-1].endswith("docker-compose-kata-runtime.json")
+
+
@pytest.mark.parametrize("kind", ["file", "dir"])
def test_secure_docker_main_downloads_use_compose_copy_only(
tmp_path: Path,
diff --git a/tests/test_oss_packaging.py b/tests/test_oss_packaging.py
index dd412406..4631d3a6 100644
--- a/tests/test_oss_packaging.py
+++ b/tests/test_oss_packaging.py
@@ -180,10 +180,10 @@ def test_harbor_dependency_contract_keeps_base_install_isolated() -> None:
def test_harbor_environment_extra_mapping_matches_installed_metadata() -> None:
provided_extras = set(metadata("harbor").get_all("Provides-Extra") or ())
- system_or_base_backends = {"docker", "openshift", "apple-container", "singularity"}
+ system_or_base_backends = {"docker", "kata", "openshift", "apple-container", "singularity"}
- assert len(HARBOR_ENVIRONMENTS) == 24
- assert len(HARBOR_NATIVE_ENV_MODES) == 23
+ assert len(HARBOR_ENVIRONMENTS) == 25
+ assert len(HARBOR_NATIVE_ENV_MODES) == 24
assert frozenset(HARBOR_ENVIRONMENTS) - {"local"} == HARBOR_NATIVE_ENV_MODES
assert set(HARBOR_ENVIRONMENT_EXTRAS) == HARBOR_NATIVE_ENV_MODES
assert {mode for mode, extra in HARBOR_ENVIRONMENT_EXTRAS.items() if extra is None} == system_or_base_backends
@@ -317,8 +317,8 @@ def test_public_docs_match_harbor_environment_and_kwarg_contract() -> None:
public_docs = f"{agents}\n{cli_reference}\n{configuration}\n{tier3}\n{eval_config}"
normalized_docs = " ".join(public_docs.split())
- assert "24 environment modes" in public_docs
- assert "23 Harbor-native backends" in public_docs
+ assert "25 environment modes" in public_docs
+ assert "24 Harbor-native backends" in public_docs
assert "All 16 values" not in public_docs
assert "same 16 values" not in public_docs
assert "14 additional Harbor-native backends" not in public_docs
@@ -809,7 +809,7 @@ def test_public_docs_show_external_nvidia_build_harness_paths_only() -> None:
assert "skillevaluator tier3 evaluate ./my-skill --agents claude-code --env-mode docker\n" in tier3
assert "Explicit model overrides are preserved exactly" in public_docs
assert "OpenCode renders it as" in public_docs
- assert "Docker or local compatibility bridge" in public_docs
+ assert "in-container (Docker and Kata) or local compatibility bridge" in public_docs
assert "experimental Claude Code" in public_docs
diff --git a/tests/test_tier3_config_key_parity.py b/tests/test_tier3_config_key_parity.py
index a9a8b67e..14909ad8 100644
--- a/tests/test_tier3_config_key_parity.py
+++ b/tests/test_tier3_config_key_parity.py
@@ -367,6 +367,27 @@ harbor:
assert result["run_config"]["harbor"]["base_image_mode"] == mode
+def test_kata_uses_the_shared_docker_base_image(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
+ config = """\
+schema_version: 1
+harbor:
+ task_source: evals_json
+ base_image_mode: reuse
+"""
+ builds: list[dict[str, Any]] = []
+ monkeypatch.setattr(
+ runner,
+ "build_eval_base_image",
+ lambda *_args, **kwargs: builds.append(kwargs) or "skillevaluator-base:test",
+ )
+
+ result, captured = _run_engine(monkeypatch, tmp_path, config, env_mode="kata")
+
+ assert "error" not in result
+ assert len(builds) == 1
+ assert captured["emit"]["base_image"] == "skillevaluator-base:test"
+
+
def test_base_image_mode_defaults_to_self_contained_tasks(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
diff --git a/tests/test_tier3_public_runtime.py b/tests/test_tier3_public_runtime.py
index 2be92816..476cb999 100644
--- a/tests/test_tier3_public_runtime.py
+++ b/tests/test_tier3_public_runtime.py
@@ -618,6 +618,14 @@ def test_nvidia_build_agent_import_selection_includes_local_bridge_agents() -> N
"skillevaluator.tier3.harbor.local_agents:SkillEvaluatorNvidiaBuildClaudeCode"
)
assert _nvidia_build_agent_import_path(provider, "opencode", "docker") is None
+ assert _nvidia_build_agent_import_path(provider, "codex", "kata") == (
+ "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorNvidiaBuildCodex"
+ )
+ assert _nvidia_build_agent_import_path(provider, "claude-code", "kata") == (
+ "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorNvidiaBuildClaudeCode"
+ )
+ assert _nvidia_build_agent_import_path(provider, "opencode", "kata") is None
+ assert _nvidia_build_agent_import_path(provider, "codex", "daytona") is None
assert _nvidia_build_agent_import_path(provider, "codex", "local") == (
"skillevaluator.tier3.harbor.local_agents:SkillEvaluatorLocalNvidiaBuildCodex"
)
@@ -689,7 +697,7 @@ def test_custom_agent_import_path_preserves_native_cloud_environment(env_mode: s
def test_custom_agent_import_path_is_rejected_for_native_cloud() -> None:
- with pytest.raises(ValueError, match="agent_import_path is supported only with --env docker or local"):
+ with pytest.raises(ValueError, match="agent_import_path is supported only with --env docker, kata, or local"):
build_harbor_run_command(
dataset_path="/tmp/dataset",
agent="codex",
diff --git a/tests/tier3/test_gateway_codex.py b/tests/tier3/test_gateway_codex.py
index 1ae55700..352d063e 100644
--- a/tests/tier3/test_gateway_codex.py
+++ b/tests/tier3/test_gateway_codex.py
@@ -30,7 +30,7 @@ def test_gateway_adapter_selection_preserves_other_routes(provider, mode, agent)
expected = runner._nvidia_build_agent_import_path(config, agent, mode)
if provider == "openai-compatible" and mode != "local" and agent == "codex":
expected = "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorGatewayCodex"
- if provider == "openai-compatible" and mode == "docker" and agent == "opencode":
+ if provider == "openai-compatible" and mode in {"docker", "kata"} and agent == "opencode":
expected = "skillevaluator.tier3.harbor.local_agents:SkillEvaluatorGatewayOpenCode"
assert runner._agent_import_path(config, agent, mode) == expected