Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier3): stop per-trial apt installs in Docker agent setup (#413)
Narendran Raghavan882ec054d9eb
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 384b071e..c79b32fc 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -439,6 +439,22 @@ All notable changes to SkillEvaluator are documented in this file.- PII findings clarify that required author attribution is accepted whileduplicate contact addresses need removal, replacement, or a reviewed waiver.+- Docker OpenCode containers no longer get the distribution `nodejs` and+ `npm` packages from agent setup, except on Alpine images, and generated+ images now include `procps`. Before, OpenCode, the default agent with NVIDIA+ Build, installed Debian's Node.js and npm with `apt-get` in every trial and+ then its own Node.js with nvm, and Claude Code installed `procps` (for+ `pgrep`) in every trial. Both took time from the 360-second agent setup+ timeout, and a package mirror error failed the trial. Like Claude Code and+ Codex containers since 0.6.0, OpenCode containers on Debian-based images now+ have no `gcc`, `g++`, or `make`, and OpenCode's Node.js is on `PATH` only for+ the OpenCode process and the commands it runs, including its stdio MCP+ servers. Skills whose scripts, tasks, or verifier need these tools, or whose+ stdio MCP servers need `gcc`, `g++`, or `make`, must install them through+ `scripts/apt-packages.txt` or a custom `evals/environment/Dockerfile`.+ Generated images rebuild once for the new package list: the per-task images,+ or the shared base image when `harbor.base_image_mode` is `reuse`.+- A `tier3 evaluate --skip-with-skill` run with two or more agents now recordsthe dataset-size status in `result.json` and the `tier3 evaluate` summary. Itis taken from the agent with the best baseline score, as in the reports.diff --git a/docs/eval-datasets.mdx b/docs/eval-datasets.mdxindex d870136c..922ca07a 100644--- a/docs/eval-datasets.mdx+++ b/docs/eval-datasets.mdx@@ -451,16 +451,28 @@ transport = "stdio"```<Note>- In Docker mode, Claude Code and Codex setup no longer installs the+ In Docker mode, Claude Code, Codex, and OpenCode setup no longer installs thedistribution `nodejs` and `npm` packages, except on Alpine images. OnDebian-based images, including the default `python:3.12-slim` image, thosepackages also brought `npx` and a C/C++ toolchain (`gcc`, `g++`, `make`).Claude Code containers now have no Node.js unless the image provides it.- Codex installs its own Node.js, which is on `PATH` only for the Codex process- and the commands it runs. If your skill's scripts, its tasks, or a stdio MCP- server (for example `command = "npx ..."`) need these tools, install them in- the image: list the packages in `scripts/apt-packages.txt`, one per line, for- the auto-generated image, or install them in `evals/environment/Dockerfile`.+ Codex and OpenCode install their own Node.js with nvm. Its `node`, `npm`, and+ `npx` are on `PATH` only for the agent process and the commands it runs,+ including stdio MCP servers, not for the verifier (`test.sh`) or task+ healthchecks. If your skill's scripts, its tasks, or the verifier need these+ tools, or a stdio MCP server does (Node.js with Claude Code, for example+ `command = "npx ..."`, or `gcc`, `g++`, or `make` with any agent), install+ them in the image: list the packages in `scripts/apt-packages.txt`, one per+ line, for the auto-generated image, or install them in+ `evals/environment/Dockerfile`.++ The auto-generated image and the `init-harbor-task` starter Dockerfile also+ include `procps`, so Claude Code setup no longer installs it (for `pgrep`)+ with `apt-get` in every trial. A custom `evals/environment/Dockerfile` keeps+ its own base image, so it should install `procps` too, unless+ `harbor.base_image_mode` is `reuse` or `rebuild` and+ `harbor.custom_dockerfile_mode` is `rebase`. Then it is rebased onto the+ shared base image, which has `procps`.</Note>```text title="my-skill/scripts/apt-packages.txt"diff --git a/src/skillevaluator/tier3/commands.py b/src/skillevaluator/tier3/commands.pyindex 036bf536..4423e527 100644--- a/src/skillevaluator/tier3/commands.py+++ b/src/skillevaluator/tier3/commands.py@@ -624,7 +624,7 @@ skills_dir = {toml_quote("/workspace/skills")}"""FROM python:3.12-slimRUN apt-get -o Acquire::Retries=3 update && \\- apt-get -o Acquire::Retries=3 install -y --no-install-recommends bash curl git jq ripgrep && \\+ apt-get -o Acquire::Retries=3 install -y --no-install-recommends bash curl git jq procps ripgrep && \\rm -rf /var/lib/apt/lists/*RUN mkdir -p /workspace/skills /logs/agent /logs/verifierdiff --git a/src/skillevaluator/tier3/harbor/__init__.py b/src/skillevaluator/tier3/harbor/__init__.pyindex e406c930..927aea84 100644--- a/src/skillevaluator/tier3/harbor/__init__.py+++ b/src/skillevaluator/tier3/harbor/__init__.py@@ -44,11 +44,13 @@ LOCAL_CODEX_HOME_DIRNAME = "codex-home"LOCAL_CODEX_SECRETS_DIRNAME = "codex-secrets"# Docker wrappers retain Harbor's installers while avoiding redundant Node/npm-# packages on the branches that bootstrap their own runtime. Claude Code uses the+# packages on the branches that bootstrap their own runtime. Codex and OpenCode+# install their own Node with NVM on glibc images. Claude Code uses the# container wrapper, which builds on that installer fix and adds Vertex AI# routing and rich MCP server declarations.DOCKER_AGENT_IMPORT_PATHS = {"codex": "skillevaluator.tier3.harbor.installed_agents:SkillEvaluatorCodex",+ "opencode": "skillevaluator.tier3.harbor.installed_agents:SkillEvaluatorOpenCode","claude-code": CONTAINER_AGENT_IMPORT_PATHS["claude-code"],"hermes": "skillevaluator.tier3.harbor.hermes_agent:SkillEvaluatorHermes",}diff --git a/src/skillevaluator/tier3/harbor/adapter.py b/src/skillevaluator/tier3/harbor/adapter.pyindex 9ea0a14a..3fdc6923 100644--- a/src/skillevaluator/tier3/harbor/adapter.py+++ b/src/skillevaluator/tier3/harbor/adapter.py@@ -87,6 +87,12 @@ _VERIFIER_FILES = (_EVAL_CORE_DIR.parent.parent / "evidence.py",)_EVALUATOR_TESTS_SUBDIR = "skill_evaluator"+# Harbor's agent setup checks each prerequisite command (curl, bash, pgrep, rg,+# stdbuf) in every trial and apt-installs the missing ones. Claude Code's setup+# needs pgrep, so procps in every generated image removes that per-trial apt+# step and its package mirror dependency. The starter Dockerfile that+# `tier3 init-harbor-task` writes (tier3/commands.py) lists the same packages.+_BASE_APT_PACKAGES = "bash curl git jq procps ripgrep"_BASE_IMAGE_PREFIX = "skillevaluator-base"# Compose 2 treats HEALTHCHECK NONE as a configured check, then fails its# readiness wait because Docker supplies no health state. Replace authored@@ -1535,7 +1541,7 @@ def build_eval_base_image("","RUN apt-get -o Acquire::Retries=3 update && \\"," apt-get -o Acquire::Retries=3 install -y --no-install-recommends \\",- " bash curl git jq ripgrep \\",+ f" {_BASE_APT_PACKAGES} \\",]if extra_apt:lines[-1] += " \\"@@ -3932,8 +3938,8 @@ def _rebase_custom_dockerfile_content() -> tuple[str, str]:"""Return custom Dockerfile content layered on top of the eval base image.- The base image already contains: python:3.12-slim, system packages- (bash/curl/git/jq), verifier dependencies, and the+ The base image already contains: python:3.12-slim, the system packages in+ ``_BASE_APT_PACKAGES``, verifier dependencies, and thestandard directory structure (/workspace/skills, /logs, etc.).The custom Dockerfile's FROM line is replaced so the skill author's@@ -4367,7 +4373,7 @@ def _write_dockerfile("","RUN apt-get -o Acquire::Retries=3 update && \\"," apt-get -o Acquire::Retries=3 install -y --no-install-recommends \\",- " bash curl git jq ripgrep \\",+ f" {_BASE_APT_PACKAGES} \\",]if apt_pkgs:dockerfile_lines[-1] += " \\"@@ -4503,7 +4509,7 @@ def _write_default_environment_dockerfile("","RUN apt-get -o Acquire::Retries=3 update && \\"," apt-get -o Acquire::Retries=3 install -y --no-install-recommends \\",- " bash curl git jq ripgrep \\",+ f" {_BASE_APT_PACKAGES} \\",]if apt_pkgs:dockerfile_lines[-1] += " \\"diff --git a/src/skillevaluator/tier3/harbor/installed_agents.py b/src/skillevaluator/tier3/harbor/installed_agents.pyindex 552ffc5d..2a9b6709 100644--- a/src/skillevaluator/tier3/harbor/installed_agents.py+++ b/src/skillevaluator/tier3/harbor/installed_agents.py@@ -3,8 +3,10 @@"""Keep Harbor's bootstrap dependencies aligned with its installer branches.-The wrappers also retry a failed agent CLI install, since a dropped download or-a package mirror error otherwise fails the trial before the agent starts.+The Claude Code, Codex and OpenCode wrappers request distribution Node/npm only+on the installer branch that uses it. The Claude Code and Codex wrappers also+retry a failed agent CLI install, since a dropped download or a package mirror+error otherwise fails the trial before the agent starts."""from __future__ import annotations@@ -17,6 +19,7 @@ from typing import TYPE_CHECKINGfrom harbor.agents.installed.base import NonZeroAgentExitCodeErrorfrom harbor.agents.installed.claude_code import ClaudeCodefrom harbor.agents.installed.codex import Codex+from harbor.agents.installed.opencode import OpenCodefrom skillevaluator.utils.redaction import redact_sensitive_text@@ -113,8 +116,8 @@ async def _installer_dependencies(return dependencies# Match the pinned Harbor installer's branch before removing prerequisites.- # Codex supplies Node via NVM on glibc; Claude's native installer needs no- # Node. Both still need system npm on their respective fallback branches.+ # Codex and OpenCode supply Node via NVM on glibc; Claude's native installer+ # needs no Node. All three still need system npm on their fallback branches.result = await environment.exec(command=(f"if {system_npm_predicate}; then printf '%s' system; "@@ -170,3 +173,21 @@ class SkillEvaluatorClaudeCode(ClaudeCode):system_npm_predicate="command -v apk >/dev/null 2>&1",)await super().ensure_system_dependencies(environment, dependencies)+++class SkillEvaluatorOpenCode(OpenCode):+ """Avoid a redundant distribution Node/npm install before Harbor's NVM."""++ async def ensure_system_dependencies(+ self,+ environment: BaseEnvironment,+ dependencies: tuple[str, ...],+ ) -> None:+ # The predicate is the one Harbor's OpenCode install uses to choose+ # between distribution Node (musl) and NVM (glibc).+ dependencies = await _installer_dependencies(+ environment,+ dependencies,+ system_npm_predicate="ldd --version 2>&1 | grep -qi musl || [ -f /etc/alpine-release ]",+ )+ await super().ensure_system_dependencies(environment, dependencies)diff --git a/src/skillevaluator/tier3/harbor/local_agents.py b/src/skillevaluator/tier3/harbor/local_agents.pyindex 8d847f32..70b2b977 100644--- a/src/skillevaluator/tier3/harbor/local_agents.py+++ b/src/skillevaluator/tier3/harbor/local_agents.py@@ -30,7 +30,7 @@ from harbor.utils.env import parse_bool_env_valuefrom skillevaluator.provider_config import is_claude_5_5_or_later, is_gpt6_modelfrom skillevaluator.tier3.harbor import LOCAL_CODEX_HOME_DIRNAME, LOCAL_CODEX_SECRETS_DIRNAMEfrom skillevaluator.tier3.harbor.installed_agents import SkillEvaluatorClaudeCode as _InstalledClaudeCode-from skillevaluator.tier3.harbor.installed_agents import SkillEvaluatorCodex+from skillevaluator.tier3.harbor.installed_agents import SkillEvaluatorCodex, SkillEvaluatorOpenCodefrom skillevaluator.tier3.harbor.nvidia_build_bridge import (MAX_REQUESTS_PER_BRIDGE,RunningBridge,@@ -499,7 +499,7 @@ class SkillEvaluatorLocalCodex(SkillEvaluatorGatewayCodex):return await super().exec_as_agent(environment, command=command, env=env, cwd=cwd, timeout_sec=timeout_sec)-class SkillEvaluatorGatewayOpenCode(OpenCode):+class SkillEvaluatorGatewayOpenCode(SkillEvaluatorOpenCode):"""Use Chat Completions for an explicitly OpenAI-compatible gateway."""def __init__(self, *args: Any, **kwargs: Any) -> None:diff --git a/src/skillevaluator/tier3/harbor/native_agents.py b/src/skillevaluator/tier3/harbor/native_agents.pyindex a24225ac..5ffcd388 100644--- a/src/skillevaluator/tier3/harbor/native_agents.py+++ b/src/skillevaluator/tier3/harbor/native_agents.py@@ -54,6 +54,7 @@ from skillevaluator.tier3.harbor.local_agents import (SkillEvaluatorLocalOpenCode,SkillEvaluatorNvidiaBuildClaudeCode,SkillEvaluatorNvidiaBuildCodex,+ SkillEvaluatorOpenCode,)from skillevaluator.tier3.plugin_native import SETUP_SCRIPT, ClaudeCodeAdapter, OpenCodeAdapter@@ -283,6 +284,10 @@ class NativeOpenCode(_NativeOpenCodeMixin, OpenCode):"""OpenCode with ``OPENCODE_CONFIG`` and ``OPENCODE_CONFIG_DIR`` pointing at the staged plugin."""+class NativeInstalledOpenCode(_NativeOpenCodeMixin, SkillEvaluatorOpenCode):+ """Installer-aware OpenCode (the Docker-engine baselines' wrapper) with the staged plugin config."""++class NativeGatewayOpenCode(_NativeOpenCodeMixin, SkillEvaluatorGatewayOpenCode):"""Gateway-routed OpenCode with the staged plugin config."""@@ -329,6 +334,10 @@ class PrivateOpenCode(SetupNetworkErrorAgent, OpenCode):"""Stock OpenCode with setup-only network error classification."""+class PrivateInstalledOpenCode(SetupNetworkErrorAgent, SkillEvaluatorOpenCode):+ """Installer-aware OpenCode with setup-only network error classification."""++class PrivateGatewayOpenCode(SetupNetworkErrorAgent, SkillEvaluatorGatewayOpenCode):"""Gateway OpenCode with setup-only network error classification."""diff --git a/src/skillevaluator/tier3/plugin_native.py b/src/skillevaluator/tier3/plugin_native.pyindex 138491d3..36d9efbb 100644--- a/src/skillevaluator/tier3/plugin_native.py+++ b/src/skillevaluator/tier3/plugin_native.py@@ -2294,6 +2294,12 @@ NATIVE_AGENT_IMPORT_PATHS: dict[tuple[str, str | None], str] = {("codex", f"{_LOCAL_AGENTS}:SkillEvaluatorGatewayCodex"): f"{_NATIVE_AGENTS}:NativeGatewayCodex",("codex", f"{_LOCAL_AGENTS}:SkillEvaluatorNvidiaBuildCodex"): f"{_NATIVE_AGENTS}:NativeNvidiaBuildCodex",("opencode", None): f"{_NATIVE_AGENTS}:NativeOpenCode",+ # The installer-aware OpenCode wrapper the Docker-engine baselines run; GKE and+ # Harbor-native cloud baselines keep the stock agent above.+ (+ "opencode",+ "skillevaluator.tier3.harbor.installed_agents:SkillEvaluatorOpenCode",+ ): f"{_NATIVE_AGENTS}:NativeInstalledOpenCode",("opencode", f"{_LOCAL_AGENTS}:SkillEvaluatorGatewayOpenCode"): f"{_NATIVE_AGENTS}:NativeGatewayOpenCode",("hermes", None): f"{_NATIVE_AGENTS}:NativeHermes",(diff --git a/tests/test_harbor_secure_docker_environment.py b/tests/test_harbor_secure_docker_environment.pyindex 6358941b..9c6afc2b 100644--- a/tests/test_harbor_secure_docker_environment.py+++ b/tests/test_harbor_secure_docker_environment.py@@ -29,6 +29,7 @@ from harbor.environments.docker.docker import DockerEnvironmentfrom harbor.models.task.config import EnvironmentConfigfrom harbor.models.trial.paths import TrialPaths+from skillevaluator.tier3.harbor import DOCKER_AGENT_IMPORT_PATHSfrom skillevaluator.tier3.harbor import stream_redaction as stream_redaction_modulefrom skillevaluator.tier3.harbor.adapter import generate_harbor_tasksfrom skillevaluator.tier3.harbor.runner import build_harbor_run_command@@ -1601,7 +1602,7 @@ def test_docker_command_uses_secure_environment_import_path() -> None:assert "--agent-import-path" not in commandassert "--environment-import-path" not in command- assert command[command.index("--agent") + 1] == "opencode"+ assert command[command.index("--agent") + 1] == DOCKER_AGENT_IMPORT_PATHS["opencode"]assert command[command.index("--env") + 1] == SECURE_DOCKER_ENV_IMPORT_PATHdiff --git a/tests/tier3/test_installed_agent_bootstrap.py b/tests/tier3/test_installed_agent_bootstrap.pyindex c11055ca..279fc387 100644--- a/tests/tier3/test_installed_agent_bootstrap.py+++ b/tests/tier3/test_installed_agent_bootstrap.py@@ -4,6 +4,8 @@"""Exercise the inherited Harbor bootstrap, including package selection."""import asyncio+import re+import subprocessfrom pathlib import Pathimport pytest@@ -11,16 +13,25 @@ import pytestpytest.importorskip("harbor")from harbor.agents.factory import AgentFactory+from harbor.agents.installed.base import BaseInstalledAgentfrom harbor.agents.installed.claude_code import ClaudeCodefrom harbor.agents.installed.codex import Codex+from harbor.agents.installed.opencode import OpenCodefrom harbor.environments.base import ExecResultfrom harbor.models.trial.config import AgentConfig--from skillevaluator.tier3.harbor import DOCKER_AGENT_IMPORT_PATHS-from skillevaluator.tier3.harbor.installed_agents import SkillEvaluatorClaudeCode, SkillEvaluatorCodex+from harbor.utils.import_path import import_symbol++from skillevaluator.tier3 import commands+from skillevaluator.tier3.harbor import DOCKER_AGENT_IMPORT_PATHS, adapter+from skillevaluator.tier3.harbor.installed_agents import (+ SkillEvaluatorClaudeCode,+ SkillEvaluatorCodex,+ SkillEvaluatorOpenCode,+)from skillevaluator.tier3.harbor.local_agents import SkillEvaluatorClaudeCode as ContainerClaudeCodefrom skillevaluator.tier3.harbor.local_agents import (SkillEvaluatorGatewayCodex,+ SkillEvaluatorGatewayOpenCode,SkillEvaluatorLocalClaudeCode,SkillEvaluatorLocalCodex,SkillEvaluatorNvidiaBuildClaudeCode,@@ -94,6 +105,11 @@ class _BootstrapEnvironment:return ExecResult(return_code=0, stdout="", stderr="")+def _vendor_install(agent_class: type) -> str:+ """The text of the agent CLI install in Harbor's installer command."""+ return "npm i -g opencode-ai" if issubclass(agent_class, OpenCode) else "npm install -g"++def test_gateway_codex_debian_bootstrap_does_not_install_system_node(tmp_path: Path) -> None:environment = _BootstrapEnvironment()agent = SkillEvaluatorGatewayCodex(logs_dir=tmp_path, model_name="openai/test-model")@@ -104,7 +120,7 @@ def test_gateway_codex_debian_bootstrap_does_not_install_system_node(tmp_path: Passert any("nvm install 22" in command for command in environment.commands)[email protected]("agent_class", [SkillEvaluatorCodex, SkillEvaluatorClaudeCode])[email protected]("agent_class", [SkillEvaluatorCodex, SkillEvaluatorClaudeCode, SkillEvaluatorOpenCode])@pytest.mark.parametrize("manager", ["apt-get", "dnf"])def test_native_bootstrap_avoids_system_node_and_keeps_vendor_installer(tmp_path: Path, agent_class: type, manager: str@@ -122,12 +138,16 @@ def test_native_bootstrap_avoids_system_node_and_keeps_vendor_installer(assert "ldd --version 2>&1 | grep -qi musl || [ -f /etc/alpine-release ]" in environment.probes[0]assert any("nvm install 22" in command for command in environment.commands)assert any("ln -sf" in command for command in environment.commands)+ elif agent_class is SkillEvaluatorOpenCode:+ assert "ldd --version 2>&1 | grep -qi musl || [ -f /etc/alpine-release ]" in environment.probes[0]+ assert any("nvm install 22" in command for command in environment.commands)+ assert any("npm i -g opencode-ai" in command for command in environment.commands)else:assert "command -v apk" in environment.probes[0]assert any("https://downloads.claude.ai/claude-code-releases/bootstrap.sh" in c for c in environment.commands)[email protected]("agent_class", [SkillEvaluatorCodex, SkillEvaluatorClaudeCode])[email protected]("agent_class", [SkillEvaluatorCodex, SkillEvaluatorClaudeCode, SkillEvaluatorOpenCode])@pytest.mark.parametrize(("manager", "expected_install"),[@@ -150,7 +170,7 @@ def test_bundled_installer_without_ca_bundle_installs_ca_certificates(assert environment.package_installs == [expected_install][email protected]("agent_class", [SkillEvaluatorCodex, SkillEvaluatorClaudeCode])[email protected]("agent_class", [SkillEvaluatorCodex, SkillEvaluatorClaudeCode, SkillEvaluatorOpenCode])@pytest.mark.parametrize("manager", ["apt-get", "apk"])def test_system_npm_installer_still_provisions_node_and_npm(tmp_path: Path, agent_class: type, manager: str) -> None:environment = _BootstrapEnvironment(manager=manager, system_node=True)@@ -163,12 +183,16 @@ def test_system_npm_installer_still_provisions_node_and_npm(tmp_path: Path, agenassert environment.package_installs[0].startswith("set -o pipefail; apt-get update" if manager == "apt-get" else "set -o pipefail; apk add")- assert any("npm install -g" in command for command in environment.commands)+ assert any(_vendor_install(agent_class) in command for command in environment.commands)@pytest.mark.parametrize(("agent_class", "missing_command", "expected_package"),- [(SkillEvaluatorCodex, "rg", "ripgrep"), (SkillEvaluatorClaudeCode, "pgrep", "procps")],+ [+ (SkillEvaluatorCodex, "rg", "ripgrep"),+ (SkillEvaluatorClaudeCode, "pgrep", "procps"),+ (SkillEvaluatorOpenCode, "stdbuf", "coreutils"),+ ],)def test_native_bootstrap_preserves_other_system_dependencies(tmp_path: Path, agent_class: type, missing_command: str, expected_package: str@@ -209,7 +233,7 @@ def test_mismatching_cli_version_runs_bootstrap_with_requested_version(tmp_path:assert any(expected in command for command in environment.commands)[email protected]("agent_class", [SkillEvaluatorCodex, SkillEvaluatorClaudeCode])[email protected]("agent_class", [SkillEvaluatorCodex, SkillEvaluatorClaudeCode, SkillEvaluatorOpenCode])@pytest.mark.parametrize(("return_code", "stdout"),[(1, "bundled"), (137, ""), (0, ""), (0, "unrecognized")],@@ -224,10 +248,10 @@ def test_unknown_installer_platform_stops_without_installing(asyncio.run(agent.install(environment))assert environment.package_installs == []- assert not any("npm install -g" in command for command in environment.commands)+ assert not any(_vendor_install(agent_class) in command for command in environment.commands)[email protected]("agent_class", [SkillEvaluatorCodex, SkillEvaluatorClaudeCode])[email protected]("agent_class", [SkillEvaluatorCodex, SkillEvaluatorClaudeCode, SkillEvaluatorOpenCode])def test_platform_probe_transport_errors_propagate(tmp_path: Path, agent_class: type) -> None:environment = _BootstrapEnvironment(probe_error=TimeoutError("probe transport timeout"))agent = agent_class(logs_dir=tmp_path, model_name="openai/test-model")@@ -238,7 +262,7 @@ def test_platform_probe_transport_errors_propagate(tmp_path: Path, agent_class:assert environment.package_installs == [][email protected]("agent_class", [SkillEvaluatorCodex, SkillEvaluatorClaudeCode])[email protected]("agent_class", [SkillEvaluatorCodex, SkillEvaluatorClaudeCode, SkillEvaluatorOpenCode])def test_non_node_dependency_requests_skip_platform_detection(tmp_path: Path, agent_class: type) -> None:environment = _BootstrapEnvironment(missing_commands=("git",), probe_error=AssertionError("unneeded probe"))agent = agent_class(logs_dir=tmp_path, model_name="openai/test-model")@@ -249,7 +273,7 @@ def test_non_node_dependency_requests_skip_platform_detection(tmp_path: Path, agassert environment.probes == [][email protected]("agent_class", [SkillEvaluatorCodex, SkillEvaluatorClaudeCode])[email protected]("agent_class", [SkillEvaluatorCodex, SkillEvaluatorClaudeCode, SkillEvaluatorOpenCode])def test_native_bootstrap_preserves_unknown_dependency_validation(tmp_path: Path, agent_class: type) -> None:environment = _BootstrapEnvironment()agent = agent_class(logs_dir=tmp_path, model_name="openai/test-model")@@ -262,15 +286,23 @@ def test_native_bootstrap_preserves_unknown_dependency_validation(tmp_path: Path@pytest.mark.parametrize(- ("agent_name", "agent_class", "model_name"),+ ("agent_name", "agent_class", "model_name", "reported_model_name", "reported_provider"),[- ("codex", SkillEvaluatorCodex, "openai/test-model"),- ("claude-code", ContainerClaudeCode, "anthropic/test-model"),+ ("codex", SkillEvaluatorCodex, "openai/test-model", "test-model", "openai"),+ ("claude-code", ContainerClaudeCode, "anthropic/test-model", "test-model", "anthropic"),+ # Harbor 0.25 keeps OpenCode's native provider/model selector intact.+ ("opencode", SkillEvaluatorOpenCode, "nvidia/test-model", "nvidia/test-model", None),],)@pytest.mark.parametrize("config_field", ["name", "import_path"])def test_docker_import_routes_resolve_and_preserve_report_identity(- tmp_path: Path, agent_name: str, agent_class: type, model_name: str, config_field: str+ tmp_path: Path,+ agent_name: str,+ agent_class: type,+ model_name: str,+ reported_model_name: str,+ reported_provider: str | None,+ config_field: str,) -> None:config = AgentConfig(**{config_field: DOCKER_AGENT_IMPORT_PATHS[agent_name]},@@ -287,8 +319,8 @@ def test_docker_import_routes_resolve_and_preserve_report_identity(assert info.name == agent_nameassert info.version == "1.2.3"assert info.model_info is not None- assert info.model_info.name == "test-model"- assert info.model_info.provider == model_name.split("/", maxsplit=1)[0]+ assert info.model_info.name == reported_model_name+ assert info.model_info.provider == reported_provider@pytest.mark.parametrize(@@ -298,6 +330,7 @@ def test_docker_import_routes_resolve_and_preserve_report_identity(SkillEvaluatorNvidiaBuildCodex,SkillEvaluatorNvidiaBuildClaudeCode,ContainerClaudeCode,+ SkillEvaluatorGatewayOpenCode,],)def test_gateway_and_nvidia_wrappers_use_repaired_bootstrap(tmp_path: Path, agent_class: type) -> None:@@ -308,7 +341,7 @@ def test_gateway_and_nvidia_wrappers_use_repaired_bootstrap(tmp_path: Path, agenassert environment.package_installs == []assert len(environment.probes) == 1- assert any("npm install -g" in command for command in environment.commands)+ assert any(_vendor_install(agent_class) in command for command in environment.commands)@pytest.mark.parametrize("agent_class", [SkillEvaluatorLocalCodex, SkillEvaluatorLocalClaudeCode])@@ -324,3 +357,171 @@ def test_local_agents_keep_existing_version_check_without_install(tmp_path: Path)assert environment.package_installs == []assert environment.probes == []+++# Debian marks coreutils Essential and the python:3.12-slim base installs+# ca-certificates, so every generated image has both without listing them.+_SLIM_BASE_APT_PACKAGES = frozenset({"coreutils", "ca-certificates"})+++def _skill(root: Path) -> Path:+ root.mkdir()+ (root / "SKILL.md").write_text("---\nname: skill\ndescription: Test fixture.\n---\n", encoding="utf-8")+ return root+++def _apt_packages(dockerfile: str) -> set[str]:+ """The packages named by a generated Dockerfile's ``apt-get install`` command."""+ flags = "install -y --no-install-recommends"+ command = next(line for line in dockerfile.replace("\\\n", " ").splitlines() if flags in line)+ return set(command.split(flags, 1)[1].split("&&", 1)[0].replace("\\", " ").split())+++def _generated_dockerfiles(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> dict[str, str]:+ """Every Dockerfile SkillEvaluator writes on the python:3.12-slim base, by its writer."""+ built: list[str] = []++ def fake_run(cmd: list[str], **_kwargs: object) -> subprocess.CompletedProcess[str]:+ if cmd[:3] == ["docker", "image", "inspect"]:+ return subprocess.CompletedProcess(cmd, 1, "", "")+ built.append((Path(cmd[-1]) / "Dockerfile").read_text(encoding="utf-8"))+ return subprocess.CompletedProcess(cmd, 0, "", "")++ monkeypatch.setattr(adapter.subprocess, "run", fake_run)+ assert adapter.build_eval_base_image(_skill(tmp_path / "base-skill")).startswith("skillevaluator-base:")++ task = tmp_path / "task"+ adapter._write_dockerfile(+ task_dir=task,+ skill_path=_skill(tmp_path / "task-skill"),+ reference_skills_dir=None,+ workspace_skill_paths=None,+ has_skill=True,+ )++ environment = tmp_path / "environment"+ environment.mkdir()+ adapter._write_default_environment_dockerfile(+ environment, base_image="", agent_config_lines=[], include_input=False+ )++ starter_skill = _skill(tmp_path / "starter-skill")+ assert (+ commands.init_harbor_task(+ starter_skill, force=False, case_id="case-001", mode="default", language="python", with_config=False+ )+ == 0+ )++ return {+ "build_eval_base_image": built[0],+ "_write_dockerfile": (task / "environment" / "Dockerfile").read_text(encoding="utf-8"),+ "_write_default_environment_dockerfile": (environment / "Dockerfile").read_text(encoding="utf-8"),+ "init-harbor-task": (starter_skill / "evals/harbor/case-001/environment/Dockerfile").read_text(+ encoding="utf-8"+ ),+ }++[email protected]("agent_name", ["claude-code", "codex", "opencode"])+def test_generated_images_satisfy_docker_agent_setup_dependencies(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, agent_name: str+) -> None:+ """Harbor's agent setup apt-installs, in every trial, each prerequisite the image lacks."""+ requested: list[tuple[str, ...]] = []++ async def record(self: BaseInstalledAgent, environment: object, dependencies: tuple[str, ...]) -> None:+ requested.append(tuple(dependencies))++ # Record what reaches Harbor's package step after the wrapper's installer-branch filter.+ monkeypatch.setattr(BaseInstalledAgent, "ensure_system_dependencies", record)+ agent_class = import_symbol(DOCKER_AGENT_IMPORT_PATHS[agent_name])+ agent = agent_class(logs_dir=tmp_path / "logs", model_name="openai/test-model")+ asyncio.run(agent.install(_BootstrapEnvironment()))+ assert requested+ needed = {+ package+ for dependencies in requested+ for dependency in dependencies+ for package in BaseInstalledAgent.SYSTEM_PACKAGES[dependency].packages["apt-get"]+ }+ assert not {"nodejs", "npm"} & needed++ for writer, dockerfile in _generated_dockerfiles(tmp_path, monkeypatch).items():+ assert dockerfile.startswith("FROM python:3.12-slim\n"), writer+ image = _apt_packages(dockerfile) | _SLIM_BASE_APT_PACKAGES+ assert needed - image == set(), writer+++def _docker_setup_note_paragraphs() -> list[str]:+ """The paragraphs of the eval-datasets note on Docker agent setup packages, whitespace collapsed."""+ docs = (Path(__file__).resolve().parents[2] / "docs" / "eval-datasets.mdx").read_text(encoding="utf-8")+ notes = [block.split("</Note>", 1)[0] for block in docs.split("<Note>")[1:]]+ (note,) = [note for note in notes if "`procps`" in note]+ paragraphs = re.split(r"\n[ \t]*\n", note.strip())+ return [" ".join(paragraph.split()) for paragraph in paragraphs]+++def _docker_setup_note() -> str:+ """The eval-datasets note on Docker agent setup packages, with whitespace collapsed."""+ return " ".join(_docker_setup_note_paragraphs())+++def test_custom_dockerfiles_reach_procps_only_through_the_shared_base_image_rebase(tmp_path: Path) -> None:+ """A custom Dockerfile keeps its own base, and so needs procps, unless it is rebased."""+ custom_from = "FROM debian:bookworm-slim"+ first_lines: dict[str, str] = {}+ for name, kwargs in {+ # The runner passes an empty base_image unless harbor.base_image_mode is reuse or rebuild.+ "default": {},+ "preserve": {"base_image": "skillevaluator-base:test", "custom_dockerfile_mode": "preserve"},+ "rebase": {"base_image": "skillevaluator-base:test", "custom_dockerfile_mode": "rebase"},+ }.items():+ skill = _skill(tmp_path / f"{name}-skill")+ (skill / "evals" / "environment").mkdir(parents=True)+ (skill / "evals" / "environment" / "Dockerfile").write_text(f"{custom_from}\nRUN true\n", encoding="utf-8")+ task = tmp_path / f"{name}-task"+ adapter._write_dockerfile(+ task_dir=task,+ skill_path=skill,+ reference_skills_dir=None,+ workspace_skill_paths=None,+ has_skill=True,+ **kwargs,+ )+ first_lines[name] = (task / "environment" / "Dockerfile").read_text(encoding="utf-8").splitlines()[0]++ assert first_lines == {+ "default": custom_from,+ "preserve": custom_from,+ "rebase": "FROM skillevaluator-base:test",+ }+ note = _docker_setup_note()+ assert "A custom `evals/environment/Dockerfile` keeps its own base image, so it should install `procps` too" in note+ assert (+ "unless `harbor.base_image_mode` is `reuse` or `rebuild` and `harbor.custom_dockerfile_mode` is `rebase`"+ in note+ )+++def test_docker_setup_note_gives_procps_guidance_its_own_paragraph() -> None:+ """Custom Dockerfile authors find the procps guidance without reading the Node.js scope first."""+ node_scope, procps = _docker_setup_note_paragraphs()++ assert "`nodejs` and `npm`" in node_scope+ assert "`scripts/apt-packages.txt`" in node_scope+ assert "procps" not in node_scope+ assert procps.startswith(+ "The auto-generated image and the `init-harbor-task` starter Dockerfile also include `procps`"+ )+ assert "nvm" not in procps+ assert "`harbor.custom_dockerfile_mode` is `rebase`" in procps+++def test_docker_setup_note_scopes_the_npx_mcp_example_to_claude_code() -> None:+ """Codex and OpenCode stdio MCP servers inherit nvm's npx; only Claude Code has no Node.js."""+ note = _docker_setup_note()++ assert "including stdio MCP servers" in note+ assert 'Node.js with Claude Code, for example `command = "npx ..."`' in note+ assert "or `gcc`, `g++`, or `make` with any agent" in notediff --git a/tests/tier3/test_private_setup_classification.py b/tests/tier3/test_private_setup_classification.pyindex 808553e5..247f984b 100644--- a/tests/tier3/test_private_setup_classification.py+++ b/tests/tier3/test_private_setup_classification.py@@ -15,12 +15,15 @@ from types import SimpleNamespaceimport pytestfrom harbor.agents.installed.base import BaseInstalledAgent, NetworkConnectionError+from harbor.agents.installed.opencode import OpenCodefrom harbor.utils.import_path import import_symbol+from skillevaluator.tier3.harbor import CONTAINER_AGENT_IMPORT_PATHS, DOCKER_AGENT_IMPORT_PATHS, runnerfrom skillevaluator.tier3.harbor.local_agents import AgentSetupNetworkError, SetupNetworkErrorAgentfrom skillevaluator.tier3.plugin_native import (NATIVE_AGENT_IMPORT_PATHS,PRIVATE_AGENT_IMPORT_PATHS,+ native_agent_import_path,private_agent_import_path,)@@ -49,6 +52,47 @@ def test_native_wrapper_is_not_wrapped_twice(key: tuple, path: str) -> None:assert issubclass(import_symbol(path), SetupNetworkErrorAgent)+class _NativeHooks:+ """The private runtime's agent routing, with every agent loading the plugin natively."""++ def with_agent_import_path(self, agent: str, base_import_path: str | None) -> str:+ return native_agent_import_path(agent, base_import_path)++ def baseline_agent_import_path(self, agent: str, base_import_path: str | None) -> str | None:+ return private_agent_import_path(agent, base_import_path)+++ ("env_mode", "default_paths", "baseline", "with_arm"),+ [+ ("docker", DOCKER_AGENT_IMPORT_PATHS, "PrivateInstalledOpenCode", "NativeInstalledOpenCode"),+ ("gke", CONTAINER_AGENT_IMPORT_PATHS, "PrivateOpenCode", "NativeOpenCode"),+ ],+)+def test_opencode_plugin_arms_share_the_environment_default_agent(+ monkeypatch: pytest.MonkeyPatch,+ env_mode: str,+ default_paths: dict[str, str],+ baseline: str,+ with_arm: str,+) -> None:+ # Both plugin arms must set OpenCode up as an ordinary run in that+ # environment does: through its default import path, else Harbor's stock agent.+ monkeypatch.setattr(runner, "_private_runtime_hooks", _NativeHooks)+ default_path = default_paths.get("opencode")+ default_agent = import_symbol(default_path) if default_path is not None else OpenCode++ baseline_path, with_path = runner._private_agent_paths(["opencode"], {}, env_mode)["opencode"]++ assert baseline_path == f"skillevaluator.tier3.harbor.native_agents:{baseline}"+ assert with_path == f"skillevaluator.tier3.harbor.native_agents:{with_arm}"+ for path in (baseline_path, with_path):+ cls = import_symbol(path)+ assert issubclass(cls, default_agent)+ assert cls.install is default_agent.install+ assert cls.ensure_system_dependencies is default_agent.ensure_system_dependencies++def test_unknown_private_wrapper_does_not_silently_replace_custom_routing() -> None:assert private_agent_import_path("codex", "custom.module:MyAgent") == "custom.module:MyAgent"