Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier3): propagate judge model overrides
Christopher Kevinec59f5a07ebe
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex cc2d4dd3..516251ba 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -90,6 +90,11 @@ All notable changes to SkillEvaluator are documented in this file.### Fixed+- Tier 3 now forwards host-configured `LLM_JUDGE_MODEL` and+ `SKILL_EVAL_JUDGE_MODEL` overrides to generated and native Harbor verifiers+ while keeping them out of agent environments. Skill-authored `runtime_env`+ and native task environments cannot set or alias either operator-controlled+ judge override.- Quality scoring now uses boundary-aware and context-aware matching for XML tags,reserved names, MCP guidance, README references, time references, exclusivitylanguage, instruction action verbs, and nested Markdown links, avoidingdiff --git a/docs/environment-variables.mdx b/docs/environment-variables.mdxindex ed2f4a46..6e9aae0f 100644--- a/docs/environment-variables.mdx+++ b/docs/environment-variables.mdx@@ -101,7 +101,7 @@ execution environments are covered in| `SKILLEVALUATOR_LOCAL_INHERIT_AGENT_KEYS` | `false` | When true, local-mode trials inherit the host's agent credential variables into their ambient environment. Off by default so a hostile skill command cannot read credentials — agents and verifiers receive credentials per-exec instead. || `SKILLEVALUATOR_RUNTIME_DIR` | `~/.local/share/skillevaluator/runtimes` | Where local mode looks for managed agent CLIs. Must be a dedicated subdirectory — not the home directory or one of its parents. || `SKILL_EVAL_HARBOR_PREBUILD_TASK_ENVS` | unset | Set to exactly `1` to opt in to pre-building Harbor task environment images before trials start (other truthy spellings are not honored). Opt-in because concurrent trial cleanup can remove a shared prebuilt image that sibling trials still need. |-| `SKILL_EVAL_JUDGE_MODEL` | falls back to `SKILL_EVAL_LLM_MODEL` | Model override for the Tier 3 dimension judge in standard grading. `LLM_JUDGE_MODEL` is honored as a higher-precedence spelling; when neither is set, the judge uses `SKILL_EVAL_LLM_MODEL`, then the built-in judge default. |+| `SKILL_EVAL_JUDGE_MODEL` | falls back to `SKILL_EVAL_LLM_MODEL` | Operator-controlled model override for the Tier 3 dimension judge in standard grading. `LLM_JUDGE_MODEL` is honored as a higher-precedence spelling; when neither is set, the judge uses `SKILL_EVAL_LLM_MODEL`, then the built-in judge default. Both spellings change only the model used with the configured `SKILL_EVAL_LLM_PROVIDER`; they do not select a separate judge provider, so the model must be supported by that provider. The override is staged only for the verifier, never the evaluated agent; attempts to set or reference it from skill-authored `runtime_env` or native task environments are rejected. || `SKILLEVALUATOR_HARBOR_DIAGNOSTIC_ARTIFACT_MAX_BYTES` | `5242880` (5 MiB) | Size cap per diagnostic artifact copied from a Harbor trial into the results directory; oversized artifacts are skipped and recorded in the artifact manifest. |## In-container grader contractdiff --git a/src/skillevaluator/tier3/harbor/adapter.py b/src/skillevaluator/tier3/harbor/adapter.pyindex 82beb3ca..dc4d0a2b 100644--- a/src/skillevaluator/tier3/harbor/adapter.py+++ b/src/skillevaluator/tier3/harbor/adapter.py@@ -201,40 +201,44 @@ _COMPOSE_ALLOWED_BUILD_KEYS = frozenset()_COMPOSE_ALLOWED_NETWORK_KEYS = frozenset({"attachable", "enable_ipv4", "enable_ipv6", "internal", "labels"})_COMPOSE_ALLOWED_VOLUME_KEYS = frozenset({"labels"})-_VERIFIER_PROVIDER_ENV_VARS = frozenset(- {- "SKILL_EVAL_LLM_PROVIDER",- "SKILL_EVAL_LLM_MODEL",- "SKILL_EVAL_LLM_API_KEY",- "SKILL_EVAL_LLM_BASE_URL",- "OPENAI_API_KEY",- "OPENAI_BASE_URL",- "ANTHROPIC_API_KEY",- "ANTHROPIC_BASE_URL",- "NVIDIA_API_KEY",- "AWS_REGION",- "AWS_ACCESS_KEY_ID",- "AWS_BEARER_TOKEN_BEDROCK",- "AWS_CONFIG_FILE",- "AWS_CONTAINER_AUTHORIZATION_TOKEN",- "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE",- "AWS_CONTAINER_CREDENTIALS_FULL_URI",- "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI",- "AWS_DEFAULT_REGION",- "AWS_PROFILE",- "AWS_ROLE_ARN",- "AWS_ROLE_SESSION_NAME",- "AWS_SDK_LOAD_CONFIG",- "AWS_SECRET_ACCESS_KEY",- "AWS_SESSION_TOKEN",- "AWS_SHARED_CREDENTIALS_FILE",- "AWS_WEB_IDENTITY_TOKEN_FILE",- }+_VERIFIER_JUDGE_MODEL_ENV_VARS = frozenset({"LLM_JUDGE_MODEL", "SKILL_EVAL_JUDGE_MODEL"})+_VERIFIER_PROVIDER_ENV_VARS = (+ frozenset(+ {+ "SKILL_EVAL_LLM_PROVIDER",+ "SKILL_EVAL_LLM_MODEL",+ "SKILL_EVAL_LLM_API_KEY",+ "SKILL_EVAL_LLM_BASE_URL",+ "OPENAI_API_KEY",+ "OPENAI_BASE_URL",+ "ANTHROPIC_API_KEY",+ "ANTHROPIC_BASE_URL",+ "NVIDIA_API_KEY",+ "AWS_REGION",+ "AWS_ACCESS_KEY_ID",+ "AWS_BEARER_TOKEN_BEDROCK",+ "AWS_CONFIG_FILE",+ "AWS_CONTAINER_AUTHORIZATION_TOKEN",+ "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE",+ "AWS_CONTAINER_CREDENTIALS_FULL_URI",+ "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI",+ "AWS_DEFAULT_REGION",+ "AWS_PROFILE",+ "AWS_ROLE_ARN",+ "AWS_ROLE_SESSION_NAME",+ "AWS_SDK_LOAD_CONFIG",+ "AWS_SECRET_ACCESS_KEY",+ "AWS_SESSION_TOKEN",+ "AWS_SHARED_CREDENTIALS_FILE",+ "AWS_WEB_IDENTITY_TOKEN_FILE",+ }+ )+ | _VERIFIER_JUDGE_MODEL_ENV_VARS)def _verifier_env_vars(runtime_env: dict[str, str] | None = None) -> tuple[str, ...]:- """Return public provider variables explicitly staged for the verifier."""+ """Return evaluator-controlled variables explicitly staged for the verifier."""return tuple(sorted(set(runtime_env or {}).intersection(_VERIFIER_PROVIDER_ENV_VARS)))@@ -4215,6 +4219,108 @@ def _native_entry_id(task_dir: Path) -> str:return task_dir.name+def _environment_reference_names(value: object) -> set[str]:+ """Return portable shell-style environment references from a TOML value."""+ if not isinstance(value, str):+ return set()+ dollar_references = {+ braced or plain+ for braced, plain in re.findall(+ r"\$\{([A-Za-z_][A-Za-z0-9_]*)(?::-[^}]*)?\}|\$([A-Za-z_][A-Za-z0-9_]*)\b",+ value,+ )+ }+ percent_references = set(re.findall(r"%([A-Za-z_][A-Za-z0-9_]*)%", value))+ return dollar_references | percent_references+++def _judge_model_env_controls(environment: dict[str, Any]) -> set[str]:+ """Return reserved judge-model names used as keys or value references."""+ authored_keys = {name for name in environment if name.upper() in _VERIFIER_JUDGE_MODEL_ENV_VARS}+ authored_references = {+ reference+ for value in environment.values()+ for reference in _environment_reference_names(value)+ if reference.upper() in _VERIFIER_JUDGE_MODEL_ENV_VARS+ }+ return authored_keys | authored_references+++def _native_env_table(container: dict[str, Any], *, path: str, task_toml: Path) -> dict[str, Any]:+ environment = container.get("env", {})+ if not isinstance(environment, dict):+ raise ValueError(f"Native Harbor task [{path}] must be a table: {task_toml}")+ return environment+++def _native_verifier_env_tables(+ verifier: dict[str, Any],+ *,+ path: str,+ task_toml: Path,+) -> list[tuple[str, dict[str, Any]]]:+ """Return every process or persistent environment owned by a verifier."""+ environments = [(f"{path}.env", _native_env_table(verifier, path=f"{path}.env", task_toml=task_toml))]+ persistent = verifier.get("environment")+ if persistent is None:+ return environments+ if not isinstance(persistent, dict):+ raise ValueError(f"Native Harbor task [{path}.environment] must be a table: {task_toml}")+ environments.append(+ (+ f"{path}.environment.env",+ _native_env_table(persistent, path=f"{path}.environment.env", task_toml=task_toml),+ )+ )+ return environments+++def _validate_native_judge_model_controls(+ task_toml: Path,+ *,+ data: dict[str, Any],+ environment_env: dict[str, Any],+) -> None:+ agent_controls = sorted(_judge_model_env_controls(environment_env))+ if agent_controls:+ raise ValueError(+ f"Native Harbor task [environment.env] cannot name or reference evaluator-controlled judge model "+ f"variable(s): {', '.join(agent_controls)}: {task_toml}"+ )++ verifier = data.get("verifier", {})+ if not isinstance(verifier, dict):+ raise ValueError(f"Native Harbor task [verifier] must be a table: {task_toml}")+ verifier_tables = _native_verifier_env_tables(verifier, path="verifier", task_toml=task_toml)++ steps = data.get("steps", [])+ if steps is None:+ steps = []+ if not isinstance(steps, list):+ raise ValueError(f"Native Harbor task [[steps]] must be an array of tables: {task_toml}")+ for index, step in enumerate(steps):+ if not isinstance(step, dict):+ raise ValueError(f"Native Harbor task [[steps]] entry {index} must be a table: {task_toml}")+ step_verifier = step.get("verifier", {})+ if not isinstance(step_verifier, dict):+ raise ValueError(f"Native Harbor task [steps.verifier] entry {index} must be a table: {task_toml}")+ verifier_tables.extend(+ _native_verifier_env_tables(+ step_verifier,+ path=f"steps[{index}].verifier",+ task_toml=task_toml,+ )+ )++ for path, environment in verifier_tables:+ controls = sorted(_judge_model_env_controls(environment))+ if controls:+ raise ValueError(+ f"Native Harbor task [{path}] cannot name or reference evaluator-controlled judge model "+ f"variable(s): {', '.join(controls)}: {task_toml}"+ )++def _native_task_workdir(task_dir: Path, *, allow_docker_image: bool = False) -> str | None:"""Read and validate the workdir Harbor will use for a native task."""try:@@ -4229,6 +4335,14 @@ def _native_task_workdir(task_dir: Path, *, allow_docker_image: bool = False) ->environment_env = environment.get("env", {})if not isinstance(environment_env, dict):raise ValueError(f"Native Harbor task [environment.env] must be a table: {task_dir / 'task.toml'}")+ verifier = data.get("verifier", {}) if isinstance(data, dict) else {}+ if not isinstance(verifier, dict):+ raise ValueError(f"Native Harbor task [verifier] must be a table: {task_dir / 'task.toml'}")+ _validate_native_judge_model_controls(+ task_dir / "task.toml",+ data=data,+ environment_env=environment_env,+ )_validate_runtime_discovery_env(environment_env)_validate_runtime_loader_env(environment_env)skills_dir = environment.get("skills_dir")@@ -4269,40 +4383,71 @@ def _ensure_native_skills_dir(task_dir: Path) -> None:task_toml.write_text("\n".join(lines) + "\n", encoding="utf-8")+def _toml_table_header_indices(lines: list[str], *segments: str) -> tuple[int, ...]:+ """Locate raw candidates for a standard bare or quoted TOML table header."""+ segment_patterns = [+ rf"(?:{re.escape(segment)}|\"{re.escape(segment)}\"|'{re.escape(segment)}')" for segment in segments+ ]+ pattern = re.compile(r"^\s*\[\s*" + r"\s*\.\s*".join(segment_patterns) + r"\s*\]\s*(?:#.*)?$")+ return tuple(index for index, line in enumerate(lines) if pattern.fullmatch(line))+++def _has_expected_verifier_env(content: str, expected: dict[str, str]) -> bool:+ """Return whether parsed TOML contains the exact managed verifier values."""+ try:+ parsed = tomllib.loads(content)+ except tomllib.TOMLDecodeError:+ return False+ verifier = parsed.get("verifier", {}) if isinstance(parsed, dict) else {}+ environment = verifier.get("env", {}) if isinstance(verifier, dict) else {}+ return isinstance(environment, dict) and all(environment.get(name) == value for name, value in expected.items())++def _ensure_skill_evaluator_verifier_env(task_dir: Path, *, verifier_env: dict[str, str] | None) -> None:- """Ensure staged native tasks forward configured public provider variables."""+ """Ensure staged native tasks forward configured evaluator variables."""task_toml = task_dir / "task.toml"content = task_toml.read_text(encoding="utf-8")- env_lines = [f'{name} = "${{{name}}}"' for name in _verifier_env_vars(verifier_env)]- if all(line in content for line in env_lines):+ managed_names = _verifier_env_vars(verifier_env)+ if not managed_names:return+ parsed = tomllib.loads(content)+ parsed_verifier = parsed.get("verifier", {}) if isinstance(parsed, dict) else {}+ if not isinstance(parsed_verifier, dict):+ raise ValueError(f"Native Harbor task [verifier] must be a table: {task_toml}")+ expected = {name: f"${{{name}}}" for name in managed_names}lines = content.splitlines()- if "[verifier.env]" in lines:- idx = lines.index("[verifier.env]") + 1- existing = set(lines)- for line in reversed(env_lines):- if line not in existing:- lines.insert(idx, line)- task_toml.write_text("\n".join(lines) + "\n", encoding="utf-8")- return-- env_block = ["[verifier.env]", *env_lines]- if "[verifier]" in lines:- start = lines.index("[verifier]") + 1- insert_at = len(lines)- for idx in range(start, len(lines)):- line = lines[idx].strip()- if line.startswith("[") and line.endswith("]"):- insert_at = idx+ if "env" in parsed_verifier:+ parsed_verifier_env = parsed_verifier["env"]+ if not isinstance(parsed_verifier_env, dict):+ raise ValueError(f"Native Harbor task [verifier.env] must be a table: {task_toml}")+ env_lines = [+ f'{name} = "{value}"' for name, value in expected.items() if parsed_verifier_env.get(name) != value+ ]+ if not env_lines:+ return+ updated_content = ""+ for verifier_env_index in _toml_table_header_indices(lines, "verifier", "env"):+ candidate_lines = lines.copy()+ candidate_lines[verifier_env_index + 1 : verifier_env_index + 1] = env_lines+ candidate = "\n".join(candidate_lines) + "\n"+ if _has_expected_verifier_env(candidate, expected):+ updated_content = candidatebreak- lines[insert_at:insert_at] = ["", *env_block]- task_toml.write_text("\n".join(lines) + "\n", encoding="utf-8")- return+ if not updated_content:+ raise ValueError(f"Native Harbor task [verifier.env] must use a standard table header: {task_toml}")+ else:+ env_lines = [f'{name} = "{value}"' for name, value in expected.items()]+ env_block = ["[verifier.env]", *env_lines]+ if "verifier" in parsed:+ candidate_lines = [*lines, "", *env_block]+ else:+ candidate_lines = [*lines, "", "[verifier]", "timeout_sec = 180.0", "", *env_block]+ updated_content = "\n".join(candidate_lines) + "\n"- insert_at = lines.index("[environment]") if "[environment]" in lines else len(lines)- lines[insert_at:insert_at] = ["[verifier]", "timeout_sec = 180.0", "", *env_block, ""]- task_toml.write_text("\n".join(lines) + "\n", encoding="utf-8")+ if not _has_expected_verifier_env(updated_content, expected):+ raise ValueError(f"Cannot inject evaluator-controlled [verifier.env] into native task: {task_toml}")+ task_toml.write_text(updated_content, encoding="utf-8")def _insert_table_block(lines: list[str], anchor: str, block: list[str]) -> None:diff --git a/src/skillevaluator/tier3/harbor/runner.py b/src/skillevaluator/tier3/harbor/runner.pyindex 2c08f3d7..26c38b8a 100644--- a/src/skillevaluator/tier3/harbor/runner.py+++ b/src/skillevaluator/tier3/harbor/runner.py@@ -32,6 +32,7 @@ from skillevaluator.evaluation.tier3_report import render_agent_eval_html_reportfrom skillevaluator.provider_config import ProviderConfig, ProviderConfigurationError, resolve_llm_providerfrom skillevaluator.tier3.evals_config import EvalsConfigError, load_evals_configfrom skillevaluator.tier3.harbor.adapter import (+ _VERIFIER_JUDGE_MODEL_ENV_VARS,_prevalidate_baseline_skill_candidates,build_eval_base_image,find_evals_file,@@ -232,6 +233,7 @@ _RUNTIME_ENV_HOST_CONTROL_NAMES = (})| _BEDROCK_HOST_ENV_VARS+ | _VERIFIER_JUDGE_MODEL_ENV_VARS| frozenset().union(*_HARBOR_ENV_MODE_VARS.values()))_RUNTIME_ENV_HOST_CONTROL_PREFIXES = (@@ -422,11 +424,14 @@ def build_harbor_run_command(def _provider_environment(config: ProviderConfig) -> dict[str, str]:- """Map a public provider config to evaluator-owned verifier variables."""+ """Build evaluator-owned verifier variables from provider config and host overrides."""environment = {"SKILL_EVAL_LLM_PROVIDER": config.provider,"SKILL_EVAL_LLM_MODEL": config.model,}+ environment.update(+ {name: value for name in _VERIFIER_JUDGE_MODEL_ENV_VARS if (value := os.environ.get(name, "").strip())}+ )if config.provider == "anthropic":environment["ANTHROPIC_API_KEY"] = config.api_key or ""if config.base_url:@@ -719,19 +724,28 @@ def _resolve_runtime_env(templates: dict[str, str] | None) -> tuple[dict[str, sterrors.append(f"harbor.runtime_env.{name} controls the host process and is not allowed")continuetemplate_value = str(template)- references = {+ dollar_references = {braced or plainfor braced, plain in re.findall(- r"\$(?:\{([A-Za-z_][A-Za-z0-9_]*)\}|([A-Za-z_][A-Za-z0-9_]*))",+ r"\$\{([A-Za-z_][A-Za-z0-9_]*)(?::-[^}]*)?\}|\$([A-Za-z_][A-Za-z0-9_]*)\b",template_value,)}+ percent_references = set(re.findall(r"%([A-Za-z_][A-Za-z0-9_]*)%", template_value))+ references = dollar_references | percent_referencesowned_references = sorted(reference for reference in references if _is_operator_owned_runtime_name(reference))if owned_references:errors.append(f"harbor.runtime_env.{name} references operator-owned credential(s): " + ", ".join(owned_references))continue+ if percent_references:+ errors.append(+ f"harbor.runtime_env.{name} uses unsupported %VAR% reference syntax for: "+ + ", ".join(sorted(percent_references))+ + "; use ${VAR}"+ )+ continuevalue = os.path.expandvars(template_value)if "$" in value:errors.append(f"harbor.runtime_env.{name} references an unset environment variable")diff --git a/tests/test_harbor_runner_environment.py b/tests/test_harbor_runner_environment.pyindex c8c76a95..017227cd 100644--- a/tests/test_harbor_runner_environment.py+++ b/tests/test_harbor_runner_environment.py@@ -161,6 +161,8 @@ def test_skill_config_cannot_override_operator_owned_agent_credentials(owned_nam"OPENAI_BASE_URL","ANTHROPIC_API_KEY","ANTHROPIC_BASE_URL",+ "LLM_JUDGE_MODEL",+ "SKILL_EVAL_JUDGE_MODEL","AWS_SECRET_ACCESS_KEY","E2B_API_KEY","DOCKER_HOST",@@ -179,17 +181,83 @@ def test_skill_config_cannot_alias_operator_owned_credentials(assert source_name in errors[0][email protected]("source_name", ["LLM_JUDGE_MODEL", "SKILL_EVAL_JUDGE_MODEL"])+def test_skill_config_cannot_alias_judge_model_with_default_expansion(+ monkeypatch: pytest.MonkeyPatch,+ source_name: str,+) -> None:+ monkeypatch.setenv(source_name, "operator-model")++ resolved, errors = runner._resolve_runtime_env({"INNOCENT_NAME": f"${{{source_name}:-skill-fallback}}"})++ assert resolved == {}+ assert errors and "operator-owned" in errors[0]+ assert source_name in errors[0]++@pytest.mark.parametrize("name",- ["SKILL_EVAL_LLM_BASE_URL", "SKILL_EVAL_LLM_PROVIDER", "SKILL_EVAL_LLM_API_KEY"],+ [+ "LLM_JUDGE_MODEL",+ "SKILL_EVAL_JUDGE_MODEL",+ "SKILL_EVAL_LLM_BASE_URL",+ "SKILL_EVAL_LLM_PROVIDER",+ "SKILL_EVAL_LLM_API_KEY",+ ],)-def test_skill_config_cannot_control_evaluator_provider_routing(name: str) -> None:+def test_skill_config_cannot_control_evaluator_or_judge_routing(name: str) -> None:resolved, errors = runner._resolve_runtime_env({name: "https://attacker.example/v1"})assert resolved == {}assert errors and "host process" in errors[0][email protected]("name", ["LLM_JUDGE_MODEL", "SKILL_EVAL_JUDGE_MODEL"])[email protected]("provider_name", ["openai", "anthropic", "bedrock", "nv_build"])+ ("configured", "expected"),+ [+ ("judge-model", "judge-model"),+ (" judge-model ", "judge-model"),+ ("", None),+ (" ", None),+ ],+)+def test_provider_environment_forwards_each_host_judge_model_override(+ monkeypatch: pytest.MonkeyPatch,+ name: str,+ provider_name: str,+ configured: str,+ expected: str | None,+) -> None:+ monkeypatch.setattr(runner.os, "environ", {name: configured})++ environment = runner._provider_environment(_provider(provider_name))++ assert environment.get(name) == expected+++ ("source_name", "expected_error"),+ [+ ("LLM_JUDGE_MODEL", "operator-owned"),+ ("HOST_AGENT_TOKEN", "unsupported"),+ ],+)+def test_runtime_env_rejects_windows_style_variable_references(+ monkeypatch: pytest.MonkeyPatch,+ source_name: str,+ expected_error: str,+) -> None:+ monkeypatch.setenv(source_name, "host-value")++ resolved, errors = runner._resolve_runtime_env({"SAFE_ALIAS": f"%{source_name}%"})++ assert resolved == {}+ assert errors and source_name in errors[0]+ assert expected_error in errors[0]++def test_mixed_agents_receive_disjoint_credentials(monkeypatch: pytest.MonkeyPatch) -> None:monkeypatch.setattr(runner.os,@@ -391,9 +459,12 @@ def test_run_harbor_eval_stages_per_agent_credential_trees((skill / "evals").mkdir(parents=True)(skill / "evals" / "evals.json").write_text("[]\n", encoding="utf-8")provider = _provider("nv_build")- emitted: list[tuple[str, bool, dict[str, str]]] = []+ emitted: list[tuple[str, bool, dict[str, str], dict[str, str]]] = []launched: dict[str, tuple[str, str, dict[str, str]]] = {}+ monkeypatch.setenv("LLM_JUDGE_MODEL", "legacy-judge-model")+ monkeypatch.setenv("SKILL_EVAL_JUDGE_MODEL", "judge-model")+monkeypatch.setattr(runner, "resolve_llm_provider", lambda: provider)monkeypatch.setattr(runner,@@ -406,7 +477,14 @@ def test_run_harbor_eval_stages_per_agent_credential_trees(def emit(_skill, target, *, with_skill, runtime_env, **_kwargs):task = target / "case-001"task.mkdir(parents=True)- emitted.append((str(target.relative_to(tmp_path / "results")), with_skill, dict(runtime_env)))+ emitted.append(+ (+ str(target.relative_to(tmp_path / "results")),+ with_skill,+ dict(runtime_env),+ dict(_kwargs["verifier_env"]),+ )+ )return [task]def launch(**kwargs):@@ -439,17 +517,30 @@ def test_run_harbor_eval_stages_per_agent_credential_trees()assert "error" not in result- assert {(path.split("/")[-2], path.split("/")[-1], with_skill) for path, with_skill, _env in emitted} == {+ assert {+ (path.split("/")[-2], path.split("/")[-1], with_skill)+ for path, with_skill, _runtime_env, _verifier_env in emitted+ } == {("opencode", "with", True),("opencode", "without", False),("claude-code", "with", True),("claude-code", "without", False),}- staged = {(path.split("/")[-2], path.split("/")[-1]): env for path, _with_skill, env in emitted}+ staged = {+ (path.split("/")[-2], path.split("/")[-1]): runtime_env+ for path, _with_skill, runtime_env, _verifier_env in emitted+ }assert staged[("opencode", "with")] == {"NVIDIA_API_KEY": "${NVIDIA_API_KEY}"}assert staged[("claude-code", "with")] == {}+ for _path, _with_skill, runtime_env, verifier_env in emitted:+ assert "LLM_JUDGE_MODEL" not in runtime_env+ assert "SKILL_EVAL_JUDGE_MODEL" not in runtime_env+ assert verifier_env["LLM_JUDGE_MODEL"] == "${LLM_JUDGE_MODEL}"+ assert verifier_env["SKILL_EVAL_JUDGE_MODEL"] == "${SKILL_EVAL_JUDGE_MODEL}"assert launched["opencode"][0].endswith("_harbor-tasks/opencode/with")assert launched["claude-code"][1].endswith("_harbor-tasks/claude-code/without")+ assert launched["opencode"][2]["LLM_JUDGE_MODEL"] == "legacy-judge-model"+ assert launched["opencode"][2]["SKILL_EVAL_JUDGE_MODEL"] == "judge-model"assert "ANTHROPIC_API_KEY" not in launched["opencode"][2]assert "OPENAI_API_KEY" not in launched["opencode"][2]assert launched["claude-code"][2]["NVIDIA_API_KEY"] == "provider-key"diff --git a/tests/test_harbor_runtime_skill_isolation.py b/tests/test_harbor_runtime_skill_isolation.pyindex cae41dd4..668c9ffd 100644--- a/tests/test_harbor_runtime_skill_isolation.py+++ b/tests/test_harbor_runtime_skill_isolation.py@@ -12,6 +12,7 @@ import statimport subprocessimport sysimport tempfile+import tomllibfrom pathlib import Pathfrom types import SimpleNamespace@@ -198,6 +199,10 @@ def test_native_tasks_project_runtime_skills_without_root_evals(tmp_path: Path)reference_skills_dir=references_dir,workspace_skill_paths=[workspace],copy_repo=True,+ verifier_env={+ "LLM_JUDGE_MODEL": "${LLM_JUDGE_MODEL}",+ "SKILL_EVAL_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}",+ },)[0]_assert_runtime_projection(task, target, workspace)@@ -215,6 +220,194 @@ def test_native_tasks_project_runtime_skills_without_root_evals(tmp_path: Path)assert (staged_nested_prebundled / "references" / "evals" / "keep.md").is_file()assert (task / "tests" / "entry.json").is_file()assert (task / "tests" / "grader.py").is_file()+ task_config = tomllib.loads((task / "task.toml").read_text(encoding="utf-8"))+ assert task_config["verifier"]["env"] == {+ "LLM_JUDGE_MODEL": "${LLM_JUDGE_MODEL}",+ "SKILL_EVAL_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}",+ }+ assert "LLM_JUDGE_MODEL" not in task_config["environment"].get("env", {})+ assert "SKILL_EVAL_JUDGE_MODEL" not in task_config["environment"].get("env", {})++[email protected]("name", ["LLM_JUDGE_MODEL", "SKILL_EVAL_JUDGE_MODEL"])+def test_native_tasks_reject_authored_judge_model_in_verifier_env(tmp_path: Path, name: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_minimal_native_task(target)+ task_toml = target / "evals" / "harbor" / "case-001" / "task.toml"+ content = task_toml.read_text(encoding="utf-8").replace(+ "[environment]",+ f'[verifier]\ntimeout_sec = 180.0\n\n[verifier.env]\n{name} = "skill-model"\n\n[environment]',+ )+ task_toml.write_text(content, encoding="utf-8")++ with pytest.raises(ValueError, match=rf"verifier\.env.*{name}|{name}.*verifier\.env"):+ stage_native_harbor_tasks(+ target,+ tmp_path / f"native-verifier-{name.lower()}",+ verifier_env={name: f"${{{name}}}"},+ )+++ "assignment",+ [+ 'LLM_JUDGE_MODEL = "skill-model"',+ 'SKILL_EVAL_JUDGE_MODEL = "skill-model"',+ 'JUDGE_HINT = "$LLM_JUDGE_MODEL"',+ 'JUDGE_HINT = "${SKILL_EVAL_JUDGE_MODEL}"',+ 'JUDGE_HINT = "${LLM_JUDGE_MODEL:-skill-fallback}"',+ 'JUDGE_HINT = "${SKILL_EVAL_JUDGE_MODEL:-skill-fallback}"',+ 'JUDGE_HINT = "%LLM_JUDGE_MODEL%"',+ ],+)+def test_native_tasks_reject_judge_model_controls_in_agent_environment(tmp_path: Path, assignment: str) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_minimal_native_task(target)+ task_toml = target / "evals" / "harbor" / "case-001" / "task.toml"+ task_toml.write_text(+ task_toml.read_text(encoding="utf-8") + f"\n[environment.env]\n{assignment}\n",+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match=r"environment.env.*judge|judge.*environment.env"):+ stage_native_harbor_tasks(target, tmp_path / "native-agent-judge-control")+++def test_native_task_comments_cannot_suppress_judge_model_injection(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_minimal_native_task(target)+ task_toml = target / "evals" / "harbor" / "case-001" / "task.toml"+ content = task_toml.read_text(encoding="utf-8").replace(+ "[environment]",+ '# LLM_JUDGE_MODEL = "${LLM_JUDGE_MODEL}"\n'+ '# SKILL_EVAL_JUDGE_MODEL = "${SKILL_EVAL_JUDGE_MODEL}"\n\n'+ "[environment]",+ )+ task_toml.write_text(content, encoding="utf-8")++ task = stage_native_harbor_tasks(+ target,+ tmp_path / "native-comment-placeholder",+ verifier_env={+ "LLM_JUDGE_MODEL": "${LLM_JUDGE_MODEL}",+ "SKILL_EVAL_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}",+ },+ )[0]++ task_config = tomllib.loads((task / "task.toml").read_text(encoding="utf-8"))+ assert task_config["verifier"]["env"] == {+ "LLM_JUDGE_MODEL": "${LLM_JUDGE_MODEL}",+ "SKILL_EVAL_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}",+ }+++def test_native_task_preserves_noncanonical_verifier_env_table_when_injecting(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_minimal_native_task(target)+ task_toml = target / "evals" / "harbor" / "case-001" / "task.toml"+ content = task_toml.read_text(encoding="utf-8").replace(+ "[environment]",+ '[verifier]\ntimeout_sec = 180.0\n\n[ verifier . env ]\nCUSTOM = "preserved"\n\n[environment]',+ )+ task_toml.write_text(content, encoding="utf-8")++ task = stage_native_harbor_tasks(+ target,+ tmp_path / "native-noncanonical-verifier-table",+ verifier_env={"SKILL_EVAL_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}"},+ )[0]++ task_config = tomllib.loads((task / "task.toml").read_text(encoding="utf-8"))+ assert task_config["verifier"]["env"] == {+ "CUSTOM": "preserved",+ "SKILL_EVAL_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}",+ }+++def test_native_task_reuses_existing_exact_provider_placeholder_when_injecting(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_minimal_native_task(target)+ task_toml = target / "evals" / "harbor" / "case-001" / "task.toml"+ content = task_toml.read_text(encoding="utf-8").replace(+ "[environment]",+ "[verifier]\ntimeout_sec = 180.0\n\n[verifier.env]\n"+ 'SKILL_EVAL_LLM_PROVIDER = "${SKILL_EVAL_LLM_PROVIDER}"\n\n[environment]',+ )+ task_toml.write_text(content, encoding="utf-8")++ task = stage_native_harbor_tasks(+ target,+ tmp_path / "native-existing-provider-placeholder",+ verifier_env={+ "SKILL_EVAL_LLM_MODEL": "${SKILL_EVAL_LLM_MODEL}",+ "SKILL_EVAL_LLM_PROVIDER": "${SKILL_EVAL_LLM_PROVIDER}",+ },+ )[0]++ task_config = tomllib.loads((task / "task.toml").read_text(encoding="utf-8"))+ assert task_config["verifier"]["env"] == {+ "SKILL_EVAL_LLM_MODEL": "${SKILL_EVAL_LLM_MODEL}",+ "SKILL_EVAL_LLM_PROVIDER": "${SKILL_EVAL_LLM_PROVIDER}",+ }++[email protected]("decoy_header", ["[verifier]", "[verifier.env]"])+def test_native_task_ignores_verifier_header_inside_multiline_string(+ tmp_path: Path,+ decoy_header: str,+) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_minimal_native_task(target)+ task_toml = target / "evals" / "harbor" / "case-001" / "task.toml"+ content = task_toml.read_text(encoding="utf-8").replace(+ '[metadata]\nentry_id = "case-001"',+ f'[metadata]\nentry_id = "case-001"\ndescription = """\n{decoy_header}\ndecoy\n"""',+ )+ task_toml.write_text(content, encoding="utf-8")++ task = stage_native_harbor_tasks(+ target,+ tmp_path / "native-multiline-decoy",+ verifier_env={"SKILL_EVAL_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}"},+ )[0]++ task_config = tomllib.loads((task / "task.toml").read_text(encoding="utf-8"))+ assert task_config["metadata"]["description"] == f"{decoy_header}\ndecoy\n"+ assert task_config["verifier"]["env"] == {+ "SKILL_EVAL_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}",+ }++[email protected]("name", ["LLM_JUDGE_MODEL", "SKILL_EVAL_JUDGE_MODEL"])+ "authored_config",+ [+ '[verifier]\ntimeout_sec = 180.0\n\n[verifier.environment]\nworkdir = "/workspace"\n\n'+ '[verifier.environment.env]\n{name} = "skill-model"\n',+ '[[steps]]\nname = "step-one"\n\n[steps.verifier.env]\n{name} = "skill-model"\n',+ '[[steps]]\nname = "step-one"\n\n[steps.verifier.environment]\nworkdir = "/workspace"\n\n'+ '[steps.verifier.environment.env]\n{name} = "skill-model"\n',+ ],+)+def test_native_tasks_reject_judge_model_controls_in_nested_verifier_envs(+ tmp_path: Path,+ name: str,+ authored_config: str,+) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_minimal_native_task(target)+ task_toml = target / "evals" / "harbor" / "case-001" / "task.toml"+ task_toml.write_text(+ task_toml.read_text(encoding="utf-8") + "\n" + authored_config.format(name=name),+ encoding="utf-8",+ )++ with pytest.raises(ValueError, match=rf"verifier.*{name}|{name}.*verifier"):+ stage_native_harbor_tasks(+ target,+ tmp_path / "native-nested-verifier-control",+ verifier_env={"SKILL_EVAL_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}"},+ )def test_native_task_staging_uses_one_private_evals_snapshot(diff --git a/tests/test_tier3_public_runtime.py b/tests/test_tier3_public_runtime.pyindex c4293fe0..5b45436e 100644--- a/tests/test_tier3_public_runtime.py+++ b/tests/test_tier3_public_runtime.py@@ -356,14 +356,18 @@ def test_generated_task_keeps_evaluator_provider_variables_out_of_agent_environmhas_skill=True,runtime_env={"SERVICE_API_TOKEN": "${SERVICE_API_TOKEN}"},verifier_env={+ "LLM_JUDGE_MODEL": "${LLM_JUDGE_MODEL}","SKILL_EVAL_LLM_PROVIDER": "${SKILL_EVAL_LLM_PROVIDER}",+ "SKILL_EVAL_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}","NVIDIA_API_KEY": "${NVIDIA_API_KEY}",},)task = tomllib.loads((tmp_path / "task.toml").read_text(encoding="utf-8"))assert task["verifier"]["env"] == {+ "LLM_JUDGE_MODEL": "${LLM_JUDGE_MODEL}","NVIDIA_API_KEY": "${NVIDIA_API_KEY}",+ "SKILL_EVAL_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}","SKILL_EVAL_LLM_PROVIDER": "${SKILL_EVAL_LLM_PROVIDER}",}assert task["environment"]["env"] == {