Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

fix(tier3): normalize judge model aliases

Christopher Kevin2e34871bf842
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f3951892..12fbbff5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -90,16 +90,18 @@ All notable changes to SkillEvaluator are documented in this file.
### Fixed
-- Tier 3 now forwards host-configured `LLM_JUDGE_MODEL` and
- `SKILL_EVAL_JUDGE_MODEL` overrides through Harbor's verifier-only job layer
- for standard grading. They are not injected into the evaluated agent's
- initial environment; skill-authored `runtime_env` and native task
- `[environment.env]` tables cannot set or alias either operator-controlled
- override.
+- Tier 3 now normalizes host-configured `LLM_JUDGE_MODEL` and
+ `SKILL_EVAL_JUDGE_MODEL` overrides in Harbor's parent process and forwards
+ the selected value through its verifier-only job layer for standard grading.
+ This lets native separate-verifier placeholders resolve without injecting
+ either name into the evaluated agent's initial environment. Skill-authored
+ `runtime_env` and native task `[environment.env]` tables cannot set or alias
+ either operator-controlled override.
Native verifier declarations remain compatible, while the job-level value
takes precedence during standard grading. Tier 3 results now record the
- resolved judge provider, model, source, and whether a dedicated job-wide
- override was applied, separately from agent models.
+ configured judge provider, model, source, and whether a dedicated job-wide
+ override was applied, separately from agent models. A provider fallback may
+ still use a different model for an individual judge call.
- Quality scoring now uses boundary-aware and context-aware matching for XML tags,
reserved names, MCP guidance, README references, time references, exclusivity
language, instruction action verbs, and nested Markdown links, avoiding
diff --git a/docs/environment-variables.mdx b/docs/environment-variables.mdx
index 335645ca..6c6aebc2 100644
--- a/docs/environment-variables.mdx
+++ b/docs/environment-variables.mdx
@@ -101,7 +101,7 @@ execution environments are covered in
| `SKILLEVALUATOR_LOCAL_INHERIT_AGENT_KEYS` | `false` | When true, local-mode trials inherit the host's agent credential variables into their ambient environment. Off by default so a hostile skill command cannot read credentials — agents and verifiers receive credentials per-exec instead. |
| `SKILLEVALUATOR_RUNTIME_DIR` | `~/.local/share/skillevaluator/runtimes` | Where local mode looks for managed agent CLIs. Must be a dedicated subdirectory — not the home directory or one of its parents. |
| `SKILL_EVAL_HARBOR_PREBUILD_TASK_ENVS` | unset | Set to exactly `1` to opt in to pre-building Harbor task environment images before trials start (other truthy spellings are not honored). Opt-in because concurrent trial cleanup can remove a shared prebuilt image that sibling trials still need. |
-| `SKILL_EVAL_JUDGE_MODEL` | falls back to `SKILL_EVAL_LLM_MODEL` | Operator-controlled model override for the Tier 3 dimension judge in standard grading. `LLM_JUDGE_MODEL` is honored as a higher-precedence spelling; when neither is set, the judge uses `SKILL_EVAL_LLM_MODEL`, then the built-in judge default. Both spellings change only the model used with the configured `SKILL_EVAL_LLM_PROVIDER`; they do not select a separate judge provider, so the model must be supported by that provider. Standard grading passes a dedicated override through Harbor's verifier-only job layer, which takes precedence over task and step verifier declarations. It is not injected into the evaluated agent's initial environment; attempts to set or reference it from skill-authored `runtime_env` or native `[environment.env]` tables are rejected. Native custom-only graders keep ownership of their verifier configuration. Shared agent/verifier containers are a process boundary, not a confidentiality boundary; use a separate verifier environment when the model name itself must be hidden from untrusted agent processes. `run_config.json` records the resolved judge provider, model, source, and `override_applied` separately from agent models. When `override_applied` is false for a native Harbor task, an authored task or step verifier declaration may supersede that recorded fallback. |
+| `SKILL_EVAL_JUDGE_MODEL` | falls back to `SKILL_EVAL_LLM_MODEL` | Operator-controlled model override for the Tier 3 dimension judge in standard grading. `LLM_JUDGE_MODEL` is honored as a higher-precedence spelling; when neither is set, the judge uses `SKILL_EVAL_LLM_MODEL`, then the built-in judge default. Both spellings change only the model used with the configured `SKILL_EVAL_LLM_PROVIDER`; they do not select a separate judge provider, so the model must be supported by that provider. Standard grading makes both spellings resolve to the selected override in Harbor's parent process, then forwards them through its verifier-only job layer, which takes precedence over task and step verifier declarations. Neither name is injected into the evaluated agent's initial environment; attempts to set or reference one from skill-authored `runtime_env` or native `[environment.env]` tables are rejected. Native custom-only graders keep ownership of their verifier configuration. Shared agent/verifier containers are a process boundary, not a confidentiality boundary; use a separate verifier environment when the model name itself must be hidden from untrusted agent processes. `run_config.json` records the configured judge provider, model, source, and `override_applied` separately from agent models; an allowed provider fallback may use a different model for an individual judge call. When `override_applied` is false for a native Harbor task, an authored task or step verifier declaration may supersede that recorded fallback. |
| `SKILLEVALUATOR_HARBOR_DIAGNOSTIC_ARTIFACT_MAX_BYTES` | `5242880` (5 MiB) | Size cap per diagnostic artifact copied from a Harbor trial into the results directory; oversized artifacts are skipped and recorded in the artifact manifest. |
## In-container grader contract
diff --git a/src/skillevaluator/tier3/harbor/runner.py b/src/skillevaluator/tier3/harbor/runner.py
index 691f71ee..a9019336 100644
--- a/src/skillevaluator/tier3/harbor/runner.py
+++ b/src/skillevaluator/tier3/harbor/runner.py
@@ -831,18 +831,35 @@ def _judge_model_config(
}
-def _job_judge_verifier_env(provider_env: Mapping[str, str], grading_mode: str) -> dict[str, str]:
- """Return host-resolved judge overrides for Harbor's verifier-only job layer."""
+def _job_judge_override(provider_env: Mapping[str, str], grading_mode: str) -> tuple[str, str] | None:
+ """Return the selected dedicated host override name and value, if enabled."""
if grading_mode == "custom_only":
+ return None
+ for name in ("LLM_JUDGE_MODEL", "SKILL_EVAL_JUDGE_MODEL"):
+ if value := provider_env.get(name):
+ return name, value
+ return None
+
+
+def _job_judge_verifier_env(provider_env: Mapping[str, str], grading_mode: str) -> dict[str, str]:
+ """Return placeholder-based judge overrides for Harbor's verifier job layer."""
+ selected = _job_judge_override(provider_env, grading_mode)
+ if selected is None:
return {}
- if provider_env.get("LLM_JUDGE_MODEL"):
- return {"LLM_JUDGE_MODEL": "${LLM_JUDGE_MODEL}"}
- if provider_env.get("SKILL_EVAL_JUDGE_MODEL"):
- # The bundled verifier reads LLM_JUDGE_MODEL first. Place either host
- # override at that highest-precedence key so a task-authored legacy
- # value cannot win after Harbor merges task, step, and job env.
- return {"LLM_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}"}
- return {}
+ source, _value = selected
+ # Harbor resolves every task-authored placeholder before verifier startup.
+ # Override both spellings from the selected host source so a stale alias
+ # cannot fail resolution or survive task/step/job environment merging.
+ return dict.fromkeys(sorted(_VERIFIER_JUDGE_MODEL_ENV_VARS), f"${{{source}}}")
+
+
+def _job_judge_subprocess_env(provider_env: Mapping[str, str], grading_mode: str) -> dict[str, str]:
+ """Make both aliases resolvable while Harbor constructs verifier environments."""
+ selected = _job_judge_override(provider_env, grading_mode)
+ if selected is None:
+ return {}
+ _source, value = selected
+ return dict.fromkeys(_VERIFIER_JUDGE_MODEL_ENV_VARS, value)
def _agent_credentials(
@@ -1889,6 +1906,7 @@ def _run_harbor_eval_impl(
verifier_env = {**configured_runtime_env, **provider_env}
staged_verifier_env = {name: f"${{{name}}}" for name in verifier_env if name not in _VERIFIER_JUDGE_MODEL_ENV_VARS}
job_judge_verifier_env = _job_judge_verifier_env(provider_env, grading_mode)
+ job_judge_subprocess_env = _job_judge_subprocess_env(provider_env, grading_mode)
include_values = [*workspace_config.get("include", []), *(include_skills or [])]
if include_values and workspace_mode != "group":
@@ -2140,7 +2158,7 @@ def _run_harbor_eval_impl(
model=model_resolution[agent]["model"],
env_mode=env_mode,
jobs_dir=jobs_dir,
- run_env=runtime_plans[agent].subprocess_env,
+ run_env={**runtime_plans[agent].subprocess_env, **job_judge_subprocess_env},
timeout_multiplier=float(timeout_multiplier),
override_cpus=override_cpus,
override_memory_mb=override_memory_mb,
@@ -2190,7 +2208,7 @@ def _run_harbor_eval_impl(
with_skill=agent_task_dirs[agent][0],
baseline=agent_task_dirs[agent][1],
jobs_dir=jobs_dir,
- run_env=dict(runtime_plans[agent].subprocess_env),
+ run_env={**runtime_plans[agent].subprocess_env, **job_judge_subprocess_env},
n_attempts=n_attempts,
n_concurrent=n_concurrent,
timeout_multiplier=float(timeout_multiplier),
diff --git a/tests/test_harbor_runner_environment.py b/tests/test_harbor_runner_environment.py
index 1cf79c6d..f0d561f3 100644
--- a/tests/test_harbor_runner_environment.py
+++ b/tests/test_harbor_runner_environment.py
@@ -8,6 +8,7 @@ from __future__ import annotations
import os
import pytest
+from harbor.utils.env import resolve_env_vars
from skillevaluator.provider_config import ProviderConfig
from skillevaluator.tier3.harbor import runner
@@ -276,6 +277,7 @@ def test_custom_only_does_not_force_the_standard_judge_model_into_custom_verifie
}
assert runner._job_judge_verifier_env(provider_env, "custom_only") == {}
+ assert runner._job_judge_subprocess_env(provider_env, "custom_only") == {}
assert runner._judge_model_config(_provider("nv_build"), provider_env, "custom_only") == {
"enabled": False,
}
@@ -286,11 +288,17 @@ def test_custom_only_does_not_force_the_standard_judge_model_into_custom_verifie
[
(
{"SKILL_EVAL_JUDGE_MODEL": "operator-model"},
- {"LLM_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}"},
+ {
+ "LLM_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}",
+ "SKILL_EVAL_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}",
+ },
),
(
{"LLM_JUDGE_MODEL": "legacy-model", "SKILL_EVAL_JUDGE_MODEL": "operator-model"},
- {"LLM_JUDGE_MODEL": "${LLM_JUDGE_MODEL}"},
+ {
+ "LLM_JUDGE_MODEL": "${LLM_JUDGE_MODEL}",
+ "SKILL_EVAL_JUDGE_MODEL": "${LLM_JUDGE_MODEL}",
+ },
),
],
)
@@ -301,6 +309,78 @@ def test_standard_judge_override_occupies_the_highest_precedence_verifier_key(
assert runner._job_judge_verifier_env(provider_env, "default") == expected
+ "host_env",
+ [
+ {"LLM_JUDGE_MODEL": "host-legacy"},
+ {"SKILL_EVAL_JUDGE_MODEL": "host-canonical"},
+ {"LLM_JUDGE_MODEL": "host-legacy", "SKILL_EVAL_JUDGE_MODEL": "host-canonical"},
+ ],
+)
+def test_standard_judge_override_replaces_both_native_aliases_before_harbor_resolution(
+ monkeypatch: pytest.MonkeyPatch,
+ host_env: dict[str, str],
+) -> None:
+ for name in ("LLM_JUDGE_MODEL", "SKILL_EVAL_JUDGE_MODEL"):
+ monkeypatch.delenv(name, raising=False)
+ for name, value in host_env.items():
+ monkeypatch.setenv(name, value)
+ provider_env = runner._provider_environment(_provider())
+ task_env = {
+ "LLM_JUDGE_MODEL": "task-legacy",
+ "SKILL_EVAL_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}",
+ }
+
+ merged = {**task_env, **runner._job_judge_verifier_env(provider_env, "default")}
+ resolved = resolve_env_vars(merged)
+ selected = host_env.get("LLM_JUDGE_MODEL") or host_env["SKILL_EVAL_JUDGE_MODEL"]
+
+ assert resolved == {
+ "LLM_JUDGE_MODEL": selected,
+ "SKILL_EVAL_JUDGE_MODEL": selected,
+ }
+
+
[email protected]("grading_mode", ["default", "default_plus_custom"])
+ "host_env",
+ [
+ {"LLM_JUDGE_MODEL": "host-legacy"},
+ {"SKILL_EVAL_JUDGE_MODEL": "host-canonical"},
+ {"LLM_JUDGE_MODEL": "host-legacy", "SKILL_EVAL_JUDGE_MODEL": "host-canonical"},
+ ],
+)
+def test_standard_judge_override_resolves_both_aliases_for_separate_verifier_startup(
+ monkeypatch: pytest.MonkeyPatch,
+ grading_mode: str,
+ host_env: dict[str, str],
+) -> None:
+ for name in ("LLM_JUDGE_MODEL", "SKILL_EVAL_JUDGE_MODEL"):
+ monkeypatch.delenv(name, raising=False)
+ for name, value in host_env.items():
+ monkeypatch.setenv(name, value)
+ provider_env = runner._provider_environment(_provider())
+ subprocess_aliases = runner._job_judge_subprocess_env(provider_env, grading_mode)
+
+ with monkeypatch.context() as resolver_env:
+ for name in ("LLM_JUDGE_MODEL", "SKILL_EVAL_JUDGE_MODEL"):
+ resolver_env.delenv(name, raising=False)
+ for name, value in subprocess_aliases.items():
+ resolver_env.setenv(name, value)
+ resolved = resolve_env_vars(
+ {
+ "LLM_JUDGE_MODEL": "${LLM_JUDGE_MODEL}",
+ "SKILL_EVAL_JUDGE_MODEL": "${SKILL_EVAL_JUDGE_MODEL}",
+ }
+ )
+
+ selected = host_env.get("LLM_JUDGE_MODEL") or host_env["SKILL_EVAL_JUDGE_MODEL"]
+ assert resolved == {
+ "LLM_JUDGE_MODEL": selected,
+ "SKILL_EVAL_JUDGE_MODEL": selected,
+ }
+
+
@pytest.mark.parametrize(
("configured_model", "expected_source"),
[(None, "provider default"), ("operator-model", "SKILL_EVAL_LLM_MODEL")],
@@ -610,11 +690,14 @@ def test_run_harbor_eval_stages_per_agent_credential_trees(
assert launched["opencode"][0].endswith("_harbor-tasks/opencode/with")
assert launched["claude-code"][1].endswith("_harbor-tasks/claude-code/without")
assert launched["opencode"][2]["LLM_JUDGE_MODEL"] == "legacy-judge-model"
- assert launched["opencode"][2]["SKILL_EVAL_JUDGE_MODEL"] == "judge-model"
+ assert launched["opencode"][2]["SKILL_EVAL_JUDGE_MODEL"] == "legacy-judge-model"
assert "ANTHROPIC_API_KEY" not in launched["opencode"][2]
assert "OPENAI_API_KEY" not in launched["opencode"][2]
assert launched["claude-code"][2]["NVIDIA_API_KEY"] == "provider-key"
- assert launched["opencode"][3] == {"LLM_JUDGE_MODEL": "${LLM_JUDGE_MODEL}"}
+ assert launched["opencode"][3] == {
+ "LLM_JUDGE_MODEL": "${LLM_JUDGE_MODEL}",
+ "SKILL_EVAL_JUDGE_MODEL": "${LLM_JUDGE_MODEL}",
+ }
assert launched["claude-code"][3] == launched["opencode"][3]
assert result["run_config"]["judge"] == {
"enabled": True,
diff --git a/tests/test_harbor_runtime_preflight.py b/tests/test_harbor_runtime_preflight.py
index cbb0eb71..acab056a 100644
--- a/tests/test_harbor_runtime_preflight.py
+++ b/tests/test_harbor_runtime_preflight.py
@@ -691,6 +691,9 @@ def test_runtime_preflight_failure_stops_full_matrix(monkeypatch, tmp_path: Path
return [task]
full_matrix = Mock(return_value=[])
+ preflight_run_env: dict[str, str] = {}
+ monkeypatch.setenv("LLM_JUDGE_MODEL", "host-legacy")
+ monkeypatch.delenv("SKILL_EVAL_JUDGE_MODEL", raising=False)
monkeypatch.setattr(runner, "resolve_llm_provider", lambda: provider)
monkeypatch.setattr(
runner,
@@ -701,17 +704,18 @@ def test_runtime_preflight_failure_stops_full_matrix(monkeypatch, tmp_path: Path
monkeypatch.setattr(runner, "_check_prerequisites", lambda **_kwargs: [])
monkeypatch.setattr(runner, "generate_harbor_tasks", emit)
monkeypatch.setattr(runner, "_run_agent_pair", full_matrix)
- monkeypatch.setattr(
- runtime_preflight,
- "run_agent_runtime_preflight",
- lambda **_kwargs: runtime_preflight.PreflightResult(
+
+ def fail_preflight(**kwargs):
+ preflight_run_env.update(kwargs["run_env"])
+ return runtime_preflight.PreflightResult(
False,
"opencode",
"nvidia/meta/llama-3.1-8b-instruct",
"401 Unauthorized",
"runtime-preflight-opencode",
- ),
- )
+ )
+
+ monkeypatch.setattr(runtime_preflight, "run_agent_runtime_preflight", fail_preflight)
result = runner.run_harbor_eval(
skill,
@@ -723,6 +727,8 @@ def test_runtime_preflight_failure_stops_full_matrix(monkeypatch, tmp_path: Path
assert result["execution_status"] == "failed"
assert result["execution_errors"] == ["opencode runtime preflight failed: 401 Unauthorized"]
+ assert preflight_run_env["LLM_JUDGE_MODEL"] == "host-legacy"
+ assert preflight_run_env["SKILL_EVAL_JUDGE_MODEL"] == "host-legacy"
full_matrix.assert_not_called()
result_path = Path(result["run_dir"]) / "result.json"
assert result["result_path"] == str(result_path)