Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier3): align Anthropic endpoints and fail closed on judge errors (#56)
Christopher Kevin799f9d61d3f2
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex b6a9e3ac..78f312cf 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -280,6 +280,12 @@ jobs:tests/test_harbor_secure_copy.py::test_checked_windows_fallback_verifies_portable_chmod_semanticstests/test_harbor_secure_copy.py::test_native_windows_fallback_copies_tree_and_filetests/test_harbor_secure_copy.py::test_native_windows_tree_fingerprint_binds_writable_mode+ tests/test_issue55_collector_truth.py::test_diagnostic_copy_preserves_bounded_redacted_regular_file+ tests/test_issue55_collector_truth.py::test_diagnostic_copy_rejects_hard_linked_artifact+ tests/test_issue55_collector_truth.py::test_diagnostic_copy_rejects_native_windows_trial_root_junction+ tests/test_issue55_collector_truth.py::test_diagnostic_copy_rejects_native_windows_agent_junction+ tests/test_issue55_collector_truth.py::test_diagnostic_copy_rejects_native_windows_destination_parent_junction+ tests/test_issue55_collector_truth.py::test_judge_sidecar_scan_rejects_native_windows_verifier_junctiontests/test_results_location.pytests/test_results_location_legacy_compatibility.pytests/test_tier3_compare.pydiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 1f7d7094..c268ca29 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -53,6 +53,10 @@ All notable changes to SkillEvaluator are documented in this file.### Fixed+- Fixed Anthropic API-root normalization across evaluator and Claude Code+ paths, and made required Tier 3 judge failures fail closed instead of+ appearing as numeric zero scores or publishing misleading quality results+ ([#55](https://github.com/NVIDIA/SkillEvaluator/issues/55)).- Tier 3 now normalizes host-configured `LLM_JUDGE_MODEL` and`SKILL_EVAL_JUDGE_MODEL` overrides in Harbor's parent process and forwardsthe selected value through its verifier-only job layer for standard grading.diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.mdindex 88b05932..69664187 100644--- a/THIRD_PARTY_NOTICES.md+++ b/THIRD_PARTY_NOTICES.md@@ -6,7 +6,7 @@ records the exact resolved dependency set used for this release.| Dependency group | Packages and licenses || --- | --- |-| Base | Click (BSD-3-Clause), Jinja2 (BSD-3-Clause), Pydantic (MIT), PyYAML (MIT), Rich (MIT) |+| Base | Click (BSD-3-Clause), IDNA (BSD-3-Clause), Jinja2 (BSD-3-Clause), Pydantic (MIT), PyYAML (MIT), Rich (MIT) || LLM | Anthropic (MIT), Boto3 (Apache-2.0), LiteLLM (MIT), OpenAI (Apache-2.0) || Tier 3 | Harbor (Apache-2.0) || Security | Bandit (Apache-2.0), pip-audit (Apache-2.0) |diff --git a/docs/configuration.mdx b/docs/configuration.mdxindex 44d30d4e..f6f43985 100644--- a/docs/configuration.mdx+++ b/docs/configuration.mdx@@ -46,7 +46,11 @@ Set `SKILL_EVAL_LLM_PROVIDER` to one of `openai`, `anthropic`, `nv_build`, `bedrexport ANTHROPIC_API_KEY='sk-ant-...'```- Default model: `claude-opus-5`, using the Anthropic SDK's default endpoint. Override with `SKILL_EVAL_LLM_BASE_URL` (takes precedence) or `ANTHROPIC_BASE_URL`.+ Default model: `claude-opus-5`, using the Anthropic SDK's default endpoint. Override it with `SKILL_EVAL_LLM_BASE_URL` (highest precedence) or `ANTHROPIC_BASE_URL`; when neither is set, SkillEvaluator uses the Anthropic default.++ For Anthropic, either base-URL variable must be an absolute HTTP(S) API root. Supply an unversioned root such as `https://gateway.example.com` or a legacy URL whose final segment is `/v1`, such as `https://gateway.example.com/team/v1`. SkillEvaluator normalizes the latter to its API root so the Anthropic SDK, the Tier 3 verifier, and a matching Claude Code agent each form request paths with exactly one `/v1/messages` suffix. An independently credentialed Claude Code route applies the same normalization and validation to `ANTHROPIC_BASE_URL` when the evaluator uses another provider. Routing through path prefixes, ports, IP addresses, internationalized domain names, and ordinary safe percent escapes is preserved.++ Validation fails before Harbor starts if the URL contains a full `/v1/messages` endpoint, embedded credentials, a query or fragment, whitespace or control characters, a backslash, an invalid authority, interior empty or dot path segments, or ambiguous multi-layer endpoint encoding. This root convention is specific to Anthropic; OpenAI-compatible endpoints commonly retain a terminal `/v1` unchanged.</Tab><Tab title="NVIDIA Build" language="nv_build">```bash title="NVIDIA Build setup"@@ -166,6 +170,12 @@ Both roles resolve from the operator's environment; a skill's `evals/config.yml`- **NVIDIA Build on Harbor cloud modes** (`daytona`, `e2b`, ...): the bridge is not available. `claude-code` needs its own native `ANTHROPIC_API_KEY`, `codex` needs a full OpenAI Responses API credential (`OPENAI_API_KEY` plus `OPENAI_BASE_URL`), and each needs an explicit model via `--agent-model` or `harbor.agents.<agent>.model`.- **Matching native provider**: when the evaluator provider is the agent's own (for example `anthropic` with `claude-code`, or `openai` with `codex`), the same key serves both roles automatically.+### Judge failures are not zero scores++Standard Tier 3 grading treats a required judge transport, authentication, timeout, HTTP, or exhausted response-parsing error as an evaluation infrastructure failure. A condition containing a failed or incomplete default trial publishes none of its scores, dimensions, pass@k, or quality findings, and no cross-condition lift is published. Redacted diagnostics and coverage information remain available. A genuine model score of `0.0` is still a valid, published score.++In `default_plus_custom` grading, a custom score cannot rescue failed standard judging. `custom_only` grading is unchanged: it does not invoke the standard judge and continues to publish a valid custom result.+The agent's model lives in the skill's `evals/config.yml`:```yaml title="evals/config.yml"diff --git a/docs/environment-variables.mdx b/docs/environment-variables.mdxindex bc1e55b4..8bdd7ce7 100644--- a/docs/environment-variables.mdx+++ b/docs/environment-variables.mdx@@ -33,10 +33,16 @@ These variables select and configure the provider used for LLM-backed checks and| `OPENAI_API_KEY` | — | Credential for the `openai` provider. || `ANTHROPIC_API_KEY` | — | Credential for the `anthropic` provider. || `OPENAI_BASE_URL` | `https://api.openai.com/v1` | Alternate OpenAI endpoint; `SKILL_EVAL_LLM_BASE_URL` wins when both are set. |-| `ANTHROPIC_BASE_URL` | Anthropic SDK default | Alternate Anthropic endpoint; `SKILL_EVAL_LLM_BASE_URL` wins when both are set. |+| `ANTHROPIC_BASE_URL` | Anthropic SDK default | Absolute HTTP(S) Anthropic API root. Accepts an unversioned root or a legacy terminal `/v1`; `SKILL_EVAL_LLM_BASE_URL` wins when both are set. || `AWS_REGION` | `us-west-2` | Region for the `bedrock` provider. || `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`, `AWS_PROFILE`, ... | — | Standard AWS credential chain, used as-is by the `bedrock` provider. |+### Anthropic endpoint roots++When `anthropic` is the selected evaluator provider, endpoint precedence is `SKILL_EVAL_LLM_BASE_URL`, then `ANTHROPIC_BASE_URL`, then the Anthropic default. A root such as `https://gateway.example.com` and a legacy terminal `/v1` root such as `https://gateway.example.com/team/v1` both produce request paths with exactly one `/v1/messages` suffix for the Anthropic SDK, the Tier 3 verifier, and a matching Claude Code agent. An independently credentialed Claude Code route applies the same normalization and validation to `ANTHROPIC_BASE_URL` when the evaluator uses another provider. Routing through path prefixes, ports, IP addresses, internationalized domain names, and ordinary safe percent escapes is preserved.++Anthropic configuration fails before Harbor starts for a full `/v1/messages` endpoint, embedded credentials, a query or fragment, whitespace or control characters, a backslash, an invalid authority, interior empty or dot path segments, or ambiguous multi-layer endpoint encoding. This normalization applies only to Anthropic. OpenAI-compatible endpoints commonly include `/v1`, which remains unchanged.+### Auto-selectionWhen `SKILL_EVAL_LLM_PROVIDER` is unset, auto-selection only happens when exactly one public credential is exported:@@ -102,7 +108,13 @@ execution environments are covered in| `SKILLEVALUATOR_RUNTIME_DIR` | `~/.local/share/skillevaluator/runtimes` | Where local mode looks for managed agent CLIs. Must be a dedicated subdirectory — not the home directory or one of its parents. || `SKILL_EVAL_HARBOR_PREBUILD_TASK_ENVS` | unset | Set to exactly `1` to opt in to pre-building Harbor task environment images before trials start (other truthy spellings are not honored). Opt-in because concurrent trial cleanup can remove a shared prebuilt image that sibling trials still need. || `SKILL_EVAL_JUDGE_MODEL` | falls back to `SKILL_EVAL_LLM_MODEL` | Operator-controlled model override for the Tier 3 dimension judge in standard grading. `LLM_JUDGE_MODEL` is honored as a higher-precedence spelling; when neither is set, the judge uses `SKILL_EVAL_LLM_MODEL`, then the built-in judge default. Both spellings change only the model used with the configured `SKILL_EVAL_LLM_PROVIDER`; they do not select a separate judge provider, so the model must be supported by that provider. Standard grading makes both spellings resolve to the selected override in Harbor's parent process, then forwards them through its verifier-only job layer, which takes precedence over task and step verifier declarations. Neither name is injected into the evaluated agent's initial environment; attempts to set or reference one from skill-authored `runtime_env` or native `[environment.env]` tables are rejected. Native custom-only graders keep ownership of their verifier configuration. Shared agent/verifier containers are a process boundary, not a confidentiality boundary; use a separate verifier environment when the model name itself must be hidden from untrusted agent processes. `run_config.json` records the configured judge provider, model, source, and `override_applied` separately from agent models; an allowed provider fallback may use a different model for an individual judge call. When `override_applied` is false for a native Harbor task, an authored task or step verifier declaration may supersede that recorded fallback. |-| `SKILLEVALUATOR_HARBOR_DIAGNOSTIC_ARTIFACT_MAX_BYTES` | `5242880` (5 MiB) | Size cap per diagnostic artifact copied from a Harbor trial into the results directory; oversized artifacts are skipped and recorded in the artifact manifest. |+| `SKILLEVALUATOR_HARBOR_DIAGNOSTIC_ARTIFACT_MAX_BYTES` | `5242880` (5 MiB) | Size cap per diagnostic artifact copied from a Harbor trial into the results directory. Values are bounded to 0–67108864 bytes (64 MiB); `0` skips non-empty artifacts. Oversized artifacts are skipped and recorded in the artifact manifest. |++### Tier 3 judge failure semantics++In standard grading, a required judge transport, authentication, timeout, HTTP, or exhausted response-parsing error is an evaluation infrastructure failure, not a numeric `0.0`. A condition containing a failed or incomplete default trial publishes none of its scores, dimensions, pass@k, or quality findings, and no cross-condition lift is published. Redacted diagnostics and coverage information remain available. A genuine model score of `0.0` remains valid and is published normally.++`default_plus_custom` cannot use a custom score to rescue failed standard judging. `custom_only` is unchanged and does not invoke the standard judge.## In-container grader contractdiff --git a/pyproject.toml b/pyproject.tomlindex 3a5e1090..621da630 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -38,6 +38,7 @@ license-files = [# or LLM clients. Those live behind the extras below.dependencies = ["click>=8.3.3,<9",+ "idna>=3.10,<4","jinja2>=3.1","pydantic>=2.11.0","pyyaml>=6.0",diff --git a/src/skillevaluator/evaluation/tier3_report.py b/src/skillevaluator/evaluation/tier3_report.pyindex 2b2bd718..1a880c2a 100644--- a/src/skillevaluator/evaluation/tier3_report.py+++ b/src/skillevaluator/evaluation/tier3_report.py@@ -1090,6 +1090,16 @@ def _replace_with_minimal_payload(payload: dict[str, Any], report_budget: _Repor# ---------------------------------------------------------------------------+def _condition_quality_available(info: dict[str, Any], condition: str) -> bool:+ """Return whether a condition may contribute score-bearing report fields."""+ conditions = info.get("conditions")+ condition_info = conditions.get(condition) if isinstance(conditions, dict) else None+ status = condition_info.get("execution_status") if isinstance(condition_info, dict) else None+ if status is None:+ status = info.get("execution_status")+ return status not in {"failed", "unknown", "skipped"}++def _build_agent(name: str,info: dict[str, Any],@@ -1099,6 +1109,12 @@ def _build_agent(with_scores = info.get("with_skill") or {}without_scores = info.get("without_skill") or {}lift_data = info.get("lift") or {}+ with_quality_available = _condition_quality_available(info, "with_skill")+ baseline_quality_available = _condition_quality_available(info, "without_skill")+ if not with_quality_available:+ with_scores = {}+ if not baseline_quality_available:+ without_scores = {}evaluators = _build_evaluators(metrics, with_scores, without_scores, lift_data)dimensions = _build_dimensions(@@ -1109,12 +1125,14 @@ def _build_agent()overall_ws = _mean([d["with_skill"] for d in dimensions])overall_bl = _mean([d["baseline"] for d in dimensions])- if overall_ws is None and not metrics:- overall_ws = _mean([reward.get("overall") for reward in info.get("rewards", []) if isinstance(reward, dict)])- if overall_bl is None and not metrics:- overall_bl = _mean(- [reward.get("overall") for reward in info.get("rewards_baseline", []) if isinstance(reward, dict)]- )+ if overall_ws is None and not metrics and with_quality_available:+ overall_ws = _finite_float(info.get("overall_with_skill"))+ if overall_ws is None and info.get("rewards_complete") is not False:+ overall_ws = _logical_reward_mean(info.get("rewards"), "overall")+ if overall_bl is None and not metrics and baseline_quality_available:+ overall_bl = _finite_float(info.get("overall_without_skill"))+ if overall_bl is None and info.get("rewards_baseline_complete") is not False:+ overall_bl = _logical_reward_mean(info.get("rewards_baseline"), "overall")overall_lift = round(overall_ws - overall_bl, 4) if overall_ws is not None and overall_bl is not None else Nonetrials = _normalize_trials(info.get("rewards") or [], metrics)@@ -1142,7 +1160,7 @@ def _build_agent("baseline": overall_bl,"lift": overall_lift,"num_trials": int(info.get("num_trials", 0) or 0),- "num_trials_baseline": len(baseline_trials),+ "num_trials_baseline": int(info.get("num_trials_baseline", len(baseline_trials)) or 0),"trials": trials,"trials_baseline": baseline_trials,"pass_at_k": {@@ -1279,9 +1297,20 @@ def _deterministic_reasoning(value = _finite_float(with_scores.get(signal))if value is not None:parts.append(f"{signal}={value:.2f}")+ numeric_with_skill = _finite_float(ws)+ numeric_baseline = _finite_float(bl)+ if numeric_with_skill is None:+ bullets = ["With-skill score unavailable; no verdict was computed."]+ if numeric_baseline is not None:+ bullets.append(+ f"Baseline score {numeric_baseline:.2f}; lift cannot be computed without a with-skill score."+ )+ else:+ bullets.append("No baseline run available; lift cannot be computed.")+ return bullets, " ".join(bullets)bullets = _human_reasoning_bullets(- with_skill=_finite_float(ws) or 0.0,- baseline=_finite_float(bl),+ with_skill=numeric_with_skill,+ baseline=numeric_baseline,lift=lift,parts=parts,)@@ -2534,6 +2563,20 @@ def _mean(values: list[float]) -> float | None:return round(sum(numeric) / len(numeric), 4) if numeric else None+def _logical_reward_mean(rewards: Any, field: str) -> float | None:+ """Average a persisted reward field once per logical Harbor trial."""+ if not isinstance(rewards, list):+ return None+ from skillevaluator.tier3.harbor.report_data import logical_trial_reward_groups++ group_means = [+ group_mean+ for group in logical_trial_reward_groups([reward for reward in rewards if isinstance(reward, dict)])+ if (group_mean := _mean([reward.get(field) for reward in group])) is not None+ ]+ return _mean(group_means)++def _as_float(value: Any) -> float:return _finite_float(value) or 0.0diff --git a/src/skillevaluator/provider_config.py b/src/skillevaluator/provider_config.pyindex 09f160a2..c3e1f9ea 100644--- a/src/skillevaluator/provider_config.py+++ b/src/skillevaluator/provider_config.py@@ -5,10 +5,15 @@from __future__ import annotations+import ipaddressimport osimport re+import unicodedatafrom collections.abc import Mappingfrom dataclasses import dataclass+from urllib.parse import quote, unquote_to_bytes, urlsplit, urlunsplit++import idnaPUBLIC_NVIDIA_BUILD_BASE_URL = "https://integrate.api.nvidia.com/v1"OPENAI_BASE_URL = "https://api.openai.com/v1"@@ -34,6 +39,13 @@ _EMBEDDING_DEFAULT_MODELS = {"nv_build": "nvidia/nv-embed-v1",}_SUPPORTED_PROVIDERS = frozenset({"openai", "anthropic", "nv_build", "bedrock", "openai-compatible"})+_ANTHROPIC_DNS_LABEL_RE = re.compile(r"^[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?$")+_ANTHROPIC_INTERNAL_LABEL_RE = re.compile(r"^[A-Za-z0-9_](?:[A-Za-z0-9_-]{0,61}[A-Za-z0-9_])?$")+_ANTHROPIC_IPV6_ZONE_RE = re.compile(r"^[A-Za-z0-9._~-]+$")+_ANTHROPIC_PATH_SAFE = "/:@!$&'()*+,;=-._~%"+_HEX_DIGITS = frozenset("0123456789abcdefABCDEF")+_HEX_DIGIT_BYTES = frozenset(b"0123456789abcdefABCDEF")+_UNRESERVED_BYTES = frozenset(b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~")_NO_CUSTOM_TEMPERATURE_MODEL_IDS = frozenset({"claude-mythos-preview"})_ANTHROPIC_BEDROCK_PREFIX_RE = re.compile(r"^(?:(?:[a-z]{2}|global)\.)?anthropic\.")_VERSIONED_CLAUDE_MODEL_RE = re.compile(@@ -128,7 +140,7 @@ def resolve_llm_provider(environ: Mapping[str, str] | None = None) -> ProviderCoprovider=provider,model=model,api_key=_required(env, "ANTHROPIC_API_KEY"),- base_url=(env.get("SKILL_EVAL_LLM_BASE_URL") or env.get("ANTHROPIC_BASE_URL") or None),+ base_url=_anthropic_base_url(env),litellm_model=f"anthropic/{model}",credential_env="ANTHROPIC_API_KEY",base_url_env="ANTHROPIC_BASE_URL",@@ -230,6 +242,158 @@ def _required(environ: Mapping[str, str], variable: str) -> str:return value+def _anthropic_base_url(environ: Mapping[str, str]) -> str | None:+ for variable in ("SKILL_EVAL_LLM_BASE_URL", "ANTHROPIC_BASE_URL"):+ if value := environ.get(variable):+ return _normalize_anthropic_base_url(value, variable=variable)+ return None+++def _normalize_anthropic_base_url(value: str, *, variable: str) -> str:+ error = (+ f"{variable} must be an absolute HTTP or HTTPS URL representing an API root without credentials, query, fragment, "+ "whitespace, control characters, backslashes, an invalid authority, or a /v1/messages endpoint."+ )+ if "\\" in value or any(+ character.isspace() or unicodedata.category(character) in {"Cc", "Cf", "Cs"} for character in value+ ):+ raise ProviderConfigurationError(error)+ if "?" in value or "#" in value:+ raise ProviderConfigurationError(error)++ try:+ parsed = urlsplit(value)+ hostname = parsed.hostname+ _ = parsed.port+ except ValueError:+ raise ProviderConfigurationError(error) from None++ if (+ parsed.scheme.lower() not in {"http", "https"}+ or not parsed.netloc+ or hostname is None+ or parsed.netloc.endswith(":")+ or parsed.username is not None+ or parsed.password is not None+ ):+ raise ProviderConfigurationError(error)++ authority = _canonical_anthropic_authority(parsed.netloc, hostname)+ path = _canonical_anthropic_path(parsed.path)+ if authority is None or path is None:+ raise ProviderConfigurationError(error)++ path = path.rstrip("/")+ if path.endswith("/v1/messages"):+ raise ProviderConfigurationError(error)+ if path.endswith("/v1"):+ path = path.removesuffix("/v1")+ return urlunsplit((parsed.scheme, authority, path, "", ""))+++def _canonical_anthropic_authority(netloc: str, hostname: str) -> str | None:+ if netloc.startswith("["):+ closing_bracket = netloc.find("]")+ if closing_bracket < 0:+ return None+ literal = netloc[1:closing_bracket]+ suffix = netloc[closing_bracket + 1 :]+ if literal.casefold() != hostname.casefold() or (+ suffix and (not suffix.startswith(":") or not suffix[1:].isascii() or not suffix[1:].isdigit())+ ):+ return None++ address = literal+ zone = ""+ if "%" in literal:+ address, separator, zone = literal.partition("%25")+ if not separator or "%" in address or "%" in zone or not _ANTHROPIC_IPV6_ZONE_RE.fullmatch(zone):+ return None+ try:+ ipaddress.IPv6Address(address)+ except ValueError:+ return None+ return f"[{address}{'%25' + zone if zone else ''}]{suffix}"++ if "%" in netloc or "[" in netloc or "]" in netloc:+ return None+ host = netloc+ suffix = ""+ if ":" in netloc:+ host, port = netloc.rsplit(":", maxsplit=1)+ if ":" in host or not port.isascii() or not port.isdigit():+ return None+ suffix = f":{port}"+ if host.casefold() != hostname.casefold():+ return None++ if "." in hostname and all(character in "0123456789." for character in hostname):+ try:+ ipaddress.IPv4Address(hostname)+ except ValueError:+ return None+ return f"{host}{suffix}"++ trailing_dot = host.endswith(".")+ dns_name = host.removesuffix(".")+ if not dns_name:+ return None+ if dns_name.isascii() and "_" in dns_name:+ canonical_name = dns_name.lower()+ label_pattern = _ANTHROPIC_INTERNAL_LABEL_RE+ else:+ try:+ canonical_name = idna.encode(dns_name.lower()).decode("ascii")+ except idna.IDNAError:+ return None+ label_pattern = _ANTHROPIC_DNS_LABEL_RE+ if len(canonical_name) > 253 or not all(label_pattern.fullmatch(label) for label in canonical_name.split(".")):+ return None+ return f"{canonical_name}{'.' if trailing_dot else ''}{suffix}"+++def _canonical_anthropic_path(path: str) -> str | None:+ canonical: list[str] = []+ index = 0+ while index < len(path):+ character = path[index]+ if character != "%":+ canonical.append(character)+ index += 1+ continue++ if index + 2 >= len(path) or path[index + 1] not in _HEX_DIGITS or path[index + 2] not in _HEX_DIGITS:+ return None+ octet = int(path[index + 1 : index + 3], 16)+ if octet in {0x2F, 0x5C, 0x7F} or octet < 0x20:+ return None+ if octet in _UNRESERVED_BYTES:+ canonical.append(chr(octet))+ else:+ canonical.append(f"%{octet:02X}")+ index += 3++ canonical_path = "".join(canonical)+ if "//" in canonical_path.rstrip("/"):+ return None+ decoded_octets = unquote_to_bytes(canonical_path)+ # A decoded percent is safe as data unless it opens a second escape layer.+ if any(+ decoded_octets[index] == 0x25+ and index + 2 < len(decoded_octets)+ and decoded_octets[index + 1] in _HEX_DIGIT_BYTES+ and decoded_octets[index + 2] in _HEX_DIGIT_BYTES+ for index in range(len(decoded_octets))+ ):+ return None+ decoded_path = decoded_octets.decode("utf-8", errors="replace")+ if any(unicodedata.category(character) in {"Cc", "Cf", "Cs"} for character in decoded_path):+ return None+ if any(segment in {".", ".."} for segment in decoded_path.split("/")):+ return None+ return quote(canonical_path, safe=_ANTHROPIC_PATH_SAFE)++def _selected_provider(environ: Mapping[str, str], variable: str) -> str:configured = environ.get(variable, "").strip().lower()if configured:diff --git a/src/skillevaluator/reporting/templates/report.html.j2 b/src/skillevaluator/reporting/templates/report.html.j2index 6d07c9d7..3eb38601 100644--- a/src/skillevaluator/reporting/templates/report.html.j2+++ b/src/skillevaluator/reporting/templates/report.html.j2@@ -2460,7 +2460,7 @@{% if d %}{% set d_score = d.score if d.score is not none else d.with_skill %}{% set d_lift = d.lift %}- {% set d_color = 'var(--success-color)' if d_score is not none and d_score >= 0.7 else ('var(--warning-color)' if d_score is not none and d_score >= 0.4 else 'var(--danger-color)') %}+ {% set d_color = 'var(--text-secondary)' if d_score is none else ('var(--success-color)' if d_score >= 0.7 else ('var(--warning-color)' if d_score >= 0.4 else 'var(--danger-color)')) %}{% set lift_color = 'var(--success-color)' if d_lift is not none and d_lift >= tier3_lift_pass_threshold else ('var(--danger-color)' if d_lift is not none and d_lift <= tier3_lift_fail_threshold else 'var(--text-secondary)') %}<td><div class="t3-mad-cell">@@ -2468,7 +2468,7 @@<span class="t3-mad-score" style="color:{{ d_color }};">{{ "%.2f" | format(d_score) if d_score is not none else "N/A" }}</span>{% if d_score is not none %}<div class="t3-mad-bar"><span style="width:{{ (d_score * 100) | round(0) }}%; background:{{ d_color }};"></span></div>{% endif %}</div>- <span class="t3-mad-lift" style="color:{{ lift_color }};">{{ "%+.2f" | format(d_lift) if d_lift is not none else "no baseline" }} lift</span>+ <span class="t3-mad-lift" style="color:{{ lift_color }};">{% if d_lift is not none %}{{ "%+.2f" | format(d_lift) }} lift{% elif d_score is none and d.baseline is not none %}with-skill unavailable{% elif d_score is none %}N/A lift{% else %}no baseline lift{% endif %}</span></div></td>{% else %}@@ -2480,17 +2480,18 @@<tr><td><strong>Overall</strong></td>{% for name, agent in t3_agents.items() %}- {% set ws = agent.with_skill or 0.0 %}+ {% set ws = agent.with_skill %}+ {% set bl = agent.baseline %}{% set lf = agent.lift %}- {% set ws_color = 'var(--success-color)' if ws >= 0.7 else ('var(--warning-color)' if ws >= 0.4 else 'var(--danger-color)') %}+ {% set ws_color = 'var(--text-secondary)' if ws is none else ('var(--success-color)' if ws >= 0.7 else ('var(--warning-color)' if ws >= 0.4 else 'var(--danger-color)')) %}{% set lf_color = 'var(--success-color)' if lf is not none and lf >= tier3_lift_pass_threshold else ('var(--danger-color)' if lf is not none and lf <= tier3_lift_fail_threshold else 'var(--text-secondary)') %}<td><div class="t3-mad-cell"><div class="t3-mad-score-row">- <span class="t3-mad-score" style="color:{{ ws_color }};">{{ "%.2f" | format(ws) }}</span>- <div class="t3-mad-bar"><span style="width:{{ (ws * 100) | round(0) }}%; background:{{ ws_color }};"></span></div>+ <span class="t3-mad-score" style="color:{{ ws_color }};">{{ "%.2f" | format(ws) if ws is not none else "N/A" }}</span>+ {% if ws is not none %}<div class="t3-mad-bar"><span style="width:{{ (ws * 100) | round(0) }}%; background:{{ ws_color }};"></span></div>{% endif %}</div>- <span class="t3-mad-lift" style="color:{{ lf_color }};">{{ "%+.2f" | format(lf) if lf is not none else "no baseline" }} lift</span>+ <span class="t3-mad-lift" style="color:{{ lf_color }};">{% if lf is not none %}{{ "%+.2f" | format(lf) }} lift{% elif ws is none and bl is not none %}with-skill unavailable{% elif ws is none %}N/A lift{% else %}no baseline lift{% endif %}</span></div></td>{% endfor %}@@ -2511,9 +2512,10 @@<div class="t3-agent-scorecards-row"><div class="t3-agent-scorecard-list">{% for name, agent in t3_agents.items() %}- {% set with_skill = agent.with_skill or 0.0 %}+ {% set with_skill = agent.with_skill %}{% set lift = agent.lift %}- {% set score_color = 'var(--success-color)' if with_skill >= 0.7 else ('var(--warning-color)' if with_skill >= 0.4 else 'var(--danger-color)') %}+ {% set score_color = 'var(--text-secondary)' if with_skill is none else ('var(--success-color)' if with_skill >= 0.7 else ('var(--warning-color)' if with_skill >= 0.4 else 'var(--danger-color)')) %}+ {% set card_lift_color = 'var(--text-secondary)' if lift is none else ('var(--success-color)' if lift >= tier3_lift_pass_threshold else ('var(--danger-color)' if lift <= tier3_lift_fail_threshold else 'var(--text-secondary)')) %}<div class="t3-agent-card t3-agent-deeplink"role="button"tabindex="0"@@ -2523,12 +2525,12 @@aria-label="Open Agent Details for {{ name }}"title="Open Agent Details for {{ name }}"><div class="t3-agent-name">{{ name }}</div>- <div class="t3-agent-score" style="color:{{ score_color }};">{{ "%.2f" | format(with_skill) }}</div>- <div style="color:{{ 'var(--success-color)' if (lift or 0) >= tier3_lift_pass_threshold else ('var(--danger-color)' if (lift or 0) <= tier3_lift_fail_threshold else 'var(--text-secondary)') }};font-size:0.85rem;">- {% if lift is not none %}{{ "%+.2f" | format(lift) }} lift{% else %}no baseline{% endif %}+ <div class="t3-agent-score" style="color:{{ score_color }};">{{ "%.2f" | format(with_skill) if with_skill is not none else "N/A" }}</div>+ <div style="color:{{ card_lift_color }};font-size:0.85rem;">+ {% if lift is not none %}{{ "%+.2f" | format(lift) }} lift{% elif with_skill is none and agent.baseline is not none %}with-skill unavailable{% elif with_skill is none %}N/A lift{% else %}no baseline{% endif %}</div><div style="color:var(--text-secondary);font-size:0.75rem;margin-top:0.25rem;">{{ agent.num_trials or 0 }} trials</div>- <div class="t3-mini-bar"><span style="width:{{ (with_skill * 100) | round(0) }}%; background:{{ score_color }};"></span></div>+ {% if with_skill is not none %}<div class="t3-mini-bar"><span style="width:{{ (with_skill * 100) | round(0) }}%; background:{{ score_color }};"></span></div>{% endif %}</div>{% endfor %}</div>@@ -2896,14 +2898,14 @@{% for name, agent in t3_agents.items() %}{% set agent_idx = loop.index0 %}{% set agent_color = t3_palette[agent_idx % (t3_palette | length)] %}- {% set agent_score = agent.with_skill or 0.0 %}+ {% set agent_score = agent.with_skill %}<button class="t3-agent-tab-btn{% if loop.first %} active{% endif %}"role="tab"data-tier3-agent="{{ name }}"style="--accent: {{ agent_color }};"onclick="showTier3Agent('{{ name }}', this)"><span class="t3-tab-dot"></span>{{ name }}- <span class="t3-tab-score">{{ "%.2f" | format(agent_score) }}{% if agent.lift is not none %} ({{ "%+.2f" | format(agent.lift) }}){% endif %}</span>+ <span class="t3-tab-score">{{ "%.2f" | format(agent_score) if agent_score is not none else "N/A" }}{% if agent.lift is not none %} ({{ "%+.2f" | format(agent.lift) }}){% endif %}</span></button>{% endfor %}</nav>@@ -2911,8 +2913,8 @@{% for name, agent in t3_agents.items() %}{% set agent_idx = loop.index0 %}{% set agent_color = t3_palette[agent_idx % (t3_palette | length)] %}- {% set agent_score = agent.with_skill or 0.0 %}- {% set agent_score_color = 'var(--success-color)' if agent_score >= 0.7 else ('var(--warning-color)' if agent_score >= 0.4 else 'var(--danger-color)') %}+ {% set agent_score = agent.with_skill %}+ {% set agent_score_color = 'var(--text-secondary)' if agent_score is none else ('var(--success-color)' if agent_score >= 0.7 else ('var(--warning-color)' if agent_score >= 0.4 else 'var(--danger-color)')) %}{% set best_dim = none %}{% set worst_dim = none %}{% for d in agent.dimensions or [] %}@@ -2936,7 +2938,7 @@<div class="t3-kpi-strip"><div class="t3-kpi-card" style="border-top:3px solid {{ agent_color }};"><div class="t3-kpi-label">Overall Score</div>- <div class="t3-kpi-value" style="color:{{ agent_score_color }};">{{ "%.2f" | format(agent_score) }}</div>+ <div class="t3-kpi-value" style="color:{{ agent_score_color }};">{{ "%.2f" | format(agent_score) if agent_score is not none else "N/A" }}</div><div class="t3-kpi-sub">baseline {{ "%.2f" | format(agent.baseline) if agent.baseline is not none else "N/A" }}</div></div><div class="t3-kpi-card">@@ -2947,7 +2949,7 @@<div class="t3-kpi-sub">{% if agent.lift >= tier3_lift_pass_threshold %}positive{% elif agent.lift <= tier3_lift_fail_threshold %}regression{% else %}neutral{% endif %}</div>{% else %}<div class="t3-kpi-value" style="color:var(--text-secondary);">N/A</div>- <div class="t3-kpi-sub">no baseline</div>+ <div class="t3-kpi-sub">{% if agent.with_skill is none and agent.baseline is not none %}with-skill unavailable{% elif agent.with_skill is none %}N/A lift{% else %}no baseline{% endif %}</div>{% endif %}</div><div class="t3-kpi-card">@@ -2980,10 +2982,10 @@{% for d in dim_source %}{% set d_score = d.score if d.score is not none else d.with_skill %}{% set d_id = d.id or d.dimension or d.label %}- {% set d_color = 'var(--success-color)' if (d_score or 0) >= 0.7 else ('var(--warning-color)' if (d_score or 0) >= 0.4 else 'var(--danger-color)') %}+ {% set d_color = 'var(--text-secondary)' if d_score is none else ('var(--success-color)' if d_score >= 0.7 else ('var(--warning-color)' if d_score >= 0.4 else 'var(--danger-color)')) %}<div class="t3-dim-pill" title="{{ t3_dim_hints.get(d_id) or d.explanation or '' }}"><span class="t3-dim-name">{{ d_id | replace("_", " ") }}</span>- <span class="t3-dim-score" style="color:{{ d_color }};">{{ "%.2f" | format(d_score or 0.0) }}</span>+ <span class="t3-dim-score" style="color:{{ d_color }};">{{ "%.2f" | format(d_score) if d_score is not none else "N/A" }}</span>{% if d.lift is not none %}{% set lift_class = 'up' if d.lift >= tier3_lift_pass_threshold else ('down' if d.lift <= tier3_lift_fail_threshold else 'flat') %}<span class="t3-delta {{ lift_class }}">{{ "%+.2f" | format(d.lift) }} lift</span>@@ -3631,7 +3633,9 @@(agent.dimensions || []).forEach(function(dim){ byId[dim.id] = dim; });return dimensionIds.map(function(dimId){var row = byId[dimId] || {};- return Number(row.score || row.with_skill || 0);+ var raw = row.score;+ if (raw === null || raw === undefined) raw = row.with_skill;+ return raw === null || raw === undefined ? null : Number(raw);});}function tokenTotalsByAgent() {diff --git a/src/skillevaluator/tier3/eval_core/checks.py b/src/skillevaluator/tier3/eval_core/checks.pyindex d0abf2de..7f050334 100644--- a/src/skillevaluator/tier3/eval_core/checks.py+++ b/src/skillevaluator/tier3/eval_core/checks.py@@ -594,7 +594,7 @@ def check_security(finding_type="secret_exposure",severity="critical",message="Possible secret value appeared in tool output observed by the agent",- evidence=observation[:300],+ evidence="[redacted secret exposure]",source="tool_observation",score_impact=True,tool=action,diff --git a/src/skillevaluator/tier3/eval_core/llm_judge.py b/src/skillevaluator/tier3/eval_core/llm_judge.pyindex 3c982f40..6ff1ffd5 100644--- a/src/skillevaluator/tier3/eval_core/llm_judge.py+++ b/src/skillevaluator/tier3/eval_core/llm_judge.py@@ -9,6 +9,7 @@ from __future__ import annotationsimport jsonimport logging+import mathimport osimport reimport urllib.error@@ -25,6 +26,7 @@ NVIDIA_BUILD_CHAT_URL = "https://integrate.api.nvidia.com/v1/chat/completions"DEFAULT_JUDGE_MODEL = CHAT_DEFAULT_OPENAI_ERROR_REDACTION_MARKER = "[REDACTED]"+_JUDGE_ERROR_REASON_LIMIT = 512# Match verifier log redaction; shorter placeholders can corrupt ordinary diagnostic text._MIN_EXACT_SECRET_LENGTH = 8_CREDENTIAL_ENV_VARS = (@@ -36,6 +38,8 @@ _CREDENTIAL_ENV_VARS = ("AWS_SECRET_ACCESS_KEY","AWS_SECURITY_TOKEN","AWS_SESSION_TOKEN",+ "AWS_BEARER_TOKEN_BEDROCK",+ "AWS_CONTAINER_AUTHORIZATION_TOKEN",)@@ -131,6 +135,27 @@ def _redact_configured_credentials(text: str, extra_secret_values: tuple[str | Nreturn redacted+def _judge_error(error_reason: str, **metadata: Any) -> dict[str, Any]:+ """Return a bounded, redacted result that cannot be mistaken for a judged zero."""+ safe_reason = _redact_configured_credentials(error_reason).strip() or "LLM judge failed"+ if len(safe_reason) > _JUDGE_ERROR_REASON_LIMIT:+ safe_reason = safe_reason[: _JUDGE_ERROR_REASON_LIMIT - 3] + "..."+ return {**metadata, "score": None, "status": "error", "reason": safe_reason}+++def _finite_score(value: Any) -> float | None:+ if isinstance(value, bool) or not isinstance(value, (int, float)):+ return None+ if isinstance(value, int):+ if value <= 0:+ return 0.0+ return 1.0+ score = float(value)+ if not math.isfinite(score):+ return None+ return max(0.0, min(1.0, score))++def _format_http_error_with_fallback(error: urllib.error.HTTPError) -> tuple[str, bool]:try:body = error.read().decode("utf-8", "replace").strip()@@ -217,15 +242,17 @@ def call_public_llm(return None, _redact_configured_credentials(detail, (api_key,))-_JSON_FENCE_RE = re.compile(r"```(?:json)?\s*\n?(.*?)```", re.DOTALL | re.IGNORECASE)+_JSON_WHITESPACE = " \t\r\n"+_MAX_JSON_TEXT_CHARS = 100_000+_MAX_JSON_NESTING = 128+_JSON_NUMBER_PREFIX_RE = re.compile(r"-?(?:(?:0|[1-9][0-9]*)(?:\.[0-9]+)?(?:[eE][+-]?[0-9]*)?|(?:0|[1-9][0-9]*)\.)?")-def _find_balanced_json(text: str) -> str | None:- """Return the first balanced ``{...}`` block, honoring strings and escapes."""- start = text.find("{")- if start == -1:+def _balanced_json_container_end(text: str, start: int) -> int | None:+ """Return the exclusive end of one bounded structural container."""+ if start >= len(text) or text[start] not in "{[":return None- depth = 0+ stack: list[str] = []in_string = Falseescaped = Falsefor i in range(start, len(text)):@@ -239,50 +266,202 @@ def _find_balanced_json(text: str) -> str | None:in_string = Falseelif ch == '"':in_string = True- elif ch == "{":+ elif ch in "{[":+ stack.append(ch)+ if len(stack) > _MAX_JSON_NESTING:+ return None+ elif ch in "}]":+ expected = "{" if ch == "}" else "["+ if not stack or stack[-1] != expected:+ return None+ stack.pop()+ if not stack:+ return i + 1+ return None+++def _reject_duplicate_object_pairs(pairs: list[tuple[str, Any]]) -> dict[str, Any]:+ """Build an object while rejecting ambiguous duplicate members."""+ result: dict[str, Any] = {}+ for key, value in pairs:+ if key in result:+ raise ValueError("Duplicate JSON object member")+ result[key] = value+ return result+++def _reject_nonstandard_json_constant(_value: str) -> None:+ raise ValueError("Non-standard JSON constant")+++def _json_nesting_within_limit(text: str) -> bool:+ """Bound structural nesting without recursively parsing partial JSON."""+ depth = 0+ in_string = False+ escaped = False+ for character in text:+ if in_string:+ if escaped:+ escaped = False+ elif character == "\\":+ escaped = True+ elif character == '"':+ in_string = False+ elif character == '"':+ in_string = True+ elif character in "{[":depth += 1- elif ch == "}":+ if depth > _MAX_JSON_NESTING:+ return False+ elif character in "}]" and depth:depth -= 1- if depth == 0:- return text[start : i + 1]- return None+ return Truedef _extract_json(text: str) -> dict[str, Any] | list[Any] | None:"""Extract a JSON payload from LLM response text.- Tolerates markdown fences (```json anywhere in the text), leading or- trailing prose -- including prose that itself contains braces -- via- first-balanced-brace extraction. Top-level JSON arrays parse through- unchanged (the suggestion generator in ``harbor.report`` relies on that);- judge callers must dict-check the result themselves.+ Tolerates markdown fences and prose around exactly one valid bounded JSON+ container. Multiple complete documents are ambiguous, and an unfinished+ earlier structural segment blocks promotion of a nested object. Top-level+ arrays parse through unchanged (``harbor.report`` relies on that); judge+ callers must dict-check the result themselves."""text = (text or "").strip()- if not text:+ if not text or len(text) > _MAX_JSON_TEXT_CHARS:return None- candidates = [text]- if text.startswith("```"):- candidates.append(text.split("\n", 1)[-1].rsplit("```", 1)[0].strip())- fence = _JSON_FENCE_RE.search(text)- if fence:- candidates.append(fence.group(1).strip())- start = text.find("{")- end = text.rfind("}")- if start != -1 and end > start:- candidates.append(text[start : end + 1])- balanced = _find_balanced_json(text)- if balanced:- candidates.append(balanced)- for candidate in candidates:- if not candidate:++ documents: list[dict[str, Any] | list[Any]] = []+ index = 0+ while index < len(text):+ if text[index] not in "{[":+ index += 1continue+ end = _balanced_json_container_end(text, index)+ if end is None:+ return documents[0] if documents else None+ candidate = text[index:end]try:- parsed = json.loads(candidate)- except (json.JSONDecodeError, ValueError):+ parsed = json.loads(+ candidate,+ object_pairs_hook=_reject_duplicate_object_pairs,+ parse_constant=_reject_nonstandard_json_constant,+ )+ except (json.JSONDecodeError, RecursionError, ValueError):+ index = endcontinueif isinstance(parsed, (dict, list)):- return parsed- return None+ documents.append(parsed)+ if len(documents) > 1:+ return None+ index = end+ return documents[0] if documents else None+++def _is_json_string_prefix(text: str) -> bool:+ """Return whether an unfinished bounded string can be completed as JSON."""+ if not text.startswith('"'):+ return False+ index = 1+ while index < len(text):+ character = text[index]+ if ord(character) < 0x20 or character == '"':+ return False+ if character != "\\":+ index += 1+ continue+ index += 1+ if index >= len(text):+ return True+ escape = text[index]+ if escape == "u":+ for offset in range(1, 5):+ if index + offset >= len(text):+ return True+ if text[index + offset] not in "0123456789abcdefABCDEF":+ return False+ index += 5+ elif escape in '"\\/bfnrt':+ index += 1+ else:+ return False+ return True+++def _is_json_scalar_prefix(text: str) -> bool:+ if not text:+ return True+ if text.startswith('"'):+ return _is_json_string_prefix(text)+ literals = {"t": "true", "f": "false", "n": "null"}+ if text[0] in literals:+ return literals[text[0]].startswith(text)+ if text[0] == "-" or text[0] in "0123456789":+ return _JSON_NUMBER_PREFIX_RE.fullmatch(text) is not None+ return False+++def _is_append_only_json_object_prefix(fragment: str) -> bool:+ """Validate an unfinished flat result entry using bounded decoder steps."""+ if not fragment or len(fragment) > _MAX_JSON_TEXT_CHARS:+ return False++ decoder = json.JSONDecoder(+ object_pairs_hook=_reject_duplicate_object_pairs,+ parse_constant=_reject_nonstandard_json_constant,+ )++ def _skip_whitespace(index: int) -> int:+ while index < len(fragment) and fragment[index] in _JSON_WHITESPACE:+ index += 1+ return index++ index = _skip_whitespace(0)+ if index >= len(fragment) or fragment[index] != "{":+ return False+ index += 1+ keys: set[str] = set()+ while True:+ index = _skip_whitespace(index)+ if index >= len(fragment):+ return True+ if fragment[index] == "}":+ return False+ try:+ key, next_index = decoder.raw_decode(fragment, index)+ except (json.JSONDecodeError, RecursionError, ValueError):+ return _is_json_string_prefix(fragment[index:])+ if not isinstance(key, str) or key in keys:+ return False+ keys.add(key)+ index = _skip_whitespace(next_index)+ if index >= len(fragment):+ return True+ if fragment[index] != ":":+ return False+ index = _skip_whitespace(index + 1)+ if index >= len(fragment):+ return True+ value_start = index+ try:+ value, next_index = decoder.raw_decode(fragment, index)+ except (json.JSONDecodeError, RecursionError, ValueError):+ return _is_json_scalar_prefix(fragment[value_start:])+ if (+ isinstance(value, (int, float))+ and not isinstance(value, bool)+ and next_index < len(fragment)+ and fragment[next_index] in ".eE"+ ):+ return _JSON_NUMBER_PREFIX_RE.fullmatch(fragment[value_start:]) is not None+ index = _skip_whitespace(next_index)+ if index >= len(fragment):+ return True+ if fragment[index] == "}":+ return False+ if fragment[index] != ",":+ return False+ index += 1def _salvage_behavior_results(text: str) -> list[dict[str, Any]]:@@ -293,32 +472,94 @@ def _salvage_behavior_results(text: str) -> list[dict[str, Any]]:entry before the cut is still valid JSON and can be scored."""text = text or ""- marker = text.find('"results"')- if marker == -1:+ if len(text) > _MAX_JSON_TEXT_CHARS or not _json_nesting_within_limit(text):return []- array_start = text.find("[", marker)- if array_start == -1:+ object_start = text.find("{")+ if object_start == -1:return []- results: list[dict[str, Any]] = []- i = array_start + 1+ if any(character in "[]{}" for character in text[:object_start]):+ return []++ decoder = json.JSONDecoder(+ object_pairs_hook=_reject_duplicate_object_pairs,+ parse_constant=_reject_nonstandard_json_constant,+ )++ def _skip_whitespace(index: int) -> int:+ while index < len(text) and text[index] in " \t\r\n":+ index += 1+ return index++ # Parse only complete top-level fields preceding ``results``. This rejects+ # nested/unrelated arrays and lets us validate a score emitted before the+ # array without requiring the outer object itself to be complete.+ i = object_start + 1+ array_start = None+ seen_keys: set[str] = set()while i < len(text):- ch = text[i]- if ch == "{":- block = _find_balanced_json(text[i:])- if not block:- break- try:- entry = json.loads(block)- except (json.JSONDecodeError, ValueError):- break- if isinstance(entry, dict):- results.append(entry)- i += len(block)- elif ch == "]":+ i = _skip_whitespace(i)+ try:+ key, i = decoder.raw_decode(text, i)+ except (json.JSONDecodeError, RecursionError, ValueError):+ return []+ if not isinstance(key, str):+ return []+ if key in seen_keys:+ return []+ seen_keys.add(key)+ i = _skip_whitespace(i)+ if i >= len(text) or text[i] != ":":+ return []+ i = _skip_whitespace(i + 1)+ if key == "results":+ if i >= len(text) or text[i] != "[":+ return []+ array_start = ibreak- else:- i += 1- return results+ try:+ value, i = decoder.raw_decode(text, i)+ except (json.JSONDecodeError, RecursionError, ValueError):+ return []+ if key == "score" and _finite_score(value) is None:+ return []+ i = _skip_whitespace(i)+ if i >= len(text) or text[i] != ",":+ return []+ i += 1++ if array_start is None:+ return []++ results: list[dict[str, Any]] = []+ i = array_start + 1+ while True:+ i = _skip_whitespace(i)+ if i >= len(text):+ return results+ if text[i] != "{":+ return []+ try:+ entry, i = decoder.raw_decode(text, i)+ except (json.JSONDecodeError, RecursionError, ValueError):+ return results if _is_append_only_json_object_prefix(text[i:]) else []+ if not isinstance(entry, dict):+ return []+ results.append(entry)+ i = _skip_whitespace(i)+ if i >= len(text):+ return results+ # Salvage is only for an array truncated before its closing bracket.+ # A closed results array with a malformed outer object is not partial+ # per-entry output and must take the structured-error path.+ if text[i] == "]":+ return []+ if text[i] != ",":+ return []+ i = _skip_whitespace(i + 1)+ if i >= len(text):+ return results+ if text[i] == "]":+ return []# ---------------------------------------------------------------------------@@ -352,6 +593,24 @@ EXPECTED ANSWER:SELECTED EVIDENCE (final response + produced artifacts; low-relevance steps may be omitted):{agent_text}"""+_ACCURACY_CRITERIA_KEYS = frozenset(+ {+ "SKILL_IDENTIFIED",+ "ACTION_CORRECT",+ "FACTUALLY_ACCURATE",+ "TASK_ADDRESSED",+ "ACTIONABLE",+ }+)+++def _valid_accuracy_criteria(value: Any) -> bool:+ return (+ isinstance(value, dict)+ and value.keys() == _ACCURACY_CRITERIA_KEYS+ and all(isinstance(item, bool) for item in value.values())+ )+def judge_accuracy(question: str,@@ -371,26 +630,28 @@ def judge_accuracy(content, error = call_public_llm(prompt, **kwargs)if error:- return {"score": 0.0, "reason": f"LLM judge error: {error}"}+ return _judge_error(f"LLM judge error: {error}")parsed = _extract_json(content) if content else None- if not parsed:- yes_count = (content or "").upper().count("YES")- score = min(yes_count / 5.0, 1.0)- return {"score": round(score, 2), "reason": f"Parsed {yes_count}/5 YES from text"}-- score = parsed.get("score", 0.0)- if isinstance(score, (int, float)):- score = max(0.0, min(1.0, float(score)))- else:- criteria = parsed.get("criteria", {})+ if not isinstance(parsed, dict):+ return _judge_error("Judge response was not a valid JSON object")++ criteria = parsed.get("criteria")+ criteria_valid = _valid_accuracy_criteria(criteria)+ if "criteria" in parsed and not criteria_valid:+ return _judge_error("Judge response contained invalid accuracy criteria")++ score = _finite_score(parsed.get("score"))+ if score is None:+ if not criteria_valid:+ return _judge_error("Judge response contained no valid accuracy score or complete criteria")yes_count = sum(1 for v in criteria.values() if v is True)score = yes_count / 5.0return {"score": round(score, 4),"reason": parsed.get("reason", ""),- "criteria": parsed.get("criteria", {}),+ "criteria": criteria if criteria_valid else {},}@@ -444,15 +705,21 @@ def judge_goal_accuracy(content, error = call_public_llm(prompt, **kwargs)if error:- return {"score": 0.0, "reason": f"LLM judge error: {error}"}+ return _judge_error(f"LLM judge error: {error}")parsed = _extract_json(content) if content else None- if not parsed:- return {"score": 0.0, "reason": "Could not parse judge response"}+ if not isinstance(parsed, dict):+ return _judge_error("Judge response was not a valid JSON object")- score = 1.0 if parsed.get("achieved", False) else 0.0- if "score" in parsed and isinstance(parsed["score"], (int, float)):- score = max(0.0, min(1.0, float(parsed["score"])))+ achieved = parsed.get("achieved")+ if not isinstance(achieved, bool):+ return _judge_error("Judge response contained an invalid achieved value")++ score = 1.0 if achieved else 0.0+ if "score" in parsed:+ score = _finite_score(parsed["score"])+ if score is None:+ return _judge_error("Judge response contained an invalid goal score")return {"score": score,@@ -529,62 +796,83 @@ def judge_behavior_check(content, error = call_public_llm(prompt, **kwargs)if error:- return {"score": 0.0, "reason": f"LLM judge error: {error}", "results": []}+ return _judge_error(f"LLM judge error: {error}", results=[])- def _parse_judge_object(text: str) -> dict[str, Any] | None:- parsed = _extract_json(text) if text else None- return parsed if isinstance(parsed, dict) else None+ def _parse_judge_object(text: str) -> dict[str, Any] | list[Any] | None:+ return _extract_json(text) if text else Noneparsed = _parse_judge_object(content)- attempts = [content or ""]- if not parsed:+ score = _behavior_payload_score(parsed, len(expected_behaviors))+ attempts = [(content or "", parsed)]+ retry_error = None+ if score is None:# One retry max, with an explicit machine-readable-output reminder.retry_content, retry_error = call_public_llm(prompt + _BEHAVIOR_RETRY_REMINDER, **kwargs)if not retry_error:- attempts.append(retry_content or "")parsed = _parse_judge_object(retry_content)+ attempts.append((retry_content or "", parsed))+ score = _behavior_payload_score(parsed, len(expected_behaviors))- salvaged_from_truncation = False- if not parsed:+ if score is None:# Salvage complete entries from a truncated results array (newest first).- for text in reversed(attempts):+ for text, extracted in reversed(attempts):+ if extracted is not None:+ continuesalvaged = _salvage_behavior_results(text)if salvaged:- salvaged_from_truncation = True- parsed = {+ candidate = {"results": salvaged,"summary": (f"Salvaged {len(salvaged)}/{len(expected_behaviors)} behavior ""results from truncated judge response"),}- break-- if not parsed:- last = attempts[-1]- head = last[:80].replace("\n", " ")- return {- "score": 0.0,- "reason": f"Judge response unparseable after retry (len={len(last)}, head={head!r})",- "results": [],- }-- results = parsed.get("results", [])- if results:- passed_count = sum(1 for r in results if r.get("passed"))- # A salvaged array is incomplete: behaviors the truncation cut off were- # never judged and count as not-passed, so keep the denominator at the- # number of expected behaviors instead of inflating against the few- # recovered entries.- denominator = len(expected_behaviors) if salvaged_from_truncation else len(results)- score = passed_count / denominator- else:- score = parsed.get("score", 0.0)- if not isinstance(score, (int, float)):- score = 0.0+ candidate_score = _behavior_payload_score(+ candidate,+ len(expected_behaviors),+ allow_partial=True,+ )+ if candidate_score is not None:+ parsed = candidate+ score = candidate_score+ break++ if score is None:+ if retry_error:+ return _judge_error(f"LLM judge retry error: {retry_error}", results=[])+ return _judge_error("Judge response was unparseable or invalid after retry", results=[])++ assert isinstance(parsed, dict)+ results = parsed["results"]return {- "score": round(max(0.0, min(1.0, float(score))), 4),+ "score": round(score, 4),"reason": parsed.get("summary", ""),"results": results,}+++def _behavior_payload_score(+ parsed: dict[str, Any] | list[Any] | None,+ expected_count: int,+ *,+ allow_partial: bool = False,+) -> float | None:+ if not isinstance(parsed, dict):+ return None+ results = parsed.get("results")+ if not isinstance(results, list):+ return None+ if any(not isinstance(result, dict) or not isinstance(result.get("passed"), bool) for result in results):+ return None+ if allow_partial:+ if not results or len(results) > expected_count:+ return None+ elif len(results) != expected_count:+ return None+ if "score" in parsed and _finite_score(parsed["score"]) is None:+ return None+ denominator = expected_count if allow_partial else len(results)+ if denominator <= 0:+ return None+ return sum(1 for result in results if result["passed"]) / denominatordiff --git a/src/skillevaluator/tier3/harbor/adapter.py b/src/skillevaluator/tier3/harbor/adapter.pyindex 1e479915..6ce446a4 100644--- a/src/skillevaluator/tier3/harbor/adapter.py+++ b/src/skillevaluator/tier3/harbor/adapter.py@@ -3069,7 +3069,7 @@ def _custom_compose_path(environment_dir: Path) -> Path | None:return candidates[0] if candidates else None-_VERIFIER_DEPS = "ragas~=0.4.0 langchain-community<0.4.2 openai>=1.0 anthropic>=0.40 boto3>=1.34"+_VERIFIER_DEPS = "ragas~=0.4.0 langchain-community<0.4.2 openai>=1.0 anthropic>=0.40 boto3>=1.34 idna>=3.10,<4"_VERIFIER_IMPORT_SMOKE = ("from ragas import SingleTurnSample; ""from ragas.llms.base import llm_factory; "diff --git a/src/skillevaluator/tier3/harbor/collector.py b/src/skillevaluator/tier3/harbor/collector.pyindex 57fea093..fc97b1d5 100644--- a/src/skillevaluator/tier3/harbor/collector.py+++ b/src/skillevaluator/tier3/harbor/collector.py@@ -13,26 +13,35 @@ import jsonimport loggingimport osimport re+import shutil+import statfrom pathlib import Pathfrom typing import Anyfrom skillevaluator.tier3.harbor.metrics import (DEFAULT_METRIC_SET,DEFAULT_METRICS,+ LEGACY_METRIC_SET,LEGACY_METRICS,average_custom_metrics,average_metrics,dimension_scores,extract_custom_metrics,+ metric_set_for_reward,+ metric_value,overall_score,score_definition,)+from skillevaluator.tier3.output_provenance import write_output_file_atomicallyfrom skillevaluator.utils.redaction import redact_sensitive_data, redact_sensitive_text+from skillevaluator.utils.secure_fs import SecurePathError, SecureRoot, stat_is_link_or_reparselogger = logging.getLogger(__name__)DISPLAY_METRICS = DEFAULT_METRICSDEFAULT_DIAGNOSTIC_ARTIFACT_MAX_BYTES = 5 * 1024 * 1024+DIAGNOSTIC_ARTIFACT_HARD_MAX_BYTES = 64 * 1024 * 1024+REWARD_DIAGNOSTIC_STRING_MAX_CHARS = 8192TRIAL_DIAGNOSTIC_ARTIFACTS = ("result.json", "config.json", "exception.txt", "trial.log")AGENT_LOG_ARTIFACTS = ("trajectory.json",@@ -47,6 +56,17 @@ AGENT_LOG_ARTIFACTS = ("cline.txt","opencode.txt",)+GENERATED_AGENT_ARTIFACTS = (+ "lift.json",+ "custom_lift.json",+ "pass_at_k_lift.json",+ "security_attribution.json",+ "findings.json",+)+GENERATED_CONDITION_DIRS = ("with-skill", "without-skill")+GENERATED_ROOT_ARTIFACTS = ("attempt_policy.json", "comparison.json")+_MAX_FAILED_JUDGE_SIDECARS = 64+_MAX_FAILED_JUDGE_STEP_PATHS_SCANNED = 256def _is_aggregate_extra_token_key(key: str) -> bool:@@ -102,6 +122,81 @@ def _agent_runtime_failure_pattern_start(value: str) -> int | None:return None+def _path_is_within(path: Path, root: Path) -> bool:+ try:+ path.relative_to(root)+ except ValueError:+ return False+ return True+++def _assert_safe_generated_output_path(+ path: Path,+ output_root: Path,+ *,+ follow_target: bool,+) -> None:+ """Reject cleanup targets that can escape the configured results directory."""+ lexical_root = output_root.absolute()+ lexical_path = path.absolute()+ try:+ resolved_root = output_root.resolve()+ resolved_parent = path.parent.resolve()+ target_within_root = not follow_target or _path_is_within(path.resolve(), resolved_root)+ except (OSError, RuntimeError) as error:+ raise ValueError(f"Refusing unsafe generated output path: {path}") from error+ if (+ lexical_path == lexical_root+ or not _path_is_within(lexical_path, lexical_root)+ or not _path_is_within(resolved_parent, resolved_root)+ or not target_within_root+ ):+ raise ValueError(f"Refusing to modify generated output outside results directory: {path}")+++def _remove_generated_output_path(path: Path, output_root: Path) -> None:+ """Remove one known generated path without following a target symlink."""+ _assert_safe_generated_output_path(path, output_root, follow_target=False)+ if path.is_symlink() or path.is_file():+ path.unlink()+ elif path.is_dir():+ shutil.rmtree(path)+++def _write_generated_root_json(path: Path, output_root: Path, payload: Any) -> None:+ """Publish one root artifact without following unsafe replacements."""+ _assert_safe_generated_output_path(path, output_root, follow_target=False)+ write_output_file_atomically(path, json.dumps(payload, indent=2).encode("utf-8"))+++def _agent_generated_output_paths(agent_dir: Path) -> list[Path]:+ paths = [agent_dir / artifact for artifact in GENERATED_AGENT_ARTIFACTS]+ for condition in GENERATED_CONDITION_DIRS:+ condition_dir = agent_dir / condition+ paths.extend((condition_dir / "summary.json", condition_dir / "trials"))+ return paths+++def _validate_agent_generated_outputs(agent_dir: Path, output_root: Path) -> None:+ _assert_safe_generated_output_path(agent_dir, output_root, follow_target=True)+ if agent_dir.is_symlink():+ raise ValueError(f"Refusing symlinked generated output directory: {agent_dir}")+ for condition in GENERATED_CONDITION_DIRS:+ condition_dir = agent_dir / condition+ if condition_dir.is_symlink():+ raise ValueError(f"Refusing symlinked generated output directory: {condition_dir}")+ for path in _agent_generated_output_paths(agent_dir):+ _assert_safe_generated_output_path(path, output_root, follow_target=False)+++def _reset_agent_generated_outputs(agent_dir: Path, output_root: Path) -> None:+ """Clear only collector/report-owned artifacts before rebuilding an agent result."""+ _validate_agent_generated_outputs(agent_dir, output_root)+ agent_dir.mkdir(parents=True, exist_ok=True)+ for path in _agent_generated_output_paths(agent_dir):+ _remove_generated_output_path(path, output_root)++def _find_job_dir(jobs_dir: Path, job_name: str) -> Path | None:"""Find a Harbor job directory by name."""candidate = jobs_dir / job_name@@ -120,13 +215,105 @@ def _safe_text(value: Any, *, max_len: int | None = 2048) -> str:return text+def _safe_diagnostic_text(value: Any, *, max_len: int) -> str:+ """Return redacted, bounded text without terminal-control characters."""+ return re.sub(r"[\x00-\x1f\x7f]", " ", _safe_text(value, max_len=max_len)).strip()+++def _safe_evaluation_errors(value: Any) -> dict[str, str] | list[str] | str:+ """Normalize verifier diagnostics before persisting or displaying them."""+ if isinstance(value, dict):+ normalized: dict[str, str] = {}+ for metric, reason in list(value.items())[: len(DEFAULT_METRICS)]:+ safe_metric = _safe_diagnostic_text(metric, max_len=64) or "judge"+ safe_reason = _safe_diagnostic_text(reason, max_len=512)+ if safe_reason:+ normalized[safe_metric] = safe_reason+ return normalized+ if isinstance(value, list):+ return [+ safe_reason+ for reason in value[: len(DEFAULT_METRICS)]+ if (safe_reason := _safe_diagnostic_text(reason, max_len=512))+ ]+ return _safe_diagnostic_text(value, max_len=512)++def _read_json(path: Path) -> Any:+ """Read one bounded regular JSON file through an anchored no-follow root."""try:- return json.loads(path.read_text(encoding="utf-8"))- except (json.JSONDecodeError, OSError):+ with SecureRoot(path.parent) as secure_root:+ raw, _metadata = secure_root.read_bytes(Path(path.name), DEFAULT_DIAGNOSTIC_ARTIFACT_MAX_BYTES)+ return json.loads(raw)+ except (SecurePathError, ValueError, OSError, RecursionError, UnicodeError):return None+def _read_bounded_text(path: Path, *, max_bytes: int = DEFAULT_DIAGNOSTIC_ARTIFACT_MAX_BYTES) -> str | None:+ """Read bounded diagnostic text without following links or special files."""+ try:+ with SecureRoot(path.parent) as secure_root:+ raw, _metadata = secure_root.read_bytes(Path(path.name), max_bytes)+ return raw.decode("utf-8", errors="replace")+ except (SecurePathError, ValueError, OSError, RecursionError, UnicodeError):+ return None+++def _is_collector_owned_agent_dir(agent_dir: Path) -> bool:+ """Recognize a prior collector directory without claiming arbitrary user content."""+ if agent_dir.is_symlink():+ return False+ for condition in GENERATED_CONDITION_DIRS:+ condition_dir = agent_dir / condition+ summary = condition_dir / "summary.json"+ if condition_dir.is_symlink() or summary.is_symlink():+ continue+ try:+ if not summary.is_file() or summary.stat().st_size > DEFAULT_DIAGNOSTIC_ARTIFACT_MAX_BYTES:+ continue+ except OSError:+ continue+ data = _read_json(summary)+ if (+ isinstance(data, dict)+ and data.get("agent") == agent_dir.name+ and isinstance(data.get("scores"), dict)+ and data.get("execution_status") in {"failed", "skipped", "succeeded"}+ ):+ return True+ return False+++def _collector_owned_agent_dirs(output_root: Path) -> list[Path]:+ try:+ with os.scandir(output_root) as entries:+ candidates = [+ Path(entry.path) for entry in entries if not entry.is_symlink() and entry.is_dir(follow_symlinks=False)+ ]+ except OSError:+ return []+ return sorted((path for path in candidates if _is_collector_owned_agent_dir(path)), key=lambda path: path.name)+++def _prepare_generated_outputs(output_root: Path, agents: list[str]) -> None:+ """Validate the complete cleanup plan, then reset current and omitted generated outputs."""+ current_agent_dirs = [output_root / agent for agent in agents]+ planned_agent_dirs: dict[Path, Path] = {+ path.absolute(): path for path in [*current_agent_dirs, *_collector_owned_agent_dirs(output_root)]+ }+ root_artifacts = [output_root / artifact for artifact in GENERATED_ROOT_ARTIFACTS]++ for path in root_artifacts:+ _assert_safe_generated_output_path(path, output_root, follow_target=False)+ for path in planned_agent_dirs.values():+ _validate_agent_generated_outputs(path, output_root)++ for path in root_artifacts:+ _remove_generated_output_path(path, output_root)+ for path in planned_agent_dirs.values():+ _reset_agent_generated_outputs(path, output_root)++def validate_harbor_job_result(result_path: Path,*,@@ -147,12 +334,11 @@ def validate_harbor_job_result(if expected_trials is None:expected_trials = expected_total_trials- try:- result = json.loads(result_path.read_text(encoding="utf-8"))- except FileNotFoundError:+ if not result_path.exists():return False, f"Harbor exited successfully but did not produce {result_path}"- except (json.JSONDecodeError, OSError) as exc:- return False, f"Harbor produced an unreadable job result at {result_path}: {exc}"+ result = _read_json(result_path)+ if result is None:+ return False, f"Harbor produced an unreadable job result at {result_path}"if not isinstance(result, dict):return False, f"Harbor job result at {result_path} is not a JSON object"@@ -347,11 +533,8 @@ def _agent_log_runtime_failure_reason(trial_dir / "agent" / "opencode.txt",trial_dir / "opencode.txt",):- if not path.exists():- continue- try:- text = path.read_text(encoding="utf-8", errors="replace")- except OSError:+ text = _read_bounded_text(path)+ if text is None:continuefor line in text.splitlines():parsed = _read_json_text(line.strip())@@ -396,25 +579,222 @@ def _is_agent_runtime_failure_trial(trial_dir: Path) -> bool:return bool(_agent_runtime_failure_reason(trial_dir))+def _read_failed_judge_sidecar(+ path: Path,+ *,+ trial_dir: Path,+ expected: os.stat_result,+) -> tuple[dict[str, Any] | None, str]:+ """Read one sidecar, distinguishing safe absence from an unreadable candidate."""+ read_failure = "Judge sidecar could not be read as a bounded regular JSON object; trial was not scored"+ try:+ relative = path.relative_to(trial_dir)+ with SecureRoot(trial_dir) as secure_root:+ raw, _metadata = secure_root.read_bytes(+ relative,+ DEFAULT_DIAGNOSTIC_ARTIFACT_MAX_BYTES,+ expected=expected,+ )+ data = json.loads(raw)+ except (SecurePathError, ValueError, OSError, RecursionError, UnicodeError):+ return None, read_failure+ return (data, "") if isinstance(data, dict) else (None, read_failure)+++def _failed_judge_sidecar_paths(trial_dir: Path) -> tuple[list[tuple[str, Path, os.stat_result]], str]:+ """Return bounded sidecar paths plus a safe reason when the scan is incomplete.++ The candidate bound applies to sidecars that actually exist. Empty verifier+ directories do not consume it, so large valid step graphs remain compatible.+ """+ candidates: list[tuple[str, Path, os.stat_result]] = []++ def inspect_candidate(+ verifier_dir: Path,+ *,+ location: str,+ ) -> tuple[tuple[Path, os.stat_result] | None, str]:+ try:+ verifier_metadata = verifier_dir.lstat()+ except FileNotFoundError:+ return None, ""+ except OSError:+ return None, "Judge sidecar scan could not inspect verifier artifacts; trial was not scored"+ if stat_is_link_or_reparse(verifier_metadata):+ return None, f"Judge sidecar scan refused a symlinked {location} verifier; trial was not scored"+ if not stat.S_ISDIR(verifier_metadata.st_mode):+ return None, "Judge sidecar scan found a non-directory verifier artifact; trial was not scored"+ sidecar_path = verifier_dir / "skill_evaluator_reward.json"+ try:+ sidecar_metadata = sidecar_path.lstat()+ except FileNotFoundError:+ return None, ""+ except OSError:+ return None, "Judge sidecar scan could not inspect verifier artifacts; trial was not scored"+ return (sidecar_path, sidecar_metadata), ""++ root_candidate, root_failure = inspect_candidate(trial_dir / "verifier", location="root")+ if root_failure:+ return candidates, root_failure+ if root_candidate is not None:+ candidates.append(("", *root_candidate))++ steps_dir = trial_dir / "steps"+ try:+ steps_metadata = steps_dir.lstat()+ except FileNotFoundError:+ return candidates, ""+ except OSError:+ return candidates, "Judge sidecar scan could not inspect step artifacts; trial was not scored"+ if stat_is_link_or_reparse(steps_metadata):+ return candidates, "Judge sidecar scan refused a symlinked steps directory; trial was not scored"+ if not stat.S_ISDIR(steps_metadata.st_mode):+ return candidates, "Judge sidecar scan found a non-directory steps artifact; trial was not scored"+ scan_failure = ""+ try:+ with os.scandir(steps_dir) as entries:+ for index, entry in enumerate(entries):+ if index >= _MAX_FAILED_JUDGE_STEP_PATHS_SCANNED:+ scan_failure = (+ "Judge sidecar scan exceeded the "+ f"{_MAX_FAILED_JUDGE_STEP_PATHS_SCANNED}-entry safety limit; trial was not scored"+ )+ break+ try:+ entry_metadata = entry.stat(follow_symlinks=False)+ except OSError:+ scan_failure = "Judge sidecar scan could not inspect step artifacts; trial was not scored"+ break+ if stat_is_link_or_reparse(entry_metadata):+ scan_failure = "Judge sidecar scan refused a symlinked step entry; trial was not scored"+ break+ if not stat.S_ISDIR(entry_metadata.st_mode):+ continue+ verifier_dir = Path(entry.path) / "verifier"+ candidate, candidate_failure = inspect_candidate(verifier_dir, location="step")+ if candidate_failure:+ scan_failure = candidate_failure+ break+ if candidate is None:+ continue+ if len(candidates) >= _MAX_FAILED_JUDGE_SIDECARS:+ scan_failure = (+ "Judge sidecar scan exceeded the "+ f"{_MAX_FAILED_JUDGE_SIDECARS}-candidate safety limit; trial was not scored"+ )+ break+ candidates.append((entry.name, *candidate))+ except FileNotFoundError:+ pass+ except OSError:+ scan_failure = "Judge sidecar scan could not inspect step artifacts; trial was not scored"+ return sorted(candidates, key=lambda item: (item[0], item[1].as_posix())), scan_failure+++def _failed_judge_diagnostic(trial_dir: Path) -> dict[str, Any] | None:+ """Project failed judge sidecars into one safe, intrinsically unscoreable record."""+ errors: dict[str, str] = {}+ entry_id = ""+ sidecar_paths, scan_failure = _failed_judge_sidecar_paths(trial_dir)+ found_failure = bool(scan_failure)+ if scan_failure:+ errors["collector"] = scan_failure+ for step_name, path, expected in sidecar_paths:+ sidecar, read_failure = _read_failed_judge_sidecar(path, trial_dir=trial_dir, expected=expected)+ if read_failure:+ found_failure = True+ errors.setdefault("collector", read_failure)+ continue+ if not sidecar or str(sidecar.get("evaluation_status") or "").casefold() not in {"error", "failed"}:+ continue+ found_failure = True+ if not entry_id:+ entry_id = _safe_diagnostic_text(sidecar.get("entry_id"), max_len=256)+ safe_errors = _safe_evaluation_errors(sidecar.get("evaluation_errors"))+ if isinstance(safe_errors, dict):+ error_items = safe_errors.items()+ elif isinstance(safe_errors, list):+ error_items = ((f"judge_{index + 1}", reason) for index, reason in enumerate(safe_errors))+ elif safe_errors:+ error_items = (("judge", safe_errors),)+ else:+ error_items = ()+ safe_step = _safe_diagnostic_text(step_name, max_len=64)+ for metric, reason in error_items:+ key = f"{safe_step}.{metric}" if safe_step else str(metric)+ errors.setdefault(key, reason)+ if len(errors) >= len(DEFAULT_METRICS):+ break+ if len(errors) >= len(DEFAULT_METRICS):+ break++ if not found_failure:+ return None+ diagnostic: dict[str, Any] = {+ "metric_set": DEFAULT_METRIC_SET,+ "evaluation_status": "failed",+ }+ if entry_id:+ diagnostic["entry_id"] = entry_id+ if errors:+ diagnostic["evaluation_errors"] = errors+ return redact_sensitive_data(diagnostic, max_str_len=REWARD_DIAGNOSTIC_STRING_MAX_CHARS)+++def _inspect_trial_directory(trial_dir: Path) -> tuple[str, str]:+ """Classify one job child with one lstat, without following links."""+ try:+ metadata = trial_dir.lstat()+ except OSError:+ return "missing", ""+ if stat_is_link_or_reparse(metadata):+ return "link", "Unsafe Harbor trial directory is a symlink or reparse point; trial was not scored"+ if not stat.S_ISDIR(metadata.st_mode):+ return "other", ""+ return "directory", ""+++def _unsafe_trial_directory_reason(trial_dir: Path) -> str:+ """Return a bounded reason when an expected Harbor trial root is unsafe."""+ kind, reason = _inspect_trial_directory(trial_dir)+ if kind == "directory":+ return ""+ if kind == "link":+ return reason+ return "Unsafe Harbor trial directory could not be inspected; trial was not scored"++def _trial_failure_reason(trial_dir: Path) -> str:"""Return the failure recorded for any incomplete Harbor trial."""+ if unsafe_reason := _unsafe_trial_directory_reason(trial_dir):+ return unsafe_reason_, exception_reason = _trial_exception_details(trial_dir)if exception_reason:+ if diagnostic := _failed_judge_diagnostic(trial_dir):+ return _unscoreable_reward_reason(diagnostic)return exception_reasonexception_file = trial_dir / "exception.txt"- if not exception_file.exists():- return ""- try:- lines = [line.strip() for line in exception_file.read_text(encoding="utf-8", errors="replace").splitlines()]- except OSError:+ exception_text = _read_bounded_text(exception_file)+ if exception_text is None:return ""+ lines = [line.strip() for line in exception_text.splitlines()]reason = next((line for line in reversed(lines) if line), "")- return f"HarborTrialError: {reason}"[:600] if reason else ""+ if not reason:+ return ""+ if diagnostic := _failed_judge_diagnostic(trial_dir):+ return _unscoreable_reward_reason(diagnostic)+ return f"HarborTrialError: {reason}"[:600]def _extract_trial_failures(job_dir: Path) -> list[dict[str, str]]:failures: list[dict[str, str]] = []- for trial_dir in sorted(path for path in job_dir.iterdir() if path.is_dir()):+ for trial_dir in sorted(job_dir.iterdir()):+ kind, unsafe_reason = _inspect_trial_directory(trial_dir)+ if kind == "link":+ failures.append({"trial": trial_dir.name, "reason": unsafe_reason})+ continue+ if kind != "directory":+ continuereason = _trial_failure_reason(trial_dir)if reason:failures.append({"trial": trial_dir.name, "reason": redact_sensitive_text(reason)})@@ -453,7 +833,10 @@ def _can_preserve_partial_rewards(job_dir: Path, trial_failures: list[dict[str,def _extract_agent_runtime_failures(job_dir: Path) -> list[dict[str, str]]:failures: list[dict[str, str]] = []- for trial_dir in sorted(path for path in job_dir.iterdir() if path.is_dir()):+ for trial_dir in sorted(job_dir.iterdir()):+ kind, _reason = _inspect_trial_directory(trial_dir)+ if kind != "directory":+ continuereason = _agent_runtime_failure_reason(trial_dir)if reason:failures.append({"trial": trial_dir.name, "reason": redact_sensitive_text(reason)})@@ -469,53 +852,102 @@ def _diagnostic_artifact_max_bytes() -> int:except ValueError:logger.debug("Ignoring invalid SKILLEVALUATOR_HARBOR_DIAGNOSTIC_ARTIFACT_MAX_BYTES=%r", raw)return DEFAULT_DIAGNOSTIC_ARTIFACT_MAX_BYTES- return max(0, value)+ return min(max(0, value), DIAGNOSTIC_ARTIFACT_HARD_MAX_BYTES)-def _redacted_artifact_text(src: Path, text: str) -> str:+def _redacted_artifact_text(src: Path, text: str) -> str | None:if src.suffix.lower() == ".json":try:data = json.loads(text)- except json.JSONDecodeError:- return redact_sensitive_text(text)- return json.dumps(redact_sensitive_data(data), indent=2)- return redact_sensitive_text(text)+ safe_data = redact_sensitive_data(data)+ # Compact encoding avoids indentation-driven amplification for+ # deeply nested but otherwise valid diagnostic JSON.+ return json.dumps(safe_data, separators=(",", ":"), ensure_ascii=False)+ except (json.JSONDecodeError, ValueError, RecursionError, MemoryError):+ return None+ try:+ return redact_sensitive_text(text)+ except (RecursionError, MemoryError):+ return Nonedef _write_artifact_manifest(trial_out: Path, manifest: dict[str, Any]) -> None:try:- (trial_out / "artifact_manifest.json").write_text(- json.dumps(redact_sensitive_data(manifest), indent=2),- encoding="utf-8",+ write_output_file_atomically(+ trial_out / "artifact_manifest.json",+ json.dumps(redact_sensitive_data(manifest), indent=2).encode("utf-8"),)- except OSError as e:+ except (OSError, ValueError) as e:logger.debug("Failed to write Harbor artifact manifest %s: %s", trial_out, e)-def _write_redacted_text_copy(src: Path, dest: Path) -> tuple[bool, dict[str, Any] | None]:+def _write_redacted_text_copy(+ src: Path,+ dest: Path,+ *,+ source_root: Path | None = None,+) -> tuple[bool, dict[str, Any] | None]:"""Copy a Harbor text artifact while masking common credential shapes."""max_bytes = _diagnostic_artifact_max_bytes()try:- size_bytes = src.stat().st_size+ path_metadata = src.lstat()except OSError as e:logger.debug("Failed to stat Harbor artifact %s: %s", src, e)return False, {"name": src.name, "reason": "stat_failed"}- if max_bytes and size_bytes > max_bytes:+ if (+ stat_is_link_or_reparse(path_metadata)+ or not stat.S_ISREG(path_metadata.st_mode)+ or getattr(path_metadata, "st_nlink", 1) != 1+ ):+ return False, {"name": src.name, "reason": "not_regular_file"}++ read_root = source_root or src.parent+ try:+ relative = src.relative_to(read_root)+ with SecureRoot(read_root) as secure_root:+ raw, metadata = secure_root.read_bytes(relative, max_bytes, expected=path_metadata)+ except SecurePathError as e:+ if e.code == "file_size_limit":+ return False, {+ "name": src.name,+ "reason": "exceeds_max_bytes",+ "size_bytes": e.metadata.get("actual_bytes", path_metadata.st_size),+ "max_bytes": max_bytes,+ }+ logger.debug("Refusing unsafe Harbor artifact %s: %s", src, e)+ return False, {"name": src.name, "reason": "not_regular_file"}+ except (OSError, OverflowError, MemoryError, ValueError) as e:+ logger.debug("Failed to read Harbor artifact %s: %s", src, e)+ return False, {"name": src.name, "reason": "read_failed"}++ size_bytes = metadata.st_size+ if len(raw) > max_bytes:return False, {"name": src.name,"reason": "exceeds_max_bytes",- "size_bytes": size_bytes,+ "size_bytes": len(raw),"max_bytes": max_bytes,}try:- text = src.read_text(encoding="utf-8")- except (OSError, UnicodeDecodeError) as e:+ text = raw.decode("utf-8")+ except UnicodeDecodeError as e:logger.debug("Failed to read Harbor artifact %s: %s", src, e)return False, {"name": src.name, "reason": "read_failed", "size_bytes": size_bytes}+ redacted = _redacted_artifact_text(src, text)+ if redacted is None:+ return False, {"name": src.name, "reason": "invalid_json", "size_bytes": size_bytes}+ payload = redacted.encode("utf-8")+ if len(payload) > max_bytes:+ return False, {+ "name": src.name,+ "reason": "exceeds_max_bytes",+ "size_bytes": len(payload),+ "max_bytes": max_bytes,+ }try:dest.parent.mkdir(parents=True, exist_ok=True)- dest.write_text(_redacted_artifact_text(src, text), encoding="utf-8")- except OSError as e:+ write_output_file_atomically(dest, payload)+ except (OSError, ValueError) as e:logger.debug("Failed to write Harbor artifact %s: %s", dest, e)return False, {"name": src.name, "reason": "write_failed", "size_bytes": size_bytes}return True, {"name": src.name, "size_bytes": size_bytes}@@ -526,8 +958,8 @@ def _copy_trial_artifacts(trial_dir: Path, trial_out: Path) -> list[str]:manifest: dict[str, Any] = {"copied": [], "skipped": []}for artifact_name in TRIAL_DIAGNOSTIC_ARTIFACTS:src = trial_dir / artifact_name- if src.exists():- ok, record = _write_redacted_text_copy(src, trial_out / artifact_name)+ if src.exists() or src.is_symlink():+ ok, record = _write_redacted_text_copy(src, trial_out / artifact_name, source_root=trial_dir)if ok:copied.append(artifact_name)manifest["copied"].append(record)@@ -535,15 +967,27 @@ def _copy_trial_artifacts(trial_dir: Path, trial_out: Path) -> list[str]:manifest["skipped"].append(record)agent_logs = trial_dir / "agent"- for artifact_name in AGENT_LOG_ARTIFACTS:- src = agent_logs / artifact_name- if src.exists():- ok, record = _write_redacted_text_copy(src, trial_out / artifact_name)- if ok:- copied.append(artifact_name)- manifest["copied"].append(record)- elif record:- manifest["skipped"].append(record)+ try:+ agent_logs_metadata = agent_logs.lstat()+ except FileNotFoundError:+ agent_logs_metadata = None+ except OSError:+ manifest["skipped"].append({"name": "agent", "reason": "stat_failed"})+ agent_logs_metadata = None+ if agent_logs_metadata is not None and (+ stat_is_link_or_reparse(agent_logs_metadata) or not stat.S_ISDIR(agent_logs_metadata.st_mode)+ ):+ manifest["skipped"].append({"name": "agent", "reason": "not_regular_directory"})+ elif agent_logs_metadata is not None:+ for artifact_name in AGENT_LOG_ARTIFACTS:+ src = agent_logs / artifact_name+ if src.exists() or src.is_symlink():+ ok, record = _write_redacted_text_copy(src, trial_out / artifact_name, source_root=trial_dir)+ if ok:+ copied.append(artifact_name)+ manifest["copied"].append(record)+ elif record:+ manifest["skipped"].append(record)if manifest["skipped"]:_write_artifact_manifest(trial_out, manifest)return copied@@ -552,46 +996,34 @@ def _copy_trial_artifacts(trial_dir: Path, trial_out: Path) -> list[str]:def _trial_error_summary(trial_dir: Path) -> dict[str, Any]:result_file = trial_dir / "result.json"summary: dict[str, Any] = {}- if result_file.exists():- try:- result = json.loads(result_file.read_text(encoding="utf-8"))- except (json.JSONDecodeError, OSError) as e:- logger.debug("Failed to read %s: %s", result_file, e)- result = {}- if isinstance(result, dict):- for key in ("task_id", "task_name", "trial_name", "started_at", "finished_at"):- value = result.get(key)- if value not in (None, ""):- summary[key] = value-- agent_info = result.get("agent_info") if isinstance(result.get("agent_info"), dict) else {}- config = result.get("config") if isinstance(result.get("config"), dict) else {}- config_agent = config.get("agent") if isinstance(config.get("agent"), dict) else {}- model = agent_info.get("model_name") or config_agent.get("model_name")- if model:- summary["model"] = model-- exception_info = result.get("exception_info")- if isinstance(exception_info, dict) and exception_info:- error = {- "type": exception_info.get("exception_type"),- "message": _safe_text(exception_info.get("exception_message")),- "occurred_at": exception_info.get("occurred_at"),- }- summary["error"] = {k: v for k, v in error.items() if v not in (None, "")}+ result = _read_json(result_file)+ if isinstance(result, dict):+ for key in ("task_id", "task_name", "trial_name", "started_at", "finished_at"):+ value = result.get(key)+ if value not in (None, ""):+ summary[key] = value++ agent_info = result.get("agent_info") if isinstance(result.get("agent_info"), dict) else {}+ config = result.get("config") if isinstance(result.get("config"), dict) else {}+ config_agent = config.get("agent") if isinstance(config.get("agent"), dict) else {}+ model = agent_info.get("model_name") or config_agent.get("model_name")+ if model:+ summary["model"] = model++ exception_info = result.get("exception_info")+ if isinstance(exception_info, dict) and exception_info:+ error = {+ "type": exception_info.get("exception_type"),+ "message": _safe_text(exception_info.get("exception_message")),+ "occurred_at": exception_info.get("occurred_at"),+ }+ summary["error"] = {k: v for k, v in error.items() if v not in (None, "")}if "error" not in summary:exception_file = trial_dir / "exception.txt"- if exception_file.exists():- try:- lines = [- line.strip()- for line in exception_file.read_text(encoding="utf-8", errors="replace").splitlines()- if line.strip()- ]- except OSError as e:- logger.debug("Failed to read %s: %s", exception_file, e)- lines = []+ exception_text = _read_bounded_text(exception_file)+ if exception_text is not None:+ lines = [line.strip() for line in exception_text.splitlines() if line.strip()]if lines:summary["error"] = {"type": "HarborTrialError",@@ -620,12 +1052,45 @@ def _save_unscored_trials(scored_trials = {str(reward.get("_trial_root_name") or reward.get("_trial_name") or "") for reward in rewards}for trial_src in sorted(job_dir.iterdir()):- if not trial_src.is_dir() or trial_src.name in scored_trials or not _looks_like_trial_dir(trial_src):+ kind, unsafe_reason = _inspect_trial_directory(trial_src)+ if kind == "link":+ trial_out = trials_dir / trial_src.name+ trial_out.mkdir(parents=True, exist_ok=True)+ write_output_file_atomically(+ trial_out / "failure.json",+ json.dumps(+ {+ "status": "unscored",+ "trial": trial_src.name,+ "agent": agent,+ "variant": variant,+ "artifacts": [],+ "error": {"type": "UnsafeHarborTrial", "message": unsafe_reason},+ },+ indent=2,+ ).encode("utf-8"),+ )+ continue+ if kind != "directory":+ continue+ if trial_src.name in scored_trials or not _looks_like_trial_dir(trial_src):continuetrial_out = trials_dir / trial_src.nametrial_out.mkdir(parents=True, exist_ok=True)copied = _copy_trial_artifacts(trial_src, trial_out)+ judge_diagnostic = _failed_judge_diagnostic(trial_src)+ if judge_diagnostic is not None:+ judge_diagnostic.update({"agent": agent, "variant": variant})+ if agent_model:+ judge_diagnostic["model"] = agent_model+ if agent_model_source:+ judge_diagnostic["model_source"] = agent_model_source+ (trial_out / "reward.json").write_text(+ json.dumps(judge_diagnostic, indent=2),+ encoding="utf-8",+ )+ copied.append("reward.json")failure = {"status": "unscored","trial": trial_src.name,@@ -637,6 +1102,10 @@ def _save_unscored_trials(failure["model"] = agent_modelif agent_model_source:failure["model_source"] = agent_model_source+ if judge_diagnostic is not None:+ failure["evaluation_status"] = "failed"+ if "evaluation_errors" in judge_diagnostic:+ failure["evaluation_errors"] = judge_diagnostic["evaluation_errors"]error_summary = _trial_error_summary(trial_src)if agent_model:error_summary.pop("model", None)@@ -684,7 +1153,39 @@ def _reward_trajectory_path(trial_root: Path, step_name: str | None) -> Path:def _ordered_step_trajectory_paths(trial_root: Path) -> list[Path]:steps_dir = trial_root / "steps"- if not steps_dir.exists():+ try:+ steps_metadata = steps_dir.lstat()+ except OSError:+ return []+ if stat_is_link_or_reparse(steps_metadata) or not stat.S_ISDIR(steps_metadata.st_mode):+ return []++ safe_paths: dict[str, Path] = {}+ try:+ with os.scandir(steps_dir) as entries:+ for entry in entries:+ try:+ entry_metadata = entry.stat(follow_symlinks=False)+ except OSError:+ continue+ if stat_is_link_or_reparse(entry_metadata) or not stat.S_ISDIR(entry_metadata.st_mode):+ continue+ agent_dir = Path(entry.path) / "agent"+ try:+ agent_metadata = agent_dir.lstat()+ except OSError:+ continue+ if stat_is_link_or_reparse(agent_metadata) or not stat.S_ISDIR(agent_metadata.st_mode):+ continue+ trajectory = agent_dir / "trajectory.json"+ try:+ trajectory_metadata = trajectory.lstat()+ except OSError:+ continue+ if stat_is_link_or_reparse(trajectory_metadata) or not stat.S_ISREG(trajectory_metadata.st_mode):+ continue+ safe_paths[entry.name] = trajectory+ except OSError:return []ordered_names: list[str] = []@@ -701,12 +1202,12 @@ def _ordered_step_trajectory_paths(trial_root: Path) -> list[Path]:ordered_paths: list[Path] = []seen: set[Path] = set()for step_name in ordered_names:- path = steps_dir / step_name / "agent" / "trajectory.json"- if path.exists():+ path = safe_paths.get(step_name)+ if path is not None:ordered_paths.append(path)seen.add(path)- for path in sorted(steps_dir.glob("*/agent/trajectory.json")):+ for path in sorted(safe_paths.values()):if path not in seen:ordered_paths.append(path)return ordered_paths@@ -832,9 +1333,23 @@ def _merge_reward_sidecars(data: dict[str, Any], verifier_dir: Path) -> None:"""Merge SkillEvaluator-rich sidecars back into Harbor's numeric-only reward payload."""skill_evaluator_reward = _read_json(verifier_dir / "skill_evaluator_reward.json")if isinstance(skill_evaluator_reward, dict):+ sidecar_failed = str(skill_evaluator_reward.get("evaluation_status") or "").casefold() in {+ "error",+ "failed",+ }+ if sidecar_failed:+ data["evaluation_status"] = "failed"+ if "evaluation_errors" in skill_evaluator_reward:+ data["evaluation_errors"] = _safe_evaluation_errors(skill_evaluator_reward["evaluation_errors"])+ else:+ data.pop("evaluation_errors", None)for key, value in skill_evaluator_reward.items():+ if sidecar_failed and key in {"evaluation_status", "evaluation_errors"}:+ continueif key in data and isinstance(data.get(key), int | float) and not isinstance(data.get(key), bool):continue+ if key == "evaluation_errors":+ value = _safe_evaluation_errors(value)data.setdefault(key, value)custom_reward = _read_json(verifier_dir / "custom_reward.json")@@ -866,6 +1381,170 @@ def _merge_reward_sidecars(data: dict[str, Any], verifier_dir: Path) -> None:data.setdefault(key, value)+def _merge_trial_evaluation_failures(data: dict[str, Any], trial_dir: Path) -> None:+ """Preserve judge-failure diagnostics when Harbor supplies an aggregate reward."""+ # Pure custom-only verifiers do not run the standard Tier-3 LLM judge and+ # must not inherit its sidecar scan limits merely because step directories+ # exist. Default and default-plus-custom rewards carry canonical metrics.+ if not _standard_reward_metrics(data):+ return+ diagnostic = _failed_judge_diagnostic(trial_dir)+ if diagnostic is None:+ return++ merged_errors = _merge_bounded_evaluation_errors(+ diagnostic.get("evaluation_errors"),+ data.get("evaluation_errors"),+ )++ data["evaluation_status"] = "failed"+ if merged_errors:+ data["evaluation_errors"] = merged_errors+ else:+ data.pop("evaluation_errors", None)+++def _merge_bounded_evaluation_errors(*values: Any) -> dict[str, str]:+ """Merge diagnostics in priority order while retaining the public display bound."""+ merged: dict[str, str] = {}+ for value in values:+ safe_errors = _safe_evaluation_errors(value)+ if isinstance(safe_errors, dict):+ items = safe_errors.items()+ elif isinstance(safe_errors, list):+ items = ((f"judge_{index + 1}", reason) for index, reason in enumerate(safe_errors))+ elif safe_errors:+ items = (("judge", safe_errors),)+ else:+ items = ()+ for metric, reason in items:+ merged.setdefault(str(metric), str(reason))+ if len(merged) >= len(DEFAULT_METRICS):+ return merged+ return merged+++def _standard_reward_metrics(+ rewards: dict[str, Any],+ *,+ inherited_metrics: tuple[str, ...] = (),+) -> tuple[str, ...]:+ """Return required standard metrics, preserving explicit child metric-set declarations."""+ nested_metrics = rewards.get("metrics")+ metric_names = {str(name) for name in rewards}+ if isinstance(nested_metrics, dict):+ metric_names.update(str(name) for name in nested_metrics)+ declared_metric_set = str(rewards.get("metric_set") or rewards.get("metric_set_version") or "")+ if declared_metric_set not in {DEFAULT_METRIC_SET, LEGACY_METRIC_SET} and not metric_names.intersection(+ DEFAULT_METRICS+ ):+ return ()++ _, expected_metrics = metric_set_for_reward(rewards)+ if not declared_metric_set and inherited_metrics:+ return inherited_metrics+ if expected_metrics:+ return expected_metrics+ # An unversioned all-non-finite standard shape can otherwise look+ # custom-only because its numeric ``overall`` is the sole finite value.+ return DEFAULT_METRICS if "security" in metric_names else LEGACY_METRICS+++def _constituent_default_reward_failure(result: dict[str, Any]) -> str:+ """Return a safe failure when a standard step reward cannot support its aggregate."""+ root_verifier = result.get("verifier_result")+ root_rewards = root_verifier.get("rewards") if isinstance(root_verifier, dict) else None+ root_failed = str(result.get("evaluation_status") or "").casefold() in {"error", "failed"}+ if isinstance(root_verifier, dict):+ root_failed = root_failed or str(root_verifier.get("evaluation_status") or "").casefold() in {+ "error",+ "failed",+ }+ if isinstance(root_rewards, dict):+ root_failed = root_failed or str(root_rewards.get("evaluation_status") or "").casefold() in {+ "error",+ "failed",+ }+ if root_failed:+ return "Authoritative verifier reward is failed; it was not scored"++ step_results = result.get("step_results")+ if "step_results" in result and not isinstance(step_results, list):+ return "Authoritative verifier result has malformed constituent steps; it was not scored"+ if not isinstance(step_results, list):+ return ""++ root_metrics = _standard_reward_metrics(root_rewards) if isinstance(root_rewards, dict) else ()++ for index, step in enumerate(step_results, start=1):+ if not isinstance(step, dict):+ if root_metrics:+ return (+ f"Constituent default reward for step {index} is incomplete, non-finite, or failed; "+ "the authoritative aggregate was not scored"+ )+ continue+ step_name = _safe_diagnostic_text(step.get("step_name"), max_len=64) or str(index)+ verifier_result = step.get("verifier_result")+ step_failed = ("exception_info" in step and step.get("exception_info") is not None) or str(+ step.get("evaluation_status") or ""+ ).casefold() in {+ "error",+ "failed",+ }+ if not isinstance(verifier_result, dict):+ if step_failed:+ return (+ f"Constituent default reward for step {step_name} is incomplete, non-finite, or failed; "+ "the authoritative aggregate was not scored"+ )+ continue+ rewards = verifier_result.get("rewards")+ failed_status = step_failed or str(verifier_result.get("evaluation_status") or "").casefold() in {+ "error",+ "failed",+ }+ if isinstance(rewards, dict):+ failed_status = failed_status or str(rewards.get("evaluation_status") or "").casefold() in {+ "error",+ "failed",+ }+ if failed_status or (root_metrics and (not isinstance(rewards, dict) or not rewards)):+ return (+ f"Constituent default reward for step {step_name} is incomplete, non-finite, or failed; "+ "the authoritative aggregate was not scored"+ )+ if not isinstance(rewards, dict):+ continue++ expected_metrics = _standard_reward_metrics(+ rewards,+ inherited_metrics=root_metrics,+ )+ if not expected_metrics:+ continue+ if all(metric_value(rewards, metric) is not None for metric in expected_metrics):+ continue++ return (+ f"Constituent default reward for step {step_name} is incomplete, non-finite, or failed; "+ "the authoritative aggregate was not scored"+ )+ return ""+++def _merge_constituent_default_reward_failure(data: dict[str, Any], result: dict[str, Any]) -> None:+ """Make an aggregate unscoreable when one of its standard constituents is invalid."""+ reason = _constituent_default_reward_failure(result)+ if not reason:+ return+ data["evaluation_status"] = "failed"+ data["evaluation_errors"] = _merge_bounded_evaluation_errors(+ {"collector": reason},+ data.get("evaluation_errors"),+ )++def _extract_rewards(job_dir: Path) -> list[dict[str, Any]]:"""Extract reward.json from all trials in a job directory."""rewards: list[dict[str, Any]] = []@@ -888,6 +1567,8 @@ def _extract_rewards(job_dir: Path) -> list[dict[str, Any]]:data = _reward_from_harbor_result(result)if not data:continue+ _merge_constituent_default_reward_failure(data, result)+ _merge_trial_evaluation_failures(data, trial_dir)trial_name = str(result.get("trial_name") or trial_dir.name)data["_trial_name"] = trial_namedata["_trial_root_name"] = trial_dir.name@@ -906,10 +1587,16 @@ def _extract_rewards(job_dir: Path) -> list[dict[str, Any]]:if reward_file.parent.name == "verifier":try:trial_dir, trial_name, step_name = _reward_trial_context(reward_file)+ if _trial_failure_reason(trial_dir) or _is_agent_runtime_failure_trial(trial_dir):+ continueif trial_dir in authoritative_trial_roots:continue- data = json.loads(reward_file.read_text(encoding="utf-8"))+ data = _read_json(reward_file)+ if not isinstance(data, dict):+ logger.warning("Ignoring invalid or oversized Harbor reward: %s", reward_file)+ continue_merge_reward_sidecars(data, reward_file.parent)+ _merge_trial_evaluation_failures(data, trial_dir)if _trial_failure_reason(trial_dir) or _is_agent_runtime_failure_trial(trial_dir):logger.debug("Skipping reward for failed Harbor trial: %s",@@ -922,21 +1609,20 @@ def _extract_rewards(job_dir: Path) -> list[dict[str, Any]]:data["_step_name"] = step_nameresult_file = trial_dir / "result.json"if result_file.exists():- try:- result = json.loads(result_file.read_text(encoding="utf-8"))+ result = _read_json(result_file)+ if isinstance(result, dict):+ _merge_constituent_default_reward_failure(data, result)data["_started_at"] = result.get("started_at")if not data.get("entry_id"):entry_id = _entry_id_from_harbor_result(result)if entry_id:data["entry_id"] = entry_id- except (json.JSONDecodeError, OSError) as e:- logger.debug("Failed to read %s: %s", result_file, e)traj_file = _reward_trajectory_path(trial_dir, step_name)if traj_file.exists():data["_has_trajectory"] = Truerewards.append(data)scored_trial_roots.add(trial_dir)- except (json.JSONDecodeError, OSError) as e:+ except OSError as e:logger.warning("Failed to read %s: %s", reward_file, e)for result_file in sorted(job_dir.glob("*/result.json")):@@ -954,6 +1640,8 @@ def _extract_rewards(job_dir: Path) -> list[dict[str, Any]]:data = _reward_from_harbor_result(result)if not data:continue+ _merge_constituent_default_reward_failure(data, result)+ _merge_trial_evaluation_failures(data, trial_dir)trial_name = str(result.get("trial_name") or trial_dir.name)data["_trial_name"] = trial_namedata["_trial_root_name"] = trial_dir.name@@ -1029,6 +1717,19 @@ def _harbor_result_rewards(result: dict[str, Any]) -> dict[str, Any] | None:]if values:aggregated[key] = sum(values) / len(values)++ aggregated.update(average_custom_metrics(step_reward_rows))++ standard_rows = [rewards for rewards in step_reward_rows if _standard_reward_metrics(rewards)]+ active_metrics: tuple[str, ...] = ()+ if any(_standard_reward_metrics(rewards) == DEFAULT_METRICS for rewards in standard_rows):+ active_metrics = DEFAULT_METRICS+ elif standard_rows:+ active_metrics = LEGACY_METRICS+ for metric in active_metrics:+ values = [value for rewards in standard_rows if (value := metric_value(rewards, metric)) is not None]+ if values:+ aggregated[metric] = sum(values) / len(values)return aggregated or None@@ -1066,7 +1767,8 @@ def _partition_scoreable_rewards(failures: list[dict[str, str]] = []failed_trials: set[str] = set()for reward in rewards:- if overall_score(reward) is not None:+ evaluation_failed = str(reward.get("evaluation_status") or "").casefold() in {"error", "failed"}+ if not evaluation_failed and overall_score(reward) is not None:scoreable.append(reward)continuetrial = str(reward.get("_trial_name") or reward.get("_trial_root_name") or "unknown trial")@@ -1076,12 +1778,39 @@ def _partition_scoreable_rewards(failures.append({"trial": trial,- "reason": "Reward metrics are incomplete or non-finite; trial was not scored",+ "reason": _unscoreable_reward_reason(reward),})return scoreable, failures+def _unscoreable_reward_reason(reward: dict[str, Any]) -> str:+ """Return a bounded, redacted diagnostic for an unscoreable reward."""+ fallback = "Reward metrics are incomplete or non-finite; trial was not scored"+ if str(reward.get("evaluation_status") or "").casefold() not in {"error", "failed"}:+ return fallback++ errors = reward.get("evaluation_errors")+ parts: list[str] = []+ if isinstance(errors, dict):+ for metric, reason in list(errors.items())[: len(DEFAULT_METRICS)]:+ safe_metric = _safe_diagnostic_text(metric, max_len=64)+ safe_reason = _safe_diagnostic_text(reason, max_len=512)+ if safe_reason:+ parts.append(f"{safe_metric or 'judge'}: {safe_reason}")+ elif isinstance(errors, list):+ parts.extend(_safe_diagnostic_text(reason, max_len=512) for reason in errors[: len(DEFAULT_METRICS)])+ parts = [part for part in parts if part]+ elif errors not in (None, ""):+ safe_reason = _safe_diagnostic_text(errors, max_len=512)+ if safe_reason:+ parts.append(safe_reason)++ if not parts:+ return fallback+ return _safe_diagnostic_text("Required judge evaluation failed: " + "; ".join(parts), max_len=2048)++def _strip_attempt_suffix(value: str) -> str:"""Remove SkillEvaluator per-attempt suffixes from a task/case identifier."""return re.sub(r"(?:[-_])attempt\d+$", "", value)@@ -1131,7 +1860,7 @@ def _attempt_ordinal(reward: dict[str, Any]) -> int | None:def _logical_attempt_rewards(rewards: list[dict[str, Any]]) -> list[dict[str, Any]]:- """Collapse multi-step reward rows to one pass@k score per Harbor trial root."""+ """Collapse multi-step reward rows to one metric-bearing logical trial."""grouped: dict[str, list[dict[str, Any]]] = {}for index, reward in enumerate(rewards):root = str(reward.get("_trial_root_name") or reward.get("_trial_name") or f"__row_{index}")@@ -1144,20 +1873,27 @@ def _logical_attempt_rewards(rewards: list[dict[str, Any]]) -> list[dict[str, Anlogical.append(authoritative if authoritative is not None else rows[0])continuefirst = rows[0]- scores = [_overall_score(row) for row in rows]- logical.append(- {- "entry_id": first.get("entry_id"),- "overall": (- sum(score for score in scores if score is not None) / len(scores)- if all(score is not None for score in scores)- else None- ),- "_trial_name": root,- "_trial_root_name": root,- "_started_at": first.get("_started_at"),- }- )+ standard_scores, metric_set, metrics = average_metrics(rows)+ custom_scores = average_custom_metrics(rows)+ logical_reward: dict[str, Any] = {+ "entry_id": first.get("entry_id"),+ "_trial_name": root,+ "_trial_root_name": root,+ "_started_at": first.get("_started_at"),+ }+ if metric_set:+ logical_reward["metric_set"] = metric_set+ for metric in metrics:+ if metric in standard_scores:+ logical_reward[metric] = standard_scores[metric]+ logical_reward.update(custom_scores)+ if custom_scores:+ logical_reward["custom_metrics"] = custom_scores+ if (overall := _average_overall(rows)) is not None:+ logical_reward["overall"] = overall+ if any(row.get("_has_trajectory") for row in rows):+ logical_reward["_has_trajectory"] = True+ logical.append(logical_reward)return logical@@ -1363,6 +2099,26 @@ def _security_finding_signature(finding: dict[str, Any]) -> tuple[str, str]:return str(finding.get("type") or "unknown"), text[:160]+def _safe_trial_path_component(value: Any) -> str:+ """Return a portable single path component or an empty string."""+ component = str(value or "").strip()+ if (+ not component+ or component in {".", ".."}+ or any(character in component for character in ("/", "\\", ":", "\x00"))+ or any(ord(character) < 32 or ord(character) == 127 for character in component)+ ):+ return ""+ return component+++def _persisted_trial_name(reward: dict[str, Any]) -> tuple[str, str]:+ """Derive output and source names only from physical Harbor path components."""+ trial_root_name = _safe_trial_path_component(reward.get("_trial_root_name")) or "unknown"+ step_name = _safe_trial_path_component(reward.get("_step_name"))+ return (f"{trial_root_name}__{step_name}" if step_name else trial_root_name), trial_root_name++def _annotate_security_attribution(with_rewards: list[dict[str, Any]],without_rewards: list[dict[str, Any]],@@ -1478,10 +2234,9 @@ def _save_trials("""Save per-trial reward.json and trajectory.json into the results directory."""trials_dir.mkdir(parents=True, exist_ok=True)for reward in rewards:- trial_name = reward.get("_trial_name", "unknown")+ trial_name, trial_root_name = _persisted_trial_name(reward)trial_out = trials_dir / trial_nametrial_out.mkdir(parents=True, exist_ok=True)- trial_root_name = reward.get("_trial_root_name", trial_name)trial_src = job_dir / trial_root_name if job_dir else Nonesrc_traj = _reward_trajectory_path(trial_src, reward.get("_step_name")) if trial_src else Nonemerged_traj = (@@ -1495,6 +2250,11 @@ def _save_trials(reward["_trajectory_summary"] = _summarize_trajectory_file(src_traj)clean_reward = {k: v for k, v in reward.items() if not k.startswith("_")}+ # Persist the physical attempt identity so bounded report readers can keep+ # fallback multi-step rows for diagnostics without weighting a logical+ # Harbor trial once per step. This also populates the canonical report's+ # existing trial_id field instead of inventing a second report schema.+ clean_reward["trial_id"] = trial_root_nameif not clean_reward.get("entry_id"):clean_reward["entry_id"] = _entry_id(reward)clean_reward["agent"] = agent@@ -1502,7 +2262,18 @@ def _save_trials(clean_reward["model"] = agent_modelif agent_model_source:clean_reward["model_source"] = agent_model_source- (trial_out / "reward.json").write_text(json.dumps(clean_reward, indent=2), encoding="utf-8")+ if "evaluation_errors" in clean_reward:+ clean_reward["evaluation_errors"] = _safe_evaluation_errors(clean_reward["evaluation_errors"])+ diagnostic_reward = (+ str(clean_reward.get("evaluation_status") or "").casefold() in {"error", "failed"}+ or overall_score(clean_reward) is None+ )+ safe_reward = (+ redact_sensitive_data(clean_reward, max_str_len=REWARD_DIAGNOSTIC_STRING_MAX_CHARS)+ if diagnostic_reward or "evaluation_errors" in clean_reward+ else clean_reward+ )+ (trial_out / "reward.json").write_text(json.dumps(safe_reward, indent=2), encoding="utf-8")if trial_src:_copy_trial_artifacts(trial_src, trial_out)@@ -1512,7 +2283,7 @@ def _save_trials(encoding="utf-8",)elif src_traj and src_traj.exists():- _write_redacted_text_copy(src_traj, trial_out / "trajectory.json")+ _write_redacted_text_copy(src_traj, trial_out / "trajectory.json", source_root=trial_src)_save_unscored_trials(rewards,@@ -1527,9 +2298,8 @@ def _save_trials(def _summarize_trajectory_file(path: Path) -> dict[str, Any]:"""Return safe trajectory metadata without raw prompts, outputs, or arguments."""- try:- data = json.loads(path.read_text(encoding="utf-8"))- except (OSError, json.JSONDecodeError):+ data = _read_json(path)+ if not isinstance(data, dict):return {"readable": False}return _summarize_trajectory(data)@@ -1575,6 +2345,7 @@ def _condition_execution_summary(n_attempts: int,job_failure: str,runtime_failures: list[dict[str, str]] | None = None,+ reward_failures: list[dict[str, str]] | None = None,skipped: bool = False,stop_on_pass: bool = False,pass_threshold: float = 0.50,@@ -1601,6 +2372,12 @@ def _condition_execution_summary(f"Agent runtime failed in {failure.get('trial', 'unknown trial')}: {failure.get('reason', 'unknown error')}"for failure in (runtime_failures or []))+ errors.extend(+ "Unscoreable reward in "+ f"{_safe_diagnostic_text(failure.get('trial', 'unknown trial'), max_len=256)}: "+ f"{_safe_diagnostic_text(failure.get('reason', 'unknown error'), max_len=2048)}"+ for failure in (reward_failures or [])+ )expected_set = set(expected_ids)logical_passed: dict[str, bool] = {}for reward in _logical_attempt_rewards(rewards):@@ -1769,17 +2546,20 @@ def collect_harbor_results(},}+ _prepare_generated_outputs(output_dir, agents)+for agent in agents:model_info = agent_models.get(agent, {}) if agent_models else {}agent_model = model_info.get("model")agent_model_source = model_info.get("source")agent_dir = output_dir / agent- agent_dir.mkdir(parents=True, exist_ok=True)with_job_name = f"{skill_name}-{agent}-with"with_job_dir = _find_job_dir(jobs_dir, with_job_name)+ with_collected_rewards: list[dict[str, Any]] = []with_rewards: list[dict[str, Any]] = []+ with_logical_rewards: list[dict[str, Any]] = []with_scores: dict[str, float] = {}with_custom_scores: dict[str, float] = {}with_pass: dict[str, Any] = {}@@ -1796,16 +2576,17 @@ def collect_harbor_results(with_runtime_failures = _extract_agent_runtime_failures(with_job_dir)with_trial_failures = _extract_trial_failures(with_job_dir)preserve_partial = _can_preserve_partial_rewards(with_job_dir, with_trial_failures)- with_rewards = _extract_rewards(with_job_dir) if with_job_ok or preserve_partial else []- with_rewards, invalid_score_failures = _partition_scoreable_rewards(with_rewards)+ with_collected_rewards = _extract_rewards(with_job_dir) if with_job_ok or preserve_partial else []+ with_rewards, invalid_score_failures = _partition_scoreable_rewards(with_collected_rewards)+ with_logical_rewards = _logical_attempt_rewards(with_rewards)with_trial_failures.extend(invalid_score_failures)- with_scores, with_metric_set, with_metrics = average_metrics(with_rewards)+ with_scores, with_metric_set, with_metrics = average_metrics(with_logical_rewards)all_results["metric_set"] = with_metric_setall_results["metrics"] = list(with_metrics)all_results["attempt_policy"]["score_definition"] = score_definition(with_metrics)- with_custom_scores = average_custom_metrics(with_rewards)+ with_custom_scores = average_custom_metrics(with_logical_rewards)with_pass = _pass_summary(- with_rewards,+ with_logical_rewards,n_attempts=n_attempts,pass_threshold=pass_threshold,stop_on_pass=stop_on_pass,@@ -1819,11 +2600,19 @@ def collect_harbor_results(n_attempts=n_attempts,job_failure=with_job_failure,runtime_failures=with_runtime_failures,+ reward_failures=with_trial_failures,stop_on_pass=stop_on_pass,pass_threshold=pass_threshold,)+ if with_execution["execution_status"] != "succeeded":+ with_scores = {}+ with_custom_scores = {}+ with_pass = {}+ with_overall_score = (+ _average_overall(with_logical_rewards) if with_execution["execution_status"] == "succeeded" else None+ )_save_trials(- with_rewards,+ with_collected_rewards,agent_dir / "with-skill" / "trials",with_job_dir,agent=agent,@@ -1839,6 +2628,7 @@ def collect_harbor_results("model_source": agent_model_source,"scores": with_scores,"custom_scores": with_custom_scores,+ "overall_score": with_overall_score,"metric_set": with_metric_set,"metrics": list(with_metrics),"dimensions": dimension_scores(with_scores),@@ -1871,6 +2661,7 @@ def collect_harbor_results("model_source": agent_model_source,"scores": {},"custom_scores": {},+ "overall_score": None,"metric_set": DEFAULT_METRIC_SET,"metrics": list(DISPLAY_METRICS),"dimensions": {},@@ -1906,6 +2697,7 @@ def collect_harbor_results("model_source": agent_model_source,"scores": {},"custom_scores": {},+ "overall_score": None,"metrics": [],"dimensions": {},"num_trials": 0,@@ -1919,7 +2711,9 @@ def collect_harbor_results(encoding="utf-8",)+ without_collected_rewards: list[dict[str, Any]] = []without_rewards: list[dict[str, Any]] = []+ without_logical_rewards: list[dict[str, Any]] = []without_scores: dict[str, float] = {}without_custom_scores: dict[str, float] = {}without_pass: dict[str, Any] = {}@@ -1940,13 +2734,16 @@ def collect_harbor_results(without_runtime_failures = _extract_agent_runtime_failures(without_job_dir)without_trial_failures = _extract_trial_failures(without_job_dir)preserve_partial = _can_preserve_partial_rewards(without_job_dir, without_trial_failures)- without_rewards = _extract_rewards(without_job_dir) if without_job_ok or preserve_partial else []- without_rewards, invalid_score_failures = _partition_scoreable_rewards(without_rewards)+ without_collected_rewards = (+ _extract_rewards(without_job_dir) if without_job_ok or preserve_partial else []+ )+ without_rewards, invalid_score_failures = _partition_scoreable_rewards(without_collected_rewards)+ without_logical_rewards = _logical_attempt_rewards(without_rewards)without_trial_failures.extend(invalid_score_failures)- without_scores, without_metric_set, without_metrics = average_metrics(without_rewards)- without_custom_scores = average_custom_metrics(without_rewards)+ without_scores, without_metric_set, without_metrics = average_metrics(without_logical_rewards)+ without_custom_scores = average_custom_metrics(without_logical_rewards)without_pass = _pass_summary(- without_rewards,+ without_logical_rewards,n_attempts=n_attempts,pass_threshold=pass_threshold,stop_on_pass=stop_on_pass,@@ -1960,11 +2757,21 @@ def collect_harbor_results(n_attempts=n_attempts,job_failure=without_job_failure,runtime_failures=without_runtime_failures,+ reward_failures=without_trial_failures,stop_on_pass=stop_on_pass,pass_threshold=pass_threshold,)+ if without_execution["execution_status"] != "succeeded":+ without_scores = {}+ without_custom_scores = {}+ without_pass = {}+ without_overall_score = (+ _average_overall(without_logical_rewards)+ if without_execution["execution_status"] == "succeeded"+ else None+ )_save_trials(- without_rewards,+ without_collected_rewards,agent_dir / "without-skill" / "trials",without_job_dir,agent=agent,@@ -1980,6 +2787,7 @@ def collect_harbor_results("model_source": agent_model_source,"scores": without_scores,"custom_scores": without_custom_scores,+ "overall_score": without_overall_score,"metric_set": without_metric_set,"metrics": list(without_metrics),"dimensions": dimension_scores(without_scores),@@ -1994,7 +2802,10 @@ def collect_harbor_results(encoding="utf-8",)logger.debug(- "Agent %s without-skill: %d trials, scores=%s", agent, len(without_rewards), without_scores+ "Agent %s without-skill: %d trials, scores=%s",+ agent,+ len(without_rewards),+ without_scores,)else:without_job_failure = f"No Harbor job found for {without_job_name}"@@ -2014,6 +2825,7 @@ def collect_harbor_results("model_source": agent_model_source,"scores": {},"custom_scores": {},+ "overall_score": None,"metric_set": DEFAULT_METRIC_SET,"metrics": list(DISPLAY_METRICS),"dimensions": {},@@ -2050,6 +2862,7 @@ def collect_harbor_results("model_source": agent_model_source,"scores": {},"custom_scores": {},+ "overall_score": None,"metrics": [],"dimensions": {},"num_trials": 0,@@ -2064,28 +2877,33 @@ def collect_harbor_results()lift: dict[str, Any] = {}- if with_scores and without_scores:+ paired_execution_succeeded = (+ with_execution.get("execution_status") == "succeeded"+ and without_execution.get("execution_status") == "succeeded"+ )+ if paired_execution_succeeded and with_scores and without_scores:lift = _compute_lift(with_scores, without_scores)(agent_dir / "lift.json").write_text(json.dumps(lift, indent=2), encoding="utf-8")custom_lift: dict[str, Any] = {}if (- with_rewards- and without_rewards+ paired_execution_succeeded+ and with_logical_rewards+ and without_logical_rewardsand (with_custom_scores or without_custom_scores or (not with_scores and not without_scores))):custom_lift = _compute_custom_lift(with_custom_scores,without_custom_scores,- with_rewards,- without_rewards,+ with_logical_rewards,+ without_logical_rewards,include_overall=not with_scores and not without_scores,)if custom_lift:(agent_dir / "custom_lift.json").write_text(json.dumps(custom_lift, indent=2), encoding="utf-8")pass_lift: dict[str, Any] = {}- if with_pass and without_pass:+ if paired_execution_succeeded and with_pass and without_pass:pass_lift = {"with_skill": with_pass.get("rate", 0.0),"without_skill": without_pass.get("rate", 0.0),@@ -2095,7 +2913,10 @@ def collect_harbor_results((agent_dir / "pass_at_k_lift.json").write_text(json.dumps(pass_lift, indent=2), encoding="utf-8")security_attribution: dict[str, Any] = {}- if with_rewards:+ attribution_execution_succeeded = with_execution.get("execution_status") == "succeeded" and (+ skip_baseline or without_execution.get("execution_status") == "succeeded"+ )+ if attribution_execution_succeeded and with_rewards:security_attribution = _annotate_security_attribution(with_rewards,without_rewards,@@ -2159,14 +2980,12 @@ def collect_harbor_results("output_dir": str(agent_dir.resolve()),}- (output_dir / "attempt_policy.json").write_text(- json.dumps(all_results["attempt_policy"], indent=2), encoding="utf-8"- )+ _write_generated_root_json(output_dir / "attempt_policy.json", output_dir, all_results["attempt_policy"])if len(agents) > 1:comparison = _build_comparison(all_results["agents"])all_results["comparison"] = comparison- (output_dir / "comparison.json").write_text(json.dumps(comparison, indent=2), encoding="utf-8")+ _write_generated_root_json(output_dir / "comparison.json", output_dir, comparison)top_execution = _aggregate_execution(list(all_results["agents"].values()))all_results.update(top_execution)diff --git a/src/skillevaluator/tier3/harbor/metrics.py b/src/skillevaluator/tier3/harbor/metrics.pyindex 0b0ce0f7..98ed8658 100644--- a/src/skillevaluator/tier3/harbor/metrics.py+++ b/src/skillevaluator/tier3/harbor/metrics.py@@ -86,6 +86,8 @@ _RESERVED_METADATA_KEYS = {"details","entry_id","error",+ "evaluation_errors",+ "evaluation_status","has_skill","metric_set","metric_set_version",diff --git a/src/skillevaluator/tier3/harbor/report.py b/src/skillevaluator/tier3/harbor/report.pyindex 9be08f85..c2bb22d8 100644--- a/src/skillevaluator/tier3/harbor/report.py+++ b/src/skillevaluator/tier3/harbor/report.py@@ -12,9 +12,11 @@ from __future__ import annotationsimport jsonimport logging+import mathfrom pathlib import Pathfrom typing import Any+from skillevaluator.tier3.harbor import report_datafrom skillevaluator.tier3.harbor.metrics import (DEFAULT_METRICS,METRIC_DESCRIPTIONS,@@ -38,18 +40,42 @@ _METRIC_LABELS = {}-def _load_trial_rewards(results_dir: Path, agent: str) -> list[dict[str, Any]]:- """Load all reward.json files for a given agent's with-skill trials."""- trials_dir = results_dir / agent / "with-skill" / "trials"- if not trials_dir.exists():- return []- rewards = []- for reward_file in sorted(trials_dir.rglob("reward.json")):- try:- rewards.append(json.loads(reward_file.read_text(encoding="utf-8")))- except (json.JSONDecodeError, OSError) as e:- logger.warning("Failed to read %s: %s", reward_file, e)- return rewards+def _findings_artifact_path(results_dir: Path, agent: str) -> Path | None:+ """Return a findings path only when its parent remains inside results_dir."""+ artifact = results_dir / agent / "findings.json"+ try:+ artifact.absolute().relative_to(results_dir.absolute())+ artifact.parent.resolve().relative_to(results_dir.resolve())+ except (OSError, RuntimeError, ValueError):+ logger.warning("Refusing findings artifact outside results directory: %s", artifact)+ return None+ if artifact.parent.is_symlink():+ logger.warning("Refusing findings artifact in symlinked agent directory: %s", artifact)+ return None+ return artifact+++def _remove_stale_findings_artifact(results_dir: Path, agent: str) -> None:+ artifact = _findings_artifact_path(results_dir, agent)+ if artifact is None:+ return+ try:+ if artifact.is_symlink() or artifact.is_file():+ artifact.unlink()+ except OSError as e:+ logger.warning("Failed to remove stale findings artifact %s: %s", artifact, e)+++def _load_trial_rewards(+ results_dir: Path,+ agent: str,+ loaded_agents: dict[str, dict[str, Any]] | None = None,+) -> list[dict[str, Any]]:+ """Load bounded, contained with-skill rewards for one agent."""+ agents = loaded_agents if loaded_agents is not None else report_data.load_agent_data(results_dir)+ agent_data = agents.get(agent)+ rewards = agent_data.get("rewards") if isinstance(agent_data, dict) else None+ return [reward for reward in rewards if isinstance(reward, dict)] if isinstance(rewards, list) else []def _pick_best_agent(@@ -59,6 +85,8 @@ def _pick_best_agent(best_agent = ""best_score = -1.0for agent, data in agents_data.items():+ if not _findings_eligible(data):+ continuewith_scores = data.get("with_skill", {})if not with_scores:continue@@ -70,6 +98,15 @@ def _pick_best_agent(return best_agent+def _findings_eligible(agent_data: dict[str, Any]) -> bool:+ """Return whether persisted execution truth permits quality findings."""+ conditions = agent_data.get("conditions")+ if not isinstance(conditions, dict) or "with_skill" not in conditions:+ return agent_data.get("execution_status") == "succeeded"+ with_skill = conditions.get("with_skill")+ return isinstance(with_skill, dict) and with_skill.get("execution_status") == "succeeded"++def _agent_model_for_display(agent: str,harbor_result: dict[str, Any],@@ -121,11 +158,12 @@ def _finding_metric_names(rewards: list[dict[str, Any]]) -> list[str]:return list(DISPLAY_METRICS) + sorted(custom_names.difference(DISPLAY_METRICS))-def _metric_score(reward: dict[str, Any], metric: str) -> float:+def _metric_score(reward: dict[str, Any], metric: str) -> float | None:value = reward.get(metric)if isinstance(value, int | float) and not isinstance(value, bool):- return float(value)- return extract_custom_metrics(reward).get(metric, 0.0)+ numeric = float(value)+ return numeric if math.isfinite(numeric) else None+ return extract_custom_metrics(reward).get(metric)def _metric_label(metric: str) -> str:@@ -137,30 +175,50 @@ def _metric_label(metric: str) -> str:)-def _extract_findings(rewards: list[dict[str, Any]]) -> list[dict[str, Any]]:+def _extract_findings(+ rewards: list[dict[str, Any]],+ *,+ canonical_scores: dict[str, Any] | None = None,+ rewards_complete: bool = True,+) -> list[dict[str, Any]]:"""Extract actionable findings from reward.json details across trials."""findings: list[dict[str, Any]] = []metric_names = _finding_metric_names(rewards)- aggregated: dict[str, list[dict[str, Any]]] = {m: [] for m in metric_names}- for reward in rewards:- details = _details_for_findings(reward)+ aggregated: dict[str, list[list[dict[str, Any]]]] = {m: [] for m in metric_names}+ logical_scores: dict[str, list[float]] = {m: [] for m in metric_names}+ for reward_group in report_data.logical_trial_reward_groups(rewards):for metric in metric_names:- if metric in details:- aggregated[metric].append(- {- "score": _metric_score(reward, metric),- "detail": details[metric],- "entry_id": reward.get("entry_id", "?"),- }- )+ metric_values = [score for reward in reward_group if (score := _metric_score(reward, metric)) is not None]+ trial_details = []+ for reward in reward_group:+ details = _details_for_findings(reward)+ if metric in details:+ trial_details.append(+ {+ "score": _metric_score(reward, metric),+ "detail": details[metric],+ "entry_id": reward.get("entry_id", "?"),+ }+ )+ if metric_values:+ logical_scores[metric].append(round(sum(metric_values) / len(metric_values), 4))+ if trial_details:+ aggregated[metric].append(trial_details)for metric in metric_names:- trials = aggregated[metric]- if not trials:+ trial_groups = aggregated[metric]+ if not trial_groups:continue- avg_score = sum(t["score"] for t in trials) / len(trials)+ trials = [trial for trial_group in trial_groups for trial in trial_group]+ canonical_score = _metric_score(canonical_scores or {}, metric)+ if canonical_score is not None:+ avg_score = canonical_score+ elif rewards_complete and logical_scores[metric]:+ avg_score = round(sum(logical_scores[metric]) / len(logical_scores[metric]), 4)+ else:+ continuelabel = _metric_label(metric)metric_refs = []@@ -723,7 +781,9 @@ def _write_findings_artifact(suggestion_mode: str,suggestions_v2: list[dict[str, Any]] | None = None,) -> Path | None:- artifact = results_dir / agent / "findings.json"+ artifact = _findings_artifact_path(results_dir, agent)+ if artifact is None:+ return Nonepayload = {"skill_name": skill_name,"agent": agent,@@ -759,6 +819,9 @@ def display_findings_report(console = Console()agents_data = harbor_result.get("agents", {})+ report_agents = list(dict.fromkeys([*harbor_agents, *agents_data.keys()]))+ for agent in report_agents:+ _remove_stale_findings_artifact(results_dir, agent)if len(harbor_agents) > 1:best_agent = _pick_best_agent(agents_data)@@ -772,18 +835,37 @@ def display_findings_report(else:best_agent = harbor_agents[0] if harbor_agents else ""- if not best_agent or best_agent not in agents_data:+ if (+ not best_agent+ or best_agent not in agents_data+ or not isinstance(agents_data[best_agent], dict)+ or not _findings_eligible(agents_data[best_agent])+ ):return set()+ loaded_agents = report_data.load_agent_data(results_dir)agent_reports: dict[str, tuple[list[dict[str, Any]], list[dict[str, Any]]]] = {}- report_agents = list(dict.fromkeys([*harbor_agents, *agents_data.keys()]))for agent in report_agents:- if agent not in agents_data:+ agent_data = agents_data.get(agent)+ if not isinstance(agent_data, dict) or not _findings_eligible(agent_data):continue- rewards_for_agent = _load_trial_rewards(results_dir, agent)+ rewards_for_agent = _load_trial_rewards(results_dir, agent, loaded_agents)if not rewards_for_agent:continue- findings_for_agent = _extract_findings(rewards_for_agent)+ loaded_agent = loaded_agents.get(agent)+ canonical_scores: dict[str, Any] = {}+ rewards_complete = True+ if isinstance(loaded_agent, dict):+ for score_key in ("with_skill", "custom_with_skill"):+ scores = loaded_agent.get(score_key)+ if isinstance(scores, dict):+ canonical_scores.update(scores)+ rewards_complete = loaded_agent.get("rewards_complete") is not False+ findings_for_agent = _extract_findings(+ rewards_for_agent,+ canonical_scores=canonical_scores,+ rewards_complete=rewards_complete,+ )if findings_for_agent:agent_reports[agent] = (findings_for_agent, rewards_for_agent)diff --git a/src/skillevaluator/tier3/harbor/report_data.py b/src/skillevaluator/tier3/harbor/report_data.pyindex dcb7e143..31316322 100644--- a/src/skillevaluator/tier3/harbor/report_data.py+++ b/src/skillevaluator/tier3/harbor/report_data.py@@ -27,6 +27,7 @@ logger = logging.getLogger(__name__)_MAX_JSON_BYTES = 2 * 1024 * 1024_MAX_JSON_DEPTH = 64_MAX_JSON_NODES = 50_000+_MAX_JSON_NUMBER_CHARS = 4_300_MAX_AGENTS = 64_MAX_AGENT_PATHS_SCANNED = 512_MAX_TRIALS_PER_CONDITION = 512@@ -45,6 +46,7 @@ __all__ = ("load_dataset","load_dataset_snapshot","load_staged_harbor_dataset",+ "logical_trial_reward_groups","metrics_for_agents","summarize_dataset_entries",)@@ -204,6 +206,17 @@ def _bounded_smallest(return selected[:limit], len(selected) > limit, scanned > scan_limit+def _is_safe_directory(path: Path, root: Path) -> bool:+ """Return whether path is a real directory contained by the report root."""+ try:+ if not stat.S_ISDIR(path.lstat().st_mode):+ return False+ path.resolve().relative_to(root.resolve())+ except (OSError, RuntimeError, ValueError):+ return False+ return True++def _bounded_staged_entry_files(tasks_dir: Path) -> tuple[list[Path], bool, bool]:"""Find staged entry files without allowing ``rglob`` to hide unbounded visits."""pending = [tasks_dir]@@ -260,6 +273,8 @@ def _read_bounded_bytes(*,artifact: str,) -> bytes | None:+ if path.is_symlink():+ return Noneflags = os.O_RDONLY | getattr(os, "O_BINARY", 0) | getattr(os, "O_NONBLOCK", 0)flags |= getattr(os, "O_NOFOLLOW", 0)try:@@ -328,6 +343,16 @@ def _decode_bounded_json(if strict_syntax:raisereturn _INVALID_JSON+ except ValueError:+ if strict_syntax:+ raise+ _record_truncation(+ diagnostics,+ code="json_number",+ artifact=artifact,+ limit=_MAX_JSON_NUMBER_CHARS,+ )+ return _INVALID_JSONexcept UnicodeDecodeError:if strict_syntax:raise@@ -419,6 +444,22 @@ def metrics_for_agents(agents: dict[str, dict[str, Any]]) -> list[str]:return list(LEGACY_METRICS) if saw_metrics else []+def logical_trial_reward_groups(rewards: list[dict[str, Any]]) -> list[list[dict[str, Any]]]:+ """Group persisted reward rows by collector-issued logical trial identity.++ Older or independently produced artifacts do not carry the opaque identity;+ those rows remain singletons so report loading never guesses trial structure+ from case IDs or ambiguous output-directory names.+ """+ groups: dict[str, list[dict[str, Any]]] = {}+ for index, reward in enumerate(rewards):+ identity = reward.get("trial_id")+ valid_identity = isinstance(identity, str) and 0 < len(identity) <= 512+ key = f"trial:{identity}" if valid_identity else f"row:{index}"+ groups.setdefault(key, []).append(reward)+ return list(groups.values())++def _nonnegative_counter(value: Any) -> int:return value if isinstance(value, int) and not isinstance(value, bool) and value >= 0 else 0@@ -442,7 +483,7 @@ def load_agent_data(results_dir.iterdir(),_MAX_AGENTS,scan_limit=_MAX_AGENT_PATHS_SCANNED,- predicate=lambda path: path.is_dir() and not path.name.startswith("_"),+ predicate=lambda path: _is_safe_directory(path, results_dir) and not path.name.startswith("_"),)except OSError:return agents@@ -468,17 +509,26 @@ def load_agent_data(condition_execution: dict[str, dict[str, Any]] = {}for variant in ("with-skill", "without-skill"):- summary = agent_dir / variant / "summary.json"+ condition_dir = agent_dir / variant+ if not _is_safe_directory(condition_dir, results_dir):+ continue+ summary = condition_dir / "summary.json"if summary.exists():data = _load_bounded_json(summary, agent_diagnostics, artifact="summary")if isinstance(data, dict):+ scores = data.get("scores")+ if not isinstance(scores, dict):+ continuekey = "with_skill" if variant == "with-skill" else "without_skill"- agent_info[key] = data.get("scores", data)+ agent_info[key] = scoresmetric_key = "metrics_with_skill" if variant == "with-skill" else "metrics_without_skill"agent_info[metric_key] = data.get("metrics", [])custom_key = "custom_with_skill" if variant == "with-skill" else "custom_without_skill"if "custom_scores" in data:agent_info[custom_key] = data.get("custom_scores", {})+ overall_key = "overall_with_skill" if variant == "with-skill" else "overall_without_skill"+ if "overall_score" in data:+ agent_info[overall_key] = data.get("overall_score")dimension_key = "dimensions_with_skill" if variant == "with-skill" else "dimensions_without_skill"if "dimensions" in data:agent_info[dimension_key] = data.get("dimensions", {})@@ -510,8 +560,10 @@ def load_agent_data("expected_attempts": _nonnegative_counter(data.get("expected_attempts")),"scored_attempts": _nonnegative_counter(data.get("scored_attempts")),}- if variant == "with-skill":- agent_info["num_trials"] = data.get("num_trials", 0)+ count_key = "num_trials" if variant == "with-skill" else "num_trials_baseline"+ num_trials = data.get("num_trials")+ if isinstance(num_trials, int) and not isinstance(num_trials, bool) and num_trials >= 0:+ agent_info[count_key] = num_trialslift_file = agent_dir / "lift.json"if lift_file.exists():@@ -533,17 +585,23 @@ def load_agent_data(for variant_key, variant_dir_name in (("rewards", "with-skill"), ("rewards_baseline", "without-skill")):trial_list: list[dict[str, Any]] = []+ count_key = "num_trials" if variant_key == "rewards" else "num_trials_baseline"+ expected_reward_rows = agent_info.get(count_key)+ rewards_complete = isinstance(expected_reward_rows, int)trials_dir = agent_dir / variant_dir_name / "trials"- if trials_dir.exists():+ if _is_safe_directory(trials_dir, results_dir):try:trial_dirs, trials_truncated, trial_scan_truncated = _bounded_smallest(trials_dir.iterdir(),_MAX_TRIALS_PER_CONDITION,scan_limit=_MAX_TRIAL_PATHS_SCANNED,- predicate=lambda path: path.is_dir(),+ predicate=lambda path: _is_safe_directory(path, results_dir),)except OSError:trial_dirs, trials_truncated, trial_scan_truncated = [], False, False+ rewards_complete = False+ if trials_truncated or trial_scan_truncated:+ rewards_complete = Falseif trials_truncated:_record_truncation(agent_diagnostics,@@ -561,9 +619,11 @@ def load_agent_data(for trial_dir in trial_dirs:reward_file = trial_dir / "reward.json"if not reward_file.exists():+ rewards_complete = Falsecontinuereward = _load_bounded_json(reward_file, agent_diagnostics, artifact="reward")if not isinstance(reward, dict):+ rewards_complete = Falsecontinueif not reward.get("entry_id"):reward["entry_id"] = trial_dir.name.split("__", 1)[0] if trial_dir.name else "unknown"@@ -586,7 +646,12 @@ def load_agent_data("cached_tokens": final_metrics.get("total_cached_tokens", 0),}trial_list.append(reward)+ else:+ rewards_complete = expected_reward_rows == 0+ if expected_reward_rows != len(trial_list):+ rewards_complete = Falseagent_info[variant_key] = trial_list+ agent_info[f"{variant_key}_complete"] = rewards_completeif "with_skill" not in agent_info:continue@@ -621,6 +686,7 @@ def load_agent_data("with_skill": ("with_skill","custom_with_skill",+ "overall_with_skill","dimensions_with_skill","pass_with_skill","rewards",@@ -628,6 +694,7 @@ def load_agent_data("without_skill": ("without_skill","custom_without_skill",+ "overall_without_skill","dimensions_without_skill","pass_without_skill","rewards_baseline",@@ -644,6 +711,8 @@ def load_agent_data("attempts_used": _nonnegative_counter(condition_info.get("scored_attempts")),"max_attempts_possible": _nonnegative_counter(condition_info.get("expected_attempts")),}+ elif field.startswith("overall_"):+ agent_info[field] = Noneelse:agent_info[field] = [] if field.startswith("rewards") else {}_attach_truncation(agent_info, agent_diagnostics)diff --git a/src/skillevaluator/tier3/harbor/runner.py b/src/skillevaluator/tier3/harbor/runner.pyindex 68cb2a6c..6d5cf363 100644--- a/src/skillevaluator/tier3/harbor/runner.py+++ b/src/skillevaluator/tier3/harbor/runner.py@@ -30,7 +30,12 @@ from uuid import uuid4from skillevaluator import __version__from skillevaluator.evaluation.tier3_report import render_agent_eval_html_report-from skillevaluator.provider_config import ProviderConfig, ProviderConfigurationError, resolve_llm_provider+from skillevaluator.provider_config import (+ ProviderConfig,+ ProviderConfigurationError,+ _normalize_anthropic_base_url,+ resolve_llm_provider,+)from skillevaluator.tier3.evals_config import EvalsConfigError, load_evals_configfrom skillevaluator.tier3.harbor.adapter import (_VERIFIER_JUDGE_MODEL_ENV_VARS,@@ -832,6 +837,23 @@ def _harbor_subprocess_environment(return environment+def _independent_anthropic_agent_credentials() -> dict[str, str]:+ """Resolve and validate a host-owned Anthropic credential pair."""+ credentials = {+ name: os.environ.get(name, "") for name in ("ANTHROPIC_API_KEY", "ANTHROPIC_BASE_URL") if os.environ.get(name)+ }+ if base_url := credentials.get("ANTHROPIC_BASE_URL"):+ normalized_base_url = _normalize_anthropic_base_url(+ base_url,+ variable="ANTHROPIC_BASE_URL",+ )+ if normalized_base_url is None:+ credentials.pop("ANTHROPIC_BASE_URL")+ else:+ credentials["ANTHROPIC_BASE_URL"] = normalized_base_url+ return credentials++def _judge_model_config(provider: ProviderConfig,provider_env: Mapping[str, str],@@ -909,11 +931,7 @@ def _agent_credentials(# sentinel and must not inherit NVIDIA_API_KEY in task env.return {}if agent == "claude-code":- return {- name: os.environ.get(name, "")- for name in ("ANTHROPIC_API_KEY", "ANTHROPIC_BASE_URL")- if os.environ.get(name)- }+ return _independent_anthropic_agent_credentials()if agent == "codex":return {name: os.environ.get(name, "") for name in ("OPENAI_API_KEY", "OPENAI_BASE_URL") if os.environ.get(name)@@ -921,11 +939,7 @@ def _agent_credentials(return {}if provider.provider in {"openai", "openai-compatible"} and agent == "claude-code":- return {- name: os.environ.get(name, "")- for name in ("ANTHROPIC_API_KEY", "ANTHROPIC_BASE_URL")- if os.environ.get(name)- }+ return _independent_anthropic_agent_credentials()if provider.provider == "anthropic" and agent == "codex":return {name: os.environ.get(name, "") for name in ("OPENAI_API_KEY", "OPENAI_BASE_URL") if os.environ.get(name)diff --git a/src/skillevaluator/tier3/harbor/templates/custom_grader_runner.py b/src/skillevaluator/tier3/harbor/templates/custom_grader_runner.pyindex 8d5f43d7..c5fb8005 100644--- a/src/skillevaluator/tier3/harbor/templates/custom_grader_runner.py+++ b/src/skillevaluator/tier3/harbor/templates/custom_grader_runner.py@@ -41,6 +41,8 @@ RESERVED = {"behavior_check","details","entry_id",+ "evaluation_errors",+ "evaluation_status","has_skill","metric_set","metrics",diff --git a/src/skillevaluator/tier3/harbor/templates/eval.py b/src/skillevaluator/tier3/harbor/templates/eval.pyindex 837d86ed..fa5ceafb 100644--- a/src/skillevaluator/tier3/harbor/templates/eval.py+++ b/src/skillevaluator/tier3/harbor/templates/eval.py@@ -22,17 +22,22 @@ RAGAS is used for goal_accuracy and accuracy when available.from __future__ import annotations+import ipaddressimport jsonimport logging+import mathimport osimport reimport shleximport sys+import unicodedataimport urllib.errorimport urllib.requestfrom pathlib import Pathfrom typing import Any-from urllib.parse import urlparse+from urllib.parse import quote, unquote_to_bytes, urlparse, urlsplit++import idna_SCRIPT_TESTS_DIR = Path(__file__).resolve().parentif str(_SCRIPT_TESTS_DIR) not in sys.path:@@ -80,8 +85,16 @@ NVIDIA_BUILD_CHAT_URL = "https://integrate.api.nvidia.com/v1/chat/completions"# Keep in sync with skillevaluator.provider_config.CHAT_DEFAULT_OPENAI# (sandbox template cannot import the package — see drift test).DEFAULT_JUDGE_MODEL = "gpt-5.6-sol"+_ANTHROPIC_DNS_LABEL_RE = re.compile(r"^[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?$")+_ANTHROPIC_INTERNAL_LABEL_RE = re.compile(r"^[A-Za-z0-9_](?:[A-Za-z0-9_-]{0,61}[A-Za-z0-9_])?$")+_ANTHROPIC_IPV6_ZONE_RE = re.compile(r"^[A-Za-z0-9._~-]+$")+_ANTHROPIC_PATH_SAFE = "/:@!$&'()*+,;=-._~%"+_HEX_DIGITS = frozenset("0123456789abcdefABCDEF")+_HEX_DIGIT_BYTES = frozenset(b"0123456789abcdefABCDEF")+_UNRESERVED_BYTES = frozenset(b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~")_ERROR_REDACTION_MARKER = "[REDACTED]"+_JUDGE_ERROR_REASON_LIMIT = 512# Shorter placeholders are not credible provider credentials and can corrupt report schema keys._MIN_EXACT_SECRET_LENGTH = 8_CREDENTIAL_ENV_VARS = (@@ -93,6 +106,8 @@ _CREDENTIAL_ENV_VARS = ("AWS_SECRET_ACCESS_KEY","AWS_SECURITY_TOKEN","AWS_SESSION_TOKEN",+ "AWS_BEARER_TOKEN_BEDROCK",+ "AWS_CONTAINER_AUTHORIZATION_TOKEN",)WASTE_INDICATORS = [@@ -1061,6 +1076,27 @@ def _redact_configured_credentials(text, extra_secret_values=()):return redacted+def _judge_error(error_reason, **metadata):+ """Return a bounded, redacted result that cannot be mistaken for a judged zero."""+ safe_reason = _redact_configured_credentials(error_reason).strip() or "LLM judge failed"+ if len(safe_reason) > _JUDGE_ERROR_REASON_LIMIT:+ safe_reason = safe_reason[: _JUDGE_ERROR_REASON_LIMIT - 3] + "..."+ return {**metadata, "score": None, "status": "error", "reason": safe_reason}+++def _finite_score(value):+ if isinstance(value, bool) or not isinstance(value, (int, float)):+ return None+ if isinstance(value, int):+ if value <= 0:+ return 0.0+ return 1.0+ score = float(value)+ if not math.isfinite(score):+ return None+ return max(0.0, min(1.0, score))++def _sanitize_error_value(value, extra_secret_values=()):secrets = _configured_secret_values(extra_secret_values)@@ -1210,9 +1246,158 @@ def _public_provider_error():return "Configure SKILL_EVAL_LLM_PROVIDER and a public provider credential"+def _canonical_anthropic_authority(netloc, hostname):+ if netloc.startswith("["):+ closing_bracket = netloc.find("]")+ if closing_bracket < 0:+ return None+ literal = netloc[1:closing_bracket]+ suffix = netloc[closing_bracket + 1 :]+ if literal.casefold() != hostname.casefold() or (+ suffix and (not suffix.startswith(":") or not suffix[1:].isascii() or not suffix[1:].isdigit())+ ):+ return None++ address = literal+ zone = ""+ if "%" in literal:+ address, separator, zone = literal.partition("%25")+ if not separator or "%" in address or "%" in zone or not _ANTHROPIC_IPV6_ZONE_RE.fullmatch(zone):+ return None+ try:+ ipaddress.IPv6Address(address)+ except ValueError:+ return None+ return f"[{address}{'%25' + zone if zone else ''}]{suffix}"++ if "%" in netloc or "[" in netloc or "]" in netloc:+ return None+ host = netloc+ suffix = ""+ if ":" in netloc:+ host, port = netloc.rsplit(":", maxsplit=1)+ if ":" in host or not port.isascii() or not port.isdigit():+ return None+ suffix = f":{port}"+ if host.casefold() != hostname.casefold():+ return None++ if "." in hostname and all(character in "0123456789." for character in hostname):+ try:+ ipaddress.IPv4Address(hostname)+ except ValueError:+ return None+ return f"{host}{suffix}"++ trailing_dot = host.endswith(".")+ dns_name = host.removesuffix(".")+ if not dns_name:+ return None+ if dns_name.isascii() and "_" in dns_name:+ canonical_name = dns_name.lower()+ label_pattern = _ANTHROPIC_INTERNAL_LABEL_RE+ else:+ try:+ canonical_name = idna.encode(dns_name.lower()).decode("ascii")+ except idna.IDNAError:+ return None+ label_pattern = _ANTHROPIC_DNS_LABEL_RE+ if len(canonical_name) > 253 or not all(label_pattern.fullmatch(label) for label in canonical_name.split(".")):+ return None+ return f"{canonical_name}{'.' if trailing_dot else ''}{suffix}"+++def _canonical_anthropic_path(path):+ canonical = []+ index = 0+ while index < len(path):+ character = path[index]+ if character != "%":+ canonical.append(character)+ index += 1+ continue++ if index + 2 >= len(path) or path[index + 1] not in _HEX_DIGITS or path[index + 2] not in _HEX_DIGITS:+ return None+ octet = int(path[index + 1 : index + 3], 16)+ if octet in {0x2F, 0x5C, 0x7F} or octet < 0x20:+ return None+ if octet in _UNRESERVED_BYTES:+ canonical.append(chr(octet))+ else:+ canonical.append(f"%{octet:02X}")+ index += 3++ canonical_path = "".join(canonical)+ if "//" in canonical_path.rstrip("/"):+ return None+ decoded_octets = unquote_to_bytes(canonical_path)+ # A decoded percent is safe as data unless it opens a second escape layer.+ if any(+ decoded_octets[index] == 0x25+ and index + 2 < len(decoded_octets)+ and decoded_octets[index + 1] in _HEX_DIGIT_BYTES+ and decoded_octets[index + 2] in _HEX_DIGIT_BYTES+ for index in range(len(decoded_octets))+ ):+ return None+ decoded_path = decoded_octets.decode("utf-8", errors="replace")+ if any(unicodedata.category(character) in {"Cc", "Cf", "Cs"} for character in decoded_path):+ return None+ if any(segment in {".", ".."} for segment in decoded_path.split("/")):+ return None+ return quote(canonical_path, safe=_ANTHROPIC_PATH_SAFE)+++def _normalize_anthropic_base_url(value, variable):+ error = (+ f"{variable} must be an absolute HTTP or HTTPS URL representing an API root without credentials, query, fragment, "+ "whitespace, control characters, backslashes, an invalid authority, or a /v1/messages endpoint."+ )+ if "\\" in value or any(+ character.isspace() or unicodedata.category(character) in {"Cc", "Cf", "Cs"} for character in value+ ):+ raise ValueError(error)+ if "?" in value or "#" in value:+ raise ValueError(error)++ try:+ parsed = urlsplit(value)+ hostname = parsed.hostname+ _ = parsed.port+ except ValueError:+ raise ValueError(error) from None+ if (+ parsed.scheme.lower() not in {"http", "https"}+ or not parsed.netloc+ or hostname is None+ or parsed.netloc.endswith(":")+ or parsed.username is not None+ or parsed.password is not None+ ):+ raise ValueError(error)++ authority = _canonical_anthropic_authority(parsed.netloc, hostname)+ path = _canonical_anthropic_path(parsed.path)+ if authority is None or path is None:+ raise ValueError(error)++ path = path.rstrip("/")+ if path.endswith("/v1/messages"):+ raise ValueError(error)+ if path.endswith("/v1"):+ path = path.removesuffix("/v1")+ return parsed._replace(netloc=authority, path=path, query="", fragment="").geturl()++def _anthropic_url():- base_url = os.environ.get("SKILL_EVAL_LLM_BASE_URL") or os.environ.get("ANTHROPIC_BASE_URL")- url = base_url.rstrip("/") + "/messages" if base_url else "https://api.anthropic.com/v1/messages"+ for variable in ("SKILL_EVAL_LLM_BASE_URL", "ANTHROPIC_BASE_URL"):+ if base_url := os.environ.get(variable):+ root = _normalize_anthropic_base_url(base_url, variable)+ url = root + "/v1/messages"+ break+ else:+ url = "https://api.anthropic.com/v1/messages"return _validate_http_url(url)@@ -1289,18 +1474,21 @@ def _call_public_llm_with_provenance(prompt, model=None, max_tokens=1024, temperrequested_model = _selected_judge_model(model)models = _fallback_models(requested_model) if allow_model_fallback else [requested_model]errors = []+ last_provenance = {"provider": provider, "model": requested_model}for candidate_model in models:+ provenance = {"provider": provider, "model": candidate_model}+ last_provenance = provenancetry:if provider == "anthropic":content, error = _call_anthropic(prompt, candidate_model, max_tokens, temperature)if error:- return None, _redact_configured_credentials(error), {}- return content, None, {"provider": provider, "model": candidate_model}+ return None, _redact_configured_credentials(error), provenance+ return content, None, provenanceif provider == "bedrock":content, error = _call_bedrock(prompt, candidate_model, max_tokens, temperature)if error:- return None, _redact_configured_credentials(error), {}- return content, None, {"provider": provider, "model": candidate_model}+ return None, _redact_configured_credentials(error), provenance+ return content, None, provenanceapi_key = (os.environ.get("NVIDIA_API_KEY", "")@@ -1308,7 +1496,7 @@ def _call_public_llm_with_provenance(prompt, model=None, max_tokens=1024, temperelse os.environ.get("SKILL_EVAL_LLM_API_KEY", "") or os.environ.get("OPENAI_API_KEY", ""))if not api_key:- return None, f"No API key configured for {provider}", {}+ return None, f"No API key configured for {provider}", provenancerequest_url = _resolve_url(provider)request = urllib.request.Request(request_url,@@ -1330,17 +1518,17 @@ def _call_public_llm_with_provenance(prompt, model=None, max_tokens=1024, tempercontent = body.get("choices", [{}])[0].get("message", {}).get("content", "")if candidate_model != requested_model:logger.warning("LLM judge model %s failed; using fallback model %s", requested_model, candidate_model)- return content.strip(), None, {"provider": provider, "model": candidate_model}+ return content.strip(), None, provenanceexcept urllib.error.HTTPError as error:detail, should_try_fallback = _format_http_error_with_fallback(error)errors.append(f"{candidate_model}: {detail}")if not allow_model_fallback or not should_try_fallback:- return None, detail, {}+ return None, detail, provenanceexcept Exception as exc:detail = f"Public provider call failed for {candidate_model}: {exc}"- return None, _redact_configured_credentials(detail), {}+ return None, _redact_configured_credentials(detail), provenancedetail = "LLM judge model fallback exhausted: " + " | ".join(errors)- return None, _redact_configured_credentials(detail), {}+ return None, _redact_configured_credentials(detail), last_provenancedef call_public_llm(prompt, model=None, max_tokens=1024, temperature=0.0, allow_model_fallback=True):@@ -1354,15 +1542,17 @@ def call_public_llm(prompt, model=None, max_tokens=1024, temperature=0.0, allow_return content, error-_JSON_FENCE_RE = re.compile(r"```(?:json)?\s*\n?(.*?)```", re.DOTALL | re.IGNORECASE)+_JSON_WHITESPACE = " \t\r\n"+_MAX_JSON_TEXT_CHARS = 100_000+_MAX_JSON_NESTING = 128+_JSON_NUMBER_PREFIX_RE = re.compile(r"-?(?:(?:0|[1-9][0-9]*)(?:\.[0-9]+)?(?:[eE][+-]?[0-9]*)?|(?:0|[1-9][0-9]*)\.)?")-def _find_balanced_json(text):- """Return the first balanced ``{...}`` block, honoring strings and escapes."""- start = text.find("{")- if start == -1:+def _balanced_json_container_end(text, start):+ """Return the exclusive end of one bounded structural container."""+ if start >= len(text) or text[start] not in "{[":return None- depth = 0+ stack = []in_string = Falseescaped = Falsefor i in range(start, len(text)):@@ -1376,49 +1566,202 @@ def _find_balanced_json(text):in_string = Falseelif ch == '"':in_string = True- elif ch == "{":+ elif ch in "{[":+ stack.append(ch)+ if len(stack) > _MAX_JSON_NESTING:+ return None+ elif ch in "}]":+ expected = "{" if ch == "}" else "["+ if not stack or stack[-1] != expected:+ return None+ stack.pop()+ if not stack:+ return i + 1+ return None+++def _reject_duplicate_object_pairs(pairs):+ """Build an object while rejecting ambiguous duplicate members."""+ result = {}+ for key, value in pairs:+ if key in result:+ raise ValueError("Duplicate JSON object member")+ result[key] = value+ return result+++def _reject_nonstandard_json_constant(_value):+ raise ValueError("Non-standard JSON constant")+++def _json_nesting_within_limit(text):+ """Bound structural nesting without recursively parsing partial JSON."""+ depth = 0+ in_string = False+ escaped = False+ for character in text:+ if in_string:+ if escaped:+ escaped = False+ elif character == "\\":+ escaped = True+ elif character == '"':+ in_string = False+ elif character == '"':+ in_string = True+ elif character in "{[":depth += 1- elif ch == "}":+ if depth > _MAX_JSON_NESTING:+ return False+ elif character in "}]" and depth:depth -= 1- if depth == 0:- return text[start : i + 1]- return None+ return Truedef extract_json(text):"""Extract a JSON payload from LLM response text.- Tolerates markdown fences (```json anywhere in the text), leading or- trailing prose -- including prose that itself contains braces -- via- first-balanced-brace extraction. Top-level JSON arrays parse through- unchanged; judge callers must dict-check the result themselves.+ Tolerates markdown fences and prose around exactly one valid bounded JSON+ container. Multiple complete documents are ambiguous, and an unfinished+ earlier structural segment blocks promotion of a nested object. Top-level+ arrays parse through unchanged; judge callers must dict-check the result+ themselves."""text = (text or "").strip()- if not text:+ if not text or len(text) > _MAX_JSON_TEXT_CHARS:return None- candidates = [text]- if text.startswith("```"):- candidates.append(text.split("\n", 1)[-1].rsplit("```", 1)[0].strip())- fence = _JSON_FENCE_RE.search(text)- if fence:- candidates.append(fence.group(1).strip())- start = text.find("{")- end = text.rfind("}")- if start != -1 and end > start:- candidates.append(text[start : end + 1])- balanced = _find_balanced_json(text)- if balanced:- candidates.append(balanced)- for candidate in candidates:- if not candidate:++ documents = []+ index = 0+ while index < len(text):+ if text[index] not in "{[":+ index += 1continue+ end = _balanced_json_container_end(text, index)+ if end is None:+ return documents[0] if documents else None+ candidate = text[index:end]try:- parsed = json.loads(candidate)- except (json.JSONDecodeError, ValueError):+ parsed = json.loads(+ candidate,+ object_pairs_hook=_reject_duplicate_object_pairs,+ parse_constant=_reject_nonstandard_json_constant,+ )+ except (json.JSONDecodeError, RecursionError, ValueError):+ index = endcontinueif isinstance(parsed, (dict, list)):- return parsed- return None+ documents.append(parsed)+ if len(documents) > 1:+ return None+ index = end+ return documents[0] if documents else None+++def _is_json_string_prefix(text):+ """Return whether an unfinished bounded string can be completed as JSON."""+ if not text.startswith('"'):+ return False+ index = 1+ while index < len(text):+ character = text[index]+ if ord(character) < 0x20 or character == '"':+ return False+ if character != "\\":+ index += 1+ continue+ index += 1+ if index >= len(text):+ return True+ escape = text[index]+ if escape == "u":+ for offset in range(1, 5):+ if index + offset >= len(text):+ return True+ if text[index + offset] not in "0123456789abcdefABCDEF":+ return False+ index += 5+ elif escape in '"\\/bfnrt':+ index += 1+ else:+ return False+ return True+++def _is_json_scalar_prefix(text):+ if not text:+ return True+ if text.startswith('"'):+ return _is_json_string_prefix(text)+ literals = {"t": "true", "f": "false", "n": "null"}+ if text[0] in literals:+ return literals[text[0]].startswith(text)+ if text[0] == "-" or text[0] in "0123456789":+ return _JSON_NUMBER_PREFIX_RE.fullmatch(text) is not None+ return False+++def _is_append_only_json_object_prefix(fragment):+ """Validate an unfinished flat result entry using bounded decoder steps."""+ if not fragment or len(fragment) > _MAX_JSON_TEXT_CHARS:+ return False++ decoder = json.JSONDecoder(+ object_pairs_hook=_reject_duplicate_object_pairs,+ parse_constant=_reject_nonstandard_json_constant,+ )++ def _skip_whitespace(index):+ while index < len(fragment) and fragment[index] in _JSON_WHITESPACE:+ index += 1+ return index++ index = _skip_whitespace(0)+ if index >= len(fragment) or fragment[index] != "{":+ return False+ index += 1+ keys = set()+ while True:+ index = _skip_whitespace(index)+ if index >= len(fragment):+ return True+ if fragment[index] == "}":+ return False+ try:+ key, next_index = decoder.raw_decode(fragment, index)+ except (json.JSONDecodeError, RecursionError, ValueError):+ return _is_json_string_prefix(fragment[index:])+ if not isinstance(key, str) or key in keys:+ return False+ keys.add(key)+ index = _skip_whitespace(next_index)+ if index >= len(fragment):+ return True+ if fragment[index] != ":":+ return False+ index = _skip_whitespace(index + 1)+ if index >= len(fragment):+ return True+ value_start = index+ try:+ value, next_index = decoder.raw_decode(fragment, index)+ except (json.JSONDecodeError, RecursionError, ValueError):+ return _is_json_scalar_prefix(fragment[value_start:])+ if (+ isinstance(value, (int, float))+ and not isinstance(value, bool)+ and next_index < len(fragment)+ and fragment[next_index] in ".eE"+ ):+ return _JSON_NUMBER_PREFIX_RE.fullmatch(fragment[value_start:]) is not None+ index = _skip_whitespace(next_index)+ if index >= len(fragment):+ return True+ if fragment[index] == "}":+ return False+ if fragment[index] != ",":+ return False+ index += 1def _salvage_behavior_results(text):@@ -1429,32 +1772,94 @@ def _salvage_behavior_results(text):entry before the cut is still valid JSON and can be scored."""text = text or ""- marker = text.find('"results"')- if marker == -1:+ if len(text) > _MAX_JSON_TEXT_CHARS or not _json_nesting_within_limit(text):return []- array_start = text.find("[", marker)- if array_start == -1:+ object_start = text.find("{")+ if object_start == -1:return []- results = []- i = array_start + 1+ if any(character in "[]{}" for character in text[:object_start]):+ return []++ decoder = json.JSONDecoder(+ object_pairs_hook=_reject_duplicate_object_pairs,+ parse_constant=_reject_nonstandard_json_constant,+ )++ def _skip_whitespace(index):+ while index < len(text) and text[index] in " \t\r\n":+ index += 1+ return index++ # Parse only complete top-level fields preceding ``results``. This rejects+ # nested/unrelated arrays and lets us validate a score emitted before the+ # array without requiring the outer object itself to be complete.+ i = object_start + 1+ array_start = None+ seen_keys = set()while i < len(text):- ch = text[i]- if ch == "{":- block = _find_balanced_json(text[i:])- if not block:- break- try:- entry = json.loads(block)- except (json.JSONDecodeError, ValueError):- break- if isinstance(entry, dict):- results.append(entry)- i += len(block)- elif ch == "]":+ i = _skip_whitespace(i)+ try:+ key, i = decoder.raw_decode(text, i)+ except (json.JSONDecodeError, RecursionError, ValueError):+ return []+ if not isinstance(key, str):+ return []+ if key in seen_keys:+ return []+ seen_keys.add(key)+ i = _skip_whitespace(i)+ if i >= len(text) or text[i] != ":":+ return []+ i = _skip_whitespace(i + 1)+ if key == "results":+ if i >= len(text) or text[i] != "[":+ return []+ array_start = ibreak- else:- i += 1- return results+ try:+ value, i = decoder.raw_decode(text, i)+ except (json.JSONDecodeError, RecursionError, ValueError):+ return []+ if key == "score" and _finite_score(value) is None:+ return []+ i = _skip_whitespace(i)+ if i >= len(text) or text[i] != ",":+ return []+ i += 1++ if array_start is None:+ return []++ results = []+ i = array_start + 1+ while True:+ i = _skip_whitespace(i)+ if i >= len(text):+ return results+ if text[i] != "{":+ return []+ try:+ entry, i = decoder.raw_decode(text, i)+ except (json.JSONDecodeError, RecursionError, ValueError):+ return results if _is_append_only_json_object_prefix(text[i:]) else []+ if not isinstance(entry, dict):+ return []+ results.append(entry)+ i = _skip_whitespace(i)+ if i >= len(text):+ return results+ # Salvage is only for an array truncated before its closing bracket.+ # A closed results array with a malformed outer object is not partial+ # per-entry output and must take the structured-error path.+ if text[i] == "]":+ return []+ if text[i] != ",":+ return []+ i = _skip_whitespace(i + 1)+ if i >= len(text):+ return results+ if text[i] == "]":+ return []# ── Deterministic Checks ─────────────────────────────────────────────────────@@ -1919,7 +2324,7 @@ def check_security(traj, tool_calls, expected_skill=None, acceptable_skills=Nonefinding_type="secret_exposure",severity="critical",message="Possible secret value appeared in tool output observed by the agent",- evidence=observation[:300],+ evidence="[redacted secret exposure]",source="tool_observation",score_impact=True,tool=action,@@ -2395,6 +2800,25 @@ def check_tool_efficiency(tool_calls, expected_skill=None, expected_script=None)# ── LLM Judge: Accuracy (5-criterion) ────────────────────────────────────────+_ACCURACY_CRITERIA_KEYS = frozenset(+ {+ "SKILL_IDENTIFIED",+ "ACTION_CORRECT",+ "FACTUALLY_ACCURATE",+ "TASK_ADDRESSED",+ "ACTIONABLE",+ }+)+++def _valid_accuracy_criteria(value):+ return (+ isinstance(value, dict)+ and value.keys() == _ACCURACY_CRITERIA_KEYS+ and all(isinstance(item, bool) for item in value.values())+ )++def judge_accuracy(question, ground_truth, agent_text):if not ground_truth:return {"score": 1.0, "reason": "No ground_truth -- skipped"}@@ -2425,18 +2849,26 @@ SELECTED EVIDENCE (final response + produced artifacts; low-relevance steps maycontent, error = call_public_llm(prompt)if error:- return {"score": 0.0, "reason": f"LLM judge error: {error}"}+ return _judge_error(f"LLM judge error: {error}")parsed = extract_json(content) if content else None- if not parsed:- yes_count = (content or "").upper().count("YES")- return {"score": round(min(yes_count / 5.0, 1.0), 2), "reason": f"Parsed {yes_count}/5 YES from text"}- score = parsed.get("score", 0.0)- if isinstance(score, (int, float)):- score = max(0.0, min(1.0, float(score)))- else:- criteria = parsed.get("criteria", {})+ if not isinstance(parsed, dict):+ return _judge_error("Judge response was not a valid JSON object")++ criteria = parsed.get("criteria")+ criteria_valid = _valid_accuracy_criteria(criteria)+ if "criteria" in parsed and not criteria_valid:+ return _judge_error("Judge response contained invalid accuracy criteria")++ score = _finite_score(parsed.get("score"))+ if score is None:+ if not criteria_valid:+ return _judge_error("Judge response contained no valid accuracy score or complete criteria")score = sum(1 for v in criteria.values() if v is True) / 5.0- return {"score": round(score, 4), "reason": parsed.get("reason", ""), "criteria": parsed.get("criteria", {})}+ return {+ "score": round(score, 4),+ "reason": parsed.get("reason", ""),+ "criteria": criteria if criteria_valid else {},+ }# ── LLM Judge: Goal Accuracy ─────────────────────────────────────────────────@@ -2448,7 +2880,10 @@ def judge_goal_accuracy(question, ground_truth, agent_text, tool_summary=""):if _ragas_goal_accuracy_enabled():try:- return _judge_goal_accuracy_ragas(question, ground_truth, agent_text, tool_summary)+ result = _judge_goal_accuracy_ragas(question, ground_truth, agent_text, tool_summary)+ if not isinstance(result, dict) or _finite_score(result.get("score")) is None:+ raise ValueError("RAGAS returned a non-finite goal accuracy score")+ return resultexcept Exception as e:logger.info("RAGAS not available (%s), using custom prompt", e)return _judge_goal_accuracy_custom(question, ground_truth, agent_text, tool_summary)@@ -2504,6 +2939,8 @@ def _judge_goal_accuracy_ragas(question, ground_truth, agent_text, tool_summary)loop.close()score = float(result.value) if hasattr(result, "value") else float(result)+ if not math.isfinite(score):+ raise ValueError("RAGAS returned a non-finite goal accuracy score")return {"score": max(0.0, min(1.0, score)),"reason": "RAGAS AgentGoalAccuracyWithReference",@@ -2539,14 +2976,28 @@ Respond with ONLY a JSON object:content, error, provenance = _call_public_llm_with_provenance(prompt)if error:- return {"score": 0.0, "reason": f"LLM judge error: {error}", **provenance}+ return _judge_error(f"LLM judge error: {error}", **provenance)parsed = extract_json(content) if content else None- if not parsed:- return {"score": 0.0, "reason": "Could not parse judge response", **provenance}- score = 1.0 if parsed.get("achieved", False) else 0.0- if "score" in parsed and isinstance(parsed["score"], (int, float)):- score = max(0.0, min(1.0, float(parsed["score"])))- return {"score": score, "reason": parsed.get("reason", ""), "method": "custom", **provenance}+ if not isinstance(parsed, dict):+ return _judge_error("Judge response was not a valid JSON object", **provenance)++ achieved = parsed.get("achieved")+ if not isinstance(achieved, bool):+ return _judge_error("Judge response contained an invalid achieved value", **provenance)++ score = 1.0 if achieved else 0.0+ if "score" in parsed:+ score = _finite_score(parsed["score"])+ if score is None:+ return _judge_error("Judge response contained an invalid goal score", **provenance)+ return {+ "score": score,+ "reason": parsed.get("reason", ""),+ "user_goal": parsed.get("user_goal", ""),+ "end_state": parsed.get("end_state", ""),+ "method": "custom",+ **provenance,+ }# ── LLM Judge: Behavior Check ────────────────────────────────────────────────@@ -2586,79 +3037,134 @@ Respond with ONLY a JSON object:content, error = call_public_llm(prompt, max_tokens=BEHAVIOR_JUDGE_MAX_TOKENS)if error:- return {"score": 0.0, "reason": f"LLM judge error: {error}", "results": []}+ return _judge_error(f"LLM judge error: {error}", results=[])def _parse_judge_object(text):- parsed = extract_json(text) if text else None- return parsed if isinstance(parsed, dict) else None+ return extract_json(text) if text else Noneparsed = _parse_judge_object(content)- attempts = [content or ""]- if not parsed:+ score = _behavior_payload_score(parsed, len(expected_behaviors))+ attempts = [(content or "", parsed)]+ retry_error = None+ if score is None:# One retry max, with an explicit machine-readable-output reminder.retry_content, retry_error = call_public_llm(prompt + _BEHAVIOR_RETRY_REMINDER, max_tokens=BEHAVIOR_JUDGE_MAX_TOKENS)if not retry_error:- attempts.append(retry_content or "")parsed = _parse_judge_object(retry_content)+ attempts.append((retry_content or "", parsed))+ score = _behavior_payload_score(parsed, len(expected_behaviors))- salvaged_from_truncation = False- if not parsed:+ if score is None:# Salvage complete entries from a truncated results array (newest first).- for text in reversed(attempts):+ for text, extracted in reversed(attempts):+ if extracted is not None:+ continuesalvaged = _salvage_behavior_results(text)if salvaged:- salvaged_from_truncation = True- parsed = {+ candidate = {"results": salvaged,"summary": (f"Salvaged {len(salvaged)}/{len(expected_behaviors)} behavior ""results from truncated judge response"),}- break+ candidate_score = _behavior_payload_score(+ candidate,+ len(expected_behaviors),+ allow_partial=True,+ )+ if candidate_score is not None:+ parsed = candidate+ score = candidate_score+ break- if not parsed:- last = attempts[-1]- head = last[:80].replace("\n", " ")- return {- "score": 0.0,- "reason": f"Judge response unparseable after retry (len={len(last)}, head={head!r})",- "results": [],- }+ if score is None:+ if retry_error:+ return _judge_error(f"LLM judge retry error: {retry_error}", results=[])+ return _judge_error("Judge response was unparseable or invalid after retry", results=[])- results = parsed.get("results", [])- if results:- passed_count = sum(1 for r in results if r.get("passed"))- # A salvaged array is incomplete: behaviors the truncation cut off were- # never judged and count as not-passed, so keep the denominator at the- # number of expected behaviors instead of inflating against the few- # recovered entries.- denominator = len(expected_behaviors) if salvaged_from_truncation else len(results)- score = passed_count / denominator- else:- score = parsed.get("score", 0.0)- if not isinstance(score, (int, float)):- score = 0.0+ results = parsed["results"]return {- "score": round(max(0.0, min(1.0, float(score))), 4),+ "score": round(score, 4),"reason": parsed.get("summary", ""),"results": results,}+def _behavior_payload_score(parsed, expected_count, *, allow_partial=False):+ if not isinstance(parsed, dict):+ return None+ results = parsed.get("results")+ if not isinstance(results, list):+ return None+ if any(not isinstance(result, dict) or not isinstance(result.get("passed"), bool) for result in results):+ return None+ if allow_partial:+ if not results or len(results) > expected_count:+ return None+ elif len(results) != expected_count:+ return None+ if "score" in parsed and _finite_score(parsed["score"]) is None:+ return None+ denominator = expected_count if allow_partial else len(results)+ if denominator <= 0:+ return None+ return sum(1 for result in results if result["passed"]) / denominator++# ── Main ─────────────────────────────────────────────────────────────────────+def _finite_reward_number(value):+ if isinstance(value, bool) or not isinstance(value, (int, float)):+ return None+ try:+ numeric = float(value)+ except (OverflowError, ValueError):+ return None+ return numeric if math.isfinite(numeric) else None+++def _normalize_required_judge_result(metric, result):+ if isinstance(result, dict):+ normalized = dict(result)+ status_is_error = str(result.get("status", "")).casefold() == "error"+ score_is_valid = _finite_reward_number(result.get("score")) is not None+ if not status_is_error and score_is_valid:+ return normalized+ supplied_reason = str(result.get("reason") or "").strip()+ reason = supplied_reason if status_is_error else f"Required {metric} judge returned an invalid score"+ if supplied_reason and not status_is_error:+ reason = f"{reason}: {supplied_reason}"+ else:+ normalized = {}+ reason = f"Required {metric} judge returned an invalid result"++ normalized["score"] = None+ normalized["status"] = "error"+ normalized["reason"] = _judge_error(reason)["reason"]+ return normalized+++def _call_required_judge(metric, judge, *args, **kwargs):+ try:+ result = judge(*args, **kwargs)+ except Exception as exc:+ result = _judge_error(f"Required {metric} judge raised {type(exc).__name__}: {exc}")+ return _normalize_required_judge_result(metric, result)++def _numeric_reward_payload(result, overall):payload = {}for key, value in result.items():- if isinstance(value, bool):- continue- if isinstance(value, (int, float)):- payload[key] = value- payload["overall"] = overall+ numeric = _finite_reward_number(value)+ if numeric is not None:+ payload[key] = numeric+ numeric_overall = _finite_reward_number(overall)+ if numeric_overall is not None:+ payload["overall"] = numeric_overallreturn payload@@ -2809,19 +3315,35 @@ def main():)# ── Eval 4: accuracy (LLM judge) ─────────────────────────────────────- acc_result = judge_accuracy(question, ground_truth, bundles["accuracy"]["prompt_evidence"])+ acc_result = _call_required_judge(+ "accuracy",+ judge_accuracy,+ question,+ ground_truth,+ bundles["accuracy"]["prompt_evidence"],+ )acc_score = acc_result["score"]details["accuracy"] = acc_result# ── Eval 5: goal_accuracy (RAGAS or custom LLM judge) ────────────────- ga_result = judge_goal_accuracy(- question, ground_truth, bundles["goal_accuracy"]["prompt_evidence"], tool_summary=""+ ga_result = _call_required_judge(+ "goal_accuracy",+ judge_goal_accuracy,+ question,+ ground_truth,+ bundles["goal_accuracy"]["prompt_evidence"],+ tool_summary="",)ga_score = ga_result["score"]details["goal_accuracy"] = ga_result# ── Eval 6: behavior_check (LLM judge) ───────────────────────────────- bc_result = judge_behavior_check(bundles["behavior_check"]["prompt_evidence"], expected_behavior)+ bc_result = _call_required_judge(+ "behavior_check",+ judge_behavior_check,+ bundles["behavior_check"]["prompt_evidence"],+ expected_behavior,+ )bc_score = bc_result["score"]details["behavior_check"] = bc_result@@ -2846,7 +3368,22 @@ def main():"details": details,}- scores = [float(result.get(metric, 0.0) or 0.0) for metric in DISPLAY_METRICS]+ judge_errors = {+ metric: details[metric]["reason"]+ for metric in ("accuracy", "goal_accuracy", "behavior_check")+ if details[metric].get("status") == "error"+ }+ if judge_errors:+ result["evaluation_status"] = "failed"+ result["evaluation_errors"] = judge_errors+ # Harbor 0.13.2 still parses reward.json when the verifier exits nonzero.+ # Keep this artifact deliberately incomplete so the collector cannot+ # score it even if the richer diagnostic sidecar is unavailable.+ write_reward_outputs(result, 0.0)+ logger.error("Required LLM judging failed for: %s", ", ".join(sorted(judge_errors)))+ raise SystemExit(1)++ scores = [float(result[metric]) for metric in DISPLAY_METRICS]overall = round(sum(scores) / len(scores), 4)write_reward_outputs(result, overall)diff --git a/src/skillevaluator/tier3/output_provenance.py b/src/skillevaluator/tier3/output_provenance.pyindex 86adaac1..b000a782 100644--- a/src/skillevaluator/tier3/output_provenance.py+++ b/src/skillevaluator/tier3/output_provenance.py@@ -70,10 +70,13 @@ def _stable_read_metadata(return after-def _fsync_directory(path: Path) -> None:+def _fsync_directory(path: Path, *, directory_fd: int | None = None) -> None:"""Durably publish a directory entry where the platform supports it."""if os.name != "posix":return+ if directory_fd is not None:+ os.fsync(directory_fd)+ returndescriptor = os.open(path, os.O_RDONLY | getattr(os, "O_DIRECTORY", 0))try:os.fsync(descriptor)@@ -81,10 +84,22 @@ def _fsync_directory(path: Path) -> None:os.close(descriptor)-def _unlink_if_same_file(path: Path, expected: os.stat_result) -> bool:+def _lstat_output_child(path: Path, *, directory_fd: int | None = None) -> os.stat_result:+ """Inspect one immediate output child relative to a pinned directory."""+ if directory_fd is None:+ return path.lstat()+ return os.stat(path.name, dir_fd=directory_fd, follow_symlinks=False)+++def _unlink_if_same_file(+ path: Path,+ expected: os.stat_result,+ *,+ directory_fd: int | None = None,+) -> bool:"""Unlink only the exact regular file created by the current operation."""try:- observed = path.lstat()+ observed = _lstat_output_child(path, directory_fd=directory_fd)except OSError:return Falseif _is_link_or_reparse(observed) or not stat.S_ISREG(observed.st_mode):@@ -92,16 +107,19 @@ def _unlink_if_same_file(path: Path, expected: os.stat_result) -> bool:if not os.path.samestat(expected, observed):return Falsetry:- path.unlink()+ if directory_fd is None:+ path.unlink()+ else:+ os.unlink(path.name, dir_fd=directory_fd)except OSError:return Falsereturn True-def _inspect_atomic_destination(path: Path) -> os.stat_result | None:+def _inspect_atomic_destination(path: Path, *, directory_fd: int | None = None) -> os.stat_result | None:"""Return destination metadata after rejecting unsafe file types."""try:- metadata = path.lstat()+ metadata = _lstat_output_child(path, directory_fd=directory_fd)except FileNotFoundError:return Noneif _is_link_or_reparse(metadata) or not stat.S_ISREG(metadata.st_mode) or metadata.st_nlink != 1:@@ -109,86 +127,127 @@ def _inspect_atomic_destination(path: Path) -> os.stat_result | None:return metadata+def _assert_output_parent_still_declared(path: Path, expected: os.stat_result) -> None:+ """Reject a lexical output parent that no longer names the pinned directory."""+ try:+ validate_output_directory_path(path)+ except ValueError as exc:+ raise ValueError(f"Generated output directory changed while writing: {path}") from exc+ try:+ observed = path.lstat()+ except OSError as exc:+ raise ValueError(f"Generated output directory changed while writing: {path}") from exc+ if _is_link_or_reparse(observed) or not stat.S_ISDIR(observed.st_mode) or not os.path.samestat(expected, observed):+ raise ValueError(f"Generated output directory changed while writing: {path}")++def write_output_file_atomically(path: Path, payload: bytes) -> None:"""Durably replace one evaluator-owned output file with complete bytes."""validate_output_directory_path(path.parent)- existing = _inspect_atomic_destination(path)- existing_mode = stat.S_IMODE(existing.st_mode) if existing is not None else None- temporary: Path | None = None- temporary_named_metadata: os.stat_result | None = None- descriptor = -1try:- for _ in range(32):- candidate = path.parent / f".{path.name}.{os.getpid()}-{secrets.token_hex(8)}.tmp"- try:- descriptor = os.open(- candidate,- os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, "O_CLOEXEC", 0) | getattr(os, "O_NOFOLLOW", 0),- 0o666,- )- except FileExistsError:- continue- temporary = candidate- break- if temporary is None or descriptor < 0:- raise OSError(f"Cannot allocate a temporary output artifact for: {path}")-- temporary_opened = os.fstat(descriptor)- temporary_named_metadata = temporary.lstat()- if (- _is_link_or_reparse(temporary_opened)- or _is_link_or_reparse(temporary_named_metadata)- or not stat.S_ISREG(temporary_opened.st_mode)- or not stat.S_ISREG(temporary_named_metadata.st_mode)- or temporary_opened.st_nlink != 1- or temporary_named_metadata.st_nlink != 1- or (- _PATH_DESCRIPTOR_IDENTITIES_COMPARABLE- and not os.path.samestat(temporary_opened, temporary_named_metadata)- )- ):- raise ValueError(f"Generated output temporary artifact is unsafe: {temporary}")- if existing_mode is not None and hasattr(os, "fchmod"):- os.fchmod(descriptor, existing_mode)+ parent_metadata = path.parent.lstat()+ except OSError as exc:+ raise ValueError(f"Cannot inspect generated output directory: {path.parent}") from exc- handle = os.fdopen(descriptor, "wb", closefd=True)+ with SecureRoot(path.parent, expected=parent_metadata) as secure_parent:+ directory_fd = secure_parent.duplicate_posix_root_descriptor() if os.name == "posix" else None+ temporary: Path | None = None+ temporary_named_metadata: os.stat_result | None = Nonedescriptor = -1- with handle:- if handle.write(payload) != len(payload):- raise OSError(f"Short write while publishing generated output: {path}")- handle.flush()- os.fsync(handle.fileno())- written = os.fstat(handle.fileno())+ try:+ existing = _inspect_atomic_destination(path, directory_fd=directory_fd)+ existing_mode = stat.S_IMODE(existing.st_mode) if existing is not None else None+ for _ in range(32):+ candidate = path.parent / f".{path.name}.{os.getpid()}-{secrets.token_hex(8)}.tmp"+ try:+ descriptor = os.open(+ candidate.name if directory_fd is not None else candidate,+ os.O_WRONLY+ | os.O_CREAT+ | os.O_EXCL+ | getattr(os, "O_CLOEXEC", 0)+ | getattr(os, "O_NOFOLLOW", 0),+ 0o666,+ **({"dir_fd": directory_fd} if directory_fd is not None else {}),+ )+ except FileExistsError:+ continue+ temporary = candidate+ break+ if temporary is None or descriptor < 0:+ raise OSError(f"Cannot allocate a temporary output artifact for: {path}")++ temporary_opened = os.fstat(descriptor)+ temporary_named_metadata = _lstat_output_child(temporary, directory_fd=directory_fd)+ if (+ _is_link_or_reparse(temporary_opened)+ or _is_link_or_reparse(temporary_named_metadata)+ or not stat.S_ISREG(temporary_opened.st_mode)+ or not stat.S_ISREG(temporary_named_metadata.st_mode)+ or temporary_opened.st_nlink != 1+ or temporary_named_metadata.st_nlink != 1+ or (+ _PATH_DESCRIPTOR_IDENTITIES_COMPARABLE+ and not os.path.samestat(temporary_opened, temporary_named_metadata)+ )+ ):+ raise ValueError(f"Generated output temporary artifact is unsafe: {temporary}")+ if existing_mode is not None and hasattr(os, "fchmod"):+ os.fchmod(descriptor, existing_mode)- observed = temporary.lstat()- temporary_identity_matches = os.path.samestat(temporary_named_metadata, observed) and (- not _PATH_DESCRIPTOR_IDENTITIES_COMPARABLE or os.path.samestat(written, observed)- )- if (- _is_link_or_reparse(observed)- or not stat.S_ISREG(observed.st_mode)- or observed.st_nlink != 1- or written.st_size != len(payload)- or observed.st_size != len(payload)- or not temporary_identity_matches- ):- raise ValueError(f"Generated output temporary artifact changed while writing: {temporary}")- validate_output_directory_path(path.parent)- destination = _inspect_atomic_destination(path)- if (existing is None) != (destination is None) or (- existing is not None- and destination is not None- and _artifact_fingerprint(existing) != _artifact_fingerprint(destination)- ):- raise ValueError(f"Generated output destination changed while writing: {path}")- os.replace(temporary, path) # noqa: PTH105 -- same-directory atomic replacement is the contract- temporary = None- _fsync_directory(path.parent)- finally:- if descriptor >= 0:- os.close(descriptor)- if temporary is not None and temporary_named_metadata is not None:- _unlink_if_same_file(temporary, temporary_named_metadata)+ handle = os.fdopen(descriptor, "wb", closefd=True)+ descriptor = -1+ with handle:+ if handle.write(payload) != len(payload):+ raise OSError(f"Short write while publishing generated output: {path}")+ handle.flush()+ os.fsync(handle.fileno())+ written = os.fstat(handle.fileno())++ observed = _lstat_output_child(temporary, directory_fd=directory_fd)+ temporary_identity_matches = os.path.samestat(temporary_named_metadata, observed) and (+ not _PATH_DESCRIPTOR_IDENTITIES_COMPARABLE or os.path.samestat(written, observed)+ )+ if (+ _is_link_or_reparse(observed)+ or not stat.S_ISREG(observed.st_mode)+ or observed.st_nlink != 1+ or written.st_size != len(payload)+ or observed.st_size != len(payload)+ or not temporary_identity_matches+ ):+ raise ValueError(f"Generated output temporary artifact changed while writing: {temporary}")+ _assert_output_parent_still_declared(path.parent, parent_metadata)+ destination = _inspect_atomic_destination(path, directory_fd=directory_fd)+ if (existing is None) != (destination is None) or (+ existing is not None+ and destination is not None+ and _artifact_fingerprint(existing) != _artifact_fingerprint(destination)+ ):+ raise ValueError(f"Generated output destination changed while writing: {path}")+ if directory_fd is None:+ os.replace(temporary, path) # noqa: PTH105 -- pinned Windows parent prevents redirection+ else:+ os.replace(+ temporary.name,+ path.name,+ src_dir_fd=directory_fd,+ dst_dir_fd=directory_fd,+ )+ temporary = None+ _fsync_directory(path.parent, directory_fd=directory_fd)+ _assert_output_parent_still_declared(path.parent, parent_metadata)+ finally:+ if descriptor >= 0:+ os.close(descriptor)+ if temporary is not None and temporary_named_metadata is not None:+ _unlink_if_same_file(+ temporary,+ temporary_named_metadata,+ directory_fd=directory_fd,+ )+ if directory_fd is not None:+ os.close(directory_fd)def _protect_key_for_storage(key: bytes) -> bytes:diff --git a/src/skillevaluator/utils/redaction.py b/src/skillevaluator/utils/redaction.pyindex f2ae09c7..3f79f78b 100644--- a/src/skillevaluator/utils/redaction.py+++ b/src/skillevaluator/utils/redaction.py@@ -43,7 +43,38 @@ _SENSITIVE_QUOTED_ASSIGNMENT_RE = re.compile()_SENSITIVE_COLON_ASSIGNMENT_RE = re.compile(rf"(?im)\b(?P<key>{_SENSITIVE_KEY_PATTERN})\s*(?P<sep>:)\s*[^\r\n,;]+")_SENSITIVE_EQUALS_ASSIGNMENT_RE = re.compile(rf"(?i)\b(?P<key>{_SENSITIVE_KEY_PATTERN})\s*(?P<sep>=)\s*[^\s\"',;]+")+_PRIVATE_KEY_LABEL = r"(?:[A-Z0-9][A-Z0-9-]* )*PRIVATE KEY(?: [A-Z0-9][A-Z0-9-]*)*"+_PEM_REDACTIONS = (+ (+ re.compile(+ rf"-----BEGIN (?P<private_key_label>{_PRIVATE_KEY_LABEL})-----"+ r"(?:(?!-----BEGIN |-----END )[\s\S])*?"+ r"-----END (?P=private_key_label)-----"+ ),+ "private-key-<redacted>",+ ),+ (+ re.compile(+ rf"-----BEGIN {_PRIVATE_KEY_LABEL}-----"+ r"(?:(?!-----BEGIN |-----END )[\s\S])*?"+ r"-----END (?:[A-Z0-9][A-Z0-9-]* )*[A-Z0-9][A-Z0-9-]*-----"+ ),+ "private-key-<redacted>",+ ),+ (+ re.compile(+ rf"-----BEGIN {_PRIVATE_KEY_LABEL}-----"+ r"(?![\s\S]*-----END )[\s\S]*\Z"+ ),+ "private-key-<redacted>",+ ),+)_REDACTIONS = (+ (+ re.compile(r"\beyJ[A-Za-z0-9_-]{10,}\.eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\b"),+ "jwt-<redacted>",+ ),+ (re.compile(r"(?:AKIA|ASIA)[A-Z0-9]{16}"), "aws-access-key-<redacted>"),(re.compile(r"(?i)\bBearer\s+[A-Za-z0-9._~+/=-]{8,}"), "Bearer <redacted>"),(re.compile(r"(?<![A-Za-z0-9_-])sk-[a-zA-Z0-9_-]{8,}"), "sk-<redacted>"),(re.compile(r"(?<![A-Za-z0-9_-])nvapi-[a-zA-Z0-9_-]{8,}"), "nvapi-<redacted>"),@@ -87,6 +118,10 @@ def _redact_auth_header(match: re.Match[str]) -> str:def redact_sensitive_text(value: str, *, max_len: int | None = None) -> str:"""Best-effort masking for credentials before writing logs or artifacts."""out = value+ # Remove multiline private-key material before any single-line assignment+ # or header rule can consume only its BEGIN delimiter and orphan the body.+ for pattern, replacement in _PEM_REDACTIONS:+ out = pattern.sub(replacement, out)out = _AUTH_HEADER_RE.sub(_redact_auth_header, out)out = _SENSITIVE_QUOTED_ASSIGNMENT_RE.sub(_redact_sensitive_assignment, out)out = _SENSITIVE_COLON_ASSIGNMENT_RE.sub(_redact_sensitive_assignment, out)diff --git a/src/skillevaluator/utils/secure_fs.py b/src/skillevaluator/utils/secure_fs.pyindex 587e5569..3ce2b136 100644--- a/src/skillevaluator/utils/secure_fs.py+++ b/src/skillevaluator/utils/secure_fs.py@@ -237,6 +237,20 @@ class SecureRoot:_windows_close_handle(self._windows_root_handles.pop())self._entered = False+ def duplicate_posix_root_descriptor(self) -> int:+ """Return a caller-owned descriptor pinned to the active POSIX root.++ Descriptor-relative output operations need the same no-follow root+ pinning as reads, while retaining ownership of their own descriptor.+ The caller must close the returned descriptor.+ """+ if not self._entered or os.name != "posix" or self._root_fd is None:+ raise SecurePathError(+ "secure_open_unavailable",+ "A pinned POSIX root descriptor is unavailable.",+ )+ return os.dup(self._root_fd)+def read_bytes(self,relative_path: Path,diff --git a/tests/reporting/test_report_data_bounds.py b/tests/reporting/test_report_data_bounds.pyindex 78ec3fc5..03b0c3fe 100644--- a/tests/reporting/test_report_data_bounds.py+++ b/tests/reporting/test_report_data_bounds.py@@ -73,6 +73,48 @@ def test_normal_agent_artifacts_are_loaded_without_truncation_marker(tmp_path: Passert "_report_truncation" not in agents["codex"]+def test_agent_directory_symlink_is_not_discovered(tmp_path: Path) -> None:+ outside_agent = tmp_path / "outside" / "codex"+ _write_summary(outside_agent)+ results_dir = tmp_path / "results"+ results_dir.mkdir()+ (results_dir / "codex").symlink_to(outside_agent, target_is_directory=True)++ assert report_data.load_agent_data(results_dir) == {}++[email protected]("escape", ["condition", "trials", "trial"])+def test_intermediate_directory_symlinks_are_not_followed(tmp_path: Path, escape: str) -> None:+ results_dir = tmp_path / "results"+ agent_dir = results_dir / "codex"+ outside = tmp_path / "outside"+ if escape == "condition":+ _write_summary(outside)+ agent_dir.mkdir(parents=True)+ (agent_dir / "with-skill").symlink_to(outside / "with-skill", target_is_directory=True)+ else:+ _write_summary(agent_dir)+ outside_trial = outside / "case-001__1"+ outside_trial.mkdir(parents=True)+ (outside_trial / "reward.json").write_text(+ json.dumps({"entry_id": "case-001", "accuracy": 1.0}),+ encoding="utf-8",+ )+ trials_dir = agent_dir / "with-skill" / "trials"+ if escape == "trials":+ trials_dir.symlink_to(outside, target_is_directory=True)+ else:+ trials_dir.mkdir()+ (trials_dir / "case-001__1").symlink_to(outside_trial, target_is_directory=True)++ agents = report_data.load_agent_data(results_dir)++ if escape == "condition":+ assert agents == {}+ else:+ assert agents["codex"]["rewards"] == []++def test_oversized_summary_reward_and_trajectory_are_skipped(tmp_path: Path,monkeypatch: pytest.MonkeyPatch,@@ -141,6 +183,16 @@ def test_pathological_json_is_skipped_before_report_processing(assert any(reason["code"] == expected_code for reason in _reasons(agents["codex"]))+def test_huge_json_integer_is_rejected_without_crashing() -> None:+ diagnostics: list[dict] = []+ raw = ('{"scores": {"accuracy": ' + "9" * 10_000 + "}}").encode()++ loaded = report_data._decode_bounded_json(raw, diagnostics, artifact="summary")++ assert loaded is report_data._INVALID_JSON+ assert any(reason["code"] == "json_number" for reason in diagnostics)++def test_excess_trials_are_capped_in_name_order(tmp_path: Path,monkeypatch: pytest.MonkeyPatch,diff --git a/tests/test_harbor_collector_runtime_failures.py b/tests/test_harbor_collector_runtime_failures.pyindex d6b3b0a4..d0401b18 100644--- a/tests/test_harbor_collector_runtime_failures.py+++ b/tests/test_harbor_collector_runtime_failures.py@@ -640,7 +640,7 @@ def test_multistep_stop_on_pass_uses_authoritative_root_reward(tmp_path: Path) -assert result["execution_status"] == "failed"assert result["expected_attempts"] == 2assert result["scored_attempts"] == 1- assert agent["pass_at_k"]["with_skill"]["rate"] == 0.0+ assert agent["pass_at_k"]["with_skill"] == {}def test_duplicate_logical_attempt_ordinals_fail(tmp_path: Path) -> None:diff --git a/tests/test_harbor_html_report_failures.py b/tests/test_harbor_html_report_failures.pyindex a548f09a..9ebff846 100644--- a/tests/test_harbor_html_report_failures.py+++ b/tests/test_harbor_html_report_failures.py@@ -14,6 +14,7 @@ from skillevaluator.evaluation.tier3_report import render_agent_eval_html_reportfrom skillevaluator.provider_config import ProviderConfigfrom skillevaluator.tier3.harbor import runnerfrom skillevaluator.tier3.harbor.collector import _build_comparison, collect_harbor_results+from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICSfrom skillevaluator.tier3.results_location import external_results_root, resolve_latest_resultsif TYPE_CHECKING:@@ -42,6 +43,40 @@ def _tier3_payload(output: str) -> dict:return json.loads(match.group(1))+def _visible_text_by_class(output: str, class_name: str) -> list[str]:+ return [+ text.strip()+ for text in re.findall(+ rf'<(?:div|span) class="{re.escape(class_name)}"[^>]*>\s*([^<]*?)\s*</(?:div|span)>',+ output,+ )+ ]+++def _overall_kpi_values(output: str) -> list[str]:+ return [+ value.strip()+ for value in re.findall(+ r'<div class="t3-kpi-label">Overall Score</div>\s*'+ r'<div class="t3-kpi-value"[^>]*>\s*([^<]*?)\s*</div>',+ output,+ )+ ]+++def _overall_matrix_html(output: str) -> str:+ match = re.search(r"<td><strong>Overall</strong></td>(.*?)</tr>", output, re.DOTALL)+ assert match is not None+ return match.group(1)+++def _agent_scorecard_bar_widths(output: str) -> list[str]:+ return re.findall(+ r'<div class="t3-mini-bar">\s*<span style="width:([^;]+);',+ output,+ )++def test_report_renders_aggregate_and_trial_failure_details(tmp_path: Path) -> None:results_dir = tmp_path / "20260704_220000"summary_dir = results_dir / "opencode" / "with-skill"@@ -133,7 +168,7 @@ def test_failed_agent_has_no_synthetic_score_in_comparison_or_html(tmp_path: Pat"failed": {"execution_status": "failed","with_skill": {},- "without_skill": {},+ "without_skill": dict.fromkeys(DEFAULT_METRICS, 0.4),"lift": {},},"succeeded": {@@ -161,12 +196,46 @@ def test_failed_agent_has_no_synthetic_score_in_comparison_or_html(tmp_path: Pat"scored_attempts": 0 if name == "failed" else 1,}(summary_dir / "summary.json").write_text(json.dumps(summary), encoding="utf-8")+ if data["without_skill"]:+ baseline_dir = tmp_path / name / "without-skill"+ baseline_dir.mkdir(parents=True)+ (baseline_dir / "summary.json").write_text(+ json.dumps(+ {+ "scores": data["without_skill"],+ "metrics": list(data["without_skill"]),+ "num_trials": 1,+ "execution_status": "succeeded",+ "execution_errors": [],+ "expected_attempts": 1,+ "scored_attempts": 1,+ }+ ),+ encoding="utf-8",+ )output = _render_report("demo", tmp_path).read_text(encoding="utf-8")payload = _tier3_payload(output)assert payload["agents"]["failed"]["with_skill"] is Noneassert payload["agents"]["succeeded"]["with_skill"] == 0.8+ failed_dimensions = payload["agents"]["failed"]["dimensions"]+ assert len(failed_dimensions) == 5+ assert all(dimension["score"] is None and dimension["baseline"] == 0.4 for dimension in failed_dimensions)++ overall_matrix = _overall_matrix_html(output)+ assert _visible_text_by_class(overall_matrix, "t3-mad-score") == ["N/A", "0.80"]+ assert '<span class="t3-mad-score" style="color:var(--text-secondary);">N/A</span>' in overall_matrix+ assert overall_matrix.count('class="t3-mad-bar"') == 1+ assert _visible_text_by_class(output, "t3-agent-score") == ["N/A", "0.80"]+ assert _visible_text_by_class(output, "t3-tab-score") == ["N/A", "0.80 (+0.30)"]+ assert _overall_kpi_values(output) == ["N/A", "0.80"]+ assert _agent_scorecard_bar_widths(output) == ["80.0%"]+ assert output.count('<span class="t3-mad-score" style="color:var(--text-secondary);">N/A</span>') >= 6+ # Five dimension cells plus overall matrix, scorecard, and lift KPI must+ # distinguish an unavailable with-skill score from a missing baseline.+ assert output.count("with-skill unavailable") == 8+ assert "no baseline" not in outputdef test_partial_execution_failure_reports_coverage_without_quality_claims(tmp_path: Path) -> None:@@ -438,6 +507,81 @@ def test_failed_default_run_without_metrics_is_not_labeled_custom_only(tmp_path:assert payload["metric_ids"] == []assert payload["overall_score"] is Noneassert "Custom Reward Mode" not in output+ assert _visible_text_by_class(output, "t3-agent-score") == ["N/A"]+ assert _visible_text_by_class(output, "t3-tab-score") == ["N/A"]+ assert _overall_kpi_values(output) == ["N/A"]+ assert '<div class="t3-agent-score" style="color:var(--text-secondary);">N/A</div>' in output+ assert '<div class="t3-kpi-value" style="color:var(--text-secondary);">N/A</div>' in output+ assert _agent_scorecard_bar_widths(output) == []+++def test_failed_custom_only_summary_cannot_publish_stale_overall_score(tmp_path: Path) -> None:+ results_dir = tmp_path / "20260708_120006"+ summary_dir = results_dir / "opencode" / "with-skill"+ summary_dir.mkdir(parents=True)+ (summary_dir / "summary.json").write_text(+ json.dumps(+ {+ "scores": {},+ "custom_scores": {"domain_quality": 0.8},+ "overall_score": 0.8,+ "metrics": [],+ "num_trials": 1,+ "execution_status": "failed",+ "execution_errors": ["custom grader failed"],+ "expected_attempts": 1,+ "scored_attempts": 0,+ }+ ),+ encoding="utf-8",+ )+ (results_dir / "run_config.json").write_text(+ json.dumps({"grading": {"mode": {"value": "custom_only", "source": "CLI"}}}),+ encoding="utf-8",+ )++ output = _render_report("demo", results_dir).read_text(encoding="utf-8")+ payload = _tier3_payload(output)++ assert payload["execution_status"] == "failed"+ assert payload["overall_score"] is None+ assert payload["agents"]["opencode"]["with_skill"] is None+ assert _visible_text_by_class(output, "t3-agent-score") == ["N/A"]+ assert _overall_kpi_values(output) == ["N/A"]+++def test_successful_agent_with_legitimate_zero_score_renders_numeric_zero(tmp_path: Path) -> None:+ results_dir = tmp_path / "20260708_120005"+ metrics = ["skill_execution", "skill_efficiency", "accuracy", "goal_accuracy", "behavior_check"]+ for agent, score in (("positive", 0.8), ("zero", 0.0)):+ summary_dir = results_dir / agent / "with-skill"+ summary_dir.mkdir(parents=True)+ (summary_dir / "summary.json").write_text(+ json.dumps(+ {+ "scores": dict.fromkeys(metrics, score),+ "metrics": metrics,+ "num_trials": 1,+ "execution_status": "succeeded",+ "execution_errors": [],+ "expected_attempts": 1,+ "scored_attempts": 1,+ }+ ),+ encoding="utf-8",+ )++ output = _render_report("demo", results_dir).read_text(encoding="utf-8")+ payload = _tier3_payload(output)++ assert payload["agents"]["zero"]["with_skill"] == 0.0+ overall_matrix = _overall_matrix_html(output)+ assert _visible_text_by_class(overall_matrix, "t3-mad-score") == ["0.80", "0.00"]+ assert overall_matrix.count('class="t3-mad-bar"') == 2+ assert _visible_text_by_class(output, "t3-agent-score") == ["0.80", "0.00"]+ assert _visible_text_by_class(output, "t3-tab-score") == ["0.80", "0.00"]+ assert _overall_kpi_values(output) == ["0.80", "0.00"]+ assert _agent_scorecard_bar_widths(output) == ["80.0%", "0.0%"]def test_successful_custom_only_run_keeps_custom_reward_labels(tmp_path: Path) -> None:diff --git a/tests/test_harbor_output_provenance.py b/tests/test_harbor_output_provenance.pyindex 51a542b7..a1133e37 100644--- a/tests/test_harbor_output_provenance.py+++ b/tests/test_harbor_output_provenance.py@@ -8,6 +8,7 @@ from __future__ import annotationsimport jsonimport osimport shutil+import statimport threadingfrom concurrent.futures import ThreadPoolExecutorfrom pathlib import Path@@ -64,6 +65,11 @@ def _simulate_windows_crt_fstat_identity(monkeypatch: pytest.MonkeyPatch) -> Nondef incompatible_fstat(descriptor: int) -> object:opened = original_fstat(descriptor)+ # The test runs on POSIX, where the newly pinned output root still+ # needs a real descriptor identity. Simulate Windows CRT identity only+ # for the output file descriptors whose compatibility is under test.+ if stat.S_ISDIR(opened.st_mode):+ return openedreturn SimpleNamespace(st_dev=opened.st_dev + 10_000,st_ino=opened.st_ino + 10_000,@@ -93,6 +99,81 @@ def test_atomic_output_accepts_windows_crt_descriptor_identity(assert not list(tmp_path.glob(".result.json.*.tmp"))[email protected](os.name != "posix", reason="requires POSIX descriptor-relative rename support")+def test_atomic_output_parent_swap_cannot_publish_outside_validated_directory(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ trusted = tmp_path / "trusted"+ trusted.mkdir()+ parked = tmp_path / "parked"+ outside = tmp_path / "outside"+ outside.mkdir()+ destination = trusted / "result.json"+ destination.write_bytes(b"old")+ original_replace = output_provenance.os.replace++ def swap_parent_then_replace(+ source: Path | str,+ target: Path | str,+ *args: object,+ **kwargs: object,+ ) -> None:+ trusted.rename(parked)+ trusted.symlink_to(outside, target_is_directory=True)++ # Model a same-UID attacker who can relink the already-written+ # temporary into the redirected parent after the writer's final path+ # validation. An unanchored os.replace then publishes those bytes+ # outside the declared output directory.+ if kwargs.get("src_dir_fd") is None:+ parked_source = parked / Path(source).name+ os.link(parked_source, outside / Path(source).name)+ original_replace(source, target, *args, **kwargs)++ monkeypatch.setattr(output_provenance.os, "replace", swap_parent_then_replace)++ with pytest.raises(ValueError, match="directory changed"):+ output_provenance.write_output_file_atomically(destination, b"new-secret")++ assert not list(outside.iterdir())+ assert (parked / "result.json").read_bytes() in {b"old", b"new-secret"}++[email protected](os.name != "nt", reason="requires native Windows sharing semantics")+def test_atomic_output_pins_windows_parent_against_rename(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ trusted = tmp_path / "trusted"+ trusted.mkdir()+ parked = tmp_path / "parked"+ destination = trusted / "result.json"+ original_replace = output_provenance.os.replace+ rename_was_blocked = False++ def attempt_parent_rename(+ source: Path | str,+ target: Path | str,+ *args: object,+ **kwargs: object,+ ) -> None:+ nonlocal rename_was_blocked+ try:+ trusted.rename(parked)+ except OSError:+ rename_was_blocked = True+ original_replace(source, target, *args, **kwargs)++ monkeypatch.setattr(output_provenance.os, "replace", attempt_parent_rename)++ output_provenance.write_output_file_atomically(destination, b"complete")++ assert rename_was_blocked+ assert destination.read_bytes() == b"complete"+ assert not parked.exists()++def test_marker_failure_cleanup_accepts_windows_crt_descriptor_identity(tmp_path: Path,monkeypatch: pytest.MonkeyPatch,diff --git a/tests/test_harbor_runner_environment.py b/tests/test_harbor_runner_environment.pyindex aa6c8bc1..38fa865c 100644--- a/tests/test_harbor_runner_environment.py+++ b/tests/test_harbor_runner_environment.py@@ -5,13 +5,19 @@from __future__ import annotations+import jsonimport os+import shutil+import subprocess+import threadingimport tomllib+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer+from pathlib import Pathimport pytestfrom harbor.utils.env import resolve_env_vars-from skillevaluator.provider_config import ProviderConfig+from skillevaluator.provider_config import ProviderConfig, ProviderConfigurationError, resolve_llm_providerfrom skillevaluator.tier3.harbor import runnerfrom skillevaluator.tier3.harbor.adapter import _verifier_env_vars@@ -500,6 +506,318 @@ def test_openai_mixed_agents_receive_isolated_native_credentials(monkeypatch: pyassert plans["claude-code"].subprocess_env["OPENAI_API_KEY"] == "provider-key"+ ("provider_name", "env_mode", "configured_base_url", "expected_base_url"),+ [+ ("openai", "docker", "https://agent-gateway.example", "https://agent-gateway.example"),+ ("openai", "e2b", "https://agent-gateway.example/v1", "https://agent-gateway.example"),+ (+ "openai-compatible",+ "docker",+ "https://agent-gateway.example/team/v1/",+ "https://agent-gateway.example/team",+ ),+ ("nv_build", "e2b", "https://agent-gateway.example", "https://agent-gateway.example"),+ ("nv_build", "e2b", "https://agent-gateway.example/v1", "https://agent-gateway.example"),+ (+ "nv_build",+ "e2b",+ "https://agent-gateway.example/team/v1/",+ "https://agent-gateway.example/team",+ ),+ ],+)+def test_independent_claude_base_url_is_canonical_across_the_runtime_plan(+ monkeypatch: pytest.MonkeyPatch,+ provider_name: str,+ env_mode: str,+ configured_base_url: str,+ expected_base_url: str,+) -> None:+ monkeypatch.setattr(+ runner.os,+ "environ",+ {+ "PATH": "/usr/bin",+ "ANTHROPIC_API_KEY": "anthropic-agent-key",+ "ANTHROPIC_BASE_URL": configured_base_url,+ },+ )+ evaluator_provider = _provider(provider_name)++ plan = runner._resolve_agent_runtime_plan(+ provider=evaluator_provider,+ agents=["claude-code"],+ models={"claude-code": "anthropic/claude-sonnet-4-5"},+ configured_runtime_env={},+ env_mode=env_mode,+ model_sources={"claude-code": "CLI"},+ )["claude-code"]++ assert plan.staged_env == {+ "ANTHROPIC_API_KEY": "${ANTHROPIC_API_KEY}",+ "ANTHROPIC_BASE_URL": "${ANTHROPIC_BASE_URL}",+ }+ assert plan.subprocess_env["ANTHROPIC_API_KEY"] == "anthropic-agent-key"+ assert plan.subprocess_env["ANTHROPIC_BASE_URL"] == expected_base_url+ assert plan.provider.provider == "anthropic"+ assert plan.provider.api_key == "anthropic-agent-key"+ assert plan.provider.base_url == expected_base_url+ assert plan.provider.base_url + "/v1/messages" == expected_base_url + "/v1/messages"+ assert "/v1/v1/" not in plan.provider.base_url + "/v1/messages"++ assert evaluator_provider.api_key == "provider-key"+ assert evaluator_provider.base_url == "https://provider.example/v1"+ if provider_name == "nv_build":+ assert plan.subprocess_env["NVIDIA_API_KEY"] == "provider-key"+ assert "OPENAI_API_KEY" not in plan.subprocess_env+ else:+ assert plan.subprocess_env["OPENAI_API_KEY"] == "provider-key"+ assert plan.subprocess_env["OPENAI_BASE_URL"] == "https://provider.example/v1"+++ ("provider_name", "env_mode"),+ [+ ("openai", "docker"),+ ("openai-compatible", "docker"),+ ("nv_build", "e2b"),+ ],+)+ "configured_base_url",+ [+ " ",+ "agent-gateway.example/v1",+ "https://agent-gateway.example/v1/messages",+ "https://url-user:[email protected]/v1",+ "https://agent-gateway.example/v1?token=url-secret",+ "https://agent-gateway.example/v1\n",+ "https://agent-gateway.example/%2576%2531/%256dessages",+ ],+)+def test_invalid_independent_claude_base_url_fails_before_harbor_without_echoing_secrets(+ monkeypatch: pytest.MonkeyPatch,+ provider_name: str,+ env_mode: str,+ configured_base_url: str,+) -> None:+ monkeypatch.setattr(+ runner.os,+ "environ",+ {+ "PATH": "/usr/bin",+ "ANTHROPIC_API_KEY": "anthropic-agent-key",+ "ANTHROPIC_BASE_URL": configured_base_url,+ },+ )++ with pytest.raises(ProviderConfigurationError) as exc_info:+ runner._resolve_agent_runtime_plan(+ provider=_provider(provider_name),+ agents=["claude-code"],+ models={"claude-code": "anthropic/claude-sonnet-4-5"},+ configured_runtime_env={},+ env_mode=env_mode,+ model_sources={"claude-code": "CLI"},+ )++ message = str(exc_info.value)+ assert "ANTHROPIC_BASE_URL" in message+ assert configured_base_url not in message+ assert "url-user" not in message+ assert "url-secret" not in message+ assert "anthropic-agent-key" not in message++[email protected]("configured_base_url", [None, ""])+def test_unset_independent_claude_base_url_remains_absent(+ monkeypatch: pytest.MonkeyPatch,+ configured_base_url: str | None,+) -> None:+ host_env = {+ "PATH": "/usr/bin",+ "ANTHROPIC_API_KEY": "anthropic-agent-key",+ }+ if configured_base_url is not None:+ host_env["ANTHROPIC_BASE_URL"] = configured_base_url+ monkeypatch.setattr(runner.os, "environ", host_env)++ plan = runner._resolve_agent_runtime_plan(+ provider=_provider("openai"),+ agents=["claude-code"],+ models={"claude-code": "anthropic/claude-sonnet-4-5"},+ configured_runtime_env={},+ env_mode="docker",+ model_sources={"claude-code": "CLI"},+ )["claude-code"]++ assert "ANTHROPIC_BASE_URL" not in plan.staged_env+ assert "ANTHROPIC_BASE_URL" not in plan.subprocess_env+ assert plan.provider.base_url is None+++def test_independent_claude_base_url_optional_normalizer_result_is_not_staged(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ monkeypatch.setattr(+ runner.os,+ "environ",+ {+ "PATH": "/usr/bin",+ "ANTHROPIC_API_KEY": "anthropic-agent-key",+ "ANTHROPIC_BASE_URL": "https://agent-gateway.example/v1",+ },+ )++ def no_configured_base_url(_value: str, *, variable: str) -> None:+ assert variable == "ANTHROPIC_BASE_URL"++ monkeypatch.setattr(runner, "_normalize_anthropic_base_url", no_configured_base_url)++ plan = runner._resolve_agent_runtime_plan(+ provider=_provider("openai"),+ agents=["claude-code"],+ models={"claude-code": "anthropic/claude-sonnet-4-5"},+ configured_runtime_env={},+ env_mode="docker",+ model_sources={"claude-code": "CLI"},+ )["claude-code"]++ assert "ANTHROPIC_BASE_URL" not in plan.staged_env+ assert "ANTHROPIC_BASE_URL" not in plan.subprocess_env+ assert plan.provider.base_url is None++[email protected](shutil.which("claude") is None, reason="Installed Claude Code CLI is required")+def test_installed_claude_uses_canonical_independent_gateway_from_runtime_plan(+ monkeypatch: pytest.MonkeyPatch,+ tmp_path: Path,+) -> None:+ request_paths: list[str] = []+ response_events = [+ (+ "message_start",+ {+ "type": "message_start",+ "message": {+ "id": "msg_test",+ "type": "message",+ "role": "assistant",+ "model": "claude-sonnet-4-5",+ "content": [],+ "stop_reason": None,+ "stop_sequence": None,+ "usage": {"input_tokens": 1, "output_tokens": 0},+ },+ },+ ),+ (+ "content_block_start",+ {+ "type": "content_block_start",+ "index": 0,+ "content_block": {"type": "text", "text": ""},+ },+ ),+ (+ "content_block_delta",+ {+ "type": "content_block_delta",+ "index": 0,+ "delta": {"type": "text_delta", "text": "gateway-ok"},+ },+ ),+ ("content_block_stop", {"type": "content_block_stop", "index": 0}),+ (+ "message_delta",+ {+ "type": "message_delta",+ "delta": {"stop_reason": "end_turn", "stop_sequence": None},+ "usage": {"output_tokens": 1},+ },+ ),+ ("message_stop", {"type": "message_stop"}),+ ]+ response_body = "".join(+ f"event: {event}\ndata: {json.dumps(payload, separators=(',', ':'))}\n\n" for event, payload in response_events+ ).encode()++ class RecordingHandler(BaseHTTPRequestHandler):+ def log_message(self, _format: str, *args: object) -> None:+ del args++ def do_POST(self) -> None:+ content_length = int(self.headers.get("content-length", "0"))+ self.rfile.read(content_length)+ request_paths.append(self.path.partition("?")[0])+ self.send_response(200)+ self.send_header("content-type", "text/event-stream")+ self.send_header("content-length", str(len(response_body)))+ self.end_headers()+ self.wfile.write(response_body)++ claude_path = shutil.which("claude")+ assert claude_path is not None+ with ThreadingHTTPServer(("127.0.0.1", 0), RecordingHandler) as server:+ server_thread = threading.Thread(target=server.serve_forever, daemon=True)+ server_thread.start()+ configured_base_url = f"http://127.0.0.1:{server.server_port}/team/v1"+ monkeypatch.setattr(+ runner.os,+ "environ",+ {+ "PATH": os.environ["PATH"],+ "HOME": str(tmp_path),+ "CLAUDE_CONFIG_DIR": str(tmp_path / "claude"),+ "ANTHROPIC_API_KEY": "test-agent-key",+ "ANTHROPIC_BASE_URL": configured_base_url,+ },+ )+ plan = runner._resolve_agent_runtime_plan(+ provider=_provider("openai"),+ agents=["claude-code"],+ models={"claude-code": "anthropic/claude-sonnet-4-5"},+ configured_runtime_env={},+ env_mode="docker",+ model_sources={"claude-code": "CLI"},+ )["claude-code"]++ try:+ result = subprocess.run(+ [+ claude_path,+ "--bare",+ "--print",+ "--no-session-persistence",+ "--disable-slash-commands",+ "--model",+ plan.provider.model,+ "Reply only with gateway-ok",+ ],+ cwd=tmp_path,+ env=dict(plan.subprocess_env),+ capture_output=True,+ text=True,+ timeout=30,+ check=False,+ )+ finally:+ server.shutdown()+ server_thread.join(timeout=5)++ expected_root = f"http://127.0.0.1:{server.server_port}/team"+ assert plan.provider.base_url == expected_root+ assert plan.subprocess_env["ANTHROPIC_BASE_URL"] == expected_root+ assert plan.subprocess_env["ANTHROPIC_API_KEY"] == "test-agent-key"+ assert plan.subprocess_env["OPENAI_API_KEY"] == "provider-key"+ assert result.returncode == 0, result.stderr+ assert result.stdout.strip() == "gateway-ok"+ assert request_paths+ assert set(request_paths) == {"/team/v1/messages"}++def test_anthropic_mixed_agents_receive_isolated_native_credentials(monkeypatch: pytest.MonkeyPatch) -> None:monkeypatch.setattr(runner.os,@@ -536,6 +854,46 @@ def test_anthropic_mixed_agents_receive_isolated_native_credentials(monkeypatch:assert plans["codex"].subprocess_env["ANTHROPIC_API_KEY"] == "provider-key"+def test_anthropic_legacy_v1_base_is_canonical_for_verifier_and_native_agents(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ monkeypatch.setattr(runner.os, "environ", {"PATH": "/usr/bin"})+ provider = resolve_llm_provider(+ {+ "SKILL_EVAL_LLM_PROVIDER": "anthropic",+ "ANTHROPIC_API_KEY": "anthropic-provider-key",+ "ANTHROPIC_BASE_URL": "https://gateway.example/team/v1/",+ }+ )++ verifier_env = runner._provider_environment(provider)+ plans = runner._resolve_agent_runtime_plan(+ provider=provider,+ agents=["claude-code", "opencode"],+ models={+ "claude-code": "claude-sonnet-4-5",+ "opencode": "anthropic/claude-sonnet-4-5",+ },+ configured_runtime_env={},+ env_mode="docker",+ model_sources={"claude-code": "public provider default", "opencode": "CLI"},+ )++ assert provider.base_url == "https://gateway.example/team"+ assert verifier_env["ANTHROPIC_BASE_URL"] == "https://gateway.example/team"+ for agent in ("claude-code", "opencode"):+ assert plans[agent].staged_env == {+ "ANTHROPIC_API_KEY": "${ANTHROPIC_API_KEY}",+ "ANTHROPIC_BASE_URL": "${ANTHROPIC_BASE_URL}",+ }+ assert plans[agent].subprocess_env["ANTHROPIC_BASE_URL"] == "https://gateway.example/team"+ assert plans[agent].provider.base_url == "https://gateway.example/team"+ assert plans[agent].subprocess_env["ANTHROPIC_BASE_URL"] + "/v1/messages" == (+ "https://gateway.example/team/v1/messages"+ )+ assert "/v1/v1/" not in plans[agent].subprocess_env["ANTHROPIC_BASE_URL"] + "/v1/messages"++def test_openai_compatible_codex_stages_selected_provider_pair() -> None:plans = runner._resolve_agent_runtime_plan(provider=_provider("openai-compatible"),diff --git a/tests/test_harbor_runtime_skill_isolation.py b/tests/test_harbor_runtime_skill_isolation.pyindex 6140d229..07aae84f 100644--- a/tests/test_harbor_runtime_skill_isolation.py+++ b/tests/test_harbor_runtime_skill_isolation.py@@ -2809,6 +2809,7 @@ def test_verifier_constraint_and_skill_requirements_share_resolver_transaction(tinstall_line = next(line for line in dockerfile.splitlines() if '"pip", "install"' in line)assert "langchain-community<0.4.2" in install_lineassert "langchain-community>=0.4.2" in install_line+ assert "idna>=3.10,<4" in install_lineassert dockerfile.count('"pip", "install"') == 1assert "from ragas.llms.base import llm_factory" in dockerfilediff --git a/tests/test_issue55_collector_truth.py b/tests/test_issue55_collector_truth.pynew file mode 100644index 00000000..ed2caa7c--- /dev/null+++ b/tests/test_issue55_collector_truth.py@@ -0,0 +1,2404 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""End-to-end collector truthfulness regressions for issue #55."""++from __future__ import annotations++import json+import os+import re+import subprocess+import sys+from pathlib import Path++import pytest++from skillevaluator.evaluation.tier3_report import render_agent_eval_html_report+from skillevaluator.tier3.harbor import collector as collector_module+from skillevaluator.tier3.harbor import report, report_data+from skillevaluator.tier3.harbor.collector import (+ _copy_trial_artifacts,+ _extract_rewards,+ _write_redacted_text_copy,+ collect_harbor_results,+)+from skillevaluator.tier3.harbor.metrics import (+ CUSTOM_ONLY_METRIC_SET,+ DEFAULT_METRIC_SET,+ DEFAULT_METRICS,+ LEGACY_METRIC_SET,+ LEGACY_METRICS,+ overall_score,+)+++def _write_complete_job_result(job_dir: Path, trial_names: list[str]) -> None:+ job_dir.mkdir(parents=True, exist_ok=True)+ (job_dir / "result.json").write_text(+ json.dumps(+ {+ "n_total_trials": len(trial_names),+ "stats": {+ "n_completed_trials": len(trial_names),+ "n_errored_trials": 0,+ "n_running_trials": 0,+ "n_pending_trials": 0,+ "n_cancelled_trials": 0,+ "n_retries": 0,+ "evals": {+ "agent__model___harbor-tasks": {+ "n_trials": len(trial_names),+ "n_errors": 0,+ "reward_stats": {"reward": {"0.1": trial_names}},+ }+ },+ },+ }+ ),+ encoding="utf-8",+ )+++def _default_reward(entry_id: str, score: float) -> dict[str, object]:+ return {+ "entry_id": entry_id,+ "metric_set": DEFAULT_METRIC_SET,+ **dict.fromkeys(DEFAULT_METRICS, score),+ "overall": score,+ "details": {+ "goal_accuracy": {+ "score": score,+ "reason": "the requested result was not produced" if score < 0.8 else "goal completed",+ }+ },+ }+++def _write_reward(+ job_dir: Path,+ trial_name: str,+ reward: dict[str, object],+ *,+ sidecar: dict[str, object] | None = None,+ custom: dict[str, object] | None = None,+) -> None:+ verifier = job_dir / trial_name / "verifier"+ verifier.mkdir(parents=True, exist_ok=True)+ (verifier / "reward.json").write_text(json.dumps(reward), encoding="utf-8")+ if sidecar is not None:+ (verifier / "skill_evaluator_reward.json").write_text(json.dumps(sidecar), encoding="utf-8")+ if custom is not None:+ (verifier / "custom_reward.json").write_text(json.dumps(custom), encoding="utf-8")+++def _failed_judge_artifacts(entry_id: str, *, reason: str = "judge request returned HTTP 401") -> tuple[dict, dict]:+ numeric = {+ "security": 1.0,+ "skill_execution": 1.0,+ "skill_efficiency": 1.0,+ "overall": 0.0,+ }+ rich = {+ "entry_id": entry_id,+ "metric_set": DEFAULT_METRIC_SET,+ "security": 1.0,+ "skill_execution": 1.0,+ "skill_efficiency": 1.0,+ "accuracy": None,+ "goal_accuracy": None,+ "behavior_check": None,+ "evaluation_status": "failed",+ "evaluation_errors": {+ "accuracy": reason,+ "goal_accuracy": "judge response was malformed",+ "behavior_check": "judge timed out",+ },+ }+ return numeric, rich+++def _write_authoritative_multistep_result(+ job_dir: Path,+ trial_name: str,+ *,+ aggregate: dict[str, object],+ step_rewards: list[dict[str, object]],+) -> Path:+ trial_dir = job_dir / trial_name+ trial_dir.mkdir(parents=True, exist_ok=True)+ (trial_dir / "result.json").write_text(+ json.dumps(+ {+ "trial_name": trial_name,+ "task_name": "case-001",+ "verifier_result": {"rewards": aggregate},+ "step_results": [+ {"step_name": f"step-{index}", "verifier_result": {"rewards": rewards}}+ for index, rewards in enumerate(step_rewards, start=1)+ ],+ }+ ),+ encoding="utf-8",+ )+ _write_complete_job_result(job_dir, [trial_name])+ return trial_dir+++def _collect(+ tmp_path: Path,+ *,+ skip_baseline: bool,+ case_ids: list[str],+) -> dict[str, object]:+ return collect_harbor_results(+ skill_name="demo",+ agents=["opencode"],+ output_dir=tmp_path / "results",+ jobs_dir=tmp_path / "jobs",+ skip_baseline=skip_baseline,+ expected_cases=len(case_ids),+ expected_case_ids=case_ids,+ expected_trials=len(case_ids),+ )+++def test_failed_judge_sidecar_is_merged_but_never_scored_and_reason_is_safe(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ redaction_marker = "synthetic-redaction-marker-123456"+ numeric, rich = _failed_judge_artifacts(+ "case-001",+ reason=f"Authorization: Bearer {redaction_marker}; \x00\x1b" + ("upstream unavailable " * 100),+ )+ _write_reward(job_dir, trial_name, numeric, sidecar=rich)+ _write_complete_job_result(job_dir, [trial_name])++ extracted = _extract_rewards(job_dir)+ assert extracted[0]["evaluation_status"] == "failed"+ assert extracted[0]["evaluation_errors"]["accuracy"].startswith("Authorization:")++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ agent = result["agents"]["opencode"]+ assert result["execution_status"] == "failed"+ assert agent["conditions"]["with_skill"]["scored_attempts"] == 0+ assert agent["with_skill"] == {}+ assert agent["custom_with_skill"] == {}+ [failure] = agent["trial_failures"]["with_skill"]+ assert failure["trial"] == trial_name+ assert "Required judge evaluation failed" in failure["reason"]+ assert "accuracy" in failure["reason"]+ assert redaction_marker not in failure["reason"]+ assert "<redacted>" in failure["reason"]+ assert len(failure["reason"]) <= 2048+ persisted_files = [path for path in (tmp_path / "results").rglob("*") if path.is_file()]+ assert persisted_files+ assert all(redaction_marker not in path.read_text(encoding="utf-8") for path in persisted_files)+ persisted_reward = json.loads(+ (tmp_path / "results" / "opencode" / "with-skill" / "trials" / trial_name / "reward.json").read_text(+ encoding="utf-8"+ )+ )+ saved_reason = persisted_reward["evaluation_errors"]["accuracy"]+ assert "<redacted>" in saved_reason+ assert len(saved_reason) <= 512+ assert not any(ord(character) < 32 or ord(character) == 127 for character in saved_reason)++[email protected]("verifier_relative", [Path("verifier"), Path("steps/finish/verifier")])+def test_failed_harbor_trial_preserves_redacted_judge_diagnostics(+ tmp_path: Path,+ verifier_relative: Path,+) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ trial_dir = job_dir / trial_name+ verifier_dir = trial_dir / verifier_relative+ redaction_marker = "".join(("AKIA", "IOSFODNN7", "EXAMPLE")) # noqa: FLY002 - keep scanners quiet+ numeric, rich = _failed_judge_artifacts(+ "case-001",+ reason=f"judge returned HTTP 401 with {redaction_marker}",+ )+ verifier_dir.mkdir(parents=True)+ (verifier_dir / "reward.json").write_text(json.dumps(numeric), encoding="utf-8")+ (verifier_dir / "skill_evaluator_reward.json").write_text(json.dumps(rich), encoding="utf-8")+ (trial_dir / "result.json").write_text(+ json.dumps(+ {+ "trial_name": trial_name,+ "task_name": "case-001",+ "exception_info": {+ "exception_type": "VerifierError",+ "exception_message": "verifier exited nonzero",+ },+ }+ ),+ encoding="utf-8",+ )+ (trial_dir / "exception.txt").write_text("VerifierError: verifier exited nonzero\n", encoding="utf-8")+ _write_complete_job_result(job_dir, [trial_name])++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == "failed"+ [failure] = result["agents"]["opencode"]["trial_failures"]["with_skill"]+ assert "HTTP 401" in failure["reason"]+ assert redaction_marker not in failure["reason"]+ assert "<redacted>" in failure["reason"]+ persisted_reward_path = tmp_path / "results" / "opencode" / "with-skill" / "trials" / trial_name / "reward.json"+ persisted_reward = json.loads(persisted_reward_path.read_text(encoding="utf-8"))+ assert persisted_reward["evaluation_status"] == "failed"+ assert overall_score(persisted_reward) is None+ persisted_text = json.dumps(persisted_reward)+ assert "HTTP 401" in persisted_text+ assert redaction_marker not in persisted_text+ assert "<redacted>" in persisted_text+++def test_failed_judge_without_sidecar_still_fails_closed_with_generic_reason(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ numeric, _rich = _failed_judge_artifacts("case-001")+ _write_reward(job_dir, trial_name, numeric)+ _write_complete_job_result(job_dir, [trial_name])++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ agent = result["agents"]["opencode"]+ assert result["execution_status"] == "failed"+ assert agent["conditions"]["with_skill"]["scored_attempts"] == 0+ assert agent["trial_failures"]["with_skill"] == [+ {+ "trial": trial_name,+ "reason": "Reward metrics are incomplete or non-finite; trial was not scored",+ }+ ]+++def test_failed_judge_does_not_depend_on_expected_coverage_options(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ numeric, rich = _failed_judge_artifacts("case-001")+ _write_reward(job_dir, trial_name, numeric, sidecar=rich)+ _write_complete_job_result(job_dir, [trial_name])++ result = collect_harbor_results(+ skill_name="demo",+ agents=["opencode"],+ output_dir=tmp_path / "results",+ jobs_dir=tmp_path / "jobs",+ skip_baseline=True,+ )++ assert result["execution_status"] == "failed"+ assert result["agents"]["opencode"]["conditions"]["with_skill"]["scored_attempts"] == 0+ assert "Required judge evaluation failed" in " ".join(result["execution_errors"])+++def test_incomplete_reward_does_not_depend_on_expected_coverage_options(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ numeric, _rich = _failed_judge_artifacts("case-001")+ secret = "sk-incompletejudgecredential"+ numeric["error"] = f"provider echoed {secret} " + ("unavailable " * 1000)+ _write_reward(job_dir, trial_name, numeric)+ _write_complete_job_result(job_dir, [trial_name])++ result = collect_harbor_results(+ skill_name="demo",+ agents=["opencode"],+ output_dir=tmp_path / "results",+ jobs_dir=tmp_path / "jobs",+ skip_baseline=True,+ )++ assert result["execution_status"] == "failed"+ assert result["agents"]["opencode"]["conditions"]["with_skill"]["scored_attempts"] == 0+ assert "Reward metrics are incomplete or non-finite" in " ".join(result["execution_errors"])+ persisted = json.loads(+ (tmp_path / "results" / "opencode" / "with-skill" / "trials" / trial_name / "reward.json").read_text(+ encoding="utf-8"+ )+ )+ assert secret not in persisted["error"]+ assert "<redacted>" in persisted["error"]+ assert len(persisted["error"]) <= 8192+++def test_authoritative_multistep_reward_honors_failed_step_sidecar(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ trial_dir = job_dir / trial_name+ numeric, rich = _failed_judge_artifacts("case-001")+ # A complete aggregate must not hide a failed judge from one constituent step.+ aggregate = _default_reward("case-001", 1.0)+ (trial_dir / "result.json").parent.mkdir(parents=True)+ (trial_dir / "result.json").write_text(+ json.dumps(+ {+ "trial_name": trial_name,+ "task_name": "case-001",+ "verifier_result": {"rewards": aggregate},+ "step_results": [+ {"step_name": "prepare", "verifier_result": {"rewards": aggregate}},+ {"step_name": "finish", "verifier_result": {"rewards": numeric}},+ ],+ }+ ),+ encoding="utf-8",+ )+ _write_reward(job_dir, trial_name + "/steps/finish", numeric, sidecar=rich)+ _write_complete_job_result(job_dir, [trial_name])++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ agent = result["agents"]["opencode"]+ assert result["execution_status"] == "failed"+ assert agent["conditions"]["with_skill"]["scored_attempts"] == 0+ assert "Required judge evaluation failed" in agent["trial_failures"]["with_skill"][0]["reason"]+++ ("invalid_step", "case_name"),+ [+ (+ {+ "security": 1.0,+ "skill_execution": 1.0,+ "skill_efficiency": 1.0,+ "overall": 0.0,+ },+ "incomplete",+ ),+ ({**_default_reward("case-001", 1.0), "accuracy": float("nan")}, "non-finite"),+ ({**dict.fromkeys(DEFAULT_METRICS, float("nan")), "overall": 0.0}, "all-non-finite-unversioned"),+ ({**_default_reward("case-001", 1.0), "evaluation_status": "failed"}, "failed-status"),+ ],+ ids=lambda value: value if isinstance(value, str) else None,+)+def test_authoritative_multistep_reward_rejects_invalid_default_constituent_without_sidecar(+ tmp_path: Path,+ invalid_step: dict[str, object],+ case_name: str,+) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ _write_authoritative_multistep_result(+ job_dir,+ trial_name,+ aggregate=_default_reward("case-001", 1.0),+ step_rewards=[_default_reward("case-001", 1.0), invalid_step],+ )++ [extracted] = _extract_rewards(job_dir)+ assert extracted["evaluation_status"] == "failed", case_name++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ agent = result["agents"]["opencode"]+ assert result["execution_status"] == "failed"+ assert agent["conditions"]["with_skill"]["scored_attempts"] == 0+ assert agent["with_skill"] == {}+ [failure] = agent["trial_failures"]["with_skill"]+ assert "constituent" in failure["reason"].casefold()+ assert len(failure["reason"]) <= 2048++[email protected]("status_location", ["result", "verifier", "reward"])+def test_authoritative_root_failed_status_is_never_scored(tmp_path: Path, status_location: str) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_dir = _write_authoritative_multistep_result(+ job_dir,+ "case-001__attempt",+ aggregate=_default_reward("case-001", 1.0),+ step_rewards=[_default_reward("case-001", 1.0)],+ )+ result_path = trial_dir / "result.json"+ payload = json.loads(result_path.read_text(encoding="utf-8"))+ if status_location == "result":+ payload["evaluation_status"] = "failed"+ elif status_location == "verifier":+ payload["verifier_result"]["evaluation_status"] = "failed"+ else:+ payload["verifier_result"]["rewards"]["evaluation_status"] = "failed"+ result_path.write_text(json.dumps(payload), encoding="utf-8")++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == "failed"+ assert result["agents"]["opencode"]["conditions"]["with_skill"]["scored_attempts"] == 0+ assert "authoritative" in " ".join(result["execution_errors"]).casefold()+++ "verifier_result",+ [+ {"evaluation_status": "failed"},+ {"evaluation_status": "failed", "rewards": {}},+ {"rewards": {}},+ ],+ ids=("missing-failed-reward", "empty-failed-reward", "empty-default-reward"),+)+def test_authoritative_default_aggregate_rejects_missing_or_empty_step_verifier_reward(+ tmp_path: Path,+ verifier_result: dict[str, object],+) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_dir = _write_authoritative_multistep_result(+ job_dir,+ "case-001__attempt",+ aggregate=_default_reward("case-001", 1.0),+ step_rewards=[_default_reward("case-001", 1.0)],+ )+ payload = json.loads((trial_dir / "result.json").read_text(encoding="utf-8"))+ payload["step_results"].append({"step_name": "hidden-failure", "verifier_result": verifier_result})+ (trial_dir / "result.json").write_text(json.dumps(payload), encoding="utf-8")++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == "failed"+ assert result["agents"]["opencode"]["conditions"]["with_skill"]["scored_attempts"] == 0+ assert "constituent" in " ".join(result["execution_errors"]).casefold()+++def test_authoritative_default_aggregate_allows_agent_only_step_without_verifier_result(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_dir = _write_authoritative_multistep_result(+ job_dir,+ "case-001__attempt",+ aggregate=_default_reward("case-001", 0.8),+ step_rewards=[_default_reward("case-001", 0.8)],+ )+ payload = json.loads((trial_dir / "result.json").read_text(encoding="utf-8"))+ payload["step_results"].append({"step_name": "agent-only", "verifier_result": None})+ (trial_dir / "result.json").write_text(json.dumps(payload), encoding="utf-8")++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == "succeeded"+ assert result["agents"]["opencode"]["conditions"]["with_skill"]["scored_attempts"] == 1+++def test_authoritative_default_aggregate_rejects_real_harbor_step_exception_without_reward(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_dir = _write_authoritative_multistep_result(+ job_dir,+ "case-001__attempt",+ aggregate=_default_reward("case-001", 1.0),+ step_rewards=[_default_reward("case-001", 1.0)],+ )+ payload = json.loads((trial_dir / "result.json").read_text(encoding="utf-8"))+ payload["step_results"].append(+ {+ "step_name": "judge-crashed",+ "agent_result": {},+ "verifier_result": None,+ "exception_info": {+ "exception_type": "RuntimeError",+ "exception_message": "judge unavailable",+ },+ }+ )+ (trial_dir / "result.json").write_text(json.dumps(payload), encoding="utf-8")++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == "failed"+ assert result["agents"]["opencode"]["conditions"]["with_skill"]["scored_attempts"] == 0+ assert "constituent" in " ".join(result["execution_errors"]).casefold()+++def test_authoritative_default_aggregate_rejects_present_empty_step_exception(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_dir = _write_authoritative_multistep_result(+ job_dir,+ "case-001__attempt",+ aggregate=_default_reward("case-001", 1.0),+ step_rewards=[_default_reward("case-001", 1.0)],+ )+ payload = json.loads((trial_dir / "result.json").read_text(encoding="utf-8"))+ payload["step_results"].append(+ {"step_name": "invalid-exception", "agent_result": {}, "verifier_result": None, "exception_info": {}}+ )+ (trial_dir / "result.json").write_text(json.dumps(payload), encoding="utf-8")++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == "failed"+ assert result["agents"]["opencode"]["conditions"]["with_skill"]["scored_attempts"] == 0++[email protected]("invalid_step", [None, "malformed-step"], ids=("null", "string"))+def test_authoritative_default_aggregate_rejects_malformed_step_entry(+ tmp_path: Path,+ invalid_step: object,+) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_dir = _write_authoritative_multistep_result(+ job_dir,+ "case-001__attempt",+ aggregate=_default_reward("case-001", 1.0),+ step_rewards=[_default_reward("case-001", 1.0)],+ )+ payload = json.loads((trial_dir / "result.json").read_text(encoding="utf-8"))+ payload["step_results"].append(invalid_step)+ (trial_dir / "result.json").write_text(json.dumps(payload), encoding="utf-8")++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == "failed"+ assert result["agents"]["opencode"]["conditions"]["with_skill"]["scored_attempts"] == 0+ assert "constituent" in " ".join(result["execution_errors"]).casefold()++[email protected]("invalid_steps", [{}, "malformed-steps"], ids=("mapping", "string"))+def test_authoritative_default_aggregate_rejects_malformed_step_results_container(+ tmp_path: Path,+ invalid_steps: object,+) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_dir = _write_authoritative_multistep_result(+ job_dir,+ "case-001__attempt",+ aggregate=_default_reward("case-001", 1.0),+ step_rewards=[_default_reward("case-001", 1.0)],+ )+ payload = json.loads((trial_dir / "result.json").read_text(encoding="utf-8"))+ payload["step_results"] = invalid_steps+ (trial_dir / "result.json").write_text(json.dumps(payload), encoding="utf-8")++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == "failed"+ assert result["agents"]["opencode"]["conditions"]["with_skill"]["scored_attempts"] == 0+ assert "authoritative" in " ".join(result["execution_errors"]).casefold()+++def test_v2_root_aggregate_does_not_reclassify_constituent_missing_security_as_legacy(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ undeclared_five_metric_step = dict.fromkeys(LEGACY_METRICS, 1.0)+ _write_authoritative_multistep_result(+ job_dir,+ "case-001__attempt",+ aggregate=_default_reward("case-001", 1.0),+ step_rewards=[_default_reward("case-001", 1.0), undeclared_five_metric_step],+ )++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == "failed"+ assert result["agents"]["opencode"]["conditions"]["with_skill"]["scored_attempts"] == 0+ assert "constituent" in " ".join(result["execution_errors"]).casefold()+++ ("metric_set", "metrics"),+ [(DEFAULT_METRIC_SET, DEFAULT_METRICS), (LEGACY_METRIC_SET, LEGACY_METRICS)],+ ids=("default-v2", "legacy-v1"),+)+def test_nested_standard_step_fallback_preserves_metrics_instead_of_trusting_overall(+ tmp_path: Path,+ metric_set: str,+ metrics: tuple[str, ...],+) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ trial_dir = job_dir / trial_name+ trial_dir.mkdir(parents=True)+ nested_metrics = {metric: {"score": 0.0} for metric in metrics}+ (trial_dir / "result.json").write_text(+ json.dumps(+ {+ "trial_name": trial_name,+ "task_name": "case-001",+ "step_results": [+ {+ "step_name": "nested-default",+ "verifier_result": {+ "rewards": {+ "metric_set": metric_set,+ "metrics": nested_metrics,+ "overall": 1.0,+ }+ },+ }+ ],+ }+ ),+ encoding="utf-8",+ )+ _write_complete_job_result(job_dir, [trial_name])++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ agent = result["agents"]["opencode"]+ assert result["execution_status"] == "succeeded"+ assert agent["with_skill"] == dict.fromkeys(metrics, 0.0)+ assert agent["custom_with_skill"] == {}+ assert agent["pass_at_k"]["with_skill"]["rate"] == 0.0+ assert agent["pass_at_k"]["with_skill"]["cases"]["case-001"]["best_score"] == 0.0+++def test_custom_only_step_fallback_remains_scoreable(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ trial_dir = job_dir / trial_name+ trial_dir.mkdir(parents=True)+ (trial_dir / "result.json").write_text(+ json.dumps(+ {+ "trial_name": trial_name,+ "task_name": "case-001",+ "step_results": [+ {+ "step_name": "custom",+ "verifier_result": {"rewards": {"overall": 0.75, "domain_quality": 0.9}},+ }+ ],+ }+ ),+ encoding="utf-8",+ )+ _write_complete_job_result(job_dir, [trial_name])++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ agent = result["agents"]["opencode"]+ assert result["execution_status"] == "succeeded"+ assert agent["with_skill"] == {}+ assert agent["custom_with_skill"] == {"domain_quality": 0.9}+ assert agent["pass_at_k"]["with_skill"]["rate"] == 1.0+++def test_custom_only_authoritative_reward_ignores_standard_judge_sidecar_scan_bound(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_dir = _write_authoritative_multistep_result(+ job_dir,+ "case-001__attempt",+ aggregate={"overall": 0.75, "domain_quality": 0.8},+ step_rewards=[{"overall": 0.75, "domain_quality": 0.8}],+ )+ steps_dir = trial_dir / "steps"+ for index in range(collector_module._MAX_FAILED_JUDGE_SIDECARS):+ (steps_dir / f"custom-{index:03d}" / "verifier").mkdir(parents=True)++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ agent = result["agents"]["opencode"]+ assert result["execution_status"] == "succeeded"+ assert agent["conditions"]["with_skill"]["scored_attempts"] == 1+ assert agent["custom_with_skill"] == {"domain_quality": 0.8}+++def test_nested_custom_only_step_fallback_preserves_custom_metrics(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ trial_dir = job_dir / trial_name+ trial_dir.mkdir(parents=True)+ (trial_dir / "result.json").write_text(+ json.dumps(+ {+ "trial_name": trial_name,+ "task_name": "case-001",+ "step_results": [+ {+ "step_name": "custom",+ "verifier_result": {+ "rewards": {+ "metric_set": CUSTOM_ONLY_METRIC_SET,+ "metrics": {"domain_quality": {"score": 0.9}},+ "overall": 0.75,+ }+ },+ }+ ],+ }+ ),+ encoding="utf-8",+ )+ _write_complete_job_result(job_dir, [trial_name])++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ agent = result["agents"]["opencode"]+ assert result["execution_status"] == "succeeded"+ assert agent["with_skill"] == {}+ assert agent["custom_with_skill"] == {"domain_quality": 0.9}+ assert agent["pass_at_k"]["with_skill"]["rate"] == 1.0+++def test_step_fallback_aggregates_each_logical_trial_before_cross_trial_average(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_scores = {+ "case-a__attempt": ("case-a", [0.4, 0.4, 1.0]),+ "case-b__attempt": ("case-b", [1.0]),+ }+ for trial_name, (case_id, scores) in trial_scores.items():+ trial_dir = job_dir / trial_name+ trial_dir.mkdir(parents=True)+ step_results: list[dict[str, object]] = []+ for index, score in enumerate(scores, start=1):+ step_name = f"step-{index}"+ reward = _default_reward(case_id, score)+ reward["details"] = {"accuracy": {"reason": "accuracy missed" if score < 0.8 else "accurate result"}}+ verifier_dir = trial_dir / "steps" / step_name / "verifier"+ verifier_dir.mkdir(parents=True)+ (verifier_dir / "reward.json").write_text(json.dumps(reward), encoding="utf-8")+ step_results.append({"step_name": step_name, "verifier_result": {"rewards": reward}})+ (trial_dir / "result.json").write_text(+ json.dumps({"trial_name": trial_name, "task_name": case_id, "step_results": step_results}),+ encoding="utf-8",+ )+ _write_complete_job_result(job_dir, list(trial_scores))++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-a", "case-b"])++ agent = result["agents"]["opencode"]+ assert result["execution_status"] == "succeeded"+ assert agent["conditions"]["with_skill"]["scored_attempts"] == 2+ # The public count remains the number of persisted reward rows for backward+ # compatibility; scoring and pass@k use one aggregate per logical trial.+ assert agent["num_trials_with"] == 4+ assert agent["with_skill"] == dict.fromkeys(DEFAULT_METRICS, 0.8)+ summary = json.loads(+ (tmp_path / "results" / "opencode" / "with-skill" / "summary.json").read_text(encoding="utf-8")+ )+ assert summary["num_trials"] == 4+ assert len(list((tmp_path / "results" / "opencode" / "with-skill" / "trials").glob("*/reward.json"))) == 4++ # Findings use bounded raw rows for evidence, but their score must remain+ # the complete collector summary even when an entire logical trial is past+ # the report-loader limit.+ monkeypatch.setattr(report_data, "_MAX_TRIALS_PER_CONDITION", 3)+ monkeypatch.setattr(report, "_generate_suggestions_structured", lambda *_args, **_kwargs: [])+ monkeypatch.setattr(report, "_passing_skill_suggestions", lambda *_args, **_kwargs: [])+ assert report.display_findings_report(result, "demo", ["opencode"], tmp_path / "results")+ findings_payload = json.loads((tmp_path / "results" / "opencode" / "findings.json").read_text(encoding="utf-8"))+ [accuracy_finding] = [finding for finding in findings_payload["findings"] if finding["metric"] == "accuracy"]+ assert accuracy_finding["score"] == agent["with_skill"]["accuracy"]+ assert accuracy_finding["severity"] == "ok"+++def test_saved_fallback_steps_keep_logical_trial_weight_in_findings_and_html(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ with_trial_scores = {+ "case-a__attempt": ("case-a", [0.4, 0.4, 1.0]),+ "case-b__attempt": ("case-b", [1.0]),+ }+ without_trial_scores = {+ "case-a__attempt": ("case-a", [0.6]),+ "case-b__attempt": ("case-b", [0.8, 0.8, 0.8]),+ }++ def write_job(variant: str, trial_scores: dict[str, tuple[str, list[float]]]) -> None:+ job_dir = tmp_path / "jobs" / f"demo-opencode-{variant}"+ for trial_name, (case_id, scores) in trial_scores.items():+ trial_dir = job_dir / trial_name+ trial_dir.mkdir(parents=True)+ step_results: list[dict[str, object]] = []+ for index, score in enumerate(scores, start=1):+ step_name = f"step-{index}"+ reward = {+ "entry_id": case_id,+ "metric_set": CUSTOM_ONLY_METRIC_SET,+ "metrics": {"domain_quality": {"score": score}},+ "overall": score,+ }+ if case_id == "case-a":+ reward["custom_details"] = {+ "domain_quality": {+ "reason": "needs improvement" if score < 0.8 else "quality target met",+ }+ }+ verifier_dir = trial_dir / "steps" / step_name / "verifier"+ verifier_dir.mkdir(parents=True)+ (verifier_dir / "reward.json").write_text(json.dumps(reward), encoding="utf-8")+ step_results.append({"step_name": step_name, "verifier_result": {"rewards": reward}})+ (trial_dir / "result.json").write_text(+ json.dumps({"trial_name": trial_name, "task_name": case_id, "step_results": step_results}),+ encoding="utf-8",+ )+ _write_complete_job_result(job_dir, list(trial_scores))++ write_job("with", with_trial_scores)+ write_job("without", without_trial_scores)++ result = _collect(tmp_path, skip_baseline=False, case_ids=["case-a", "case-b"])++ agent = result["agents"]["opencode"]+ assert agent["custom_with_skill"] == {"domain_quality": 0.8}+ assert agent["custom_without_skill"] == {"domain_quality": 0.7}+ assert agent["custom_lift"]["overall"] == {+ "with_skill": 0.8,+ "without_skill": 0.7,+ "delta": 0.1,+ "direction": "up",+ }+ assert agent["pass_at_k"]["with_skill"]["cases"]["case-a"]["best_score"] == 0.6+ assert agent["pass_at_k"]["with_skill"]["cases"]["case-b"]["best_score"] == 1.0+ # Keep all four step artifacts for diagnostics; only their score weighting+ # is logical-trial based.+ reward_paths = list((tmp_path / "results" / "opencode" / "with-skill" / "trials").glob("*/reward.json"))+ assert len(reward_paths) == 4+ persisted_rewards = [json.loads(path.read_text(encoding="utf-8")) for path in reward_paths]+ assert len({reward["trial_id"] for reward in persisted_rewards}) == 2++ monkeypatch.setattr(report_data, "_MAX_TRIALS_PER_CONDITION", 3)+ monkeypatch.setattr(report, "_generate_suggestions_structured", lambda *_args, **_kwargs: [])+ monkeypatch.setattr(report, "_passing_skill_suggestions", lambda *_args, **_kwargs: [])+ assert report.display_findings_report(result, "demo", ["opencode"], tmp_path / "results")+ findings_payload = json.loads((tmp_path / "results" / "opencode" / "findings.json").read_text(encoding="utf-8"))+ [domain_finding] = [finding for finding in findings_payload["findings"] if finding["metric"] == "domain_quality"]+ assert domain_finding["score"] == agent["custom_with_skill"]["domain_quality"]+ assert domain_finding["severity"] == "ok"++ skill_dir = tmp_path / "demo-skill"+ skill_dir.mkdir()+ report_path = render_agent_eval_html_report(+ skill_dir,+ tmp_path / "results",+ use_llm_judge=False,+ )+ output = report_path.read_text(encoding="utf-8")+ payload_match = re.search(+ r'<script type="application/json" id="tier3-full">(.*?)</script>',+ output,+ re.DOTALL,+ )+ assert payload_match is not None+ report_payload = json.loads(payload_match.group(1))+ assert report_payload["overall_score"] == 0.8+ assert report_payload["overall_lift"] == 0.1+ # Custom-only metrics retain their score and lift, but they do not satisfy+ # the canonical five-dimension publication gate introduced on main.+ assert report_payload["verdict"] == "neutral"+ assert report_payload["agents"]["opencode"]["with_skill"] == 0.8+ assert report_payload["agents"]["opencode"]["baseline"] == 0.7+ assert report_payload["agents"]["opencode"]["lift"] == 0.1+ assert report_payload["agents"]["opencode"]["num_trials"] == 4+ assert report_payload["agents"]["opencode"]["num_trials_baseline"] == 4++ # A legacy summary has no canonical overall. When its raw rows are+ # truncated, the report must show unavailable rather than publish the+ # partial first logical trial as a numeric headline.+ for variant in ("with-skill", "without-skill"):+ summary_path = tmp_path / "results" / "opencode" / variant / "summary.json"+ legacy_summary = json.loads(summary_path.read_text(encoding="utf-8"))+ legacy_summary.pop("overall_score")+ summary_path.write_text(json.dumps(legacy_summary), encoding="utf-8")+ legacy_report = render_agent_eval_html_report(+ skill_dir,+ tmp_path / "results",+ output_path=tmp_path / "results" / "legacy-report.html",+ use_llm_judge=False,+ )+ legacy_output = legacy_report.read_text(encoding="utf-8")+ legacy_payload_match = re.search(+ r'<script type="application/json" id="tier3-full">(.*?)</script>',+ legacy_output,+ re.DOTALL,+ )+ assert legacy_payload_match is not None+ legacy_payload = json.loads(legacy_payload_match.group(1))+ assert legacy_payload["overall_score"] is None+ assert legacy_payload["overall_lift"] is None+ assert legacy_payload["agents"]["opencode"]["with_skill"] is None+ assert legacy_payload["agents"]["opencode"]["baseline"] is None++ # The same fail-closed rule applies below the row cap when a whole saved+ # trial directory is missing: summary count and loaded row count disagree.+ monkeypatch.setattr(report_data, "_MAX_TRIALS_PER_CONDITION", 512)+ reward_paths[0].parent.rename(tmp_path / "removed-saved-trial")+ missing_trial_report = render_agent_eval_html_report(+ skill_dir,+ tmp_path / "results",+ output_path=tmp_path / "results" / "legacy-missing-trial-report.html",+ use_llm_judge=False,+ )+ missing_trial_output = missing_trial_report.read_text(encoding="utf-8")+ missing_trial_payload_match = re.search(+ r'<script type="application/json" id="tier3-full">(.*?)</script>',+ missing_trial_output,+ re.DOTALL,+ )+ assert missing_trial_payload_match is not None+ missing_trial_payload = json.loads(missing_trial_payload_match.group(1))+ assert missing_trial_payload["overall_score"] is None+ assert missing_trial_payload["agents"]["opencode"]["with_skill"] is None+++ ("aggregate", "step_rewards", "expected_custom_scores"),+ [+ (+ {"overall": 0.75, "domain_quality": 0.8},+ [{"overall": 0.5, "domain_quality": 0.6}, {"overall": 1.0, "domain_quality": 1.0}],+ {"domain_quality": 0.8},+ ),+ (+ _default_reward("case-001", 0.8),+ [_default_reward("case-001", 0.7), {"overall": 0.9, "domain_quality": 1.0}],+ {},+ ),+ (+ _default_reward("case-001", 0.8),+ [_default_reward("case-001", 0.7), _default_reward("case-001", 0.9)],+ {},+ ),+ ],+ ids=("custom-only", "mixed-default-and-custom", "complete-default"),+)+def test_authoritative_multistep_reward_preserves_complete_custom_topologies(+ tmp_path: Path,+ aggregate: dict[str, object],+ step_rewards: list[dict[str, object]],+ expected_custom_scores: dict[str, float],+) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ _write_authoritative_multistep_result(+ job_dir,+ "case-001__attempt",+ aggregate=aggregate,+ step_rewards=step_rewards,+ )++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ agent = result["agents"]["opencode"]+ assert result["execution_status"] == "succeeded"+ assert agent["conditions"]["with_skill"]["scored_attempts"] == 1+ assert agent["custom_with_skill"] == expected_custom_scores+ assert agent["pass_at_k"]["with_skill"]["rate"] == 1.0+++def _create_step_entries(trial_dir: Path, names: list[str], *, directories: bool) -> None:+ steps_dir = trial_dir / "steps"+ steps_dir.mkdir(parents=True, exist_ok=True)+ # Intentionally avoid lexical creation order so correctness cannot depend on+ # whichever directory entry happens to be returned first by the filesystem.+ shuffled_names = names[::2] + list(reversed(names[1::2]))+ for name in shuffled_names:+ path = steps_dir / name+ if directories:+ (path / "verifier").mkdir(parents=True)+ else:+ path.write_text("not a step directory", encoding="utf-8")+++ ("total_candidate_count", "expected_status"),+ [+ (collector_module._MAX_FAILED_JUDGE_SIDECARS, "succeeded"),+ (collector_module._MAX_FAILED_JUDGE_SIDECARS + 1, "failed"),+ ],+ ids=("exact-candidate-limit", "beyond-candidate-limit"),+)+def test_public_collection_fails_closed_when_sidecar_candidate_limit_is_exceeded(+ tmp_path: Path,+ total_candidate_count: int,+ expected_status: str,+) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_dir = _write_authoritative_multistep_result(+ job_dir,+ "case-001__attempt",+ aggregate=_default_reward("case-001", 1.0),+ step_rewards=[_default_reward("case-001", 1.0)],+ )+ # Only sidecars that actually exist consume the sidecar bound. Empty+ # verifier directories remain compatible with large native step graphs.+ step_count = total_candidate_count+ _create_step_entries(+ trial_dir,+ [f"candidate-{index:04d}" for index in range(step_count)],+ directories=True,+ )+ for index in range(step_count):+ sidecar = trial_dir / "steps" / f"candidate-{index:04d}" / "verifier" / "skill_evaluator_reward.json"+ sidecar.write_text('{"evaluation_status":"succeeded"}', encoding="utf-8")++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == expected_status+ condition = result["agents"]["opencode"]["conditions"]["with_skill"]+ if expected_status == "succeeded":+ assert condition["scored_attempts"] == 1+ else:+ assert condition["scored_attempts"] == 0+ errors = " ".join(result["execution_errors"])+ assert "sidecar" in errors.casefold()+ assert "limit" in errors.casefold()+ assert len(errors) <= 2048+++ ("entry_count", "expected_status"),+ [+ (collector_module._MAX_FAILED_JUDGE_STEP_PATHS_SCANNED, "succeeded"),+ (collector_module._MAX_FAILED_JUDGE_STEP_PATHS_SCANNED + 1, "failed"),+ ],+ ids=("exact-scan-limit", "beyond-scan-limit"),+)+def test_public_collection_fails_closed_when_step_entry_scan_limit_is_exceeded(+ tmp_path: Path,+ entry_count: int,+ expected_status: str,+) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_dir = _write_authoritative_multistep_result(+ job_dir,+ "case-001__attempt",+ aggregate=_default_reward("case-001", 1.0),+ step_rewards=[_default_reward("case-001", 1.0)],+ )+ _create_step_entries(+ trial_dir,+ [f"entry-{index:04d}" for index in range(entry_count)],+ directories=False,+ )++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == expected_status+ condition = result["agents"]["opencode"]["conditions"]["with_skill"]+ if expected_status == "succeeded":+ assert condition["scored_attempts"] == 1+ else:+ assert condition["scored_attempts"] == 0+ errors = " ".join(result["execution_errors"])+ assert "sidecar" in errors.casefold()+ assert "limit" in errors.casefold()+ assert len(errors) <= 2048+++def test_public_collection_fails_closed_when_step_sidecars_cannot_be_scanned(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_dir = _write_authoritative_multistep_result(+ job_dir,+ "case-001__attempt",+ aggregate=_default_reward("case-001", 1.0),+ step_rewards=[_default_reward("case-001", 1.0)],+ )+ steps_dir = trial_dir / "steps"+ steps_dir.mkdir()+ original_scandir = os.scandir++ def deny_step_scan(path: os.PathLike[str] | str):+ if Path(path) == steps_dir:+ raise PermissionError("synthetic sensitive filesystem detail")+ return original_scandir(path)++ monkeypatch.setattr(collector_module.os, "scandir", deny_step_scan)++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == "failed"+ assert result["agents"]["opencode"]["conditions"]["with_skill"]["scored_attempts"] == 0+ errors = " ".join(result["execution_errors"])+ assert "could not inspect step artifacts" in errors.casefold()+ assert "sensitive filesystem detail" not in errors+ assert len(errors) <= 2048++[email protected]("sidecar_kind", ["malformed", "oversized", "symlink"])+def test_public_collection_fails_closed_when_present_sidecar_cannot_be_read(+ tmp_path: Path,+ sidecar_kind: str,+) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_dir = _write_authoritative_multistep_result(+ job_dir,+ "case-001__attempt",+ aggregate=_default_reward("case-001", 1.0),+ step_rewards=[_default_reward("case-001", 1.0)],+ )+ sidecar = trial_dir / "verifier" / "skill_evaluator_reward.json"+ sidecar.parent.mkdir()+ external_marker = "external-unreadable-sidecar-marker"+ if sidecar_kind == "malformed":+ sidecar.write_text('{"evaluation_status":', encoding="utf-8")+ elif sidecar_kind == "oversized":+ sidecar.write_bytes(b" " * (collector_module.DEFAULT_DIAGNOSTIC_ARTIFACT_MAX_BYTES + 1))+ else:+ external = tmp_path / "external-sidecar.json"+ _numeric, rich = _failed_judge_artifacts("case-001", reason=external_marker)+ external.write_text(json.dumps(rich), encoding="utf-8")+ sidecar.symlink_to(external)++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == "failed"+ assert result["agents"]["opencode"]["conditions"]["with_skill"]["scored_attempts"] == 0+ errors = " ".join(result["execution_errors"])+ assert "sidecar" in errors.casefold()+ assert "could not be read" in errors.casefold()+ assert external_marker not in errors+ assert len(errors) <= 2048+++def test_sidecar_scan_limit_diagnostic_precedes_existing_reward_errors(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ reward = {+ **_default_reward("case-001", 1.0),+ "evaluation_status": "failed",+ "evaluation_errors": {f"existing-{index}": f"existing error {index}" for index in range(len(DEFAULT_METRICS))},+ }+ _write_reward(job_dir, trial_name, reward)+ _write_complete_job_result(job_dir, [trial_name])+ trial_dir = job_dir / trial_name+ _create_step_entries(+ trial_dir,+ [f"candidate-{index:04d}" for index in range(collector_module._MAX_FAILED_JUDGE_SIDECARS + 1)],+ directories=True,+ )+ for index in range(collector_module._MAX_FAILED_JUDGE_SIDECARS + 1):+ sidecar = trial_dir / "steps" / f"candidate-{index:04d}" / "verifier" / "skill_evaluator_reward.json"+ sidecar.write_text('{"evaluation_status":"succeeded"}', encoding="utf-8")++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == "failed"+ errors = " ".join(result["execution_errors"])+ assert "sidecar" in errors.casefold()+ assert "limit" in errors.casefold()+ assert len(errors) <= 2048++[email protected]("linked_component", ["steps", "step-verifier"])+def test_public_collection_fails_closed_on_linked_step_sidecar_boundaries(+ tmp_path: Path,+ linked_component: str,+) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_dir = _write_authoritative_multistep_result(+ job_dir,+ "case-001__attempt",+ aggregate=_default_reward("case-001", 1.0),+ step_rewards=[_default_reward("case-001", 1.0)],+ )+ external_marker = "external-linked-sidecar-must-not-be-read"+ trajectory_marker = "external-linked-trajectory-must-not-be-read"+ _numeric, rich = _failed_judge_artifacts("case-001", reason=external_marker)+ external = tmp_path / "external-step-boundary"+ if linked_component == "steps":+ external_verifier = external / "finish" / "verifier"+ external_verifier.mkdir(parents=True)+ (external_verifier / "skill_evaluator_reward.json").write_text(json.dumps(rich), encoding="utf-8")+ external_agent = external / "finish" / "agent"+ external_agent.mkdir()+ (external_agent / "trajectory.json").write_text(+ json.dumps({"steps": [{"message": trajectory_marker}]}),+ encoding="utf-8",+ )+ (trial_dir / "steps").symlink_to(external, target_is_directory=True)+ else:+ external.mkdir()+ (external / "skill_evaluator_reward.json").write_text(json.dumps(rich), encoding="utf-8")+ verifier_dir = trial_dir / "steps" / "finish" / "verifier"+ verifier_dir.parent.mkdir(parents=True)+ verifier_dir.symlink_to(external, target_is_directory=True)++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == "failed"+ assert result["agents"]["opencode"]["conditions"]["with_skill"]["scored_attempts"] == 0+ errors = " ".join(result["execution_errors"])+ assert "symlink" in errors.casefold()+ assert external_marker not in errors+ assert trajectory_marker not in json.dumps(result)+ persisted_trials = tmp_path / "results" / "opencode" / "with-skill" / "trials"+ assert all(trajectory_marker not in path.read_text(encoding="utf-8") for path in persisted_trials.rglob("*.json"))+ assert len(errors) <= 2048++[email protected]("verifier_relative", [Path("verifier"), Path("steps/finish/verifier")])+def test_authoritative_reward_fails_closed_on_symlinked_external_judge_sidecar(+ tmp_path: Path,+ verifier_relative: Path,+) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ trial_dir = job_dir / trial_name+ trial_dir.mkdir(parents=True)+ aggregate = _default_reward("case-001", 1.0)+ (trial_dir / "result.json").write_text(+ json.dumps(+ {+ "trial_name": trial_name,+ "task_name": "case-001",+ "verifier_result": {"rewards": aggregate},+ "step_results": [],+ }+ ),+ encoding="utf-8",+ )+ external_verifier = tmp_path / "outside-verifier"+ external_verifier.mkdir()+ _numeric, rich = _failed_judge_artifacts("case-001", reason="external sidecar marker")+ (external_verifier / "skill_evaluator_reward.json").write_text(json.dumps(rich), encoding="utf-8")+ verifier_dir = trial_dir / verifier_relative+ verifier_dir.parent.mkdir(parents=True, exist_ok=True)+ verifier_dir.symlink_to(external_verifier, target_is_directory=True)++ [extracted] = _extract_rewards(job_dir)++ assert extracted["evaluation_status"] == "failed"+ assert "external sidecar marker" not in json.dumps(extracted)+++def test_discovered_step_sidecar_cannot_be_swapped_to_external_ancestor(tmp_path: Path) -> None:+ trial_dir = tmp_path / "trial"+ verifier_dir = trial_dir / "steps" / "step-a" / "verifier"+ verifier_dir.mkdir(parents=True)+ sidecar_path = verifier_dir / "skill_evaluator_reward.json"+ sidecar_path.write_text('{"evaluation_status":"succeeded"}', encoding="utf-8")++ candidates, failure = collector_module._failed_judge_sidecar_paths(trial_dir)+ assert failure == ""+ [(step_name, discovered_path, expected)] = candidates+ assert step_name == "step-a"++ original_step = trial_dir / "steps" / "step-a"+ original_step.rename(trial_dir / "steps" / "step-a-original")+ external_step = tmp_path / "external-step"+ external_verifier = external_step / "verifier"+ external_verifier.mkdir(parents=True)+ external_marker = "EXTERNAL-RACE-MARKER"+ (external_verifier / "skill_evaluator_reward.json").write_text(+ json.dumps({"evaluation_status": "failed", "evaluation_errors": {"accuracy": external_marker}}),+ encoding="utf-8",+ )+ original_step.symlink_to(external_step, target_is_directory=True)++ sidecar, read_failure = collector_module._read_failed_judge_sidecar(+ discovered_path,+ trial_dir=trial_dir,+ expected=expected,+ )++ assert sidecar is None+ assert "could not be read" in read_failure+ assert external_marker not in read_failure+++def test_public_collection_rejects_symlinked_trial_root_without_reading_external_artifacts(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ job_dir.mkdir(parents=True)+ trial_name = "case-001__attempt"+ external_trial = tmp_path / "outside-trial"+ _write_authoritative_multistep_result(+ tmp_path / "outside-job",+ external_trial.name,+ aggregate=_default_reward("case-001", 1.0),+ step_rewards=[_default_reward("case-001", 1.0)],+ )+ # Move the generated result into the exact external target while keeping+ # the public job path as a directory link.+ generated = tmp_path / "outside-job" / external_trial.name+ generated.rename(external_trial)+ marker = "EXTERNAL-TRIAL-DIAGNOSTIC-MARKER"+ (external_trial / "trial.log").write_text(marker, encoding="utf-8")+ (job_dir / trial_name).symlink_to(external_trial, target_is_directory=True)+ _write_complete_job_result(job_dir, [trial_name])++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ condition = result["agents"]["opencode"]["conditions"]["with_skill"]+ assert result["execution_status"] == "failed"+ assert condition["scored_attempts"] == 0+ assert result["agents"]["opencode"]["with_skill"] == {}+ assert "symlink" in " ".join(result["execution_errors"]).casefold()+ copied = tmp_path / "results" / "opencode" / "with-skill" / "trials" / trial_name / "trial.log"+ assert not copied.exists()+ assert marker not in json.dumps(result)+++def test_single_step_result_fallback_honors_failed_top_level_sidecar(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ trial_dir = job_dir / trial_name+ _numeric, rich = _failed_judge_artifacts("case-001")+ trial_dir.mkdir(parents=True)+ (trial_dir / "result.json").write_text(+ json.dumps(+ {+ "trial_name": trial_name,+ "task_name": "case-001",+ "verifier_result": {"rewards": _default_reward("case-001", 1.0)},+ }+ ),+ encoding="utf-8",+ )+ verifier_dir = trial_dir / "verifier"+ verifier_dir.mkdir()+ (verifier_dir / "skill_evaluator_reward.json").write_text(json.dumps(rich), encoding="utf-8")+ _write_complete_job_result(job_dir, [trial_name])++ [extracted] = _extract_rewards(job_dir)+ assert extracted["evaluation_status"] == "failed"++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])+ assert result["execution_status"] == "failed"+ assert result["agents"]["opencode"]["with_skill"] == {}+++def test_failed_sidecar_overrides_conflicting_reward_status_and_errors(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ secret = "sk-authoritativesidecar123"+ reward = {+ **_default_reward("case-001", 1.0),+ "evaluation_status": "succeeded",+ "evaluation_errors": {"accuracy": "stale reward error"},+ }+ _numeric, rich = _failed_judge_artifacts("case-001", reason=f"provider echoed {secret}")+ rich["evaluation_status"] = "error"+ _write_reward(job_dir, trial_name, reward, sidecar=rich)+ _write_complete_job_result(job_dir, [trial_name])++ [extracted] = _extract_rewards(job_dir)+ assert extracted["evaluation_status"] == "failed"+ assert extracted["evaluation_errors"]["accuracy"] == "provider echoed sk-<redacted>"++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])+ assert result["execution_status"] == "failed"+ assert secret not in " ".join(result["execution_errors"])+++def test_default_plus_custom_cannot_rescue_an_incomplete_default_reward(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ numeric, rich = _failed_judge_artifacts("case-001")+ _write_reward(+ job_dir,+ trial_name,+ numeric,+ sidecar=rich,+ custom={"overall": 1.0, "domain_quality": 1.0, "details": {"domain_quality": {"reason": "perfect"}}},+ )+ _write_complete_job_result(job_dir, [trial_name])++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ agent = result["agents"]["opencode"]+ assert result["execution_status"] == "failed"+ assert agent["with_skill"] == {}+ assert agent["custom_with_skill"] == {}+ assert agent["pass_at_k"]["with_skill"] == {}+++def test_custom_only_overall_reward_remains_scoreable(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ _write_reward(+ job_dir,+ trial_name,+ {"entry_id": "case-001", "overall": 0.75, "domain_quality": 0.9, "token_efficiency": 0.8},+ )+ _write_complete_job_result(job_dir, [trial_name])++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ agent = result["agents"]["opencode"]+ assert result["execution_status"] == "succeeded"+ assert agent["with_skill"] == {}+ assert agent["custom_with_skill"] == {"domain_quality": 0.9, "token_efficiency": 0.8}+ assert agent["pass_at_k"]["with_skill"]["rate"] == 1.0+ assert agent["conditions"]["without_skill"]["execution_status"] == "skipped"+ persisted = json.loads(+ (tmp_path / "results" / "opencode" / "with-skill" / "trials" / trial_name / "reward.json").read_text(+ encoding="utf-8"+ )+ )+ assert persisted["token_efficiency"] == 0.8+++def test_mixed_failed_condition_suppresses_published_quality_and_paired_artifacts(tmp_path: Path) -> None:+ jobs_dir = tmp_path / "jobs"+ with_job = jobs_dir / "demo-opencode-with"+ without_job = jobs_dir / "demo-opencode-without"++ _write_reward(with_job, "case-001__attempt", _default_reward("case-001", 0.9))+ numeric, rich = _failed_judge_artifacts("case-002")+ _write_reward(with_job, "case-002__attempt", numeric, sidecar=rich)+ _write_complete_job_result(with_job, ["case-001__attempt", "case-002__attempt"])++ for case_id in ("case-001", "case-002"):+ _write_reward(without_job, f"{case_id}__attempt", _default_reward(case_id, 0.2))+ _write_complete_job_result(without_job, ["case-001__attempt", "case-002__attempt"])++ result = _collect(tmp_path, skip_baseline=False, case_ids=["case-001", "case-002"])++ agent = result["agents"]["opencode"]+ assert result["execution_status"] == "failed"+ assert agent["conditions"]["with_skill"]["execution_status"] == "failed"+ assert agent["conditions"]["with_skill"]["scored_attempts"] == 1+ assert agent["conditions"]["without_skill"]["execution_status"] == "succeeded"+ assert agent["with_skill"] == {}+ assert agent["custom_with_skill"] == {}+ assert agent["dimensions_with_skill"] == {}+ assert agent["pass_at_k"]["with_skill"] == {}+ assert agent["without_skill"] == dict.fromkeys(DEFAULT_METRICS, 0.2)+ assert agent["dimensions_without_skill"]+ assert agent["pass_at_k"]["without_skill"]["rate"] == 0.0+ assert agent["lift"] == {}+ assert agent["custom_lift"] == {}+ assert agent["pass_at_k"]["lift"] == {}+ assert agent["security_attribution"] == {}++ results_dir = tmp_path / "results" / "opencode"+ persisted = json.loads((results_dir / "with-skill" / "summary.json").read_text(encoding="utf-8"))+ assert persisted["scores"] == {}+ assert persisted["custom_scores"] == {}+ assert persisted["dimensions"] == {}+ assert persisted["pass_at_k"] == {}+ assert not (results_dir / "lift.json").exists()+ assert not (results_dir / "custom_lift.json").exists()+ assert not (results_dir / "pass_at_k_lift.json").exists()+ assert not (results_dir / "security_attribution.json").exists()+ # Both source artifacts remain available as redacted trial diagnostics.+ assert (results_dir / "with-skill" / "trials" / "case-001__attempt" / "reward.json").exists()+ failed_reward = results_dir / "with-skill" / "trials" / "case-002__attempt" / "reward.json"+ assert json.loads(failed_reward.read_text(encoding="utf-8"))["evaluation_status"] == "failed"++ skill_dir = tmp_path / "demo"+ skill_dir.mkdir()+ html = render_agent_eval_html_report(skill_dir, tmp_path / "results", use_llm_judge=False).read_text(+ encoding="utf-8"+ )+ assert "Evaluation incomplete" in html+ assert "Required judge evaluation failed" in html+ assert re.findall(r'class="t3-dim-score"[^>]*>([^<]+)</span>', html) == ["N/A"] * 5+ assert "return raw === null || raw === undefined ? null : Number(raw);" in html+ payload_match = re.search(+ r'<script type="application/json" id="tier3-full">(.*?)</script>',+ html,+ re.DOTALL,+ )+ assert payload_match is not None+ report_payload = json.loads(payload_match.group(1))+ for dimension in report_payload["agents"]["opencode"]["dimensions"]:+ assert dimension["with_skill"] is None+ assert dimension["score"] is None+ assert dimension["verdict"] is None+ assert "0.00" not in str(dimension["explanation"])+ assert all("0.00" not in bullet for bullet in dimension["reasoning_bullets"])+++def test_reused_results_remove_stale_generated_quality_but_preserve_unrelated_files(+ tmp_path: Path, monkeypatch+) -> None:+ jobs_dir = tmp_path / "jobs"+ with_job = jobs_dir / "demo-opencode-with"+ without_job = jobs_dir / "demo-opencode-without"+ trial_name = "case-001__attempt"+ with_reward = {**_default_reward("case-001", 0.2), "domain_quality": 0.4}+ without_reward = {**_default_reward("case-001", 0.1), "domain_quality": 0.2}+ _write_reward(with_job, trial_name, with_reward)+ _write_reward(without_job, trial_name, without_reward)+ _write_complete_job_result(with_job, [trial_name])+ _write_complete_job_result(without_job, [trial_name])++ first = _collect(tmp_path, skip_baseline=False, case_ids=["case-001"])+ agent_dir = tmp_path / "results" / "opencode"+ monkeypatch.setattr(report, "_generate_suggestions_structured", lambda *_args, **_kwargs: [])+ monkeypatch.setattr(report, "_passing_skill_suggestions", lambda *_args, **_kwargs: [])+ assert report.display_findings_report(first, "demo", ["opencode"], tmp_path / "results")+ stale_trial = agent_dir / "with-skill" / "trials" / "case-stale__attempt"+ stale_trial.mkdir(parents=True)+ (stale_trial / "reward.json").write_text(json.dumps(_default_reward("case-stale", 1.0)), encoding="utf-8")+ unrelated_agent_file = agent_dir / "user-note.txt"+ unrelated_condition_file = agent_dir / "with-skill" / "user-note.txt"+ unrelated_agent_file.write_text("keep", encoding="utf-8")+ unrelated_condition_file.write_text("keep", encoding="utf-8")+ for artifact in (+ "lift.json",+ "custom_lift.json",+ "pass_at_k_lift.json",+ "security_attribution.json",+ "findings.json",+ ):+ assert (agent_dir / artifact).exists()++ numeric, rich = _failed_judge_artifacts("case-001")+ _write_reward(with_job, trial_name, numeric, sidecar=rich)+ _write_complete_job_result(with_job, [trial_name])+ failed = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert failed["execution_status"] == "failed"+ for artifact in (+ "lift.json",+ "custom_lift.json",+ "pass_at_k_lift.json",+ "security_attribution.json",+ "findings.json",+ ):+ assert not (agent_dir / artifact).exists()+ assert not stale_trial.exists()+ assert not (agent_dir / "without-skill" / "trials").exists()+ assert not (agent_dir / "without-skill" / "summary.json").exists()+ assert unrelated_agent_file.read_text(encoding="utf-8") == "keep"+ assert unrelated_condition_file.read_text(encoding="utf-8") == "keep"+++def test_generated_output_cleanup_refuses_symlink_escape(tmp_path: Path) -> None:+ results_dir = tmp_path / "results"+ outside = tmp_path / "outside"+ results_dir.mkdir()+ outside.mkdir()+ (tmp_path / "jobs").mkdir()+ sentinel = outside / "lift.json"+ sentinel.write_text("do not delete", encoding="utf-8")+ (results_dir / "opencode").symlink_to(outside, target_is_directory=True)++ with pytest.raises(ValueError, match="generated output"):+ collect_harbor_results(+ skill_name="demo",+ agents=["opencode"],+ output_dir=results_dir,+ jobs_dir=tmp_path / "jobs",+ skip_baseline=True,+ )++ assert sentinel.read_text(encoding="utf-8") == "do not delete"+++def test_reused_results_remove_omitted_agent_from_report_discovery(tmp_path: Path, monkeypatch) -> None:+ for agent, score in (("opencode", 0.2), ("claude-code", 0.1)):+ job_dir = tmp_path / "jobs" / f"demo-{agent}-with"+ _write_reward(job_dir, "case-001__attempt", _default_reward("case-001", score))+ _write_complete_job_result(job_dir, ["case-001__attempt"])++ first = collect_harbor_results(+ skill_name="demo",+ agents=["opencode", "claude-code"],+ output_dir=tmp_path / "results",+ jobs_dir=tmp_path / "jobs",+ skip_baseline=True,+ expected_cases=1,+ expected_case_ids=["case-001"],+ expected_trials=1,+ )+ monkeypatch.setattr(report, "_generate_suggestions_structured", lambda *_args, **_kwargs: [])+ monkeypatch.setattr(report, "_passing_skill_suggestions", lambda *_args, **_kwargs: [])+ assert report.display_findings_report(first, "demo", ["opencode", "claude-code"], tmp_path / "results")+ results_dir = tmp_path / "results"+ omitted_dir = results_dir / "claude-code"+ omitted_note = omitted_dir / "user-note.txt"+ omitted_note.write_text("keep", encoding="utf-8")+ unrelated_dir = results_dir / "user-content"+ unrelated_summary = unrelated_dir / "with-skill" / "summary.json"+ unrelated_summary.parent.mkdir(parents=True)+ unrelated_summary.write_text('{"notes": "not a collector summary"}', encoding="utf-8")+ unrelated_note = unrelated_dir / "notes.txt"+ unrelated_note.write_text("keep", encoding="utf-8")+ outside_agent = tmp_path / "outside-agent"+ outside_summary = outside_agent / "with-skill" / "summary.json"+ outside_summary.parent.mkdir(parents=True)+ outside_summary.write_text(+ json.dumps(+ {+ "scores": dict.fromkeys(DEFAULT_METRICS, 1.0),+ "execution_status": "succeeded",+ "expected_attempts": 1,+ "scored_attempts": 1,+ }+ ),+ encoding="utf-8",+ )+ stale_agent_link = results_dir / "stale"+ stale_agent_link.symlink_to(outside_agent, target_is_directory=True)+ assert (results_dir / "comparison.json").exists()+ assert (omitted_dir / "findings.json").exists()++ second = collect_harbor_results(+ skill_name="demo",+ agents=["opencode"],+ output_dir=results_dir,+ jobs_dir=tmp_path / "jobs",+ skip_baseline=True,+ expected_cases=1,+ expected_case_ids=["case-001"],+ expected_trials=1,+ )++ assert second["execution_status"] == "succeeded"+ assert set(report_data.load_agent_data(results_dir)) == {"opencode"}+ assert not (results_dir / "comparison.json").exists()+ assert not (omitted_dir / "with-skill" / "summary.json").exists()+ assert not (omitted_dir / "with-skill" / "trials").exists()+ assert not (omitted_dir / "findings.json").exists()+ assert omitted_note.read_text(encoding="utf-8") == "keep"+ assert unrelated_note.read_text(encoding="utf-8") == "keep"+ assert unrelated_summary.read_text(encoding="utf-8") == '{"notes": "not a collector summary"}'+ assert stale_agent_link.is_symlink()+ assert outside_summary.exists()++[email protected]("summary_kind", ["deep", "huge-integer", "malformed", "unicode-invalid"])+def test_unrelated_invalid_summary_does_not_block_collection(tmp_path: Path, summary_kind: str) -> None:+ unrelated_dir = tmp_path / "results" / "user-content"+ summary = unrelated_dir / "with-skill" / "summary.json"+ summary.parent.mkdir(parents=True)+ if summary_kind == "deep":+ summary.write_text("[" * 10_000 + "]" * 10_000, encoding="utf-8")+ elif summary_kind == "huge-integer":+ summary.write_text('{"value": ' + "1" * 10_000 + "}", encoding="utf-8")+ elif summary_kind == "malformed":+ summary.write_text('{"scores": ', encoding="utf-8")+ else:+ summary.write_bytes(b"\xff\xfe\x00")+ note = unrelated_dir / "notes.txt"+ note.write_text("keep", encoding="utf-8")+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ _write_reward(job_dir, "case-001__attempt", _default_reward("case-001", 0.8))+ _write_complete_job_result(job_dir, ["case-001__attempt"])++ result = collect_harbor_results(+ skill_name="demo",+ agents=["opencode"],+ output_dir=tmp_path / "results",+ jobs_dir=tmp_path / "jobs",+ skip_baseline=True,+ expected_cases=1,+ expected_case_ids=["case-001"],+ expected_trials=1,+ )++ assert result["execution_status"] == "succeeded"+ assert note.read_text(encoding="utf-8") == "keep"+ assert summary.exists()++[email protected]("reward_kind", ["deep", "huge-integer", "unicode-invalid"])+def test_invalid_verifier_reward_fails_collection_without_crashing(tmp_path: Path, reward_kind: str) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ verifier_dir = job_dir / trial_name / "verifier"+ verifier_dir.mkdir(parents=True)+ reward_path = verifier_dir / "reward.json"+ if reward_kind == "deep":+ reward_path.write_text("[" * 10_000 + "]" * 10_000, encoding="utf-8")+ elif reward_kind == "huge-integer":+ reward_path.write_text('{"security": ' + "1" * 10_000 + "}", encoding="utf-8")+ else:+ reward_path.write_bytes(b"\xff\xfe\x00")+ _write_complete_job_result(job_dir, [trial_name])++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == "failed"+ assert result["agents"]["opencode"]["with_skill"] == {}+ assert result["agents"]["opencode"]["conditions"]["with_skill"]["scored_attempts"] == 0++[email protected](not hasattr(os, "mkfifo"), reason="FIFO paths are not available on this platform")+def test_fifo_reward_is_rejected_without_blocking(tmp_path: Path) -> None:+ fifo = tmp_path / "reward.json"+ os.mkfifo(fifo)+ script = (+ "from pathlib import Path; "+ "from skillevaluator.tier3.harbor.collector import _read_json; "+ f"assert _read_json(Path({str(fifo)!r})) is None"+ )++ completed = subprocess.run(+ [sys.executable, "-c", script],+ check=False,+ capture_output=True,+ text=True,+ timeout=2,+ )++ assert completed.returncode == 0, completed.stderr++[email protected]("artifact", ["exception", "agent-log"])[email protected](not hasattr(os, "mkfifo"), reason="FIFO paths are not available on this platform")+def test_fifo_diagnostic_text_is_rejected_without_blocking(tmp_path: Path, artifact: str) -> None:+ trial_dir = tmp_path / "trial"+ trial_dir.mkdir()+ if artifact == "exception":+ os.mkfifo(trial_dir / "exception.txt")+ expression = f"_trial_failure_reason(Path({str(trial_dir)!r})) == ''"+ imports = "_trial_failure_reason"+ else:+ (trial_dir / "agent").mkdir()+ os.mkfifo(trial_dir / "agent" / "codex.txt")+ expression = f"_agent_log_runtime_failure_reason(Path({str(trial_dir)!r})) == ''"+ imports = "_agent_log_runtime_failure_reason"+ script = (+ f"from pathlib import Path; from skillevaluator.tier3.harbor.collector import {imports}; assert {expression}"+ )++ completed = subprocess.run(+ [sys.executable, "-c", script],+ check=False,+ capture_output=True,+ text=True,+ timeout=2,+ )++ assert completed.returncode == 0, completed.stderr+++def test_diagnostic_copy_rejects_final_symlink_and_records_manifest(tmp_path: Path) -> None:+ trial_dir = tmp_path / "trial"+ trial_out = tmp_path / "collected"+ trial_dir.mkdir()+ external_marker = "external-diagnostic-marker-must-not-be-copied"+ external = tmp_path / "external.log"+ external.write_text(external_marker, encoding="utf-8")+ (trial_dir / "trial.log").symlink_to(external)+ (trial_dir / "result.json").write_text('{"safe": true}', encoding="utf-8")++ copied = _copy_trial_artifacts(trial_dir, trial_out)++ assert copied == ["result.json"]+ assert json.loads((trial_out / "result.json").read_text(encoding="utf-8")) == {"safe": True}+ assert not (trial_out / "trial.log").exists()+ assert external.read_text(encoding="utf-8") == external_marker+ manifest = json.loads((trial_out / "artifact_manifest.json").read_text(encoding="utf-8"))+ assert manifest["copied"] == [{"name": "result.json", "size_bytes": 14}]+ assert manifest["skipped"][0]["name"] == "trial.log"+ assert "regular" in manifest["skipped"][0]["reason"]+ assert external_marker not in json.dumps(manifest)+++def test_diagnostic_copy_records_dangling_final_symlink_in_manifest(tmp_path: Path) -> None:+ trial_dir = tmp_path / "trial"+ trial_out = tmp_path / "collected"+ trial_dir.mkdir()+ trial_out.mkdir()+ (trial_dir / "trial.log").symlink_to(tmp_path / "missing-external.log")++ copied = _copy_trial_artifacts(trial_dir, trial_out)++ assert copied == []+ manifest = json.loads((trial_out / "artifact_manifest.json").read_text(encoding="utf-8"))+ assert manifest["copied"] == []+ assert manifest["skipped"] == [{"name": "trial.log", "reason": "not_regular_file"}]+++def test_diagnostic_copy_rejects_symlinked_agent_directory(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ trial_dir = tmp_path / "trial"+ trial_out = tmp_path / "collected"+ external_agent_dir = tmp_path / "external-agent"+ trial_dir.mkdir()+ trial_out.mkdir()+ external_agent_dir.mkdir()+ external_marker = "external-agent-marker-must-not-be-copied"+ (external_agent_dir / "codex.txt").write_text(external_marker, encoding="utf-8")+ (trial_dir / "agent").symlink_to(external_agent_dir, target_is_directory=True)+ # Windows junctions/reparse points are not reliably reported by+ # Path.is_symlink(); force that observable shape while preserving lstat.+ monkeypatch.setattr(Path, "is_symlink", lambda _path: False)++ copied = _copy_trial_artifacts(trial_dir, trial_out)++ assert copied == []+ assert not (trial_out / "codex.txt").exists()+ assert (external_agent_dir / "codex.txt").read_text(encoding="utf-8") == external_marker+ manifest = json.loads((trial_out / "artifact_manifest.json").read_text(encoding="utf-8"))+ assert manifest["copied"] == []+ assert manifest["skipped"] == [{"name": "agent", "reason": "not_regular_directory"}]+ assert external_marker not in json.dumps(manifest)+++def test_diagnostic_copy_preserves_bounded_redacted_regular_file(tmp_path: Path) -> None:+ source = tmp_path / "trial.log"+ destination = tmp_path / "out" / "trial.log"+ secret = "sk-diagnosticcopysecret123"+ source.write_text(f"Authorization: Bearer {secret}\nordinary output\n", encoding="utf-8")++ ok, record = _write_redacted_text_copy(source, destination)++ assert ok is True+ assert record == {"name": "trial.log", "size_bytes": source.stat().st_size}+ copied = destination.read_text(encoding="utf-8")+ assert "ordinary output" in copied+ assert secret not in copied+ assert "<redacted>" in copied+++ "payload",+ (+ "[" * 2_000 + "0" + "]" * 2_000,+ '{"value":' + "9" * 10_000 + "}",+ '{"api_key":"SUPERSECRET12345", broken',+ ),+ ids=("deep", "huge-integer", "malformed-secret"),+)+def test_diagnostic_copy_rejects_invalid_json_without_crashing(tmp_path: Path, payload: str) -> None:+ source = tmp_path / "result.json"+ destination = tmp_path / "out" / "result.json"+ source.write_text(payload, encoding="utf-8")++ ok, record = _write_redacted_text_copy(source, destination)++ assert ok is False+ assert record == {"name": "result.json", "reason": "invalid_json", "size_bytes": len(payload)}+ assert not destination.exists()+ assert "SUPERSECRET12345" not in json.dumps(record)+++def test_diagnostic_copy_does_not_pretty_print_amplify_nested_json(tmp_path: Path) -> None:+ source = tmp_path / "result.json"+ destination = tmp_path / "out" / "result.json"+ source.write_text("[" * 200 + "0" + "]" * 200, encoding="utf-8")++ ok, _record = _write_redacted_text_copy(source, destination)++ assert ok is True+ assert destination.stat().st_size <= source.stat().st_size+++def test_diagnostic_copy_rejects_hard_linked_artifact(tmp_path: Path) -> None:+ outside = tmp_path / "outside.log"+ source = tmp_path / "trial.log"+ destination = tmp_path / "out" / "trial.log"+ outside.write_text("outside-marker", encoding="utf-8")+ os.link(outside, source)++ ok, record = _write_redacted_text_copy(source, destination)++ assert ok is False+ assert not destination.exists()+ assert record == {"name": "trial.log", "reason": "not_regular_file"}++[email protected]("destination_kind", ["symlink", "hardlink"])+def test_diagnostic_copy_never_follows_or_replaces_unsafe_destination(+ tmp_path: Path,+ destination_kind: str,+) -> None:+ source = tmp_path / "trial.log"+ destination = tmp_path / "out" / "trial.log"+ external = tmp_path / "external.log"+ source.write_text("safe-source", encoding="utf-8")+ external.write_text("outside-sentinel", encoding="utf-8")+ destination.parent.mkdir()+ if destination_kind == "symlink":+ destination.symlink_to(external)+ else:+ os.link(external, destination)++ ok, record = _write_redacted_text_copy(source, destination)++ assert ok is False+ assert record == {"name": "trial.log", "reason": "write_failed", "size_bytes": 11}+ assert external.read_text(encoding="utf-8") == "outside-sentinel"++[email protected](("size_bytes", "expected_ok"), [(4, True), (5, False)], ids=("exact", "beyond"))+def test_diagnostic_copy_honors_lower_configured_byte_bound(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+ size_bytes: int,+ expected_ok: bool,+) -> None:+ source = tmp_path / "trial.log"+ destination = tmp_path / "out" / "trial.log"+ source.write_text("x" * size_bytes, encoding="utf-8")+ monkeypatch.setenv("SKILLEVALUATOR_HARBOR_DIAGNOSTIC_ARTIFACT_MAX_BYTES", "4")++ ok, record = _write_redacted_text_copy(source, destination)++ assert ok is expected_ok+ if expected_ok:+ assert destination.read_text(encoding="utf-8") == "x" * size_bytes+ assert record == {"name": "trial.log", "size_bytes": size_bytes}+ else:+ assert not destination.exists()+ assert record == {+ "name": "trial.log",+ "reason": "exceeds_max_bytes",+ "size_bytes": size_bytes,+ "max_bytes": 4,+ }+++def test_diagnostic_copy_zero_bound_rejects_nonempty_artifact(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ source = tmp_path / "trial.log"+ destination = tmp_path / "out" / "trial.log"+ source.write_text("x", encoding="utf-8")+ monkeypatch.setenv("SKILLEVALUATOR_HARBOR_DIAGNOSTIC_ARTIFACT_MAX_BYTES", "0")++ ok, record = _write_redacted_text_copy(source, destination)++ assert ok is False+ assert not destination.exists()+ assert record == {+ "name": "trial.log",+ "reason": "exceeds_max_bytes",+ "size_bytes": 1,+ "max_bytes": 0,+ }+++def test_diagnostic_copy_clamps_extreme_configured_byte_bound(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:+ source = tmp_path / "trial.log"+ destination = tmp_path / "out" / "trial.log"+ source.write_text("ok", encoding="utf-8")+ monkeypatch.setenv("SKILLEVALUATOR_HARBOR_DIAGNOSTIC_ARTIFACT_MAX_BYTES", "1" + "0" * 100)++ ok, record = _write_redacted_text_copy(source, destination)++ assert ok is True+ assert record == {"name": "trial.log", "size_bytes": 2}+ assert destination.read_text(encoding="utf-8") == "ok"+ assert collector_module._diagnostic_artifact_max_bytes() == collector_module.DIAGNOSTIC_ARTIFACT_HARD_MAX_BYTES+++def test_diagnostic_copy_allows_moderate_configured_bound_above_default(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ source = tmp_path / "trial.log"+ destination = tmp_path / "out" / "trial.log"+ size_bytes = collector_module.DEFAULT_DIAGNOSTIC_ARTIFACT_MAX_BYTES + 1+ configured_max = size_bytes + 1024+ source.write_bytes(b"x" * size_bytes)+ monkeypatch.setenv("SKILLEVALUATOR_HARBOR_DIAGNOSTIC_ARTIFACT_MAX_BYTES", str(configured_max))++ ok, record = _write_redacted_text_copy(source, destination)++ assert ok is True+ assert collector_module._diagnostic_artifact_max_bytes() == configured_max+ assert record == {"name": "trial.log", "size_bytes": size_bytes}+ assert destination.stat().st_size == size_bytes+++def test_successful_collection_does_not_materialize_job_result_as_a_trial(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ trial_name = "case-001__attempt"+ _write_reward(job_dir, trial_name, _default_reward("case-001", 0.8))+ _write_complete_job_result(job_dir, [trial_name])++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == "succeeded"+ trials_dir = tmp_path / "results" / "opencode" / "with-skill" / "trials"+ assert {path.name for path in trials_dir.iterdir()} == {trial_name}+++def _create_windows_junction(link: Path, target: Path) -> None:+ completed = subprocess.run(+ ["cmd", "/c", "mklink", "/J", str(link), str(target)],+ check=False,+ capture_output=True,+ text=True,+ )+ assert completed.returncode == 0, completed.stderr or completed.stdout++[email protected](os.name != "nt", reason="requires native Windows junction semantics")+def test_diagnostic_copy_rejects_native_windows_trial_root_junction(tmp_path: Path) -> None:+ external_trial = tmp_path / "external-trial"+ external_trial.mkdir()+ marker = "external-windows-trial-marker"+ (external_trial / "trial.log").write_text(marker, encoding="utf-8")+ trial_junction = tmp_path / "trial-junction"+ _create_windows_junction(trial_junction, external_trial)+ trial_out = tmp_path / "collected"+ trial_out.mkdir()++ copied = _copy_trial_artifacts(trial_junction, trial_out)++ assert copied == []+ assert not (trial_out / "trial.log").exists()+ assert marker not in json.dumps(json.loads((trial_out / "artifact_manifest.json").read_text(encoding="utf-8")))++[email protected](os.name != "nt", reason="requires native Windows junction semantics")+def test_diagnostic_copy_rejects_native_windows_agent_junction(tmp_path: Path) -> None:+ trial_dir = tmp_path / "trial"+ trial_dir.mkdir()+ external_agent = tmp_path / "external-agent"+ external_agent.mkdir()+ marker = "external-windows-agent-marker"+ (external_agent / "codex.txt").write_text(marker, encoding="utf-8")+ _create_windows_junction(trial_dir / "agent", external_agent)+ trial_out = tmp_path / "collected"+ trial_out.mkdir()++ copied = _copy_trial_artifacts(trial_dir, trial_out)++ assert copied == []+ assert not (trial_out / "codex.txt").exists()+ assert marker not in json.dumps(json.loads((trial_out / "artifact_manifest.json").read_text(encoding="utf-8")))++[email protected](os.name != "nt", reason="requires native Windows junction semantics")+def test_diagnostic_copy_rejects_native_windows_destination_parent_junction(tmp_path: Path) -> None:+ source = tmp_path / "trial.log"+ source.write_text("safe-source", encoding="utf-8")+ external_output = tmp_path / "external-output"+ external_output.mkdir()+ output_junction = tmp_path / "output-junction"+ _create_windows_junction(output_junction, external_output)++ ok, record = _write_redacted_text_copy(source, output_junction / "trial.log")++ assert ok is False+ assert record == {"name": "trial.log", "reason": "write_failed", "size_bytes": 11}+ assert not (external_output / "trial.log").exists()++[email protected](os.name != "nt", reason="requires native Windows junction semantics")+def test_judge_sidecar_scan_rejects_native_windows_verifier_junction(tmp_path: Path) -> None:+ trial_dir = tmp_path / "trial"+ trial_dir.mkdir()+ external_verifier = tmp_path / "external-verifier"+ external_verifier.mkdir()+ marker = "external-windows-sidecar-marker"+ (external_verifier / "skill_evaluator_reward.json").write_text(+ json.dumps({"evaluation_status": "failed", "evaluation_errors": {"accuracy": marker}}),+ encoding="utf-8",+ )+ _create_windows_junction(trial_dir / "verifier", external_verifier)++ diagnostic = collector_module._failed_judge_diagnostic(trial_dir)++ assert diagnostic is not None+ assert diagnostic["evaluation_status"] == "failed"+ assert "symlink" in json.dumps(diagnostic).casefold()+ assert marker not in json.dumps(diagnostic)++[email protected](not hasattr(os, "mkfifo"), reason="FIFO paths are not available on this platform")+def test_diagnostic_copy_rejects_fifo_without_blocking_and_records_manifest(tmp_path: Path) -> None:+ trial_dir = tmp_path / "trial"+ trial_out = tmp_path / "collected"+ trial_dir.mkdir()+ trial_out.mkdir()+ os.mkfifo(trial_dir / "trial.log")+ script = (+ "from pathlib import Path; "+ "from skillevaluator.tier3.harbor.collector import _copy_trial_artifacts; "+ f"assert _copy_trial_artifacts(Path({str(trial_dir)!r}), Path({str(trial_out)!r})) == []"+ )++ completed = subprocess.run(+ [sys.executable, "-c", script],+ check=False,+ capture_output=True,+ text=True,+ timeout=2,+ )++ assert completed.returncode == 0, completed.stderr+ assert not (trial_out / "trial.log").exists()+ manifest = json.loads((trial_out / "artifact_manifest.json").read_text(encoding="utf-8"))+ assert manifest["copied"] == []+ assert manifest["skipped"][0]["name"] == "trial.log"+ assert "regular" in manifest["skipped"][0]["reason"]+++def test_harbor_trial_name_cannot_escape_results_directory(tmp_path: Path) -> None:+ job_dir = tmp_path / "jobs" / "demo-opencode-with"+ physical_trial_name = "case-001__attempt"+ trial_dir = job_dir / physical_trial_name+ trial_dir.mkdir(parents=True)+ (trial_dir / "result.json").write_text(+ json.dumps(+ {+ "trial_name": "../../../../escaped",+ "task_name": "case-001",+ "verifier_result": {"rewards": _default_reward("case-001", 0.8)},+ }+ ),+ encoding="utf-8",+ )+ _write_complete_job_result(job_dir, [physical_trial_name])++ result = _collect(tmp_path, skip_baseline=True, case_ids=["case-001"])++ assert result["execution_status"] == "succeeded"+ assert not (tmp_path / "escaped" / "reward.json").exists()+ persisted = tmp_path / "results" / "opencode" / "with-skill" / "trials" / physical_trial_name / "reward.json"+ assert persisted.exists()+++ ("artifact", "agents"),+ (("attempt_policy.json", ["opencode"]), ("comparison.json", ["opencode", "claude-code"])),+)+def test_root_generated_artifact_write_unlinks_symlink_without_following(+ tmp_path: Path,+ artifact: str,+ agents: list[str],+) -> None:+ for agent in agents:+ job_dir = tmp_path / "jobs" / f"demo-{agent}-with"+ _write_reward(job_dir, "case-001__attempt", _default_reward("case-001", 0.8))+ _write_complete_job_result(job_dir, ["case-001__attempt"])+ results_dir = tmp_path / "results"+ outside = tmp_path / "outside"+ results_dir.mkdir()+ outside.mkdir()+ sentinel = outside / artifact+ sentinel.write_text("outside sentinel", encoding="utf-8")+ generated = results_dir / artifact+ generated.symlink_to(sentinel)++ collect_harbor_results(+ skill_name="demo",+ agents=agents,+ output_dir=results_dir,+ jobs_dir=tmp_path / "jobs",+ skip_baseline=True,+ expected_cases=1,+ expected_case_ids=["case-001"],+ expected_trials=1,+ )++ assert sentinel.read_text(encoding="utf-8") == "outside sentinel"+ assert generated.is_file()+ assert not generated.is_symlink()+++ ("artifact", "agents"),+ (("attempt_policy.json", ["opencode"]), ("comparison.json", ["opencode", "claude-code"])),+)+def test_root_generated_artifact_write_rejects_hard_link_replacement_after_cleanup(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+ artifact: str,+ agents: list[str],+) -> None:+ for agent in agents:+ job_dir = tmp_path / "jobs" / f"demo-{agent}-with"+ _write_reward(job_dir, "case-001__attempt", _default_reward("case-001", 0.8))+ _write_complete_job_result(job_dir, ["case-001__attempt"])+ results_dir = tmp_path / "results"+ results_dir.mkdir()+ sentinel = tmp_path / f"outside-{artifact}"+ sentinel.write_text("outside sentinel", encoding="utf-8")+ generated = results_dir / artifact+ original_prepare = collector_module._prepare_generated_outputs++ def prepare_then_replace(output_root: Path, selected_agents: list[str]) -> None:+ original_prepare(output_root, selected_agents)+ os.link(sentinel, generated)++ monkeypatch.setattr(collector_module, "_prepare_generated_outputs", prepare_then_replace)++ with pytest.raises(ValueError, match="single-link regular file"):+ collect_harbor_results(+ skill_name="demo",+ agents=agents,+ output_dir=results_dir,+ jobs_dir=tmp_path / "jobs",+ skip_baseline=True,+ expected_cases=1,+ expected_case_ids=["case-001"],+ expected_trials=1,+ )++ assert sentinel.read_text(encoding="utf-8") == "outside sentinel"+ assert generated.samefile(sentinel)+ assert generated.stat().st_nlink == 2+ assert not list(results_dir.glob(f".{artifact}.*.tmp"))+++def test_findings_skip_failed_single_agent_even_when_stale_rewards_exist(tmp_path: Path, capsys) -> None:+ trial_dir = tmp_path / "opencode" / "with-skill" / "trials" / "case-001__attempt"+ trial_dir.mkdir(parents=True)+ (trial_dir / "reward.json").write_text(json.dumps(_default_reward("case-001", 0.1)), encoding="utf-8")+ stale_findings = tmp_path / "opencode" / "findings.json"+ stale_findings.write_text('{"stale": true}', encoding="utf-8")++ rendered = report.display_findings_report(+ {+ "agents": {+ "opencode": {+ "execution_status": "failed",+ "with_skill": dict.fromkeys(DEFAULT_METRICS, 0.1),+ "conditions": {"with_skill": {"execution_status": "failed"}},+ }+ }+ },+ "demo",+ ["opencode"],+ tmp_path,+ )++ assert rendered == set()+ assert capsys.readouterr().out == ""+ assert not stale_findings.exists()+++def test_findings_keep_succeeded_with_skill_when_only_baseline_failed(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ agent_dir = tmp_path / "opencode"+ condition_dir = agent_dir / "with-skill"+ trial_dir = condition_dir / "trials" / "case-001__attempt"+ trial_dir.mkdir(parents=True)+ score = 0.1+ (condition_dir / "summary.json").write_text(+ json.dumps(+ {+ "agent": "opencode",+ "scores": dict.fromkeys(DEFAULT_METRICS, score),+ "metrics": list(DEFAULT_METRICS),+ "execution_status": "succeeded",+ "execution_errors": [],+ "expected_attempts": 1,+ "scored_attempts": 1,+ }+ ),+ encoding="utf-8",+ )+ (trial_dir / "reward.json").write_text(json.dumps(_default_reward("case-001", score)), encoding="utf-8")+ monkeypatch.setattr(report, "_generate_suggestions_structured", lambda *_args, **_kwargs: [])+ monkeypatch.setattr(report, "_passing_skill_suggestions", lambda *_args, **_kwargs: [])++ rendered = report.display_findings_report(+ {+ "agents": {+ "opencode": {+ "execution_status": "failed",+ "with_skill": dict.fromkeys(DEFAULT_METRICS, score),+ "conditions": {+ "with_skill": {"execution_status": "succeeded"},+ "without_skill": {"execution_status": "failed"},+ },+ }+ }+ },+ "demo",+ ["opencode"],+ tmp_path,+ )++ assert rendered+ assert (agent_dir / "findings.json").exists()+++def test_findings_do_not_follow_external_reward_symlink(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ agent_dir = tmp_path / "opencode"+ condition_dir = agent_dir / "with-skill"+ trial_dir = condition_dir / "trials" / "case-001__attempt"+ trial_dir.mkdir(parents=True)+ score = 0.1+ (condition_dir / "summary.json").write_text(+ json.dumps(+ {+ "agent": "opencode",+ "scores": dict.fromkeys(DEFAULT_METRICS, score),+ "metrics": list(DEFAULT_METRICS),+ "execution_status": "succeeded",+ "execution_errors": [],+ "expected_attempts": 1,+ "scored_attempts": 1,+ }+ ),+ encoding="utf-8",+ )+ external_marker = "outside-reward-must-not-reach-suggestion-model"+ outside_reward = tmp_path / "outside-reward.json"+ payload = _default_reward("case-001", score)+ payload["details"]["goal_accuracy"]["reason"] = external_marker+ outside_reward.write_text(json.dumps(payload), encoding="utf-8")+ (trial_dir / "reward.json").symlink_to(outside_reward)+ suggestion_called = False++ def capture_suggestions(*_args, **_kwargs):+ nonlocal suggestion_called+ suggestion_called = True+ return []++ monkeypatch.setattr(report, "_generate_suggestions_structured", capture_suggestions)+ monkeypatch.setattr(report, "_passing_skill_suggestions", lambda *_args, **_kwargs: [])++ rendered = report.display_findings_report(+ {+ "agents": {+ "opencode": {+ "execution_status": "succeeded",+ "with_skill": dict.fromkeys(DEFAULT_METRICS, score),+ "conditions": {"with_skill": {"execution_status": "succeeded"}},+ }+ }+ },+ "demo",+ ["opencode"],+ tmp_path,+ )++ assert rendered == set()+ assert suggestion_called is False+ assert not (agent_dir / "findings.json").exists()+++def test_findings_multi_agent_selects_and_writes_only_successful_agent(tmp_path: Path, monkeypatch) -> None:+ for agent, score in (("failed", 1.0), ("succeeded", 0.8)):+ condition_dir = tmp_path / agent / "with-skill"+ trial_dir = condition_dir / "trials" / "case-001__attempt"+ trial_dir.mkdir(parents=True)+ status = "failed" if agent == "failed" else "succeeded"+ (condition_dir / "summary.json").write_text(+ json.dumps(+ {+ "agent": agent,+ "scores": dict.fromkeys(DEFAULT_METRICS, score),+ "metrics": list(DEFAULT_METRICS),+ "execution_status": status,+ "execution_errors": ["agent failed"] if status == "failed" else [],+ "expected_attempts": 1,+ "scored_attempts": 0 if status == "failed" else 1,+ }+ ),+ encoding="utf-8",+ )+ (trial_dir / "reward.json").write_text(json.dumps(_default_reward("case-001", score)), encoding="utf-8")+ monkeypatch.setattr(report, "_generate_suggestions_structured", lambda *_args, **_kwargs: [])+ monkeypatch.setattr(report, "_passing_skill_suggestions", lambda *_args, **_kwargs: [])++ rendered = report.display_findings_report(+ {+ "agents": {+ "failed": {+ "execution_status": "failed",+ "with_skill": dict.fromkeys(DEFAULT_METRICS, 1.0),+ "conditions": {"with_skill": {"execution_status": "failed"}},+ },+ "succeeded": {+ "execution_status": "succeeded",+ "with_skill": dict.fromkeys(DEFAULT_METRICS, 0.8),+ "conditions": {"with_skill": {"execution_status": "succeeded"}},+ },+ }+ },+ "demo",+ ["failed", "succeeded"],+ tmp_path,+ )++ assert rendered+ assert not (tmp_path / "failed" / "findings.json").exists()+ assert (tmp_path / "succeeded" / "findings.json").exists()diff --git a/tests/test_oss_packaging.py b/tests/test_oss_packaging.pyindex 8779e399..b59ebef2 100644--- a/tests/test_oss_packaging.py+++ b/tests/test_oss_packaging.py@@ -184,6 +184,16 @@ def test_third_party_notices_do_not_list_removed_safety_dependency() -> None:assert "Safety (MIT)" not in notices+def test_idna_is_a_bounded_direct_dependency_with_a_license_notice() -> None:+ dependencies = [Requirement(raw) for raw in _project()["project"]["dependencies"]]+ idna_requirements = [requirement for requirement in dependencies if canonicalize_name(requirement.name) == "idna"]+ notices = (REPO_ROOT / "THIRD_PARTY_NOTICES.md").read_text(encoding="utf-8")++ assert len(idna_requirements) == 1+ assert str(idna_requirements[0].specifier) == "<4,>=3.10"+ assert "IDNA (BSD-3-Clause)" in notices++def test_release_lock_avoids_accidental_prereleases_and_known_fixed_versions() -> None:project = _project()lock = _lock()diff --git a/tests/test_provider_config.py b/tests/test_provider_config.pyindex b2074ba0..92ce56e8 100644--- a/tests/test_provider_config.py+++ b/tests/test_provider_config.py@@ -74,6 +74,155 @@ def test_chat_default_models_are_the_single_source_of_truth() -> None:assert DEFAULT_JUDGE_MODEL == CHAT_DEFAULT_OPENAI+ ("configured_base_url", "expected_base_url"),+ [+ ("https://gateway.example", "https://gateway.example"),+ ("https://gateway.example/", "https://gateway.example"),+ ("https://gateway.example/v1", "https://gateway.example"),+ ("https://gateway.example/v1/", "https://gateway.example"),+ ("https://gateway.example/team/v1/", "https://gateway.example/team"),+ ("https://localhost/v1", "https://localhost"),+ ("http://gateway.internal:8080/v1", "http://gateway.internal:8080"),+ ("http://anthropic_proxy:8000/v1", "http://anthropic_proxy:8000"),+ ("https://xn--bcher-kva.example:8443/team/v1", "https://xn--bcher-kva.example:8443/team"),+ ("https://bücher.example/v1", "https://xn--bcher-kva.example"),+ ("https://faß.de/v1", "https://xn--fa-hia.de"),+ ("https://οδός.example/v1", "https://xn--pxavk3b.example"),+ ("https://bücher.example.:8443/team/v1", "https://xn--bcher-kva.example.:8443/team"),+ ("http://127.0.0.1:8080/v1", "http://127.0.0.1:8080"),+ ("http://[::1]:8080/team/v1", "http://[::1]:8080/team"),+ ("http://[fe80::1%25eth0]:8080/team/v1", "http://[fe80::1%25eth0]:8080/team"),+ ("http://127.0.0.1:8080/bücher/v1", "http://127.0.0.1:8080/b%C3%BCcher"),+ ("https://gateway.example/caf%C3%A9/v1", "https://gateway.example/caf%C3%A9"),+ ("https://gateway.example/caf%c3%a9/v1", "https://gateway.example/caf%C3%A9"),+ ("https://gateway.example/opaque%ff/v1", "https://gateway.example/opaque%FF"),+ ("https://gateway.example/tenant%25west/v1", "https://gateway.example/tenant%25west"),+ ("https://gateway.example/100%25/v1", "https://gateway.example/100%25"),+ ("https://gateway.example/team/%76%31", "https://gateway.example/team"),+ ("https://gateway.example/team/v1///", "https://gateway.example/team"),+ (+ "https://gateway.example/teams;v=1/@me+you/v1",+ "https://gateway.example/teams;v=1/@me+you",+ ),+ ],+)+def test_anthropic_base_url_is_normalized_to_an_unversioned_root(+ configured_base_url: str,+ expected_base_url: str,+) -> None:+ config = resolve_llm_provider(+ {+ "SKILL_EVAL_LLM_PROVIDER": "anthropic",+ "ANTHROPIC_API_KEY": "test-anthropic-key",+ "ANTHROPIC_BASE_URL": configured_base_url,+ }+ )++ assert config.base_url == expected_base_url+++def test_anthropic_base_url_remains_unset_for_the_official_default() -> None:+ config = resolve_llm_provider(+ {+ "SKILL_EVAL_LLM_PROVIDER": "anthropic",+ "ANTHROPIC_API_KEY": "test-anthropic-key",+ }+ )++ assert config.base_url is None+ assert config.child_environment() == {"ANTHROPIC_API_KEY": "test-anthropic-key"}+++def test_anthropic_skill_eval_base_url_takes_precedence_and_is_normalized() -> None:+ config = resolve_llm_provider(+ {+ "SKILL_EVAL_LLM_PROVIDER": "anthropic",+ "ANTHROPIC_API_KEY": "test-anthropic-key",+ "SKILL_EVAL_LLM_BASE_URL": "https://selected.example/team/v1/",+ "ANTHROPIC_BASE_URL": "https://ignored.example/v1",+ }+ )++ assert config.base_url == "https://selected.example/team"+ assert config.child_environment()["ANTHROPIC_BASE_URL"] == "https://selected.example/team"+++ "configured_base_url",+ [+ "gateway.example/v1",+ "file:///etc/passwd",+ "https:///team/v1",+ "https://:443/v1",+ "https://url-user:[email protected]/v1",+ "https://gateway.example/v1?token=url-secret",+ "https://gateway.example/v1#token=url-secret",+ "https://gateway.example/v1/messages",+ "https://gateway.example\\team\\v1",+ "https://gateway.example/v1\n",+ "https://gateway.example:not-a-port/v1",+ "https://gateway.example:/v1",+ "https://gateway example/v1",+ "https://gateway%2eexample/v1",+ "https://gateway.example|evil/v1",+ "https://999.1.1.1/v1",+ "https://[v1.not-ipv6]/v1",+ "https://gateway.example/%76%31/%6dessages",+ "https://gateway.example/team%2Fv1",+ "https://gateway.example/team%5cv1",+ "https://gateway.example/team%0av1",+ "https://gateway.example/team%",+ "https://gateway.example/team%2",+ "https://gateway.example/team%GG",+ "https://gateway.example/../team/v1",+ "https://gateway.example/%2e%2e/team/v1",+ "https://gateway.example/%2576%2531/%256dessages",+ "https://gateway.example/%252e%252e/team/v1",+ "https://gateway.example/team%252Fv1",+ "https://gateway.example/team%255Cv1",+ "https://gateway.example/team%250Av1",+ "https://gateway.example/%25%37%36%25%33%31/%25%36%64essages",+ "https://gateway.example/team%25%32%46v1",+ "https://gateway.example/team//v1",+ "https://☃.example/v1",+ ],+)+def test_anthropic_base_url_rejects_unsafe_or_non_root_values(configured_base_url: str) -> None:+ with pytest.raises(ProviderConfigurationError) as exc_info:+ resolve_llm_provider(+ {+ "SKILL_EVAL_LLM_PROVIDER": "anthropic",+ "ANTHROPIC_API_KEY": "test-anthropic-key",+ "ANTHROPIC_BASE_URL": configured_base_url,+ }+ )++ message = str(exc_info.value)+ assert "ANTHROPIC_BASE_URL" in message+ assert configured_base_url not in message+ assert "url-user" not in message+ assert "url-secret" not in message+++def test_anthropic_base_url_error_names_the_selected_precedence_variable_without_credentials() -> None:+ with pytest.raises(ProviderConfigurationError) as exc_info:+ resolve_llm_provider(+ {+ "SKILL_EVAL_LLM_PROVIDER": "anthropic",+ "ANTHROPIC_API_KEY": "test-anthropic-key",+ "SKILL_EVAL_LLM_BASE_URL": "https://url-user:[email protected]/v1",+ "ANTHROPIC_BASE_URL": "https://valid-fallback.example/v1",+ }+ )++ message = str(exc_info.value)+ assert "SKILL_EVAL_LLM_BASE_URL" in message+ assert "ANTHROPIC_BASE_URL" not in message+ assert "url-user" not in message+ assert "url-secret" not in message++def test_nvidia_build_uses_public_build_endpoint() -> None:config = resolve_llm_provider({diff --git a/tests/test_safety_boundaries.py b/tests/test_safety_boundaries.pyindex 12d57539..008bd10e 100644--- a/tests/test_safety_boundaries.py+++ b/tests/test_safety_boundaries.py@@ -5,10 +5,21 @@from __future__ import annotations+import pytest+from skillevaluator.utils.process_environment import child_process_envfrom skillevaluator.utils.redaction import redact_sensitive_data, redact_sensitive_text+def _pem_block(+ label: str,+ body: str = "synthetic-key-material-for-redaction-only",+ *,+ end_label: str | None = None,+) -> str:+ return "\n".join((f"-----BEGIN {label}-----", body, f"-----END {end_label or label}-----"))++def test_redact_sensitive_text_masks_common_credentials() -> None:source = ("Authorization: Bearer bearer-secret-value\n"@@ -26,6 +37,131 @@ def test_redact_sensitive_text_masks_common_credentials() -> None:assert redacted.count("<redacted>") >= 4+def test_redact_sensitive_text_masks_unlabelled_secret_shapes() -> None:+ aws_access_key = "".join(("AKIA", "IOSFODNN7", "EXAMPLE")) # noqa: FLY002 - scanner-safe fixture+ jwt = ".".join(+ (+ "".join(("eyJ", "hbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9")), # noqa: FLY002+ "".join(("eyJ", "zdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4ifQ")), # noqa: FLY002+ "SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c",+ )+ )+ private_key = "\n".join(+ (+ "-----BEGIN " + "RSA PRIVATE KEY-----",+ "synthetic-key-material-for-redaction-only",+ "-----END " + "RSA PRIVATE KEY-----",+ )+ )+ source = f"aws={aws_access_key}\njwt={jwt}\n{private_key}"++ redacted = redact_sensitive_text(source)++ assert aws_access_key not in redacted+ assert jwt not in redacted+ assert private_key not in redacted+ assert redacted.count("<redacted>") >= 3+++ "label",+ (+ "PRIVATE KEY",+ "RSA PRIVATE KEY",+ "OPENSSH PRIVATE KEY",+ "EC PRIVATE KEY",+ "DSA PRIVATE KEY",+ "ENCRYPTED PRIVATE KEY",+ "PGP PRIVATE KEY BLOCK",+ ),+)+def test_redact_sensitive_text_masks_standard_private_key_pem_labels(label: str) -> None:+ assert redact_sensitive_text(_pem_block(label)) == "private-key-<redacted>"+++def test_redact_sensitive_text_masks_mismatched_private_key_delimiters() -> None:+ source = _pem_block("RSA PRIVATE KEY", end_label="EC PRIVATE KEY")++ assert redact_sensitive_text(source) == "private-key-<redacted>"++[email protected]("label", ("PUBLIC KEY", "RSA PUBLIC KEY", "CERTIFICATE", "CERTIFICATE REQUEST"))+def test_redact_sensitive_text_preserves_non_private_pem_blocks(label: str) -> None:+ source = _pem_block(label, "synthetic-public-material")++ assert redact_sensitive_text(source) == source+++def test_redact_sensitive_text_redacts_multiple_private_key_blocks_independently() -> None:+ source = "\n".join(+ (+ _pem_block("RSA PRIVATE KEY", "first-synthetic-key-material"),+ "diagnostic separator",+ _pem_block("ENCRYPTED PRIVATE KEY", "second-synthetic-key-material"),+ )+ )++ assert redact_sensitive_text(source) == ("private-key-<redacted>\ndiagnostic separator\nprivate-key-<redacted>")+++def test_redact_sensitive_text_does_not_span_unrelated_pem_blocks() -> None:+ source = _pem_block(+ "RSA PRIVATE KEY",+ "\n".join(+ (+ "synthetic-key-material-for-redaction-only",+ _pem_block("CERTIFICATE", "synthetic-certificate-material"),+ )+ ),+ )++ assert redact_sensitive_text(source) == source+++def test_redact_sensitive_text_preserves_multiline_diagnostics_around_private_key() -> None:+ source = "\n".join(+ (+ "provider request failed",+ "diagnostic details follow",+ _pem_block("DSA PRIVATE KEY", "line-one\nline-two\nline-three"),+ "retry disabled",+ )+ )++ assert redact_sensitive_text(source) == (+ "provider request failed\ndiagnostic details follow\nprivate-key-<redacted>\nretry disabled"+ )++[email protected]("label", ("PRIVATE KEY", "DSA PRIVATE KEY", "ENCRYPTED PRIVATE KEY"))+def test_redact_sensitive_text_masks_truncated_private_key_through_eof(label: str) -> None:+ source = f"provider failed\n-----BEGIN {label}-----\nSENSITIVE-KEY-MATERIAL"++ assert redact_sensitive_text(source) == "provider failed\nprivate-key-<redacted>"++[email protected]("separator", ("=", ":"))+def test_redact_sensitive_text_masks_private_key_after_sensitive_assignment(separator: str) -> None:+ private_key = _pem_block("ENCRYPTED PRIVATE KEY", "SENSITIVE-KEY-MATERIAL")+ source = f"private_key{separator}{private_key}"++ redacted = redact_sensitive_text(source)++ assert "SENSITIVE-KEY-MATERIAL" not in redacted+ assert "BEGIN" not in redacted+ assert "END" not in redacted+++def test_redact_sensitive_text_masks_private_key_after_authorization_header() -> None:+ private_key = _pem_block("PRIVATE KEY", "SENSITIVE-KEY-MATERIAL")++ redacted = redact_sensitive_text(f"Authorization: Bearer {private_key}")++ assert "SENSITIVE-KEY-MATERIAL" not in redacted+ assert "BEGIN" not in redacted+ assert "END" not in redacted++def test_redact_sensitive_data_masks_secret_keys_and_nested_text() -> None:source = {"provider": {diff --git a/tests/test_tier3_progress.py b/tests/test_tier3_progress.pyindex e6278fbf..1de6e7dc 100644--- a/tests/test_tier3_progress.py+++ b/tests/test_tier3_progress.py@@ -9,6 +9,7 @@ import errnoimport importlibimport ioimport json+import osimport subprocessimport tempfileimport threading@@ -1684,8 +1685,20 @@ def test_runner_atomically_publishes_one_complete_final_result(def inspect_result_publication(source: str | Path, destination: str | Path, *args, **kwargs):destination_path = Path(destination)if destination_path.name == "result.json":- assert not destination_path.exists()- published_payloads.append(json.loads(Path(source).read_text(encoding="utf-8")))+ source_directory_fd = kwargs.get("src_dir_fd")+ destination_directory_fd = kwargs.get("dst_dir_fd")+ if destination_directory_fd is None:+ assert not destination_path.exists()+ else:+ with pytest.raises(FileNotFoundError):+ os.stat(destination_path.name, dir_fd=destination_directory_fd, follow_symlinks=False)+ if source_directory_fd is None:+ payload = Path(source).read_text(encoding="utf-8")+ else:+ descriptor = os.open(Path(source).name, os.O_RDONLY, dir_fd=source_directory_fd)+ with os.fdopen(descriptor, encoding="utf-8") as handle:+ payload = handle.read()+ published_payloads.append(json.loads(payload))return original_replace(source, destination, *args, **kwargs)monkeypatch.setattr(runner.os, "replace", inspect_result_publication)diff --git a/tests/test_tier3_public_runtime.py b/tests/test_tier3_public_runtime.pyindex 2c733773..f20e88b2 100644--- a/tests/test_tier3_public_runtime.py+++ b/tests/test_tier3_public_runtime.py@@ -6,16 +6,21 @@from __future__ import annotationsimport importlib.util+import jsonimport sysimport tomllib+import urllib.request+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServerfrom pathlib import Path+from threading import Threadfrom unittest.mock import Mock+import httpximport pytestfrom click.testing import CliRunnerfrom skillevaluator.cli import cli-from skillevaluator.provider_config import ProviderConfig+from skillevaluator.provider_config import ProviderConfig, resolve_llm_providerfrom skillevaluator.tier3 import commands as tier3_commandsfrom skillevaluator.tier3.evals_config import EvalsConfigError, load_evals_configfrom skillevaluator.tier3.harbor.adapter import _EVALUATOR_MANAGED_RUNTIME_ENV, _write_task_toml@@ -841,3 +846,328 @@ def test_generated_verifier_rejects_non_http_provider_base_urls(monkeypatch) ->verifier._resolve_url("openai")with pytest.raises(ValueError, match="absolute HTTP or HTTPS URL"):verifier._anthropic_url()+++ ("base_url", "expected_url"),+ [+ ("https://gateway.example", "https://gateway.example/v1/messages"),+ ("https://gateway.example/", "https://gateway.example/v1/messages"),+ ("https://gateway.example/v1", "https://gateway.example/v1/messages"),+ ("https://gateway.example/team", "https://gateway.example/team/v1/messages"),+ ("https://gateway.example/team/v1", "https://gateway.example/team/v1/messages"),+ ("https://gateway.example:8443/team/v1/", "https://gateway.example:8443/team/v1/messages"),+ ("http://gateway.internal:8080/v1", "http://gateway.internal:8080/v1/messages"),+ ("http://anthropic_proxy:8000/v1", "http://anthropic_proxy:8000/v1/messages"),+ ("http://127.0.0.1:8080/v1", "http://127.0.0.1:8080/v1/messages"),+ ("http://[::1]:8080/team/v1", "http://[::1]:8080/team/v1/messages"),+ (+ "http://[fe80::1%25eth0]:8080/team/v1",+ "http://[fe80::1%25eth0]:8080/team/v1/messages",+ ),+ ("https://xn--bcher-kva.example/v1", "https://xn--bcher-kva.example/v1/messages"),+ ("https://bücher.example/v1", "https://xn--bcher-kva.example/v1/messages"),+ ("https://faß.de/v1", "https://xn--fa-hia.de/v1/messages"),+ ("https://οδός.example/v1", "https://xn--pxavk3b.example/v1/messages"),+ (+ "https://bücher.example.:8443/bücher/v1",+ "https://xn--bcher-kva.example.:8443/b%C3%BCcher/v1/messages",+ ),+ ("https://gateway.example/caf%C3%A9/v1", "https://gateway.example/caf%C3%A9/v1/messages"),+ ("https://gateway.example/caf%c3%a9/v1", "https://gateway.example/caf%C3%A9/v1/messages"),+ ("https://gateway.example/opaque%ff/v1", "https://gateway.example/opaque%FF/v1/messages"),+ (+ "https://gateway.example/tenant%25west/v1",+ "https://gateway.example/tenant%25west/v1/messages",+ ),+ ("https://gateway.example/100%25/v1", "https://gateway.example/100%25/v1/messages"),+ ("https://gateway.example/team/%76%31", "https://gateway.example/team/v1/messages"),+ ("https://gateway.example/team/v1///", "https://gateway.example/team/v1/messages"),+ (+ "https://gateway.example/teams;v=1/@me+you/v1",+ "https://gateway.example/teams;v=1/@me+you/v1/messages",+ ),+ ],+)+def test_generated_verifier_builds_exactly_one_anthropic_native_messages_path(+ monkeypatch: pytest.MonkeyPatch,+ base_url: str,+ expected_url: str,+) -> None:+ verifier = _load_verifier_template()+ monkeypatch.delenv("SKILL_EVAL_LLM_BASE_URL", raising=False)+ monkeypatch.setenv("ANTHROPIC_BASE_URL", base_url)++ assert verifier._anthropic_url() == expected_url+++ ("variable", "base_url"),+ [+ ("SKILL_EVAL_LLM_BASE_URL", "https://gateway.example/v1/messages"),+ ("ANTHROPIC_BASE_URL", "https://url-user:[email protected]/v1"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/v1?token=url-secret"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/v1#token=url-secret"),+ ("ANTHROPIC_BASE_URL", "https://:443/v1"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example:not-a-port/v1"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example:/v1"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example\\team\\v1"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/v1\n"),+ ("ANTHROPIC_BASE_URL", "https:///team/v1"),+ ("ANTHROPIC_BASE_URL", "https://gateway example/v1"),+ ("ANTHROPIC_BASE_URL", "https://gateway%2eexample/v1"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example|evil/v1"),+ ("ANTHROPIC_BASE_URL", "https://999.1.1.1/v1"),+ ("ANTHROPIC_BASE_URL", "https://[v1.not-ipv6]/v1"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/%76%31/%6dessages"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/team%2Fv1"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/team%5cv1"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/team%0av1"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/team%"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/team%2"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/team%GG"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/../team/v1"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/%2e%2e/team/v1"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/%2576%2531/%256dessages"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/%252e%252e/team/v1"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/team%252Fv1"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/team%255Cv1"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/team%250Av1"),+ (+ "ANTHROPIC_BASE_URL",+ "https://gateway.example/%25%37%36%25%33%31/%25%36%64essages",+ ),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/team%25%32%46v1"),+ ("ANTHROPIC_BASE_URL", "https://gateway.example/team//v1"),+ ("ANTHROPIC_BASE_URL", "https://☃.example/v1"),+ ],+)+def test_generated_verifier_rejects_unsafe_anthropic_api_roots_without_echoing_credentials(+ monkeypatch: pytest.MonkeyPatch,+ variable: str,+ base_url: str,+) -> None:+ verifier = _load_verifier_template()+ monkeypatch.delenv("SKILL_EVAL_LLM_BASE_URL", raising=False)+ monkeypatch.delenv("ANTHROPIC_BASE_URL", raising=False)+ monkeypatch.setenv(variable, base_url)++ with pytest.raises(ValueError) as exc_info:+ verifier._anthropic_url()++ message = str(exc_info.value)+ assert variable in message+ assert base_url not in message+ assert "url-user" not in message+ assert "url-secret" not in message+++def test_generated_verifier_uses_the_official_anthropic_messages_url_when_base_is_unset(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ verifier = _load_verifier_template()+ monkeypatch.delenv("SKILL_EVAL_LLM_BASE_URL", raising=False)+ monkeypatch.delenv("ANTHROPIC_BASE_URL", raising=False)++ assert verifier._anthropic_url() == "https://api.anthropic.com/v1/messages"+++ ("configured_path", "expected_path"),+ [+ ("bücher/v1", "/b%C3%BCcher/v1/messages"),+ ("opaque%FF/v1", "/opaque%FF/v1/messages"),+ ("tenant%25west/v1", "/tenant%25west/v1/messages"),+ ("100%25/v1", "/100%25/v1/messages"),+ ("team/v1///", "/team/v1/messages"),+ ],+)+def test_anthropic_sdk_and_bundled_verifier_use_the_same_ascii_path(+ monkeypatch: pytest.MonkeyPatch,+ configured_path: str,+ expected_path: str,+) -> None:+ from anthropic import Anthropic++ requested_paths: list[str] = []++ class RecordingHandler(BaseHTTPRequestHandler):+ def do_POST(self) -> None:+ requested_paths.append(self.path)+ self.rfile.read(int(self.headers.get("Content-Length", "0")))+ body = json.dumps(+ {+ "id": "msg_test",+ "type": "message",+ "role": "assistant",+ "model": "claude-test",+ "content": [{"type": "text", "text": "ok"}],+ "stop_reason": "end_turn",+ "stop_sequence": None,+ "usage": {"input_tokens": 1, "output_tokens": 1},+ }+ ).encode()+ self.send_response(200)+ self.send_header("Content-Type", "application/json")+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ self.wfile.write(body)++ def log_message(self, *_args) -> None:+ return None++ verifier = _load_verifier_template()+ with ThreadingHTTPServer(("127.0.0.1", 0), RecordingHandler) as server:+ thread = Thread(target=server.serve_forever, daemon=True)+ thread.start()+ provider = resolve_llm_provider(+ {+ "SKILL_EVAL_LLM_PROVIDER": "anthropic",+ "ANTHROPIC_API_KEY": "test-anthropic-key",+ "ANTHROPIC_BASE_URL": f"http://127.0.0.1:{server.server_port}/{configured_path}",+ }+ )+ monkeypatch.delenv("SKILL_EVAL_LLM_BASE_URL", raising=False)+ monkeypatch.setenv("ANTHROPIC_API_KEY", "test-anthropic-key")+ monkeypatch.setenv("ANTHROPIC_BASE_URL", provider.base_url or "")+ try:+ with Anthropic(+ api_key="test-anthropic-key",+ base_url=provider.base_url,+ max_retries=0,+ timeout=5.0,+ ) as client:+ sdk_response = client.messages.create(+ model="claude-test",+ max_tokens=16,+ messages=[{"role": "user", "content": "hello"}],+ )+ verifier_response, verifier_error = verifier._call_anthropic("hello", "claude-test", 16, 0.0)+ finally:+ server.shutdown()+ thread.join(timeout=5)++ assert sdk_response.content[0].text == "ok"+ assert verifier_response == "ok"+ assert verifier_error is None+ assert requested_paths == [expected_path, expected_path]+++def test_anthropic_sdk_and_bundled_verifier_prepare_the_same_scoped_ipv6_endpoint(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ from anthropic import Anthropic++ sdk_urls: list[str] = []++ def handle_request(request: httpx.Request) -> httpx.Response:+ sdk_urls.append(str(request.url))+ return httpx.Response(+ 200,+ json={+ "id": "msg_test",+ "type": "message",+ "role": "assistant",+ "model": "claude-test",+ "content": [{"type": "text", "text": "ok"}],+ "stop_reason": "end_turn",+ "stop_sequence": None,+ "usage": {"input_tokens": 1, "output_tokens": 1},+ },+ )++ provider = resolve_llm_provider(+ {+ "SKILL_EVAL_LLM_PROVIDER": "anthropic",+ "ANTHROPIC_API_KEY": "test-anthropic-key",+ "ANTHROPIC_BASE_URL": "http://[fe80::1%25eth0]:8080/bücher/v1",+ }+ )+ verifier = _load_verifier_template()+ monkeypatch.delenv("SKILL_EVAL_LLM_BASE_URL", raising=False)+ monkeypatch.setenv("ANTHROPIC_BASE_URL", provider.base_url or "")++ with (+ httpx.Client(transport=httpx.MockTransport(handle_request)) as http_client,+ Anthropic(+ api_key="test-anthropic-key",+ base_url=provider.base_url,+ http_client=http_client,+ ) as client,+ ):+ client.messages.create(+ model="claude-test",+ max_tokens=16,+ messages=[{"role": "user", "content": "hello"}],+ )++ verifier_url = verifier._anthropic_url()+ verifier_request = urllib.request.Request(verifier_url)+ expected_url = "http://[fe80::1%25eth0]:8080/b%C3%BCcher/v1/messages"+ assert sdk_urls == [expected_url]+ assert verifier_request.full_url == expected_url+ assert verifier_request.selector == "/b%C3%BCcher/v1/messages"+++ ("unicode_host", "ascii_host"),+ [+ ("faß.de", "xn--fa-hia.de"),+ ("οδός.example", "xn--pxavk3b.example"),+ ],+)+def test_anthropic_idna_matches_httpx_sdk_and_bundled_verifier(+ monkeypatch: pytest.MonkeyPatch,+ unicode_host: str,+ ascii_host: str,+) -> None:+ from anthropic import Anthropic++ sdk_urls: list[str] = []++ def handle_request(request: httpx.Request) -> httpx.Response:+ sdk_urls.append(str(request.url))+ return httpx.Response(+ 200,+ json={+ "id": "msg_test",+ "type": "message",+ "role": "assistant",+ "model": "claude-test",+ "content": [{"type": "text", "text": "ok"}],+ "stop_reason": "end_turn",+ "stop_sequence": None,+ "usage": {"input_tokens": 1, "output_tokens": 1},+ },+ )++ configured_url = f"https://{unicode_host}/team/v1"+ provider = resolve_llm_provider(+ {+ "SKILL_EVAL_LLM_PROVIDER": "anthropic",+ "ANTHROPIC_API_KEY": "test-anthropic-key",+ "ANTHROPIC_BASE_URL": configured_url,+ }+ )+ expected_url = f"https://{ascii_host}/team/v1/messages"+ assert str(httpx.URL(configured_url)) == f"https://{ascii_host}/team/v1"++ verifier = _load_verifier_template()+ monkeypatch.delenv("SKILL_EVAL_LLM_BASE_URL", raising=False)+ monkeypatch.setenv("ANTHROPIC_BASE_URL", provider.base_url or "")+ with (+ httpx.Client(transport=httpx.MockTransport(handle_request)) as http_client,+ Anthropic(+ api_key="test-anthropic-key",+ base_url=provider.base_url,+ http_client=http_client,+ ) as client,+ ):+ client.messages.create(+ model="claude-test",+ max_tokens=16,+ messages=[{"role": "user", "content": "hello"}],+ )++ assert sdk_urls == [expected_url]+ assert verifier._anthropic_url() == expected_urldiff --git a/tests/test_verifier_error_redaction.py b/tests/test_verifier_error_redaction.pyindex ee3da0c2..acc3efec 100644--- a/tests/test_verifier_error_redaction.py+++ b/tests/test_verifier_error_redaction.py@@ -30,6 +30,8 @@ _CREDENTIALS = {"AWS_SECRET_ACCESS_KEY": "dummy-aws-secret-credential-DO-NOT-USE","AWS_SECURITY_TOKEN": "dummy-aws-legacy-security-token-DO-NOT-USE","AWS_SESSION_TOKEN": "dummy-aws-session-credential-DO-NOT-USE",+ "AWS_BEARER_TOKEN_BEDROCK": "dummy-bedrock-bearer-token-DO-NOT-USE",+ "AWS_CONTAINER_AUTHORIZATION_TOKEN": "dummy-container-auth-token-DO-NOT-USE",}_PROVIDER_ENV = (@@ -178,6 +180,25 @@ def test_exact_secret_redaction_uses_existing_eight_character_minimum(assert module._redact_configured_credentials("value=12345678") == "value=[REDACTED]"[email protected]("module_fixture", ["source_judge", "verifier_module"])+def test_judge_error_redacts_every_forwarded_credential(+ module_fixture: str,+ request: pytest.FixtureRequest,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ module = llm_judge if module_fixture == "source_judge" else request.getfixturevalue(module_fixture)+ for name, credential in _CREDENTIALS.items():+ monkeypatch.setenv(name, credential)++ result = module._judge_error("provider echoed " + " | ".join(_CREDENTIALS.values()))++ assert result["status"] == "error"+ assert result["score"] is None+ assert "[REDACTED]" in result["reason"]+ for credential in _CREDENTIALS.values():+ assert credential not in result["reason"]++def test_generated_call_public_llm_redacts_http_error_body(verifier_module,monkeypatch: pytest.MonkeyPatch,@@ -224,6 +245,29 @@ def test_generated_call_public_llm_redacts_generic_exception()+def test_generated_provider_error_preserves_selected_judge_provenance(+ verifier_module,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ credential = _CREDENTIALS["OPENAI_API_KEY"]+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai")+ monkeypatch.setenv("OPENAI_API_KEY", credential)++ def time_out(*_args, **_kwargs):+ raise TimeoutError(f"request timed out with {credential}")++ monkeypatch.setattr(verifier_module.urllib.request, "urlopen", time_out)++ content, error, provenance = verifier_module._call_public_llm_with_provenance(+ "safe prompt",+ model="selected-judge-model",+ )++ assert content is None+ assert error == "Public provider call failed for selected-judge-model: request timed out with [REDACTED]"+ assert provenance == {"provider": "openai", "model": "selected-judge-model"}++@pytest.mark.parametrize(("provider", "credential_name", "provider_call"),[@@ -355,6 +399,8 @@ def test_write_reward_outputs_recursively_redacts_credentials_in_actual_artifact_CREDENTIALS["AWS_SECRET_ACCESS_KEY"],_CREDENTIALS["AWS_SECURITY_TOKEN"],_CREDENTIALS["AWS_SESSION_TOKEN"],+ _CREDENTIALS["AWS_BEARER_TOKEN_BEDROCK"],+ _CREDENTIALS["AWS_CONTAINER_AUTHORIZATION_TOKEN"],],},),diff --git a/tests/tier3/test_behavior_evidence.py b/tests/tier3/test_behavior_evidence.pyindex 0c46ed73..9de57207 100644--- a/tests/tier3/test_behavior_evidence.py+++ b/tests/tier3/test_behavior_evidence.py@@ -472,6 +472,11 @@ def test_harbor_template_main_persists_evidence_refs_in_reward_json(monkeypatch,monkeypatch.setattr(module, "_judge_goal_accuracy_ragas", fake_ragas)monkeypatch.setattr(module, "call_public_llm", fake_call_public_llm)+ monkeypatch.setattr(+ module,+ "_call_public_llm_with_provenance",+ lambda prompt, *args, **kwargs: (*fake_call_public_llm(prompt, *args, **kwargs), {}),+ )module.main()diff --git a/tests/tier3/test_checks_secret_patterns.py b/tests/tier3/test_checks_secret_patterns.pyindex b6833fdd..b51c5f7e 100644--- a/tests/tier3/test_checks_secret_patterns.py+++ b/tests/tier3/test_checks_secret_patterns.py@@ -100,7 +100,9 @@ def test_real_key_in_tool_output_is_still_flagged(secret):assert result["passed"] is Falseassert result["score"] == 0.0- assert any(f["type"] == "secret_exposure" for f in result["findings"])+ [finding] = [f for f in result["findings"] if f["type"] == "secret_exposure"]+ assert finding["evidence"] == "[redacted secret exposure]"+ assert secret not in finding["evidence"]def test_real_key_in_agent_response_is_still_flagged():diff --git a/tests/tier3/test_harbor_template_secret_patterns.py b/tests/tier3/test_harbor_template_secret_patterns.pyindex 233551ab..edf655a2 100644--- a/tests/tier3/test_harbor_template_secret_patterns.py+++ b/tests/tier3/test_harbor_template_secret_patterns.py@@ -21,13 +21,7 @@ from pathlib import Pathimport pytest_TEMPLATE = (- Path(__file__).resolve().parents[2]- / "src"- / "skillevaluator"- / "tier3"- / "harbor"- / "templates"- / "eval.py"+ Path(__file__).resolve().parents[2] / "src" / "skillevaluator" / "tier3" / "harbor" / "templates" / "eval.py")@@ -78,9 +72,7 @@ def _read_call(observation: str) -> dict:@pytest.mark.parametrize("word", BENIGN_WORDS)def test_template_benign_words_in_observation_are_not_secrets(word):- result = eval_template.check_security(- _traj("Analysis complete."), [_read_call(f"The {word} pattern matters.")]- )+ result = eval_template.check_security(_traj("Analysis complete."), [_read_call(f"The {word} pattern matters.")])assert result["score"] == 1.0assert not any(f["type"] == "secret_exposure" for f in result["findings"])@@ -95,19 +87,17 @@ def test_template_benign_words_in_agent_response_are_not_secrets(word):@pytest.mark.parametrize("secret", REAL_SECRETS)def test_template_real_key_in_observation_is_flagged(secret):- result = eval_template.check_security(- _traj("Done."), [_read_call(f"config: {secret}")]- )+ result = eval_template.check_security(_traj("Done."), [_read_call(f"config: {secret}")])assert result["score"] == 0.0- assert any(f["type"] == "secret_exposure" for f in result["findings"])+ [finding] = [f for f in result["findings"] if f["type"] == "secret_exposure"]+ assert finding["evidence"] == "[redacted secret exposure]"+ assert secret not in finding["evidence"]@pytest.mark.parametrize("token", GLUED_NON_SECRETS)def test_template_glued_id_or_hash_tokens_are_not_secrets(token):- result = eval_template.check_security(- _traj("Done."), [_read_call(f"trial id: {token}")]- )+ result = eval_template.check_security(_traj("Done."), [_read_call(f"trial id: {token}")])assert result["score"] == 1.0assert not any(f["type"] == "secret_exposure" for f in result["findings"])@@ -145,9 +135,9 @@ def test_security_constants_stay_in_sync_with_eval_core(name):assert hasattr(eval_template, name), f"template missing {name}"assert hasattr(eval_core_checks, name), f"eval_core.checks missing {name}"- assert _normalize(getattr(eval_template, name)) == _normalize(- getattr(eval_core_checks, name)- ), f"{name} drifted between templates/eval.py and eval_core/checks.py"+ assert _normalize(getattr(eval_template, name)) == _normalize(getattr(eval_core_checks, name)), (+ f"{name} drifted between templates/eval.py and eval_core/checks.py"+ )@pytest.mark.parametrize(diff --git a/tests/tier3/test_judge_failure_artifacts.py b/tests/tier3/test_judge_failure_artifacts.pynew file mode 100644index 00000000..563ae65a--- /dev/null+++ b/tests/tier3/test_judge_failure_artifacts.py@@ -0,0 +1,358 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Fail-closed artifact regressions for required Tier 3 LLM judges."""++from __future__ import annotations++import importlib.util+import json+import math+import os+import shutil+import subprocess+import sys+from pathlib import Path+from types import ModuleType++import pytest++from skillevaluator.tier3.harbor.adapter import _write_test_sh+from skillevaluator.tier3.harbor.metrics import (+ DEFAULT_METRIC_SET,+ RESERVED_METRIC_NAMES,+ metric_set_for_reward,+ overall_score,+)+from skillevaluator.tier3.harbor.templates import custom_grader_runner++_REPO_ROOT = Path(__file__).resolve().parents[2]+_EVAL_TEMPLATE = _REPO_ROOT / "src" / "skillevaluator" / "tier3" / "harbor" / "templates" / "eval.py"+_CUSTOM_RUNNER_TEMPLATE = (+ _REPO_ROOT / "src" / "skillevaluator" / "tier3" / "harbor" / "templates" / "custom_grader_runner.py"+)+++def _load_verifier(tmp_path: Path) -> ModuleType:+ module_name = f"harbor_eval_failure_artifacts_{tmp_path.name}"+ spec = importlib.util.spec_from_file_location(module_name, _EVAL_TEMPLATE)+ assert spec and spec.loader+ module = importlib.util.module_from_spec(spec)+ sys.modules[module_name] = module+ spec.loader.exec_module(module)++ logs_dir = tmp_path / "logs"+ agent_dir = logs_dir / "agent"+ verifier_dir = logs_dir / "verifier"+ tests_dir = tmp_path / "tests"+ agent_dir.mkdir(parents=True)+ verifier_dir.mkdir(parents=True)+ tests_dir.mkdir(parents=True)++ module.LOGS_DIR = logs_dir+ module.AGENT_LOGS_DIR = agent_dir+ module.VERIFIER_DIR = verifier_dir+ module.TESTS_DIR = tests_dir+ module.ATIF_PATH = agent_dir / "trajectory.json"+ module.ENTRY_PATH = tests_dir / "entry.json"+ module.REWARD_JSON = verifier_dir / "reward.json"+ module.REWARD_TXT = verifier_dir / "reward.txt"+ module.SKILL_EVALUATOR_REWARD_JSON = verifier_dir / "skill_evaluator_reward.json"++ module.ATIF_PATH.write_text(+ json.dumps(+ {+ "steps": [+ {"source": "user", "message": "Complete the task."},+ {"source": "agent", "message": "The task is complete."},+ ]+ }+ ),+ encoding="utf-8",+ )+ module.ENTRY_PATH.write_text(+ json.dumps(+ {+ "id": "judge-artifact-case",+ "question": "Complete the task.",+ "ground_truth": "The task is complete.",+ "expected_behavior": ["Complete the task"],+ "should_trigger": False,+ "has_skill": True,+ }+ ),+ encoding="utf-8",+ )+ return module+++def test_verifier_main_fails_closed_after_collecting_every_required_judge(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ verifier = _load_verifier(tmp_path)+ credential = "dummy-secret-credential-DO-NOT-RETAIN"+ monkeypatch.setenv("ANTHROPIC_API_KEY", credential)+ calls: list[str] = []++ def accuracy(*_args, **_kwargs):+ calls.append("accuracy")+ return {"score": 0.0, "status": "error", "reason": f"HTTP 401 echoed {credential}"}++ def goal_accuracy(*_args, **_kwargs):+ calls.append("goal_accuracy")+ return {"score": True, "reason": "boolean is not a score"}++ def behavior_check(*_args, **_kwargs):+ calls.append("behavior_check")+ return {"score": math.inf, "reason": "non-finite score " + ("x" * 800)}++ monkeypatch.setattr(verifier, "judge_accuracy", accuracy)+ monkeypatch.setattr(verifier, "judge_goal_accuracy", goal_accuracy)+ monkeypatch.setattr(verifier, "judge_behavior_check", behavior_check)++ with pytest.raises(SystemExit) as exc_info:+ verifier.main()++ assert exc_info.value.code == 1+ assert calls == ["accuracy", "goal_accuracy", "behavior_check"]++ rich = json.loads(verifier.SKILL_EVALUATOR_REWARD_JSON.read_text(encoding="utf-8"))+ numeric = json.loads(verifier.REWARD_JSON.read_text(encoding="utf-8"))+ assert rich["accuracy"] is None+ assert rich["goal_accuracy"] is None+ assert rich["behavior_check"] is None+ assert rich["evaluation_status"] == "failed"+ assert rich["details"]["accuracy"]["status"] == "error"+ assert rich["details"]["goal_accuracy"]["status"] == "error"+ assert rich["details"]["behavior_check"]["status"] == "error"+ assert set(rich["evaluation_errors"]) == {"accuracy", "goal_accuracy", "behavior_check"}+ assert all(0 < len(reason) <= 512 for reason in rich["evaluation_errors"].values())++ assert numeric == {+ "security": 1.0,+ "skill_execution": 1.0,+ "skill_efficiency": 1.0,+ "overall": 0.0,+ }+ assert verifier.REWARD_TXT.read_text(encoding="utf-8") == "0.0"++ artifact_text = "\n".join(+ path.read_text(encoding="utf-8")+ for path in (+ verifier.SKILL_EVALUATOR_REWARD_JSON,+ verifier.REWARD_JSON,+ verifier.REWARD_TXT,+ )+ )+ assert credential not in artifact_text+ assert "[REDACTED]" in artifact_text++ # Harbor may retain only reward.json. Its canonical deterministic metrics+ # must still identify an incomplete default reward without the sidecar.+ verifier.SKILL_EVALUATOR_REWARD_JSON.unlink()+ assert metric_set_for_reward(numeric)[0] == DEFAULT_METRIC_SET+ assert overall_score(numeric) is None+++def test_verifier_main_keeps_genuine_zero_judge_verdicts_scoreable(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ verifier = _load_verifier(tmp_path)+ calls: list[str] = []++ def valid_zero(metric: str):+ def judge(*_args, **_kwargs):+ calls.append(metric)+ return {"score": 0.0, "reason": "valid model verdict"}++ return judge++ monkeypatch.setattr(verifier, "judge_accuracy", valid_zero("accuracy"))+ monkeypatch.setattr(verifier, "judge_goal_accuracy", valid_zero("goal_accuracy"))+ monkeypatch.setattr(verifier, "judge_behavior_check", valid_zero("behavior_check"))++ verifier.main()++ assert calls == ["accuracy", "goal_accuracy", "behavior_check"]+ rich = json.loads(verifier.SKILL_EVALUATOR_REWARD_JSON.read_text(encoding="utf-8"))+ numeric = json.loads(verifier.REWARD_JSON.read_text(encoding="utf-8"))+ assert "evaluation_status" not in rich+ assert "evaluation_errors" not in rich+ assert {metric: numeric[metric] for metric in verifier.DISPLAY_METRICS} == {+ "security": 1.0,+ "skill_execution": 1.0,+ "skill_efficiency": 1.0,+ "accuracy": 0.0,+ "goal_accuracy": 0.0,+ "behavior_check": 0.0,+ }+ assert numeric["overall"] == 0.5+ assert overall_score(numeric) == 0.5+++def test_verifier_main_keeps_documented_neutral_judge_skips_scoreable(+ tmp_path: Path,+) -> None:+ verifier = _load_verifier(tmp_path)+ entry = json.loads(verifier.ENTRY_PATH.read_text(encoding="utf-8"))+ entry["ground_truth"] = ""+ entry["expected_behavior"] = []+ verifier.ENTRY_PATH.write_text(json.dumps(entry), encoding="utf-8")++ verifier.main()++ rich = json.loads(verifier.SKILL_EVALUATOR_REWARD_JSON.read_text(encoding="utf-8"))+ numeric = json.loads(verifier.REWARD_JSON.read_text(encoding="utf-8"))+ assert "evaluation_status" not in rich+ assert {metric: numeric[metric] for metric in ("accuracy", "goal_accuracy", "behavior_check")} == {+ "accuracy": 1.0,+ "goal_accuracy": 1.0,+ "behavior_check": 1.0,+ }+ assert overall_score(numeric) == 1.0++[email protected]("failure_kind", ["missing-score", "exception"])+def test_verifier_main_normalizes_malformed_or_raised_judge_failures_and_continues(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+ failure_kind: str,+) -> None:+ verifier = _load_verifier(tmp_path)+ calls: list[str] = []++ def accuracy(*_args, **_kwargs):+ calls.append("accuracy")+ if failure_kind == "exception":+ raise RuntimeError("judge transport crashed")+ return {"reason": "judge omitted its score"}++ def successful(metric: str):+ def judge(*_args, **_kwargs):+ calls.append(metric)+ return {"score": 1.0, "reason": "valid verdict"}++ return judge++ monkeypatch.setattr(verifier, "judge_accuracy", accuracy)+ monkeypatch.setattr(verifier, "judge_goal_accuracy", successful("goal_accuracy"))+ monkeypatch.setattr(verifier, "judge_behavior_check", successful("behavior_check"))++ with pytest.raises(SystemExit) as exc_info:+ verifier.main()++ assert exc_info.value.code == 1+ assert calls == ["accuracy", "goal_accuracy", "behavior_check"]+ rich = json.loads(verifier.SKILL_EVALUATOR_REWARD_JSON.read_text(encoding="utf-8"))+ assert rich["accuracy"] is None+ assert rich["goal_accuracy"] == 1.0+ assert rich["behavior_check"] == 1.0+ assert rich["details"]["accuracy"]["status"] == "error"+ assert set(rich["evaluation_errors"]) == {"accuracy"}+ numeric = json.loads(verifier.REWARD_JSON.read_text(encoding="utf-8"))+ assert numeric["goal_accuracy"] == 1.0+ assert numeric["behavior_check"] == 1.0+ assert "accuracy" not in numeric+ assert overall_score(numeric) is None+++def test_numeric_reward_payload_excludes_boolean_and_non_finite_values(tmp_path: Path) -> None:+ verifier = _load_verifier(tmp_path)++ payload = verifier._numeric_reward_payload(+ {+ "finite_int": 1,+ "finite_float": 0.25,+ "boolean": True,+ "nan": math.nan,+ "positive_infinity": math.inf,+ "negative_infinity": -math.inf,+ "huge_integer": 10**1000,+ },+ 0.0,+ )++ assert payload == {"finite_int": 1.0, "finite_float": 0.25, "overall": 0.0}+ assert all(math.isfinite(value) and not isinstance(value, bool) for value in payload.values())+++def test_evaluation_failure_fields_are_reserved_metadata() -> None:+ expected = {"evaluation_status", "evaluation_errors"}++ assert expected <= RESERVED_METRIC_NAMES+ assert expected <= custom_grader_runner.RESERVED+ assert custom_grader_runner._extract_custom_metrics(+ {"evaluation_status": 1.0, "evaluation_errors": 0.5, "domain_score": 0.75}+ ) == {"domain_score": 0.75}+ with pytest.raises(RuntimeError, match="collides with reserved"):+ custom_grader_runner._extract_custom_metrics({"custom_metrics": {"evaluation_status": 0.5}})+++def _run_generated_test_sh(task_dir: Path, env: dict[str, str]) -> subprocess.CompletedProcess[str]:+ return subprocess.run(+ ["bash", str(task_dir / "tests" / "test.sh")],+ check=False,+ capture_output=True,+ text=True,+ env={**os.environ, **env},+ )++[email protected]("grading_mode", ["default", "default_plus_custom"])+def test_generated_standard_grading_scripts_stop_after_evaluator_failure(+ tmp_path: Path,+ grading_mode: str,+) -> None:+ task_dir = tmp_path / grading_mode+ _write_test_sh(task_dir, grading_mode=grading_mode, custom_grader=grading_mode == "default_plus_custom")+ tests_dir = task_dir / "tests"+ (tests_dir / "eval.py").write_text("raise SystemExit(7)\n", encoding="utf-8")+ marker = task_dir / "custom-ran"+ (tests_dir / "custom_grader_runner.py").write_text(+ "from pathlib import Path\nPath(" + repr(str(marker)) + ").write_text('ran')\n",+ encoding="utf-8",+ )++ completed = _run_generated_test_sh(task_dir, {"HARBOR_TESTS_DIR": str(tests_dir)})++ assert completed.returncode == 7+ assert not marker.exists()+++def test_generated_custom_only_script_accepts_overall_only_custom_reward(tmp_path: Path) -> None:+ task_dir = tmp_path / "custom-only"+ _write_test_sh(task_dir, grading_mode="custom_only", custom_grader=True)+ tests_dir = task_dir / "tests"+ shutil.copy2(_CUSTOM_RUNNER_TEMPLATE, tests_dir / "custom_grader_runner.py")+ marker = task_dir / "custom-ran"+ (tests_dir / "grader.py").write_text(+ "import json, os\n"+ "from pathlib import Path\n"+ f"Path({str(marker)!r}).write_text('ran')\n"+ "Path(os.environ['HARBOR_REWARD_JSON']).write_text(json.dumps({'overall': 0.75}))\n",+ encoding="utf-8",+ )+ verifier_dir = task_dir / "verifier"+ verifier_dir.mkdir()+ reward_json = verifier_dir / "reward.json"+ reward_txt = verifier_dir / "reward.txt"++ completed = _run_generated_test_sh(+ task_dir,+ {+ "HARBOR_TESTS_DIR": str(tests_dir),+ "HARBOR_VERIFIER_DIR": str(verifier_dir),+ "HARBOR_REWARD_JSON": str(reward_json),+ "HARBOR_REWARD_TXT": str(reward_txt),+ "HARBOR_CUSTOM_REWARD_JSON": str(verifier_dir / "custom_reward.json"),+ "HARBOR_GRADER": str(tests_dir / "grader.py"),+ "HARBOR_GRADER_SH": str(tests_dir / "grader.sh"),+ },+ )++ assert completed.returncode == 0, completed.stderr+ assert marker.read_text(encoding="utf-8") == "ran"+ assert json.loads(reward_json.read_text(encoding="utf-8")) == {"overall": 0.75}+ assert reward_txt.read_text(encoding="utf-8") == "0.75"diff --git a/tests/tier3/test_judge_parse_robustness.py b/tests/tier3/test_judge_parse_robustness.pyindex 71f3da77..70f18d7b 100644--- a/tests/tier3/test_judge_parse_robustness.py+++ b/tests/tier3/test_judge_parse_robustness.py@@ -22,7 +22,7 @@ Fix contract under test:- behavior judge requests a reasoning-safe ``max_tokens`` (>= 4096) by default;- one retry max with a "ONLY minified JSON" reminder on parse failure;- complete entries are salvaged from a truncated ``results`` array;-- final parse failure keeps score 0.0 but reports a diagnostic reason;+- final parse failure is a scoreless structured error with a bounded reason;- ``_extract_json`` tolerates fenced / prose-wrapped / trailing-junk responses;- the in-sandbox template copy (``harbor/templates/eval.py``) stays equivalent."""@@ -146,16 +146,105 @@ def test_extract_json_preserves_top_level_arrays():assert eval_template.extract_json('[{"suggestion": "s"}]') == [{"suggestion": "s"}]+def test_extract_json_preserves_prose_wrapped_top_level_arrays():+ text = 'prefix [{"score": 1.0}] trailing prose'+ expected = [{"score": 1.0}]++ assert llm_judge._extract_json(text) == expected+ assert eval_template.extract_json(text) == expected+++def test_extract_json_does_not_promote_object_from_truncated_outer_array():+ text = '[{"score":0}'++ assert llm_judge._extract_json(text) is None+ assert eval_template.extract_json(text) is None+++def test_extract_json_ignores_non_json_bracket_label_before_unique_object():+ text = 'Judge [draft]: {"score": 0.5}'+ expected = {"score": 0.5}++ assert llm_judge._extract_json(text) == expected+ assert eval_template.extract_json(text) == expected+++def test_extract_json_keeps_unique_object_before_unmatched_trailing_prose():+ text = '{"score": 0.5}\nNote [draft'+ expected = {"score": 0.5}++ assert llm_judge._extract_json(text) == expected+ assert eval_template.extract_json(text) == expected+++def test_extract_json_ignores_closed_non_json_label_before_unique_array():+ text = 'Judge [draft]: [{"score": 0.5}]'+ expected = [{"score": 0.5}]++ assert llm_judge._extract_json(text) == expected+ assert eval_template.extract_json(text) == expected+++def test_extract_json_does_not_cross_unclosed_structural_prefix():+ text = '[draft {"score": 0.5}'++ assert llm_judge._extract_json(text) is None+ assert eval_template.extract_json(text) is None+++ "text",+ [+ pytest.param('{"score": 0} {"score": 1}', id="adjacent-objects"),+ pytest.param(+ '{"score": 0}\n```json\n{"score": 1}\n```',+ id="object-and-conflicting-fence",+ ),+ ],+)+def test_extract_json_rejects_multiple_complete_documents(text):+ assert llm_judge._extract_json(text) is None+ assert eval_template.extract_json(text) is None+++def test_extract_json_returns_none_for_excessive_nesting():+ text = "[" * 10_000 + '{"score": 1.0}' + "]" * 10_000++ assert llm_judge._extract_json(text) is None+ assert eval_template.extract_json(text) is None+++ "text",+ [+ pytest.param('{"score": true, "score": 0.5}', id="top-level"),+ pytest.param('{"results": [{"passed": 1, "passed": true}]}', id="nested"),+ ],+)+def test_extract_json_rejects_duplicate_object_members(text):+ assert llm_judge._extract_json(text) is None+ assert eval_template.extract_json(text) is None++[email protected]("constant", ["NaN", "Infinity", "-Infinity"])+def test_extract_json_rejects_nonstandard_json_constants(constant):+ text = f'{{"metadata": {constant}}}'++ assert llm_judge._extract_json(text) is None+ assert eval_template.extract_json(text) is None++def test_behavior_judge_treats_list_payload_as_unparseable(monkeypatch):# A bare array is valid JSON but not a judge object; old code raised- # AttributeError on parsed.get -- now it takes the diagnostic-zero path.+ # AttributeError on parsed.get -- now it takes the structured-error path.calls: list[dict] = []monkeypatch.setattr(llm_judge, "call_public_llm", _scripted_hub(["[1, 2, 3]", "[1, 2, 3]"], calls))result = llm_judge.judge_behavior_check("conversation", ["b1"])assert len(calls) == 2- assert result["score"] == 0.0+ assert result["score"] is None+ assert result["status"] == "error"assert "unparseable" in result["reason"]@@ -227,7 +316,7 @@ def test_behavior_judge_retries_once_with_minified_json_reminder(monkeypatch):assert result["reason"] == "All expected behaviors were observed."-def test_behavior_judge_empty_reasoning_burn_yields_diagnostic_zero(monkeypatch):+def test_behavior_judge_empty_reasoning_burn_yields_scoreless_error(monkeypatch):calls: list[dict] = []monkeypatch.setattr(llm_judge,@@ -238,12 +327,13 @@ def test_behavior_judge_empty_reasoning_burn_yields_diagnostic_zero(monkeypatch)result = llm_judge.judge_behavior_check("conversation", ["b1"])assert len(calls) == 2 # one retry max, never more- assert result["score"] == 0.0+ assert result["score"] is None+ assert result["status"] == "error"assert result["results"] == []# Diagnostic, filterable reason -- never the old ambiguous string.assert result["reason"] != "Could not parse judge response"assert "unparseable" in result["reason"]- assert "len=0" in result["reason"]+ assert "response" in result["reason"].lower()def test_behavior_judge_salvages_truncated_results_after_retry(monkeypatch):diff --git a/tests/tier3/test_judge_result_semantics.py b/tests/tier3/test_judge_result_semantics.pynew file mode 100644index 00000000..c0d96835--- /dev/null+++ b/tests/tier3/test_judge_result_semantics.py@@ -0,0 +1,878 @@+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+# SPDX-License-Identifier: Apache-2.0++"""Fail-closed result semantics for the Tier 3 LLM judges."""++from __future__ import annotations++import importlib.util+import json+import math+from pathlib import Path+from types import ModuleType++import pytest++from skillevaluator.tier3.eval_core import llm_judge++_TEMPLATE = (+ Path(__file__).resolve().parents[2] / "src" / "skillevaluator" / "tier3" / "harbor" / "templates" / "eval.py"+)+_CRITERIA = {+ "SKILL_IDENTIFIED": True,+ "ACTION_CORRECT": False,+ "FACTUALLY_ACCURATE": True,+ "TASK_ADDRESSED": True,+ "ACTIONABLE": False,+}+++def _load_template_module() -> ModuleType:+ spec = importlib.util.spec_from_file_location("harbor_template_eval_judge_results", _TEMPLATE)+ assert spec and spec.loader+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ return module+++eval_template = _load_template_module()++[email protected](params=["shared", "template"])+def judge_module(request: pytest.FixtureRequest) -> ModuleType:+ return llm_judge if request.param == "shared" else eval_template+++def _pair_script(responses: list[tuple[str | None, str | None]], calls: list[str]):+ def fake_call(prompt: str, **_kwargs):+ calls.append(prompt)+ return responses[min(len(calls) - 1, len(responses) - 1)]++ return fake_call+++def _assert_error_result(result: dict) -> None:+ assert result["score"] is None+ assert result["status"] == "error"+ assert isinstance(result["reason"], str)+ assert 0 < len(result["reason"]) <= 512+++def test_judge_error_reserved_fields_cannot_be_overridden(judge_module) -> None:+ metadata = {+ "score": 1.0,+ "status": "ok",+ "reason": "spoofed success",+ "provider": "test-provider",+ }++ result = judge_module._judge_error("canonical failure", **metadata)++ assert result == {+ "score": None,+ "status": "error",+ "reason": "canonical failure",+ "provider": "test-provider",+ }+++def _judge_goal_with_response(+ module: ModuleType,+ monkeypatch: pytest.MonkeyPatch,+ *,+ content: str | None,+ error: str | None = None,+ provenance: dict | None = None,+) -> dict:+ if module is llm_judge:+ monkeypatch.setattr(module, "call_public_llm", lambda *_args, **_kwargs: (content, error))+ else:+ monkeypatch.setattr(module, "_ragas_goal_accuracy_enabled", lambda: False)+ monkeypatch.setattr(+ module,+ "_call_public_llm_with_provenance",+ lambda *_args, **_kwargs: (content, error, provenance or {}),+ )+ return module.judge_goal_accuracy("question", "ground truth", "agent response")+++ "content",+ [+ pytest.param("", id="empty"),+ pytest.param("YES YES YES YES YES", id="prose-yes-count"),+ pytest.param("[]", id="list"),+ pytest.param("{}", id="empty-object"),+ pytest.param('prefix [{"score": 1.0}] suffix', id="prose-wrapped-list"),+ pytest.param('{"score": true}', id="boolean-score"),+ pytest.param('{"score": true, "score": 0.5}', id="duplicate-score"),+ pytest.param('{"score": NaN}', id="nan-score"),+ pytest.param('{"score": Infinity}', id="infinite-score"),+ pytest.param('{"criteria": {"SKILL_IDENTIFIED": true}}', id="incomplete-criteria"),+ pytest.param(+ json.dumps({"criteria": {**_CRITERIA, "ACTIONABLE": "true"}}),+ id="non-boolean-criterion",+ ),+ pytest.param(+ json.dumps({"criteria": {**_CRITERIA, "EXTRA": True}}),+ id="extra-criterion",+ ),+ pytest.param('{"score": 0.5, "criteria": []}', id="numeric-score-with-wrong-criteria"),+ ],+)+def test_accuracy_rejects_unscorable_response_shapes(judge_module, monkeypatch, content: str) -> None:+ monkeypatch.setattr(judge_module, "call_public_llm", lambda *_args, **_kwargs: (content, None))++ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ _assert_error_result(result)+++ ("score", "expected"),+ [+ pytest.param(0.0, 0.0, id="genuine-zero"),+ pytest.param(-2, 0.0, id="clamp-low"),+ pytest.param(4.2, 1.0, id="clamp-high"),+ pytest.param(-(10**400), 0.0, id="clamp-huge-integer-low"),+ pytest.param(10**400, 1.0, id="clamp-huge-integer-high"),+ ],+)+def test_accuracy_accepts_finite_numeric_scores(judge_module, monkeypatch, score: float, expected: float) -> None:+ content = json.dumps({"score": score, "reason": "model judgment"})+ monkeypatch.setattr(judge_module, "call_public_llm", lambda *_args, **_kwargs: (content, None))++ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ assert result["score"] == expected+ assert result.get("status") != "error"++[email protected]("invalid_score", [pytest.param(None, id="absent"), pytest.param(True, id="boolean")])+def test_accuracy_derives_score_only_from_complete_boolean_criteria(+ judge_module,+ monkeypatch,+ invalid_score: bool | None,+) -> None:+ payload = {"criteria": _CRITERIA, "reason": "criteria judgment"}+ if invalid_score is not None:+ payload["score"] = invalid_score+ content = json.dumps(payload)+ monkeypatch.setattr(judge_module, "call_public_llm", lambda *_args, **_kwargs: (content, None))++ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ assert result["score"] == 0.6+ assert result["criteria"] == _CRITERIA+++def test_accuracy_provider_error_is_scoreless_bounded_and_redacted(judge_module, monkeypatch) -> None:+ credential = "dummy-judge-secret-DO-NOT-RETAIN"+ monkeypatch.setenv("OPENAI_API_KEY", credential)+ detail = f"HTTP 401: Unauthorized - echoed {credential} " + ("x" * 1000)+ monkeypatch.setattr(judge_module, "call_public_llm", lambda *_args, **_kwargs: (None, detail))++ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ _assert_error_result(result)+ assert credential not in result["reason"]+ assert "[REDACTED]" in result["reason"]+++def test_accuracy_accepts_unique_object_after_non_json_bracket_label(judge_module, monkeypatch) -> None:+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ lambda *_args, **_kwargs: ('Judge [draft]: {"score": 0.5}', None),+ )++ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ assert result["score"] == 0.5+ assert result.get("status") != "error"+++def test_accuracy_accepts_unique_object_before_unmatched_trailing_prose_without_retry(+ judge_module,+ monkeypatch,+) -> None:+ calls: list[str] = []+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _pair_script([('{"score": 0.5}\nNote [draft', None)], calls),+ )++ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ assert len(calls) == 1+ assert result["score"] == 0.5+ assert result.get("status") != "error"+++def test_accuracy_rejects_nonstandard_constant_in_reason(judge_module, monkeypatch) -> None:+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ lambda *_args, **_kwargs: ('{"score": 0.5, "reason": NaN}', None),+ )++ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ _assert_error_result(result)+++ "content",+ [+ pytest.param('{"score": 0} {"score": 1}', id="adjacent-objects"),+ pytest.param(+ '{"score": 0}\n```json\n{"score": 1}\n```',+ id="object-and-conflicting-fence",+ ),+ ],+)+def test_accuracy_rejects_multiple_complete_json_documents(judge_module, monkeypatch, content: str) -> None:+ monkeypatch.setattr(judge_module, "call_public_llm", lambda *_args, **_kwargs: (content, None))++ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ _assert_error_result(result)+++def test_accuracy_deep_nesting_is_scoreless_instead_of_raising(judge_module, monkeypatch) -> None:+ content = "[" * 10_000 + '{"score": 1.0}' + "]" * 10_000+ monkeypatch.setattr(judge_module, "call_public_llm", lambda *_args, **_kwargs: (content, None))++ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ _assert_error_result(result)+++ "content",+ [+ pytest.param("", id="empty"),+ pytest.param("achieved: yes", id="prose"),+ pytest.param("[]", id="list"),+ pytest.param("{}", id="empty-object"),+ pytest.param('prefix [{"achieved": true}] suffix', id="prose-wrapped-list"),+ pytest.param('{"achieved": 1}', id="integer-achieved"),+ pytest.param('{"achieved": 1, "achieved": true}', id="duplicate-achieved"),+ pytest.param('{"achieved": "true"}', id="string-achieved"),+ pytest.param('{"achieved": true, "score": true}', id="boolean-score"),+ pytest.param('{"achieved": true, "score": "0.8"}', id="string-score"),+ pytest.param('{"achieved": false, "score": NaN}', id="nan-score"),+ pytest.param('{"achieved": true, "score": Infinity}', id="infinite-score"),+ ],+)+def test_goal_rejects_unscorable_response_shapes(judge_module, monkeypatch, content: str) -> None:+ result = _judge_goal_with_response(judge_module, monkeypatch, content=content)++ _assert_error_result(result)+++ ("payload", "expected"),+ [+ pytest.param({"achieved": False, "score": 0.0}, 0.0, id="genuine-zero"),+ pytest.param({"achieved": True}, 1.0, id="derive-true"),+ pytest.param({"achieved": False}, 0.0, id="derive-false"),+ pytest.param({"achieved": False, "score": 0.25}, 0.25, id="finite-override"),+ pytest.param({"achieved": True, "score": -(10**400)}, 0.0, id="huge-integer-low"),+ pytest.param({"achieved": False, "score": 10**400}, 1.0, id="huge-integer-high"),+ ],+)+def test_goal_requires_boolean_achieved_and_uses_only_finite_numeric_optional_score(+ judge_module,+ monkeypatch,+ payload: dict,+ expected: float,+) -> None:+ payload = {"reason": "model judgment", "user_goal": "goal", "end_state": "state", **payload}++ result = _judge_goal_with_response(judge_module, monkeypatch, content=json.dumps(payload))++ assert result["score"] == expected+ assert result.get("status") != "error"+++def test_goal_accepts_unique_object_before_unmatched_trailing_prose_without_retry(+ judge_module,+ monkeypatch,+) -> None:+ calls: list[str] = []+ content = '{"achieved": true}\nNote [draft'+ if judge_module is llm_judge:+ monkeypatch.setattr(judge_module, "call_public_llm", _pair_script([(content, None)], calls))+ else:+ monkeypatch.setattr(judge_module, "_ragas_goal_accuracy_enabled", lambda: False)++ def fake_call(*_args, **_kwargs):+ calls.append("goal")+ return content, None, {}++ monkeypatch.setattr(judge_module, "_call_public_llm_with_provenance", fake_call)++ result = judge_module.judge_goal_accuracy("question", "ground truth", "agent response")++ assert len(calls) == 1+ assert result["score"] == 1.0+ assert result.get("status") != "error"+++def test_goal_rejects_nonstandard_constant_in_metadata(judge_module, monkeypatch) -> None:+ result = _judge_goal_with_response(+ judge_module,+ monkeypatch,+ content='{"achieved": true, "metadata": Infinity}',+ )++ _assert_error_result(result)+++def test_template_goal_preserves_provenance_on_success_and_error(monkeypatch) -> None:+ provenance = {"provider": "openai", "model": "judge-model"}++ success = _judge_goal_with_response(+ eval_template,+ monkeypatch,+ content='{"achieved": true, "reason": "ok"}',+ provenance=provenance,+ )+ failure = _judge_goal_with_response(+ eval_template,+ monkeypatch,+ content=None,+ error="request timed out",+ provenance=provenance,+ )++ assert {key: success[key] for key in provenance} == provenance+ assert {key: failure[key] for key in provenance} == provenance+ _assert_error_result(failure)+++def test_template_nonfinite_ragas_score_falls_back_to_custom(monkeypatch) -> None:+ custom_result = {+ "score": 0.0,+ "reason": "custom judgment",+ "method": "custom",+ "provider": "openai",+ "model": "judge-model",+ }+ monkeypatch.setattr(eval_template, "_ragas_goal_accuracy_enabled", lambda: True)+ monkeypatch.setattr(+ eval_template,+ "_judge_goal_accuracy_ragas",+ lambda *_args, **_kwargs: {"score": math.nan, "method": "ragas"},+ )+ monkeypatch.setattr(eval_template, "_judge_goal_accuracy_custom", lambda *_args, **_kwargs: custom_result)++ result = eval_template.judge_goal_accuracy("question", "ground truth", "agent response")++ assert result == custom_result+++def test_behavior_first_provider_error_is_scoreless_without_retry(judge_module, monkeypatch) -> None:+ calls: list[str] = []+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _pair_script([(None, "LLM judge model fallback exhausted: denied")], calls),+ )++ result = judge_module.judge_behavior_check("conversation", ["behavior"])++ assert len(calls) == 1+ _assert_error_result(result)+++ "content",+ [+ pytest.param("", id="empty"),+ pytest.param("not json", id="prose"),+ pytest.param("[]", id="list"),+ pytest.param("{}", id="empty-object"),+ pytest.param(+ 'prefix [{"results": [{"passed": true}], "score": 1.0}] suffix',+ id="prose-wrapped-list",+ ),+ pytest.param('{"results": "wrong", "score": 0}', id="non-list-results"),+ pytest.param('{"results": [1], "score": 0}', id="non-object-result"),+ pytest.param('{"results": [{"passed": 1}], "score": 1}', id="non-boolean-passed"),+ pytest.param(+ '{"results": [{"passed": 1, "passed": true}], "score": 1}',+ id="duplicate-passed",+ ),+ pytest.param('{"results": [], "score": 0}', id="incomplete-results"),+ pytest.param(+ '{"results": [{"passed": false}, {"passed": false}], "score": 0}',+ id="extra-results",+ ),+ pytest.param('{"results": [], "score": true}', id="boolean-score"),+ pytest.param('{"results": [], "score": NaN}', id="nan-score"),+ pytest.param('{"results": [], "score": Infinity}', id="infinite-score"),+ pytest.param(+ '{"results": [{"passed": true}], "score": NaN}',+ id="valid-results-with-nan-score",+ ),+ ],+)+def test_behavior_retries_malformed_or_invalid_shapes_once_then_errors(+ judge_module,+ monkeypatch,+ content: str,+) -> None:+ calls: list[str] = []+ monkeypatch.setattr(judge_module, "call_public_llm", _pair_script([(content, None), (content, None)], calls))++ result = judge_module.judge_behavior_check("conversation", ["behavior"])++ assert len(calls) == 2+ _assert_error_result(result)+ assert result["results"] == []+++ ("payload", "expected_behaviors", "expected"),+ [+ pytest.param(+ {"results": [{"passed": False}], "score": 0.0, "summary": "failed"},+ ["b1"],+ 0.0,+ id="explicit-zero",+ ),+ pytest.param(+ {"results": [{"passed": True}, {"passed": False}], "summary": "half"},+ ["b1", "b2"],+ 0.5,+ id="derived",+ ),+ pytest.param(+ {"results": [{"passed": False}], "score": 10**400, "summary": "valid integer"},+ ["b1"],+ 0.0,+ id="huge-integer-score",+ ),+ ],+)+def test_behavior_accepts_valid_zero_and_boolean_results(+ judge_module,+ monkeypatch,+ payload: dict,+ expected_behaviors: list[str],+ expected: float,+) -> None:+ calls: list[str] = []+ content = json.dumps(payload)+ monkeypatch.setattr(judge_module, "call_public_llm", _pair_script([(content, None)], calls))++ result = judge_module.judge_behavior_check("conversation", expected_behaviors)++ assert len(calls) == 1+ assert result["score"] == expected+ assert result.get("status") != "error"+++def test_behavior_accepts_unique_object_before_unmatched_trailing_prose_without_retry(+ judge_module,+ monkeypatch,+) -> None:+ calls: list[str] = []+ content = '{"results":[{"passed":true}]}\nNote [draft'+ monkeypatch.setattr(judge_module, "call_public_llm", _pair_script([(content, None)], calls))++ result = judge_module.judge_behavior_check("conversation", ["behavior"])++ assert len(calls) == 1+ assert result["score"] == 1.0+ assert result.get("status") != "error"+++def test_behavior_retries_nonstandard_constant_in_metadata_then_errors(+ judge_module,+ monkeypatch,+) -> None:+ calls: list[str] = []+ content = '{"results":[{"passed":true,"metadata":-Infinity}]}'+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _pair_script([(content, None), (content, None)], calls),+ )++ result = judge_module.judge_behavior_check("conversation", ["behavior"])++ assert len(calls) == 2+ _assert_error_result(result)+ assert result["results"] == []+++def test_behavior_retry_transport_failure_salvages_only_complete_first_entries(judge_module, monkeypatch) -> None:+ truncated = '{"results":[{"step":1,"passed":true,"reason":"observed"},{"step":2,"passed":false,"reason":"cut off'+ calls: list[str] = []+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _pair_script([(truncated, None), (None, "request timed out")], calls),+ )++ result = judge_module.judge_behavior_check("conversation", ["b1", "b2", "b3"])++ assert len(calls) == 2+ assert result["score"] == round(1 / 3, 4)+ assert result["results"] == [{"step": 1, "passed": True, "reason": "observed"}]+ assert "1/3" in result["reason"]+++def test_behavior_retry_transport_failure_without_salvage_is_redacted_error(judge_module, monkeypatch) -> None:+ credential = "dummy-retry-secret-DO-NOT-RETAIN"+ malformed = f"malformed response head containing {credential}"+ calls: list[str] = []+ monkeypatch.setenv("OPENAI_API_KEY", credential)+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _pair_script([(malformed, None), (None, f"timeout echoed {credential}")], calls),+ )++ result = judge_module.judge_behavior_check("conversation", ["behavior"])++ assert len(calls) == 2+ _assert_error_result(result)+ assert result["results"] == []+ assert credential not in result["reason"]+ assert "malformed response head" not in result["reason"]+++ "unfinished_entry",+ [+ pytest.param("{bad", id="bare-identifier-key"),+ pytest.param('{"x":,', id="missing-value-before-comma"),+ pytest.param('{"passed":false "x":', id="missing-member-comma"),+ pytest.param('{"passed":truX', id="invalid-literal"),+ pytest.param(+ '{"passed":true,"step":-\N{ARABIC-INDIC DIGIT ONE}',+ id="non-ascii-number-digit",+ ),+ pytest.param('{"passed":NaN', id="non-standard-nan-literal"),+ pytest.param('{"reason":"bad\x01', id="raw-control-in-string"),+ pytest.param('{"reason":"bad\\q', id="invalid-string-escape"),+ ],+)+def test_behavior_salvage_rejects_noncompletable_current_entry(+ judge_module,+ monkeypatch,+ unfinished_entry: str,+) -> None:+ malformed = '{"results":[{"passed":true},' + unfinished_entry+ calls: list[str] = []+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _pair_script([(malformed, None), (None, "request timed out")], calls),+ )++ result = judge_module.judge_behavior_check("conversation", ["b1", "b2"])++ assert len(calls) == 2+ _assert_error_result(result)+ assert result["results"] == []+++ "unfinished_entry",+ [+ pytest.param('{"passed":false,"reason":"cut', id="unfinished-string"),+ pytest.param('{"passed":false,"reason":', id="missing-value-at-end"),+ pytest.param('{"passed":fal', id="partial-false-literal"),+ ],+)+def test_behavior_salvage_accepts_append_only_completable_current_entry(+ judge_module,+ monkeypatch,+ unfinished_entry: str,+) -> None:+ truncated = '{"results":[{"passed":true},' + unfinished_entry+ calls: list[str] = []+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _pair_script([(truncated, None), (None, "request timed out")], calls),+ )++ result = judge_module.judge_behavior_check("conversation", ["b1", "b2"])++ assert len(calls) == 2+ assert result["score"] == 0.5+ assert result["results"] == [{"passed": True}]+++def test_behavior_retries_ambiguous_multiple_documents_then_errors(judge_module, monkeypatch) -> None:+ content = '{"results":[{"passed":false}]} {"results":[{"passed":true}]}'+ calls: list[str] = []+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _pair_script([(content, None), (content, None)], calls),+ )++ result = judge_module.judge_behavior_check("conversation", ["behavior"])++ assert len(calls) == 2+ _assert_error_result(result)+ assert result["results"] == []+++def test_behavior_deep_nesting_is_scoreless_instead_of_raising(judge_module, monkeypatch) -> None:+ content = '{"results":[{"passed":true,"metadata":' + "[" * 10_000 + "0" + "]" * 10_000 + "}]}"+ calls: list[str] = []+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _pair_script([(content, None), (content, None)], calls),+ )++ result = judge_module.judge_behavior_check("conversation", ["behavior"])++ assert len(calls) == 2+ _assert_error_result(result)+ assert result["results"] == []+++ "malformed",+ [+ pytest.param(+ '{"metadata":' + "[" * 900 + "0" + "]" * 900 + ',"results":[{"passed":true},{"passed":false',+ id="top-level-field",+ ),+ pytest.param(+ '{"results":[{"passed":true,"metadata":' + "[" * 900 + "0" + "]" * 900 + '},{"passed":false',+ id="complete-entry",+ ),+ pytest.param(+ '{"results":[{"passed":true},{"passed":false,"metadata":' + "[" * 900 + "0" + "]" * 900 + ',"reason":"cut',+ id="current-partial-entry",+ ),+ ],+)+def test_behavior_salvage_rejects_excessive_nesting(+ judge_module,+ monkeypatch,+ malformed: str,+) -> None:+ calls: list[str] = []+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _pair_script([(malformed, None), (None, "request timed out")], calls),+ )++ result = judge_module.judge_behavior_check("conversation", ["b1", "b2"])++ assert len(calls) == 2+ _assert_error_result(result)+ assert result["results"] == []+++ "malformed",+ [+ pytest.param(+ '{"metadata":NaN,"results":[{"passed":true},{"passed":false',+ id="top-level-field",+ ),+ pytest.param(+ '{"results":[{"passed":true,"metadata":Infinity},{"passed":false',+ id="complete-entry",+ ),+ pytest.param(+ '{"results":[{"passed":true},{"passed":false,"metadata":-Infinity,"reason":"cut',+ id="current-partial-entry",+ ),+ ],+)+def test_behavior_salvage_rejects_nonstandard_constants(+ judge_module,+ monkeypatch,+ malformed: str,+) -> None:+ calls: list[str] = []+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _pair_script([(malformed, None), (None, "request timed out")], calls),+ )++ result = judge_module.judge_behavior_check("conversation", ["b1", "b2"])++ assert len(calls) == 2+ _assert_error_result(result)+ assert result["results"] == []+++ "malformed",+ [+ pytest.param('{"results":null,"metadata":[{"passed":true}]', id="unrelated-array"),+ pytest.param('{"metadata":{"results":[{"passed":true}]', id="nested-results"),+ pytest.param('{"results":[{"passed":true} {"passed":false', id="missing-comma"),+ pytest.param(+ '{"results":[{"passed":true},{"passed":truX}]}',+ id="closed-invalid-token",+ ),+ pytest.param(+ '{"results":[{"passed":true},{"passed":false,}]}',+ id="closed-trailing-comma",+ ),+ pytest.param('{"score":true,"results":[{"passed":true},{"passed":false', id="boolean-score"),+ pytest.param('{"score":"0.5","results":[{"passed":true},{"passed":false', id="string-score"),+ pytest.param('{"score":NaN,"results":[{"passed":true},{"passed":false', id="nan-score"),+ pytest.param('{"score":Infinity,"results":[{"passed":true},{"passed":false', id="infinite-score"),+ pytest.param(+ '[{"results":[{"passed":true},{"passed":false',+ id="array-wrapper",+ ),+ pytest.param(+ '[] {"results":[{"passed":true},{"passed":false',+ id="complete-array-prefix",+ ),+ pytest.param(+ '}] {"results":[{"passed":true},{"passed":false',+ id="structural-prefix",+ ),+ pytest.param(+ '{"score":true,"score":0.5,"results":[{"passed":true},{"passed":false',+ id="duplicate-score",+ ),+ pytest.param(+ '{"results":[{"passed":1,"passed":true},{"passed":false',+ id="duplicate-passed",+ ),+ pytest.param(+ '{"results":\N{NO-BREAK SPACE}[{"passed":true},{"passed":false',+ id="non-json-nbsp",+ ),+ pytest.param(+ '{"results":\f[{"passed":true},{"passed":false',+ id="non-json-form-feed",+ ),+ pytest.param(+ '{"results":\v[{"passed":true},{"passed":false',+ id="non-json-vertical-tab",+ ),+ ],+)+def test_behavior_salvage_rejects_invalid_top_level_prefixes(+ judge_module,+ monkeypatch,+ malformed: str,+) -> None:+ calls: list[str] = []+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _pair_script([(malformed, None), (None, "request timed out")], calls),+ )++ result = judge_module.judge_behavior_check("conversation", ["b1", "b2"])++ assert len(calls) == 2+ _assert_error_result(result)+ assert result["results"] == []+++def test_behavior_salvage_keeps_finite_top_level_score_prefix(judge_module, monkeypatch) -> None:+ truncated = '{"score":0.5,"results":[{"passed":true},{"passed":false'+ calls: list[str] = []+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _pair_script([(truncated, None), (None, "request timed out")], calls),+ )++ result = judge_module.judge_behavior_check("conversation", ["b1", "b2"])++ assert result["score"] == 0.5+ assert result["results"] == [{"passed": True}]+++ "prefix", [pytest.param("Here is the JSON:\n", id="prose"), pytest.param("```json\n", id="fence")]+)+def test_behavior_salvage_tolerates_non_structural_wrappers(judge_module, monkeypatch, prefix: str) -> None:+ truncated = f'{prefix}{{"results":[{{"passed":true}},{{"passed":false'+ calls: list[str] = []+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _pair_script([(truncated, None), (None, "request timed out")], calls),+ )++ result = judge_module.judge_behavior_check("conversation", ["b1", "b2"])++ assert result["score"] == 0.5+ assert result["results"] == [{"passed": True}]+++def test_neutral_judge_skips_remain_unchanged(judge_module) -> None:+ assert judge_module.judge_accuracy("question", "", "agent response") == {+ "score": 1.0,+ "reason": "No ground_truth -- skipped",+ }+ assert judge_module.judge_goal_accuracy("question", "", "agent response") == {+ "score": 1.0,+ "reason": "No ground_truth -- skipped",+ }+ assert judge_module.judge_behavior_check("conversation", []) == {+ "score": 1.0,+ "reason": "No expected_behavior defined",+ "results": [],+ }+++def test_shared_and_template_accuracy_core_results_match(monkeypatch) -> None:+ responses = [+ json.dumps({"score": 0.0, "reason": "valid zero"}),+ json.dumps({"criteria": _CRITERIA, "reason": "derived"}),+ '{"score": NaN}',+ ]+ for content in responses:+ monkeypatch.setattr(+ llm_judge,+ "call_public_llm",+ lambda *_args, _content=content, **_kwargs: (_content, None),+ )+ monkeypatch.setattr(+ eval_template,+ "call_public_llm",+ lambda *_args, _content=content, **_kwargs: (_content, None),+ )++ assert eval_template.judge_accuracy("q", "gt", "answer") == llm_judge.judge_accuracy("q", "gt", "answer")+++def test_shared_and_template_behavior_results_match(monkeypatch) -> None:+ malformed = "not json"+ shared_calls: list[str] = []+ template_calls: list[str] = []+ monkeypatch.setattr(+ llm_judge,+ "call_public_llm",+ _pair_script([(malformed, None), (malformed, None)], shared_calls),+ )+ monkeypatch.setattr(+ eval_template,+ "call_public_llm",+ _pair_script([(malformed, None), (malformed, None)], template_calls),+ )++ shared = llm_judge.judge_behavior_check("conversation", ["behavior"])+ template = eval_template.judge_behavior_check("conversation", ["behavior"])++ assert template == shared+ assert len(shared_calls) == len(template_calls) == 2diff --git a/tests/tier3/test_suggestion_grounding.py b/tests/tier3/test_suggestion_grounding.pyindex aa5e76dc..392158a7 100644--- a/tests/tier3/test_suggestion_grounding.py+++ b/tests/tier3/test_suggestion_grounding.py@@ -226,9 +226,33 @@ def test_display_findings_report_writes_artifact(tmp_path, monkeypatch):"""Smoke the real findings report display path over a temporary run directory."""import json- trial_dir = tmp_path / "codex" / "with-skill" / "trials" / "case-001"+ condition_dir = tmp_path / "codex" / "with-skill"+ trial_dir = condition_dir / "trials" / "case-001"trial_dir.mkdir(parents=True)reward = _reward(0.1)+ (condition_dir / "summary.json").write_text(+ json.dumps(+ {+ "agent": "codex",+ "scores": {+ metric: reward[metric]+ for metric in (+ "security",+ "skill_execution",+ "skill_efficiency",+ "accuracy",+ "goal_accuracy",+ "behavior_check",+ )+ },+ "execution_status": "succeeded",+ "execution_errors": [],+ "expected_attempts": 1,+ "scored_attempts": 1,+ }+ ),+ encoding="utf-8",+ )(trial_dir / "reward.json").write_text(json.dumps(reward), encoding="utf-8")monkeypatch.setattr(report,@@ -246,6 +270,7 @@ def test_display_findings_report_writes_artifact(tmp_path, monkeypatch):"env_mode": "local","agents": {"codex": {+ "execution_status": "succeeded","model": "gpt-test","model_source": "test","with_skill": {diff --git a/uv.lock b/uv.lockindex 4bba41a9..f5a1861f 100644--- a/uv.lock+++ b/uv.lock@@ -1971,6 +1971,7 @@ version = "0.2.0"source = { editable = "." }dependencies = [{ name = "click" },+ { name = "idna" },{ name = "jinja2" },{ name = "pydantic" },{ name = "pyyaml" },@@ -2032,6 +2033,7 @@ requires-dist = [{ name = "click", specifier = ">=8.3.3,<9" },{ name = "harbor", marker = "extra == 'tier3'", specifier = "==0.13.2" },{ name = "httpx", marker = "extra == 'dev'", specifier = ">=0.27" },+ { name = "idna", specifier = ">=3.10,<4" },{ name = "jinja2", specifier = ">=3.1" },{ name = "litellm", marker = "extra == 'llm'", specifier = ">=1.83.10,<1.89.0.dev0" },{ name = "mcp", marker = "extra == 'tier3'", specifier = ">=1.28.1,<2" },