Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier3): retry malformed structured judge responses (#78)
Christopher Kevin5387261f6b24
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 6889c9c8..f54c4e84 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -6,6 +6,16 @@ All notable changes to SkillEvaluator are documented in this file.### Fixed+- Tier 3 accuracy and custom goal judges now retry one malformed (including+ empty) or schema-invalid response with a 4096-token output budget before+ failing closed, preventing a transient formatting error from making an otherwise+ successful trial and its full comparison arm unscoreable. Generated and+ injected Harbor verifier configs now reserve 600 seconds for six sequential+ direct provider attempts plus fail-closed artifact writes. Explicit native+ task timeouts remain owner-controlled and are not rewritten, and whole jobs+ defer to Harbor's task-configured phase controls instead of a hidden two-hour+ cap+ ([#70](https://github.com/NVIDIA/SkillEvaluator/issues/70)).- PII scanning no longer treats Markdown ATX headings as code comments, soemails in headings such as `# Contact: ...` are flagged. Hash lines insidePython strings, YAML scalars, and shell heredocs are scanned too. Realdiff --git a/docs/superpowers/plans/2026-08-24-issue-70-judge-retry.md b/docs/superpowers/plans/2026-08-24-issue-70-judge-retry.mdnew file mode 100644index 00000000..03155062--- /dev/null+++ b/docs/superpowers/plans/2026-08-24-issue-70-judge-retry.md@@ -0,0 +1,364 @@+# Issue 70 Judge Retry Parity Implementation Plan++> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.++**Goal:** Make Tier 3 accuracy and custom goal-accuracy judges recover from one malformed (including empty) or schema-invalid response without weakening fail-closed trial and arm scoring.++**Architecture:** Add one internal call-parse-validate-retry helper to both the shared judge module and the standalone Harbor verifier template. Accuracy and goal accuracy provide strict payload validators, default to exactly 4096 output tokens, and retry only parse/schema failures. The helper accepts pair-returning and provenance-returning call functions so the template goal judge keeps the provider/model from the decisive attempt; behavior-check salvage remains isolated. SkillEvaluator-managed Harbor verifier configs reserve 600 seconds for six sequential direct 90-second provider attempts plus artifact-write overhead, while authored native verifier timeouts remain unchanged. Provider SDK work outside the direct retry helper, including OpenAI RAGAS and Bedrock SDK retries, can require an operator-supplied Harbor timeout multiplier.++**Tech Stack:** Python 3.12/3.13, pytest, Harbor 0.13.2, Ruff, Docker, NVIDIA Build for live verification.++**Scope boundary:** Do not persist raw model responses. They can echo arbitrary task or credential material into portable reports, and provider transports do not consistently expose finish reason or usage. Safe response telemetry and rescore support require separate designs; issue #70's primary retry and token-budget defect is handled here.++---++### Task 1: Test-drive retry, budget, provenance, and fail-closed contracts++**Files:**+- Modify: `tests/tier3/test_judge_result_semantics.py`+- Modify: `tests/tier3/test_judge_failure_artifacts.py`+- Verify: `tests/tier3/test_judge_parse_robustness.py`+- Verify: `tests/test_issue55_collector_truth.py`++- [ ] **Step 1: Add failing unit tests for malformed and schema-invalid recovery**++In `tests/tier3/test_judge_result_semantics.py`, define valid retry payloads:++```python+VALID_ACCURACY_RESPONSE = json.dumps(+ {"criteria": _CRITERIA, "score": 0.6, "reason": "valid retry"}+)+VALID_GOAL_RESPONSE = json.dumps(+ {+ "user_goal": "complete the task",+ "end_state": "task completed",+ "achieved": True,+ "score": 1.0,+ "reason": "valid retry",+ }+)+```++For both values of the existing `judge_module` fixture, add accuracy tests covering:++- malformed text → valid response;+- schema-invalid complete criteria (`ACTIONABLE: "true"`) → valid response;+- malformed → malformed returns a structured error after exactly two calls;+- first provider error returns a structured error after exactly one call;+- malformed → retry provider error returns a bounded, redacted structured error after two calls;+- default `max_tokens` is exactly `4096` on both attempts;+- attempt two contains `previous reply could not be parsed or validated`.++Use a real scripted function at the judge transport boundary and assert returned judge behavior, not the mock itself:++```python+def test_accuracy_retries_schema_invalid_response_once_and_recovers(judge_module, monkeypatch):+ invalid = json.dumps({"criteria": {**_CRITERIA, "ACTIONABLE": "true"}, "score": 0.6})+ calls = []++ def fake_call(prompt, **kwargs):+ calls.append((prompt, kwargs))+ return (invalid, None) if len(calls) == 1 else (VALID_ACCURACY_RESPONSE, None)++ monkeypatch.setattr(judge_module, "call_public_llm", fake_call)+ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ assert result["score"] == 0.6+ assert [kwargs["max_tokens"] for _, kwargs in calls] == [4096, 4096]+ assert "previous reply could not be parsed or validated" in calls[1][0]+```++For goal accuracy, cover both shared and template surfaces explicitly. Shared tests patch `call_public_llm` with pair responses. Template tests disable RAGAS and patch `_call_public_llm_with_provenance` with triples. Cover:++- malformed → valid;+- schema-invalid `{"achieved": "true"}` → valid;+- invalid → invalid exactly twice;+- initial provider error exactly once;+- invalid → retry provider error twice with redaction;+- exactly `4096` on both attempts;+- template success and final error use the second attempt's `provider` and `model`.++Add shared explicit-override tests for accuracy and goal:++```python[email protected]("judge_name", ["judge_accuracy", "judge_goal_accuracy"])+def test_shared_structured_judge_keeps_explicit_max_tokens_override(monkeypatch, judge_name):+ calls = []+ response = VALID_ACCURACY_RESPONSE if judge_name == "judge_accuracy" else VALID_GOAL_RESPONSE++ def fake_call(prompt, **kwargs):+ calls.append(kwargs)+ return response, None++ monkeypatch.setattr(llm_judge, "call_public_llm", fake_call)+ judge = getattr(llm_judge, judge_name)+ judge("question", "ground truth", "agent response", max_tokens=2048)++ assert calls == [{"max_tokens": 2048}]+```++- [ ] **Step 2: Add failing real-verifier recovery and terminal-failure tests**++In `tests/tier3/test_judge_failure_artifacts.py`, use `_load_verifier` so the actual standalone `main()` writes real rich and numeric reward files.++Recovery test transport sequence:++```python+pair_responses = [+ ("not-json", None),+ (+ json.dumps(+ {+ "criteria": {+ "SKILL_IDENTIFIED": True,+ "ACTION_CORRECT": True,+ "FACTUALLY_ACCURATE": True,+ "TASK_ADDRESSED": True,+ "ACTIONABLE": True,+ },+ "score": 1.0,+ "reason": "accuracy recovered",+ }+ ),+ None,+ ),+ (json.dumps({"results": [{"step": 1, "passed": True}], "score": 1.0}), None),+]+goal_responses = [+ ("not-json", None, {"provider": "nv_build", "model": "first-model"}),+ (+ json.dumps({"achieved": True, "score": 1.0, "reason": "goal recovered"}),+ None,+ {"provider": "nv_build", "model": "retry-model"},+ ),+]+```++Patch only `call_public_llm` and `_call_public_llm_with_provenance`, disable RAGAS, run `verifier.main()`, then assert:++```python+assert "evaluation_status" not in rich+assert "evaluation_errors" not in rich+assert rich["details"]["accuracy"]["score"] == 1.0+assert rich["details"]["goal_accuracy"]["model"] == "retry-model"+assert numeric["accuracy"] == numeric["goal_accuracy"] == numeric["behavior_check"] == 1.0+assert [kwargs["max_tokens"] for _, kwargs in pair_calls] == [4096, 4096, 4096]+assert [kwargs["max_tokens"] for _, kwargs in goal_calls] == [4096, 4096]+```++Add a separate terminal-failure test where accuracy returns invalid text twice, goal and behavior return valid payloads, and the invalid text contains a configured synthetic credential. Assert:++```python+with pytest.raises(SystemExit) as exc_info:+ verifier.main()+assert exc_info.value.code == 1+assert len(accuracy_attempts) == 2+assert rich["evaluation_status"] == "failed"+assert rich["accuracy"] is None+assert rich["details"]["accuracy"]["status"] == "error"+assert len(rich["evaluation_errors"]["accuracy"]) <= 512+assert synthetic_credential not in json.dumps(rich)+assert "accuracy" not in numeric+```++This proves two invalid attempts still produce the deliberately incomplete sentinel consumed by the existing collector fail-closed test.++- [ ] **Step 3: Run the new tests and verify RED**++```bash+uv run python -m pytest -q \+ tests/tier3/test_judge_result_semantics.py \+ tests/tier3/test_judge_failure_artifacts.py \+ -k 'retry or max_tokens or malformed_accuracy_and_goal'+```++Expected: recovery/default-budget assertions fail because accuracy and goal currently call once at 1024 tokens. Confirm failures are behavioral assertions, not setup errors.++---++### Task 2: Implement one validated-JSON retry for accuracy and goal++**Files:**+- Modify: `src/skillevaluator/tier3/eval_core/llm_judge.py:565-730`+- Modify: `src/skillevaluator/tier3/harbor/templates/eval.py:3280-3470`++- [ ] **Step 1: Add shared constants and payload validators in both modules**++```python+STRUCTURED_JUDGE_MAX_TOKENS = 4096++_JUDGE_RETRY_REMINDER = (+ "\n\nIMPORTANT: Your previous reply could not be parsed or validated. Respond with ONLY the "+ "minified JSON object on a single line -- no markdown fences, no prose, and keep explanations brief."+)+++def _accuracy_payload_error(parsed):+ if not isinstance(parsed, dict):+ return "Judge response was not a valid JSON object"+ criteria = parsed.get("criteria")+ criteria_valid = _valid_accuracy_criteria(criteria)+ if "criteria" in parsed and not criteria_valid:+ return "Judge response contained invalid accuracy criteria"+ if _finite_score(parsed.get("score")) is None and not criteria_valid:+ return "Judge response contained no valid accuracy score or complete criteria"+ return None+++def _goal_payload_error(parsed):+ if not isinstance(parsed, dict):+ return "Judge response was not a valid JSON object"+ if not isinstance(parsed.get("achieved"), bool):+ return "Judge response contained an invalid achieved value"+ if "score" in parsed and _finite_score(parsed["score"]) is None:+ return "Judge response contained an invalid goal score"+ return None+```++Use existing module-style types in `llm_judge.py` and dependency-free Python in the standalone template.++- [ ] **Step 2: Add the internal call-parse-validate-retry helper**++Implement this contract in both modules, passing `_extract_json` in the shared module and `extract_json` in the template:++```python+def _call_validated_json_judge(prompt, validate, call, extract, **call_kwargs):+ call_kwargs.setdefault("max_tokens", STRUCTURED_JUDGE_MAX_TOKENS)++ def invoke(call_prompt):+ content, error, *metadata = call(call_prompt, **call_kwargs)+ provenance = metadata[0] if metadata and isinstance(metadata[0], dict) else {}+ parsed = extract(content) if content else None+ validation_error = validate(parsed) if not error else None+ return parsed, error, provenance, validation_error++ parsed, error, provenance, validation_error = invoke(prompt)+ if error:+ return None, f"LLM judge error: {error}", provenance+ if validation_error is None:+ return parsed, None, provenance++ parsed, error, provenance, validation_error = invoke(prompt + _JUDGE_RETRY_REMINDER)+ if error:+ return None, f"LLM judge retry error: {error}", provenance+ if validation_error is not None:+ return None, f"{validation_error} after retry", provenance+ return parsed, None, provenance+```++Do not include raw response content in error text or metadata. `_judge_error` remains the bounded/redacted terminal constructor.++- [ ] **Step 3: Route judges through the helper without changing scoring**++- Shared and template accuracy call the helper with `call_public_llm`, then compute the already-validated criteria/score exactly as before.+- Shared goal calls the helper with `call_public_llm`.+- Template custom goal calls the helper with `_call_public_llm_with_provenance` and includes returned provenance in both success and error results.+- Classify a successful provider response with empty model content separately from transport/configuration failures so the shared path retries it as malformed, matching the standalone template. Keep all genuine provider errors one-call.+- Initial provider errors return immediately; only parse/schema validation failures retry.+- Exactly one retry is permitted.+- Define `BEHAVIOR_JUDGE_MAX_TOKENS = STRUCTURED_JUDGE_MAX_TOKENS`; do not refactor behavior parsing or salvage into the generic helper.+- Raise only SkillEvaluator-managed generated/scaffolded/injected Harbor verifier defaults from 180 to 600 seconds. Preserve explicit native-task timeout overrides; existing native 180-second values must be updated explicitly when additional provider headroom is required.++- [ ] **Step 4: Run focused GREEN verification**++```bash+uv run python -m pytest -q \+ tests/tier3/test_judge_result_semantics.py \+ tests/tier3/test_judge_parse_robustness.py \+ tests/tier3/test_judge_failure_artifacts.py \+ tests/test_issue55_collector_truth.py::test_mixed_failed_condition_suppresses_published_quality_and_paired_artifacts+```++Expected: all pass. Recovery writes complete rewards; terminal retry exhaustion remains unscoreable and the collector still suppresses incomplete arm aggregates.++- [ ] **Step 5: Run touched-file lint and commit**++```bash+uv run ruff check \+ src/skillevaluator/tier3/eval_core/llm_judge.py \+ src/skillevaluator/tier3/harbor/templates/eval.py \+ tests/tier3/test_judge_result_semantics.py \+ tests/tier3/test_judge_parse_robustness.py \+ tests/tier3/test_judge_failure_artifacts.py+git add \+ src/skillevaluator/tier3/eval_core/llm_judge.py \+ src/skillevaluator/tier3/harbor/templates/eval.py \+ tests/tier3/test_judge_result_semantics.py \+ tests/tier3/test_judge_parse_robustness.py \+ tests/tier3/test_judge_failure_artifacts.py \+ docs/superpowers/plans/2026-08-24-issue-70-judge-retry.md+git commit -m "fix(tier3): retry malformed structured judge responses"+```++---++### Task 3: Independent review and end-to-end verification++**Files:**+- Review: `origin/main..HEAD`+- Live fixture: temporary one-case copy of `src/skillevaluator/tier3/reference_skills/calculator`++- [ ] **Step 1: Complete two-stage and whole-change review**++Dispatch spec compliance first, then code quality. Fix and re-review every Critical or Important issue. After task reviews pass, dispatch a fresh whole-change reviewer focused on retry count, schema-invalid paths, max-token overrides, provenance, redaction, shared/template drift, fail-closed artifacts, collector/report correctness, and regressions.++- [ ] **Step 2: Run full automated verification**++```bash+uv run python -m pytest -q -n auto+uv run ruff check src tests+uv build+```++Record exact pass/skip counts and build artifacts. Any failure must be diagnosed before proceeding.++- [ ] **Step 3: Run controlled live recovery with NVIDIA Build**++Load only `NVIDIA_API_KEY` from `/Users/christopherk/Work/.env` without sourcing or printing the file. Set:++```bash+export NVIDIA_API_KEY="$(sed -n 's/^NVIDIA_API_KEY=//p' /Users/christopherk/Work/.env | tail -n 1)"+export SKILL_EVAL_LLM_PROVIDER=nv_build+export SKILL_EVAL_LLM_MODEL=nvidia/nemotron-3-nano-30b-a3b+```++The explicit catalog model is required because the standalone verifier's fallback model is the OpenAI default rather than the provider-specific NVIDIA Build default.++For shared accuracy and goal, wrap the real `call_public_llm` so attempt one returns the judge-specific schema-invalid payload and attempt two delegates to the original real NVIDIA Build call.++Load a fresh standalone template module for each template judge. For template accuracy, wrap its real `call_public_llm` so attempt one returns schema-invalid accuracy JSON and attempt two delegates to the original real NVIDIA Build call. For template custom goal, wrap `_call_public_llm_with_provenance` the same way: attempt one returns invalid `achieved` with `{"provider": "controlled", "model": "invalid-first"}`, and attempt two delegates to the original real provider call. Assert:++- exactly two calls per judge;+- both attempts receive `max_tokens == 4096`;+- attempt two returns a finite score without `status: error`;+- template accuracy recovers through its standalone pair-returning transport;+- the template goal result's provider/model are the real second-attempt provenance;+- template goal's second-attempt model is `nvidia/nemotron-3-nano-30b-a3b`.++Then call real behavior-check normally and assert a finite score. This combines a controlled triggering first attempt with a real provider response on the recovery path; it does not claim that NVIDIA Build naturally emitted malformed output.++- [ ] **Step 4: Run paired Docker Tier 3 E2E**++```bash+ISSUE70_E2E_DIR="$(mktemp -d /tmp/skillevaluator-issue70-e2e.XXXXXX)"+cp -R src/skillevaluator/tier3/reference_skills/calculator "$ISSUE70_E2E_DIR/calculator"+uv run python -c 'import json, pathlib, sys; path = pathlib.Path(sys.argv[1]); entries = json.loads(path.read_text()); path.write_text(json.dumps([entries[0]], indent=2) + "\n")' \+ "$ISSUE70_E2E_DIR/calculator/evals/evals.json"+uv run skillevaluator tier3 evaluate "$ISSUE70_E2E_DIR/calculator" \+ --agents opencode \+ --env-mode docker \+ --n-attempts 1 \+ --harbor-keep-jobs+```++Keep the baseline arm enabled. Inspect `result.json`, both condition summaries, retained rich reward files, and rendered report output. Require full scored-attempt coverage and non-empty aggregates for both arms. If practical, repeat with `--agents claude-code`; otherwise report the exact untested boundary.++- [ ] **Step 5: Verify final branch state**++```bash+git status --short --branch+git log --oneline --decorate origin/main..HEAD+git diff --check origin/main..HEAD+```++Expected: clean branch, intentional commits only, and no whitespace errors.diff --git a/src/skillevaluator/inference/__init__.py b/src/skillevaluator/inference/__init__.pyindex 481c04cc..ceb8b93a 100644--- a/src/skillevaluator/inference/__init__.py+++ b/src/skillevaluator/inference/__init__.py@@ -9,9 +9,10 @@ for LLM-powered analysis tasks.from skillevaluator.inference.client import LLMClientfrom skillevaluator.inference.finding_verifier import FindingVerifier-from skillevaluator.inference.types import LLMClientError, LLMConfigError, LLMVerdict+from skillevaluator.inference.types import EmptyLLMResponseError, LLMClientError, LLMConfigError, LLMVerdict__all__ = [+ "EmptyLLMResponseError","FindingVerifier","LLMClient","LLMClientError",diff --git a/src/skillevaluator/inference/client.py b/src/skillevaluator/inference/client.pyindex 7738e6a7..15865210 100644--- a/src/skillevaluator/inference/client.py+++ b/src/skillevaluator/inference/client.py@@ -25,7 +25,7 @@ from typing import Anyfrom urllib.parse import urlsplitfrom skillevaluator.constants import LLM_VERIFY_MODEL, LLM_VERIFY_TEMPERATURE-from skillevaluator.inference.types import LLMClientError+from skillevaluator.inference.types import EmptyLLMResponseError, LLMClientErrorfrom skillevaluator.logging_config import get_loggerfrom skillevaluator.provider_config import (OPENAI_BASE_URL,@@ -254,7 +254,7 @@ class LLMClient:response = client.messages.create(**call_kwargs)content = "".join(str(block.text) for block in response.content if getattr(block, "type", None) == "text")if not content:- raise LLMClientError("LLM returned empty response content")+ raise EmptyLLMResponseError("LLM returned empty response content")return content.strip()if config.provider == "bedrock":try:@@ -275,7 +275,7 @@ class LLMClient:)content = response.choices[0].message.contentif not content:- raise LLMClientError("LLM returned empty response content")+ raise EmptyLLMResponseError("LLM returned empty response content")return str(content).strip()call_kwargs: dict[str, Any] = {"model": config.model,@@ -290,7 +290,7 @@ class LLMClient:response = client.chat.completions.create(**call_kwargs)content = response.choices[0].message.contentif not content:- raise LLMClientError("LLM returned empty response content")+ raise EmptyLLMResponseError("LLM returned empty response content")return content.strip()def extract_json_from_response(self, system_prompt: str, user_prompt: str) -> dict:diff --git a/src/skillevaluator/inference/types.py b/src/skillevaluator/inference/types.pyindex ba248604..4a22315b 100644--- a/src/skillevaluator/inference/types.py+++ b/src/skillevaluator/inference/types.py@@ -12,6 +12,10 @@ class LLMClientError(Exception):"""Raised when an LLM operation fails (missing key, bad response, etc.)."""+class EmptyLLMResponseError(LLMClientError):+ """Raised when a successful provider response contains no model text."""++LLMConfigError = LLMClientErrordiff --git a/src/skillevaluator/tier3/commands.py b/src/skillevaluator/tier3/commands.pyindex 217dfe46..6a7823a8 100644--- a/src/skillevaluator/tier3/commands.py+++ b/src/skillevaluator/tier3/commands.py@@ -30,7 +30,12 @@ from skillevaluator.tier3.case_ids import safe_child, validate_case_id, validatefrom skillevaluator.tier3.dataset_utils import DATASET_EXTENSIONS, load_dataset_entries_with_formatfrom skillevaluator.tier3.evals_config import CONFIG_FILENAMES, _validate_config, load_evals_configfrom skillevaluator.tier3.evals_spec import validate_harbor_contract, validate_skillevaluators-from skillevaluator.tier3.harbor import HARBOR_AGENTS, HARBOR_AGENTS_SUPPORTED, canonical_agent_name+from skillevaluator.tier3.harbor import (+ DEFAULT_LLM_VERIFIER_TIMEOUT_SEC,+ HARBOR_AGENTS,+ HARBOR_AGENTS_SUPPORTED,+ canonical_agent_name,+)from skillevaluator.tier3.harbor.metrics import DEFAULT_METRICS, LEGACY_METRICSfrom skillevaluator.tier3.harbor.progress import (NullProgressReporter,@@ -540,7 +545,7 @@ entry_id = {toml_quote(case_id)}timeout_sec = 300.0[verifier]-timeout_sec = 180.0+timeout_sec = {DEFAULT_LLM_VERIFIER_TIMEOUT_SEC}[environment]cpus = 2diff --git a/src/skillevaluator/tier3/eval_core/llm_judge.py b/src/skillevaluator/tier3/eval_core/llm_judge.pyindex 6ff1ffd5..91d75541 100644--- a/src/skillevaluator/tier3/eval_core/llm_judge.py+++ b/src/skillevaluator/tier3/eval_core/llm_judge.py@@ -17,6 +17,7 @@ import urllib.requestfrom typing import Anyfrom urllib.parse import urlparse+from skillevaluator.inference.types import EmptyLLMResponseErrorfrom skillevaluator.provider_config import CHAT_DEFAULT_OPENAI, _model_leaf, _supports_custom_temperaturelogger = logging.getLogger(__name__)@@ -27,6 +28,7 @@ DEFAULT_JUDGE_MODEL = CHAT_DEFAULT_OPENAI_ERROR_REDACTION_MARKER = "[REDACTED]"_JUDGE_ERROR_REASON_LIMIT = 512+_JUDGE_TEXT_LIMIT = 512# Match verifier log redaction; shorter placeholders can corrupt ordinary diagnostic text._MIN_EXACT_SECRET_LENGTH = 8_CREDENTIAL_ENV_VARS = (@@ -143,6 +145,14 @@ def _judge_error(error_reason: str, **metadata: Any) -> dict[str, Any]:return {**metadata, "score": None, "status": "error", "reason": safe_reason}+def _bounded_judge_text(value: Any) -> str:+ """Normalize trusted-shape model text before it reaches artifacts and reports."""+ text = _redact_configured_credentials(value).strip() if isinstance(value, str) else ""+ if len(text) > _JUDGE_TEXT_LIMIT:+ text = text[: _JUDGE_TEXT_LIMIT - 3] + "..."+ return text++def _finite_score(value: Any) -> float | None:if isinstance(value, bool) or not isinstance(value, (int, float)):return None@@ -237,6 +247,8 @@ def call_public_llm(temperature=temperature,)return client.completions("You are a precise evaluation judge.", prompt), None+ except EmptyLLMResponseError:+ return "", Noneexcept Exception as exc:detail = f"Public provider call failed: {exc}"return None, _redact_configured_credentials(detail, (api_key,))@@ -294,6 +306,13 @@ def _reject_nonstandard_json_constant(_value: str) -> None:raise ValueError("Non-standard JSON constant")+def _parse_finite_json_float(value: str) -> float:+ parsed = float(value)+ if not math.isfinite(parsed):+ raise ValueError("JSON number overflowed to a non-finite value")+ return parsed++def _json_nesting_within_limit(text: str) -> bool:"""Bound structural nesting without recursively parsing partial JSON."""depth = 0@@ -346,6 +365,7 @@ def _extract_json(text: str) -> dict[str, Any] | list[Any] | None:candidate,object_pairs_hook=_reject_duplicate_object_pairs,parse_constant=_reject_nonstandard_json_constant,+ parse_float=_parse_finite_json_float,)except (json.JSONDecodeError, RecursionError, ValueError):index = end@@ -409,6 +429,7 @@ def _is_append_only_json_object_prefix(fragment: str) -> bool:decoder = json.JSONDecoder(object_pairs_hook=_reject_duplicate_object_pairs,parse_constant=_reject_nonstandard_json_constant,+ parse_float=_parse_finite_json_float,)def _skip_whitespace(index: int) -> int:@@ -483,6 +504,7 @@ def _salvage_behavior_results(text: str) -> list[dict[str, Any]]:decoder = json.JSONDecoder(object_pairs_hook=_reject_duplicate_object_pairs,parse_constant=_reject_nonstandard_json_constant,+ parse_float=_parse_finite_json_float,)def _skip_whitespace(index: int) -> int:@@ -562,6 +584,44 @@ def _salvage_behavior_results(text: str) -> list[dict[str, Any]]:return []+STRUCTURED_JUDGE_MAX_TOKENS = 4096++_JUDGE_RETRY_REMINDER = (+ "\n\nIMPORTANT: Your previous reply could not be parsed or validated. Respond with ONLY the "+ "minified JSON object on a single line -- no markdown fences, no prose, and keep explanations brief."+)+++def _call_validated_json_judge(+ prompt: str,+ validate: Any,+ call: Any,+ extract: Any,+ **call_kwargs: Any,+) -> tuple[Any, str | None, dict[str, Any]]:+ call_kwargs.setdefault("max_tokens", STRUCTURED_JUDGE_MAX_TOKENS)++ def invoke(call_prompt: str) -> tuple[Any, str | None, dict[str, Any], str | None]:+ content, error, *metadata = call(call_prompt, **call_kwargs)+ provenance = metadata[0] if metadata and isinstance(metadata[0], dict) else {}+ parsed = extract(content) if content else None+ validation_error = validate(parsed) if not error else None+ return parsed, error, provenance, validation_error++ parsed, error, provenance, validation_error = invoke(prompt)+ if error:+ return None, f"LLM judge error: {error}", provenance+ if validation_error is None:+ return parsed, None, provenance++ parsed, error, provenance, validation_error = invoke(prompt + _JUDGE_RETRY_REMINDER)+ if error:+ return None, f"LLM judge retry error: {error}", provenance+ if validation_error is not None:+ return None, f"{validation_error} after retry", provenance+ return parsed, None, provenance++# ---------------------------------------------------------------------------# Accuracy judge (5-criterion)# ---------------------------------------------------------------------------@@ -612,6 +672,20 @@ def _valid_accuracy_criteria(value: Any) -> bool:)+def _accuracy_payload_error(parsed: Any) -> str | None:+ if not isinstance(parsed, dict):+ return "Judge response was not a valid JSON object"+ if "reason" in parsed and not isinstance(parsed["reason"], str):+ return "Judge response contained an invalid accuracy reason"+ criteria = parsed.get("criteria")+ criteria_valid = _valid_accuracy_criteria(criteria)+ if "criteria" in parsed and not criteria_valid:+ return "Judge response contained invalid accuracy criteria"+ if _finite_score(parsed.get("score")) is None and not criteria_valid:+ return "Judge response contained no valid accuracy score or complete criteria"+ return None++def judge_accuracy(question: str,ground_truth: str,@@ -628,29 +702,29 @@ def judge_accuracy(agent_text=agent_text,)- content, error = call_public_llm(prompt, **kwargs)+ parsed, error, _provenance = _call_validated_json_judge(+ prompt,+ _accuracy_payload_error,+ call_public_llm,+ _extract_json,+ **kwargs,+ )if error:- return _judge_error(f"LLM judge error: {error}")-- parsed = _extract_json(content) if content else None- if not isinstance(parsed, dict):- return _judge_error("Judge response was not a valid JSON object")+ return _judge_error(error)+ assert isinstance(parsed, dict)criteria = parsed.get("criteria")criteria_valid = _valid_accuracy_criteria(criteria)- if "criteria" in parsed and not criteria_valid:- return _judge_error("Judge response contained invalid accuracy criteria")score = _finite_score(parsed.get("score"))if score is None:- if not criteria_valid:- return _judge_error("Judge response contained no valid accuracy score or complete criteria")+ assert criteria_validyes_count = sum(1 for v in criteria.values() if v is True)score = yes_count / 5.0return {"score": round(score, 4),- "reason": parsed.get("reason", ""),+ "reason": _bounded_judge_text(parsed.get("reason", "")),"criteria": criteria if criteria_valid else {},}@@ -685,6 +759,19 @@ Respond with ONLY a JSON object:"achieved": true/false, "score": 1.0 or 0.0, "reason": "<brief explanation>"}}"""+def _goal_payload_error(parsed: Any) -> str | None:+ if not isinstance(parsed, dict):+ return "Judge response was not a valid JSON object"+ for field in ("reason", "user_goal", "end_state"):+ if field in parsed and not isinstance(parsed[field], str):+ return f"Judge response contained an invalid {field} value"+ if not isinstance(parsed.get("achieved"), bool):+ return "Judge response contained an invalid achieved value"+ if "score" in parsed and _finite_score(parsed["score"]) is None:+ return "Judge response contained an invalid goal score"+ return None++def judge_goal_accuracy(question: str,ground_truth: str,@@ -703,29 +790,30 @@ def judge_goal_accuracy(agent_text=agent_text,)- content, error = call_public_llm(prompt, **kwargs)+ parsed, error, _provenance = _call_validated_json_judge(+ prompt,+ _goal_payload_error,+ call_public_llm,+ _extract_json,+ **kwargs,+ )if error:- return _judge_error(f"LLM judge error: {error}")-- parsed = _extract_json(content) if content else None- if not isinstance(parsed, dict):- return _judge_error("Judge response was not a valid JSON object")+ return _judge_error(error)+ assert isinstance(parsed, dict)achieved = parsed.get("achieved")- if not isinstance(achieved, bool):- return _judge_error("Judge response contained an invalid achieved value")+ assert isinstance(achieved, bool)score = 1.0 if achieved else 0.0if "score" in parsed:score = _finite_score(parsed["score"])- if score is None:- return _judge_error("Judge response contained an invalid goal score")+ assert score is not Nonereturn {"score": score,- "reason": parsed.get("reason", ""),- "user_goal": parsed.get("user_goal", ""),- "end_state": parsed.get("end_state", ""),+ "reason": _bounded_judge_text(parsed.get("reason", "")),+ "user_goal": _bounded_judge_text(parsed.get("user_goal", "")),+ "end_state": _bounded_judge_text(parsed.get("end_state", "")),}@@ -768,7 +856,7 @@ def _compact_behavior_conversation(conversation_text: str, limit: int = 8000) -># reasoning tokens before emitting the per-behavior results array; the old 1024# cap was observed live to truncate behavior_check output to EMPTY content# (finish_reason="length", reasoning_tokens=1024).-BEHAVIOR_JUDGE_MAX_TOKENS = 4096+BEHAVIOR_JUDGE_MAX_TOKENS = STRUCTURED_JUDGE_MAX_TOKENS_BEHAVIOR_RETRY_REMINDER = ("\n\nIMPORTANT: Your previous reply could not be parsed. Respond with ONLY the "diff --git a/src/skillevaluator/tier3/harbor/__init__.py b/src/skillevaluator/tier3/harbor/__init__.pyindex d3387f9f..96abfa7a 100644--- a/src/skillevaluator/tier3/harbor/__init__.py+++ b/src/skillevaluator/tier3/harbor/__init__.py@@ -46,6 +46,11 @@ HARBOR_AGENTS = HARBOR_AGENTS_SUPPORTED | HARBOR_AGENTS_EXPERIMENTALAGENT_ALIASES = {"claude": "claude-code"}+# Accuracy, custom goal, and behavior can each make two sequential 90-second+# direct public-provider attempts. The remaining minute covers verifier+# overhead and fail-closed artifact writes.+DEFAULT_LLM_VERIFIER_TIMEOUT_SEC = 600.0+def canonical_agent_name(agent: str) -> str:"""Return the Harbor agent identifier for a public CLI/config name."""@@ -55,6 +60,7 @@ def canonical_agent_name(agent: str) -> str:__all__ = ["AGENT_ALIASES","DEFAULT_ENV_MODE",+ "DEFAULT_LLM_VERIFIER_TIMEOUT_SEC","ENV_MODE_LOCAL","HARBOR_AGENTS","HARBOR_AGENTS_EXPERIMENTAL",diff --git a/src/skillevaluator/tier3/harbor/adapter.py b/src/skillevaluator/tier3/harbor/adapter.pyindex 1d15bef4..6fb88d79 100644--- a/src/skillevaluator/tier3/harbor/adapter.py+++ b/src/skillevaluator/tier3/harbor/adapter.py@@ -35,6 +35,7 @@ from typing import Anyfrom urllib.parse import unquotefrom skillevaluator.tier3.case_ids import safe_child, validate_case_ids, validate_output_directory_path+from skillevaluator.tier3.harbor import DEFAULT_LLM_VERIFIER_TIMEOUT_SECfrom skillevaluator.tier3.harbor.secure_copy import (copy_file_secure,copytree_secure,@@ -1823,7 +1824,7 @@ has_skill = {str(has_skill).lower()}timeout_sec = 300.0[verifier]-timeout_sec = 180.0+timeout_sec = {DEFAULT_LLM_VERIFIER_TIMEOUT_SEC}[verifier.env]{_verifier_env_block(verifier_env if verifier_env is not None else runtime_env)}@@ -4347,7 +4348,13 @@ def _ensure_skill_evaluator_verifier_env(task_dir: Path, *, verifier_env: dict[sreturninsert_at = lines.index("[environment]") if "[environment]" in lines else len(lines)- lines[insert_at:insert_at] = ["[verifier]", "timeout_sec = 180.0", "", *env_block, ""]+ lines[insert_at:insert_at] = [+ "[verifier]",+ f"timeout_sec = {DEFAULT_LLM_VERIFIER_TIMEOUT_SEC}",+ "",+ *env_block,+ "",+ ]task_toml.write_text("\n".join(lines) + "\n", encoding="utf-8")diff --git a/src/skillevaluator/tier3/harbor/progress.py b/src/skillevaluator/tier3/harbor/progress.pyindex 37410af7..fa2e65f1 100644--- a/src/skillevaluator/tier3/harbor/progress.py+++ b/src/skillevaluator/tier3/harbor/progress.py@@ -218,7 +218,7 @@ class PlainProgressReporter:("preflight-trials", plan.preflight_trials),("containers", plan.total_containers),(- "task-timeout",+ "max-scaled-top-level-agent-timeout",None if plan.task_timeout_seconds is None else f"{plan.task_timeout_seconds:g}s",),)@@ -368,7 +368,7 @@ class RichProgressReporter(PlainProgressReporter):("Preflight trials", plan.preflight_trials),("Containers", plan.total_containers),(- "Task timeout",+ "Max scaled top-level agent timeout",None if plan.task_timeout_seconds is None else f"{plan.task_timeout_seconds:g}s",),)diff --git a/src/skillevaluator/tier3/harbor/report.py b/src/skillevaluator/tier3/harbor/report.pyindex c2bb22d8..91d557f1 100644--- a/src/skillevaluator/tier3/harbor/report.py+++ b/src/skillevaluator/tier3/harbor/report.py@@ -16,6 +16,7 @@ import mathfrom pathlib import Pathfrom typing import Any+from skillevaluator.tier3.eval_core.llm_judge import _redact_configured_credentialsfrom skillevaluator.tier3.harbor import report_datafrom skillevaluator.tier3.harbor.metrics import (DEFAULT_METRICS,@@ -24,11 +25,12 @@ from skillevaluator.tier3.harbor.metrics import (METRIC_QUESTIONS,extract_custom_metrics,)-from skillevaluator.utils.redaction import redact_sensitive_data+from skillevaluator.utils.redaction import redact_sensitive_data, redact_sensitive_textlogger = logging.getLogger(__name__)DISPLAY_METRICS = DEFAULT_METRICS+_REPORT_REASON_LIMIT = 512_METRIC_LABELS = {"security": "SECURITY (unsafe operations, secret leakage, unauthorized access)",@@ -301,9 +303,39 @@ def _render_findings_body(findings: list[dict[str, Any]]) -> Any:return body+def _bounded_report_reason(value: Any) -> str:+ """Coerce legacy/custom artifact values into bounded display text."""+ if isinstance(value, str):+ text = value.strip()+ else:+ try:+ text = json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))+ except (TypeError, ValueError, RecursionError):+ text = str(value)+ text = text.strip()+ text = redact_sensitive_text(_redact_configured_credentials(text))+ if len(text) > _REPORT_REASON_LIMIT:+ text = text[: _REPORT_REASON_LIMIT - 3] + "..."+ return text+++def _dedupe_report_reasons(reasons: list[Any]) -> list[str]:+ seen: set[str] = set()+ deduped: list[str] = []+ for value in reasons:+ reason = _bounded_report_reason(value)+ if not reason:+ continue+ key = reason[:80].lower()+ if key not in seen:+ seen.add(key)+ deduped.append(reason)+ return deduped++def _collect_fail_reasons(metric: str, trials: list[dict[str, Any]]) -> list[str]:"""Extract human-readable failure reasons from trial details."""- reasons: list[str] = []+ reasons: list[Any] = []for trial in trials:detail = trial["detail"]@@ -365,19 +397,12 @@ def _collect_fail_reasons(metric: str, trials: list[dict[str, Any]]) -> list[strif message:reasons.append(message)- seen: set[str] = set()- deduped: list[str] = []- for r in reasons:- key = r[:80].lower()- if key not in seen:- seen.add(key)- deduped.append(r)- return deduped+ return _dedupe_report_reasons(reasons)def _collect_pass_reasons(metric: str, trials: list[dict[str, Any]]) -> list[str]:"""Extract concise success reasons."""- reasons: list[str] = []+ reasons: list[Any] = []for trial in trials:detail = trial["detail"]if not isinstance(detail, dict):@@ -423,14 +448,7 @@ def _collect_pass_reasons(metric: str, trials: list[dict[str, Any]]) -> list[strif isinstance(check_data, dict) and check_data.get("passed") and check_data.get("reason"):reasons.append(check_data["reason"])- seen: set[str] = set()- deduped: list[str] = []- for r in reasons:- key = r[:80].lower()- if key not in seen:- seen.add(key)- deduped.append(r)- return deduped+ return _dedupe_report_reasons(reasons)def _build_evidence_ref_lookup(rewards: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:diff --git a/src/skillevaluator/tier3/harbor/runner.py b/src/skillevaluator/tier3/harbor/runner.pyindex c647b390..ff6d1b7d 100644--- a/src/skillevaluator/tier3/harbor/runner.py+++ b/src/skillevaluator/tier3/harbor/runner.py@@ -1267,16 +1267,17 @@ def _run_harbor()try:handoff = _nvidia_build_key_handoff(run_env, env_mode=env_mode)+ # Harbor owns its phase deadlines, and native tasks may intentionally+ # leave the agent unbounded. An outer deadline can preempt valid jobs.result = subprocess.run(command,capture_output=True,text=True,input=handoff.stdin_text,env=handoff.subprocess_env,- timeout=7200,check=False,)- except (OSError, subprocess.TimeoutExpired) as exc:+ except OSError as exc:return False, str(exc)if result.returncode == 0:return _validate_harbor_job_result(diff --git a/src/skillevaluator/tier3/harbor/templates/eval.py b/src/skillevaluator/tier3/harbor/templates/eval.pyindex 54ff6a7b..8627908d 100644--- a/src/skillevaluator/tier3/harbor/templates/eval.py+++ b/src/skillevaluator/tier3/harbor/templates/eval.py@@ -96,6 +96,7 @@ _UNRESERVED_BYTES = frozenset(b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuv_ERROR_REDACTION_MARKER = "[REDACTED]"_JUDGE_ERROR_REASON_LIMIT = 512+_JUDGE_TEXT_LIMIT = 512# Shorter placeholders are not credible provider credentials and can corrupt report schema keys._MIN_EXACT_SECRET_LENGTH = 8_CREDENTIAL_ENV_VARS = (@@ -1085,6 +1086,14 @@ def _judge_error(error_reason, **metadata):return {**metadata, "score": None, "status": "error", "reason": safe_reason}+def _bounded_judge_text(value):+ """Normalize trusted-shape model text before it reaches artifacts and reports."""+ text = _redact_configured_credentials(value).strip() if isinstance(value, str) else ""+ if len(text) > _JUDGE_TEXT_LIMIT:+ text = text[: _JUDGE_TEXT_LIMIT - 3] + "..."+ return text++def _finite_score(value):if isinstance(value, bool) or not isinstance(value, (int, float)):return None@@ -1517,6 +1526,8 @@ def _call_public_llm_with_provenance(prompt, model=None, max_tokens=1024, temperwith urllib.request.urlopen(request, timeout=90) as response: # nosec B310body = json.loads(response.read())content = body.get("choices", [{}])[0].get("message", {}).get("content", "")+ if content is None:+ content = ""if candidate_model != requested_model:logger.warning("LLM judge model %s failed; using fallback model %s", requested_model, candidate_model)return content.strip(), None, provenance@@ -1595,6 +1606,13 @@ def _reject_nonstandard_json_constant(_value):raise ValueError("Non-standard JSON constant")+def _parse_finite_json_float(value):+ parsed = float(value)+ if not math.isfinite(parsed):+ raise ValueError("JSON number overflowed to a non-finite value")+ return parsed++def _json_nesting_within_limit(text):"""Bound structural nesting without recursively parsing partial JSON."""depth = 0@@ -1647,6 +1665,7 @@ def extract_json(text):candidate,object_pairs_hook=_reject_duplicate_object_pairs,parse_constant=_reject_nonstandard_json_constant,+ parse_float=_parse_finite_json_float,)except (json.JSONDecodeError, RecursionError, ValueError):index = end@@ -1710,6 +1729,7 @@ def _is_append_only_json_object_prefix(fragment):decoder = json.JSONDecoder(object_pairs_hook=_reject_duplicate_object_pairs,parse_constant=_reject_nonstandard_json_constant,+ parse_float=_parse_finite_json_float,)def _skip_whitespace(index):@@ -1784,6 +1804,7 @@ def _salvage_behavior_results(text):decoder = json.JSONDecoder(object_pairs_hook=_reject_duplicate_object_pairs,parse_constant=_reject_nonstandard_json_constant,+ parse_float=_parse_finite_json_float,)def _skip_whitespace(index):@@ -3265,6 +3286,38 @@ def score_skill_execution(return {"score": round(avg, 4), "details": checks}+STRUCTURED_JUDGE_MAX_TOKENS = 4096++_JUDGE_RETRY_REMINDER = (+ "\n\nIMPORTANT: Your previous reply could not be parsed or validated. Respond with ONLY the "+ "minified JSON object on a single line -- no markdown fences, no prose, and keep explanations brief."+)+++def _call_validated_json_judge(prompt, validate, call, extract, **call_kwargs):+ call_kwargs.setdefault("max_tokens", STRUCTURED_JUDGE_MAX_TOKENS)++ def invoke(call_prompt):+ content, error, *metadata = call(call_prompt, **call_kwargs)+ provenance = metadata[0] if metadata and isinstance(metadata[0], dict) else {}+ parsed = extract(content) if content else None+ validation_error = validate(parsed) if not error else None+ return parsed, error, provenance, validation_error++ parsed, error, provenance, validation_error = invoke(prompt)+ if error:+ return None, f"LLM judge error: {error}", provenance+ if validation_error is None:+ return parsed, None, provenance++ parsed, error, provenance, validation_error = invoke(prompt + _JUDGE_RETRY_REMINDER)+ if error:+ return None, f"LLM judge retry error: {error}", provenance+ if validation_error is not None:+ return None, f"{validation_error} after retry", provenance+ return parsed, None, provenance++# ── LLM Judge: Accuracy (5-criterion) ────────────────────────────────────────@@ -3287,6 +3340,33 @@ def _valid_accuracy_criteria(value):)+def _accuracy_payload_error(parsed):+ if not isinstance(parsed, dict):+ return "Judge response was not a valid JSON object"+ if "reason" in parsed and not isinstance(parsed["reason"], str):+ return "Judge response contained an invalid accuracy reason"+ criteria = parsed.get("criteria")+ criteria_valid = _valid_accuracy_criteria(criteria)+ if "criteria" in parsed and not criteria_valid:+ return "Judge response contained invalid accuracy criteria"+ if _finite_score(parsed.get("score")) is None and not criteria_valid:+ return "Judge response contained no valid accuracy score or complete criteria"+ return None+++def _goal_payload_error(parsed):+ if not isinstance(parsed, dict):+ return "Judge response was not a valid JSON object"+ for field in ("reason", "user_goal", "end_state"):+ if field in parsed and not isinstance(parsed[field], str):+ return f"Judge response contained an invalid {field} value"+ if not isinstance(parsed.get("achieved"), bool):+ return "Judge response contained an invalid achieved value"+ if "score" in parsed and _finite_score(parsed["score"]) is None:+ return "Judge response contained an invalid goal score"+ return None++def judge_accuracy(question, ground_truth, agent_text):if not ground_truth:return {"score": 1.0, "reason": "No ground_truth -- skipped"}@@ -3315,26 +3395,26 @@ EXPECTED ANSWER:SELECTED EVIDENCE (final response + produced artifacts; low-relevance steps may be omitted):{agent_text}"""- content, error = call_public_llm(prompt)+ parsed, error, _provenance = _call_validated_json_judge(+ prompt,+ _accuracy_payload_error,+ call_public_llm,+ extract_json,+ )if error:- return _judge_error(f"LLM judge error: {error}")- parsed = extract_json(content) if content else None- if not isinstance(parsed, dict):- return _judge_error("Judge response was not a valid JSON object")+ return _judge_error(error)+ assert isinstance(parsed, dict)criteria = parsed.get("criteria")criteria_valid = _valid_accuracy_criteria(criteria)- if "criteria" in parsed and not criteria_valid:- return _judge_error("Judge response contained invalid accuracy criteria")score = _finite_score(parsed.get("score"))if score is None:- if not criteria_valid:- return _judge_error("Judge response contained no valid accuracy score or complete criteria")+ assert criteria_validscore = sum(1 for v in criteria.values() if v is True) / 5.0return {"score": round(score, 4),- "reason": parsed.get("reason", ""),+ "reason": _bounded_judge_text(parsed.get("reason", "")),"criteria": criteria if criteria_valid else {},}@@ -3442,27 +3522,28 @@ Did the agent achieve the expected goal?Respond with ONLY a JSON object:{{"user_goal": "...", "end_state": "...", "achieved": true/false, "score": 1.0, "reason": "..."}}"""- content, error, provenance = _call_public_llm_with_provenance(prompt)+ parsed, error, provenance = _call_validated_json_judge(+ prompt,+ _goal_payload_error,+ _call_public_llm_with_provenance,+ extract_json,+ )if error:- return _judge_error(f"LLM judge error: {error}", **provenance)- parsed = extract_json(content) if content else None- if not isinstance(parsed, dict):- return _judge_error("Judge response was not a valid JSON object", **provenance)+ return _judge_error(error, **provenance)+ assert isinstance(parsed, dict)achieved = parsed.get("achieved")- if not isinstance(achieved, bool):- return _judge_error("Judge response contained an invalid achieved value", **provenance)+ assert isinstance(achieved, bool)score = 1.0 if achieved else 0.0if "score" in parsed:score = _finite_score(parsed["score"])- if score is None:- return _judge_error("Judge response contained an invalid goal score", **provenance)+ assert score is not Nonereturn {"score": score,- "reason": parsed.get("reason", ""),- "user_goal": parsed.get("user_goal", ""),- "end_state": parsed.get("end_state", ""),+ "reason": _bounded_judge_text(parsed.get("reason", "")),+ "user_goal": _bounded_judge_text(parsed.get("user_goal", "")),+ "end_state": _bounded_judge_text(parsed.get("end_state", "")),"method": "custom",**provenance,}@@ -3474,7 +3555,7 @@ Respond with ONLY a JSON object:# reasoning tokens before emitting the per-behavior results array; the old 1024# cap was observed live to truncate behavior_check output to EMPTY content# (finish_reason="length", reasoning_tokens=1024).-BEHAVIOR_JUDGE_MAX_TOKENS = 4096+BEHAVIOR_JUDGE_MAX_TOKENS = STRUCTURED_JUDGE_MAX_TOKENS_BEHAVIOR_RETRY_REMINDER = ("\n\nIMPORTANT: Your previous reply could not be parsed. Respond with ONLY the "diff --git a/tests/inference/test_llm_client.py b/tests/inference/test_llm_client.pyindex 31544b6d..64356424 100644--- a/tests/inference/test_llm_client.py+++ b/tests/inference/test_llm_client.py@@ -10,7 +10,7 @@ from unittest.mock import MagicMock, patchimport pytest-from skillevaluator.inference import LLMClient, LLMClientError, LLMVerdict+from skillevaluator.inference import EmptyLLMResponseError, LLMClient, LLMClientError, LLMVerdictfrom skillevaluator.inference.client import _is_native_openai_endpoint, _token_limit_kwargsfrom skillevaluator.provider_config import (CHAT_DEFAULT_ANTHROPIC,@@ -329,6 +329,40 @@ class TestCompletions:result = client.completions("system", "user")assert result == "Hello world"+ def test_openai_compatible_empty_response_uses_typed_error(self, monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "nv_build")+ monkeypatch.setenv("NVIDIA_API_KEY", "test-key")+ mock_openai = MagicMock()+ mock_openai.chat.completions.create.return_value.choices = [MagicMock(message=MagicMock(content=""))]++ with (+ patch("openai.OpenAI", return_value=mock_openai),+ pytest.raises(EmptyLLMResponseError, match="empty response"),+ ):+ LLMClient().completions("system", "user")++ def test_anthropic_empty_response_uses_typed_error(self, monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "anthropic")+ monkeypatch.setenv("ANTHROPIC_API_KEY", "test-key")+ mock_anthropic = MagicMock()+ mock_anthropic.messages.create.return_value.content = []++ with (+ patch("anthropic.Anthropic", return_value=mock_anthropic),+ pytest.raises(EmptyLLMResponseError, match="empty response"),+ ):+ LLMClient().completions("system", "user")++ def test_bedrock_empty_response_uses_typed_error(self, monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "bedrock")+ response = SimpleNamespace(choices=[SimpleNamespace(message=SimpleNamespace(content=""))])++ with (+ patch("litellm.completion", return_value=response),+ pytest.raises(EmptyLLMResponseError, match="empty response"),+ ):+ LLMClient().completions("system", "user")+def test_none_temperature_is_omitted(self, monkeypatch: pytest.MonkeyPatch) -> None:class NoTemperatureClient(LLMClient):default_temperature = Nonediff --git a/tests/test_harbor_runner_status.py b/tests/test_harbor_runner_status.pyindex 06d3de02..ad2f0439 100644--- a/tests/test_harbor_runner_status.py+++ b/tests/test_harbor_runner_status.py@@ -7,6 +7,7 @@ from __future__ import annotationsimport importlibimport json+import subprocessimport sysimport threadingimport time@@ -561,6 +562,60 @@ def test_merge_attempt_jobs_rejects_source_destination_overlap(tmp_path: Path, rassert marker.read_text(encoding="utf-8") == "source"+ "task_toml",+ [+ "[agent]\ntimeout_sec = 300.0\n\n[verifier]\ntimeout_sec = 600.0\n",+ "[agent]\ntimeout_sec = 300.0\n",+ (+ "[agent]\ntimeout_sec = 300.0\n\n"+ '[[steps]]\nname = "first"\n\n'+ '[[steps]]\nname = "second"\n[steps.agent]\ntimeout_sec = 450.0\n'+ ),+ "[verifier]\ntimeout_sec = 600.0\n",+ ],+ ids=("single-step", "native-verifier-default", "multi-step", "unbounded-agent"),+)+def test_run_harbor_defers_to_harbor_phase_timeouts_without_outer_deadline(+ monkeypatch: pytest.MonkeyPatch,+ tmp_path: Path,+ task_toml: str,+) -> None:+ dataset = tmp_path / "dataset"+ for index in range(5):+ task = dataset / f"case-{index}"+ task.mkdir(parents=True)+ (task / "task.toml").write_text(task_toml, encoding="utf-8")+ captured: dict[str, object] = {}+ monkeypatch.setattr(runner, "build_harbor_run_command", lambda **_kwargs: ["harbor", "run"])++ def completed_run(command: list[str], **kwargs: object) -> subprocess.CompletedProcess[str]:+ captured.update(kwargs)+ return subprocess.CompletedProcess(command, 1, stdout="", stderr="expected test stop")++ monkeypatch.setattr(runner.subprocess, "run", completed_run)++ ok, _detail = runner._run_harbor(+ dataset=dataset,+ agent="opencode",+ job_name="demo-opencode-with",+ env_mode="docker",+ model="nvidia/openai/gpt-oss-120b",+ jobs_dir=tmp_path / "jobs",+ run_env={},+ n_attempts=3,+ n_concurrent=2,+ timeout_multiplier=1.5,+ override_cpus=None,+ override_memory_mb=None,+ override_storage_mb=None,+ expected_trials=15,+ )++ assert ok is False+ assert "timeout" not in captured++def _run(monkeypatch: pytest.MonkeyPatch,jobs_dir: Path,diff --git a/tests/test_harbor_runtime_skill_isolation.py b/tests/test_harbor_runtime_skill_isolation.pyindex 07aae84f..e432ba8a 100644--- a/tests/test_harbor_runtime_skill_isolation.py+++ b/tests/test_harbor_runtime_skill_isolation.py@@ -257,6 +257,7 @@ def test_native_tasks_preserve_authored_judge_model_in_verifier_env(if grading_mode == "default":expected["SKILL_EVAL_LLM_PROVIDER"] = "${SKILL_EVAL_LLM_PROVIDER}"assert task_config["verifier"]["env"] == expected+ assert task_config["verifier"]["timeout_sec"] == 180.0@pytest.mark.parametrize(@@ -284,6 +285,21 @@ def test_native_tasks_reject_judge_model_controls_in_agent_environment(tmp_path:stage_native_harbor_tasks(target, tmp_path / "native-agent-judge-control")+def test_native_task_injected_verifier_timeout_covers_all_structured_judge_attempts(tmp_path: Path) -> None:+ _, target, _, _ = _write_projection_fixture(tmp_path)+ _write_minimal_native_task(target)++ task = stage_native_harbor_tasks(+ target,+ tmp_path / "native-injected-verifier-timeout",+ grading_mode="default",+ verifier_env={"SKILL_EVAL_LLM_PROVIDER": "${SKILL_EVAL_LLM_PROVIDER}"},+ )[0]+ task_config = tomllib.loads((task / "task.toml").read_text(encoding="utf-8"))++ assert task_config["verifier"]["timeout_sec"] >= (3 * 2 * 90) + 60++def test_native_task_reuses_existing_exact_provider_placeholder_when_injecting(tmp_path: Path) -> None:_, target, _, _ = _write_projection_fixture(tmp_path)_write_minimal_native_task(target)diff --git a/tests/test_harbor_scaffold_transactions.py b/tests/test_harbor_scaffold_transactions.pyindex 16ce161f..b46a44e5 100644--- a/tests/test_harbor_scaffold_transactions.py+++ b/tests/test_harbor_scaffold_transactions.py@@ -27,6 +27,25 @@ def _skill(tmp_path: Path) -> Path:return skill+def test_harbor_scaffold_verifier_timeout_covers_all_structured_judge_attempts(tmp_path: Path) -> None:+ skill = _skill(tmp_path)++ assert (+ commands.init_harbor_task(+ skill,+ force=False,+ case_id="case-001",+ mode="default",+ language="python",+ with_config=True,+ )+ == 0+ )+ task = tomllib.loads((skill / "evals" / "harbor" / "case-001" / "task.toml").read_text(encoding="utf-8"))++ assert task["verifier"]["timeout_sec"] >= (3 * 2 * 90) + 60++@pytest.mark.parametrize("case_id", ["../outside", r"..\outside", "/tmp/outside", r"C:\outside"])def test_scaffold_rejects_escaping_case_id_before_mutation(tmp_path: Path, case_id: str) -> None:skill = _skill(tmp_path)diff --git a/tests/test_llm_judge.py b/tests/test_llm_judge.pyindex 88a08b37..de28a3f2 100644--- a/tests/test_llm_judge.py+++ b/tests/test_llm_judge.py@@ -5,6 +5,7 @@from __future__ import annotations+import jsonfrom unittest.mock import MagicMock, patchimport pytest@@ -103,6 +104,69 @@ def test_call_public_llm_uses_production_gpt5_payload_without_temperature(assert "temperature" not in call_kwargs+ ("judge_name", "valid_response"),+ [+ (+ "judge_accuracy",+ json.dumps(+ {+ "criteria": {+ "SKILL_IDENTIFIED": True,+ "ACTION_CORRECT": True,+ "FACTUALLY_ACCURATE": True,+ "TASK_ADDRESSED": True,+ "ACTIONABLE": True,+ },+ "score": 1.0,+ "reason": "recovered",+ }+ ),+ ),+ (+ "judge_goal_accuracy",+ json.dumps({"achieved": True, "score": 1.0, "reason": "recovered"}),+ ),+ ],+)+def test_shared_structured_judges_retry_empty_sdk_response(+ monkeypatch: pytest.MonkeyPatch,+ judge_name: str,+ valid_response: str,+) -> None:+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "nv_build")+ monkeypatch.setenv("NVIDIA_API_KEY", "test-key")+ monkeypatch.delenv("SKILL_EVAL_LLM_MODEL", raising=False)+ mock_openai = MagicMock()+ mock_openai.chat.completions.create.side_effect = [+ MagicMock(choices=[MagicMock(message=MagicMock(content=""))]),+ MagicMock(choices=[MagicMock(message=MagicMock(content=valid_response))]),+ ]++ with patch("openai.OpenAI", return_value=mock_openai):+ result = getattr(llm_judge, judge_name)("question", "ground truth", "agent response")++ assert result["score"] == 1.0+ assert mock_openai.chat.completions.create.call_count == 2+ calls = mock_openai.chat.completions.create.call_args_list+ assert [call.kwargs["max_tokens"] for call in calls] == [4096, 4096]+ assert "previous reply could not be parsed or validated" in calls[1].kwargs["messages"][1]["content"]+++def test_shared_structured_judge_does_not_retry_sdk_transport_error(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "nv_build")+ monkeypatch.setenv("NVIDIA_API_KEY", "test-key")+ mock_openai = MagicMock()+ mock_openai.chat.completions.create.side_effect = RuntimeError("transport unavailable")++ with patch("openai.OpenAI", return_value=mock_openai):+ result = llm_judge.judge_accuracy("question", "ground truth", "agent response")++ assert result["score"] is None+ assert result["status"] == "error"+ assert mock_openai.chat.completions.create.call_count == 1++@pytest.mark.parametrize(("provider", "request_url"),[diff --git a/tests/test_tier3_progress.py b/tests/test_tier3_progress.pyindex a2734761..d3df5bfb 100644--- a/tests/test_tier3_progress.py+++ b/tests/test_tier3_progress.py@@ -104,6 +104,48 @@ def test_auto_reporter_selects_terminal_appropriate_presentation(monkeypatch: pyassert isinstance(progress.create_progress_reporter("off", stream=_TTYBuffer()), progress.NullProgressReporter)+def test_plain_plan_labels_the_maximum_scaled_top_level_agent_timeout() -> None:+ progress = _progress_module()+ output = io.StringIO()+ reporter = progress.PlainProgressReporter(stream=output, refresh_interval=60)+ plan = _plan(progress)+ plan = progress.Tier3RunPlan(+ **{field: getattr(plan, field) for field in plan.__dataclass_fields__ if field != "task_timeout_seconds"},+ task_timeout_seconds=300.0,+ )++ reporter.start(plan)+ reporter.close()++ rendered = output.getvalue()+ assert "max-scaled-top-level-agent-timeout=300s" in rendered+ assert "task-timeout" not in rendered+++def test_rich_plan_labels_the_maximum_scaled_top_level_agent_timeout(monkeypatch: pytest.MonkeyPatch) -> None:+ import rich.live+ from rich.console import Console++ progress = _progress_module()+ _FakeLive.instances.clear()+ monkeypatch.setattr(rich.live, "Live", _FakeLive)+ plan = _plan(progress)+ plan = progress.Tier3RunPlan(+ **{field: getattr(plan, field) for field in plan.__dataclass_fields__ if field != "task_timeout_seconds"},+ task_timeout_seconds=300.0,+ )+ reporter = progress.RichProgressReporter(stream=_TTYBuffer(), refresh_interval=60)++ reporter.start(plan)+ rendered = io.StringIO()+ Console(file=rendered, force_terminal=False, width=180).print(_FakeLive.instances[0].renderables[-1])+ reporter.close()++ text = rendered.getvalue()+ assert "Max scaled top-level agent timeout" in text+ assert "Task timeout" not in text++@pytest.mark.parametrize("term", ["dumb", "unknown"])def test_auto_reporter_uses_immediate_plain_output_for_non_interactive_tty(monkeypatch: pytest.MonkeyPatch,diff --git a/tests/test_tier3_public_runtime.py b/tests/test_tier3_public_runtime.pyindex 05e9fbab..a0135446 100644--- a/tests/test_tier3_public_runtime.py+++ b/tests/test_tier3_public_runtime.py@@ -375,6 +375,21 @@ def test_generated_task_stages_public_provider_variables_for_the_verifier(tmp_paassert 'OPENAI_API_KEY = "${OPENAI_API_KEY}"' in task+def test_generated_task_verifier_timeout_covers_all_structured_judge_attempts(tmp_path) -> None:+ _write_task_toml(+ tmp_path,+ {"id": "timeout-test", "expected_skill": "demo"},+ has_skill=True,+ runtime_env={},+ )++ task = tomllib.loads((tmp_path / "task.toml").read_text(encoding="utf-8"))++ # Accuracy, custom goal, and behavior can each make two sequential+ # 90-second provider attempts. Leave a full minute for verifier overhead.+ assert task["verifier"]["timeout_sec"] >= (3 * 2 * 90) + 60++def test_generated_task_keeps_evaluator_provider_variables_out_of_agent_environment(tmp_path) -> None:_write_task_toml(tmp_path,diff --git a/tests/tier3/test_judge_failure_artifacts.py b/tests/tier3/test_judge_failure_artifacts.pyindex e38d1775..fbe8225f 100644--- a/tests/tier3/test_judge_failure_artifacts.py+++ b/tests/tier3/test_judge_failure_artifacts.py@@ -17,6 +17,7 @@ from types import ModuleTypeimport pytest+from skillevaluator.tier3.harbor import collector, reportfrom skillevaluator.tier3.harbor.adapter import _write_test_shfrom skillevaluator.tier3.harbor.metrics import (DEFAULT_METRIC_SET,@@ -193,6 +194,248 @@ def test_verifier_main_keeps_genuine_zero_judge_verdicts_scoreable(assert overall_score(numeric) == 0.5+def test_verifier_main_recovers_malformed_accuracy_and_goal_judges(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ verifier = _load_verifier(tmp_path)+ monkeypatch.setattr(verifier, "_ragas_goal_accuracy_enabled", lambda: False)+ pair_calls: list[tuple[str, dict]] = []+ goal_calls: list[tuple[str, dict]] = []+ pair_responses = [+ ("not-json", None),+ (+ json.dumps(+ {+ "criteria": {+ "SKILL_IDENTIFIED": True,+ "ACTION_CORRECT": True,+ "FACTUALLY_ACCURATE": True,+ "TASK_ADDRESSED": True,+ "ACTIONABLE": True,+ },+ "score": 1.0,+ "reason": "accuracy recovered",+ }+ ),+ None,+ ),+ (json.dumps({"results": [{"step": 1, "passed": True}], "score": 1.0}), None),+ ]+ goal_responses = [+ ("not-json", None, {"provider": "nv_build", "model": "first-model"}),+ (+ json.dumps({"achieved": True, "score": 1.0, "reason": "goal recovered"}),+ None,+ {"provider": "nv_build", "model": "retry-model"},+ ),+ ]++ def pair_call(prompt: str, **kwargs):+ pair_calls.append((prompt, kwargs))+ return pair_responses[len(pair_calls) - 1]++ def goal_call(prompt: str, **kwargs):+ goal_calls.append((prompt, kwargs))+ return goal_responses[len(goal_calls) - 1]++ monkeypatch.setattr(verifier, "call_public_llm", pair_call)+ monkeypatch.setattr(verifier, "_call_public_llm_with_provenance", goal_call)++ verifier.main()++ rich = json.loads(verifier.SKILL_EVALUATOR_REWARD_JSON.read_text(encoding="utf-8"))+ numeric = json.loads(verifier.REWARD_JSON.read_text(encoding="utf-8"))+ assert "evaluation_status" not in rich+ assert "evaluation_errors" not in rich+ assert rich["details"]["accuracy"]["score"] == 1.0+ assert rich["details"]["goal_accuracy"]["score"] == 1.0+ assert rich["details"]["goal_accuracy"]["model"] == "retry-model"+ assert numeric["accuracy"] == numeric["goal_accuracy"] == numeric["behavior_check"] == 1.0+ assert [kwargs["max_tokens"] for _, kwargs in pair_calls] == [4096, 4096, 4096]+ assert [kwargs["max_tokens"] for _, kwargs in goal_calls] == [4096, 4096]+ assert "previous reply could not be parsed or validated" in pair_calls[1][0]+ assert "previous reply could not be parsed or validated" in goal_calls[1][0]+++def test_verifier_retries_non_string_judge_text_before_collector_and_report(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ verifier = _load_verifier(tmp_path)+ monkeypatch.setattr(verifier, "_ragas_goal_accuracy_enabled", lambda: False)+ accuracy_calls: list[str] = []+ goal_calls: list[str] = []++ def pair_call(prompt: str, **_kwargs):+ if "SKILL_IDENTIFIED" not in prompt:+ return json.dumps({"results": [{"step": 1, "passed": True}], "score": 1.0}), None+ accuracy_calls.append(prompt)+ if len(accuracy_calls) == 1:+ return json.dumps({"score": 1.0, "reason": {"nested": "accuracy"}}), None+ return json.dumps({"score": 1.0, "reason": "accuracy recovered"}), None++ def goal_call(prompt: str, **_kwargs):+ goal_calls.append(prompt)+ if len(goal_calls) == 1:+ return (+ json.dumps(+ {+ "achieved": True,+ "score": 1.0,+ "reason": ["nested", "goal"],+ "user_goal": {"nested": "goal"},+ "end_state": ["nested", "state"],+ }+ ),+ None,+ {"provider": "nv_build", "model": "first-model"},+ )+ return (+ json.dumps(+ {+ "achieved": True,+ "score": 1.0,+ "reason": "goal recovered",+ "user_goal": "complete the task",+ "end_state": "task completed",+ }+ ),+ None,+ {"provider": "nv_build", "model": "retry-model"},+ )++ monkeypatch.setattr(verifier, "call_public_llm", pair_call)+ monkeypatch.setattr(verifier, "_call_public_llm_with_provenance", goal_call)++ verifier.main()++ collected = json.loads(verifier.REWARD_JSON.read_text(encoding="utf-8"))+ collector._merge_reward_sidecars(collected, verifier.VERIFIER_DIR)+ findings = report._extract_findings([collected])++ assert len(accuracy_calls) == 2+ assert len(goal_calls) == 2+ assert collected["details"]["accuracy"]["reason"] == "accuracy recovered"+ assert collected["details"]["goal_accuracy"]["reason"] == "goal recovered"+ assert all(isinstance(reason, str) for finding in findings for reason in finding["reasons"])+++ ("metric", "score", "detail"),+ [+ pytest.param("accuracy", 1.0, {"reason": {"nested": "a" * 600}}, id="accuracy-pass"),+ pytest.param("accuracy", 0.0, {"reason": ["nested", "accuracy"]}, id="accuracy-fail"),+ pytest.param(+ "goal_accuracy",+ 1.0,+ {"reason": ["nested", "goal"], "end_state": {"nested": "e" * 600}},+ id="goal-pass",+ ),+ pytest.param("goal_accuracy", 0.0, {"reason": {"nested": "goal"}}, id="goal-fail"),+ pytest.param(+ "behavior_check",+ 1.0,+ {"reason": {"nested": "summary"}, "results": [{"passed": True, "reason": "ok"}]},+ id="behavior-pass",+ ),+ pytest.param(+ "behavior_check",+ 0.0,+ {"reason": "failed", "results": [{"passed": False, "reason": {"nested": "step"}}]},+ id="behavior-fail",+ ),+ ],+)+def test_report_coerces_and_bounds_non_string_reasons_from_existing_artifacts(+ metric: str,+ score: float,+ detail: dict,+) -> None:+ reward = {+ "entry_id": "legacy-judge-artifact",+ metric: score,+ "details": {metric: detail},+ }++ findings = report._extract_findings([reward])++ finding = next(item for item in findings if item["metric"] == metric)+ assert finding["reasons"]+ assert all(isinstance(reason, str) for reason in finding["reasons"])+ assert all(len(reason) <= 512 for reason in finding["reasons"])+++def test_report_redacts_configured_secret_before_bounding_reason(monkeypatch: pytest.MonkeyPatch) -> None:+ credential = "SECRET-ABCDEFGHIJKLMNOPQRSTUVWXYZ-0123456789"+ prefix = "x" * 490+ monkeypatch.setenv("OPENAI_API_KEY", credential)+ reward = {+ "entry_id": "credential-boundary-artifact",+ "accuracy": 1.0,+ "details": {"accuracy": {"reason": prefix + credential}},+ }++ findings = report._extract_findings([reward])++ accuracy = next(item for item in findings if item["metric"] == "accuracy")+ assert accuracy["reasons"] == [prefix + "[REDACTED]"]+ assert credential not in accuracy["reasons"][0]+ assert "SECRET-" not in accuracy["reasons"][0]+++def test_verifier_main_keeps_accuracy_fail_closed_after_retry_exhaustion(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ verifier = _load_verifier(tmp_path)+ credential = "dummy-verifier-retry-secret-DO-NOT-RETAIN"+ monkeypatch.setenv("NVIDIA_API_KEY", credential)+ monkeypatch.setattr(verifier, "_ragas_goal_accuracy_enabled", lambda: False)+ pair_calls: list[tuple[str, dict]] = []+ pair_responses = [+ (f"not-json containing {credential}", None),+ (f"still-not-json containing {credential}", None),+ (json.dumps({"results": [{"step": 1, "passed": True}], "score": 1.0}), None),+ ]+ goal_calls: list[tuple[str, dict]] = []++ def pair_call(prompt: str, **kwargs):+ pair_calls.append((prompt, kwargs))+ return pair_responses[len(pair_calls) - 1]++ def goal_call(prompt: str, **kwargs):+ goal_calls.append((prompt, kwargs))+ return (+ json.dumps({"achieved": True, "score": 1.0, "reason": "goal valid"}),+ None,+ {"provider": "nv_build", "model": "goal-model"},+ )++ monkeypatch.setattr(verifier, "call_public_llm", pair_call)+ monkeypatch.setattr(verifier, "_call_public_llm_with_provenance", goal_call)++ with pytest.raises(SystemExit) as exc_info:+ verifier.main()++ assert exc_info.value.code == 1+ assert len(pair_calls) == 3+ accuracy_attempts = [call for call in pair_calls if "SKILL_IDENTIFIED" in call[0]]+ assert len(accuracy_attempts) == 2+ assert [kwargs["max_tokens"] for _, kwargs in accuracy_attempts] == [4096, 4096]+ assert len(goal_calls) == 1+ assert "previous reply could not be parsed or validated" in pair_calls[1][0]+ assert "previous reply could not be parsed or validated" not in pair_calls[2][0]+ rich = json.loads(verifier.SKILL_EVALUATOR_REWARD_JSON.read_text(encoding="utf-8"))+ numeric = json.loads(verifier.REWARD_JSON.read_text(encoding="utf-8"))+ assert rich["evaluation_status"] == "failed"+ assert rich["accuracy"] is None+ assert rich["details"]["accuracy"]["status"] == "error"+ assert len(rich["evaluation_errors"]["accuracy"]) <= 512+ assert credential not in json.dumps(rich)+ assert "accuracy" not in numeric++def test_verifier_main_keeps_documented_neutral_judge_skips_scoreable(tmp_path: Path,) -> None:diff --git a/tests/tier3/test_judge_parse_robustness.py b/tests/tier3/test_judge_parse_robustness.pyindex 70f18d7b..74647b96 100644--- a/tests/tier3/test_judge_parse_robustness.py+++ b/tests/tier3/test_judge_parse_robustness.py@@ -53,6 +53,20 @@ def _load_template_module():eval_template = _load_template_module()+class _JSONResponse:+ def __init__(self, payload):+ self.payload = payload++ def __enter__(self):+ return self++ def __exit__(self, *_args):+ return False++ def read(self):+ return json.dumps(self.payload).encode()++# ---------------------------------------------------------------------------# Captured offending shapes# ---------------------------------------------------------------------------@@ -234,6 +248,18 @@ def test_extract_json_rejects_nonstandard_json_constants(constant):assert eval_template.extract_json(text) is None+ "text",+ [+ pytest.param('{"metadata": 1e999}', id="positive-overflow"),+ pytest.param('{"nested": [{"value": -1e999}]}', id="negative-nested-overflow"),+ ],+)+def test_extract_json_rejects_finite_syntax_that_overflows_to_infinity(text):+ assert llm_judge._extract_json(text) is None+ assert eval_template.extract_json(text) is None++def test_behavior_judge_treats_list_payload_as_unparseable(monkeypatch):# A bare array is valid JSON but not a judge object; old code raised# AttributeError on parsed.get -- now it takes the structured-error path.@@ -260,6 +286,13 @@ def test_salvage_recovers_complete_entries_from_truncated_results():assert [r["passed"] for r in salvaged] == [True, False]+def test_salvage_rejects_complete_entries_with_overflowed_numeric_metadata():+ text = '{"results":[{"step":1,"passed":true,"metadata":1e999},{"step":2'++ assert llm_judge._salvage_behavior_results(text) == []+ assert eval_template._salvage_behavior_results(text) == []++def test_salvage_returns_empty_for_empty_or_alien_text():assert llm_judge._salvage_behavior_results(EMPTY_REASONING_BURN_RESPONSE) == []assert llm_judge._salvage_behavior_results("plain prose") == []@@ -432,6 +465,8 @@ def test_template_behavior_judge_matches_eval_core(monkeypatch, responses):def test_template_behavior_judge_max_tokens_matches_eval_core_constant():assert eval_template.BEHAVIOR_JUDGE_MAX_TOKENS == llm_judge.BEHAVIOR_JUDGE_MAX_TOKENS+ assert eval_template.BEHAVIOR_JUDGE_MAX_TOKENS == eval_template.STRUCTURED_JUDGE_MAX_TOKENS+ assert llm_judge.BEHAVIOR_JUDGE_MAX_TOKENS == llm_judge.STRUCTURED_JUDGE_MAX_TOKENSassert llm_judge.BEHAVIOR_JUDGE_MAX_TOKENS >= 4096@@ -523,6 +558,81 @@ def test_template_bedrock_request_uses_model_compatible_temperature(monkeypatch,assert inference_config["temperature"] == expected_temperature+ ("judge_name", "valid_response"),+ [+ pytest.param(+ "judge_accuracy",+ {"score": 1.0, "reason": "recovered"},+ id="accuracy",+ ),+ pytest.param(+ "judge_goal_accuracy",+ {+ "achieved": True,+ "score": 1.0,+ "reason": "recovered",+ "user_goal": "complete the task",+ "end_state": "task completed",+ },+ id="custom-goal",+ ),+ ],+)+def test_template_structured_judges_retry_nullable_openai_content(+ monkeypatch,+ judge_name,+ valid_response,+):+ responses = iter(+ [+ {"choices": [{"message": {"content": None}}]},+ {"choices": [{"message": {"content": json.dumps(valid_response)}}]},+ ]+ )+ requests = []++ def fake_urlopen(request, timeout):+ assert timeout == 90+ requests.append(json.loads(request.data))+ return _JSONResponse(next(responses))++ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai")+ monkeypatch.setenv("OPENAI_API_KEY", "test-key")+ monkeypatch.setenv("SKILL_EVAL_LLM_BASE_URL", "https://openai-compatible.example/v1")+ monkeypatch.delenv("LLM_JUDGE_FALLBACK_MODELS", raising=False)+ monkeypatch.setattr(eval_template.urllib.request, "urlopen", fake_urlopen)++ result = getattr(eval_template, judge_name)("question", "ground truth", "agent response")++ assert result["score"] == 1.0+ assert len(requests) == 2+ assert [request["max_tokens"] for request in requests] == [4096, 4096]+ assert "previous reply could not be parsed or validated" in requests[1]["messages"][0]["content"]++[email protected]("content", [{"nested": "text"}, ["text"], 1])+def test_template_non_string_openai_content_remains_a_provider_error(monkeypatch, content):+ requests = []++ def fake_urlopen(request, timeout):+ requests.append(json.loads(request.data))+ return _JSONResponse({"choices": [{"message": {"content": content}}]})++ monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai")+ monkeypatch.setenv("OPENAI_API_KEY", "test-key")+ monkeypatch.setenv("SKILL_EVAL_LLM_BASE_URL", "https://openai-compatible.example/v1")+ monkeypatch.delenv("LLM_JUDGE_FALLBACK_MODELS", raising=False)+ monkeypatch.setattr(eval_template.urllib.request, "urlopen", fake_urlopen)++ result = eval_template.judge_accuracy("question", "ground truth", "agent response")++ assert len(requests) == 1+ assert result["score"] is None+ assert result["status"] == "error"+ assert result["reason"].startswith("LLM judge error: Public provider call failed")++@pytest.mark.parametrize("model",["gpt-5.6-sol", "openai/gpt-5.6-sol", "openai/openai/gpt-5.6-sol"],diff --git a/tests/tier3/test_judge_result_semantics.py b/tests/tier3/test_judge_result_semantics.pyindex c0d96835..5b8e18d8 100644--- a/tests/tier3/test_judge_result_semantics.py+++ b/tests/tier3/test_judge_result_semantics.py@@ -10,6 +10,7 @@ import jsonimport mathfrom pathlib import Pathfrom types import ModuleType+from typing import Anyimport pytest@@ -25,6 +26,16 @@ _CRITERIA = {"TASK_ADDRESSED": True,"ACTIONABLE": False,}+VALID_ACCURACY_RESPONSE = json.dumps({"criteria": _CRITERIA, "score": 0.6, "reason": "valid retry"})+VALID_GOAL_RESPONSE = json.dumps(+ {+ "user_goal": "complete the task",+ "end_state": "task completed",+ "achieved": True,+ "score": 1.0,+ "reason": "valid retry",+ }+)def _load_template_module() -> ModuleType:@@ -51,6 +62,37 @@ def _pair_script(responses: list[tuple[str | None, str | None]], calls: list[strreturn fake_call+def _recorded_pair_script(+ responses: list[tuple[str | None, str | None]],+ calls: list[dict[str, Any]],+):+ def fake_call(prompt: str, **kwargs: Any):+ calls.append({"prompt": prompt, "kwargs": kwargs})+ return responses[min(len(calls) - 1, len(responses) - 1)]++ return fake_call+++def _patch_goal_script(+ module: ModuleType,+ monkeypatch: pytest.MonkeyPatch,+ responses: list[tuple[str | None, str | None, dict[str, str]]],+ calls: list[dict[str, Any]],+) -> None:+ def fake_call(prompt: str, **kwargs: Any):+ calls.append({"prompt": prompt, "kwargs": kwargs})+ content, error, provenance = responses[min(len(calls) - 1, len(responses) - 1)]+ if module is llm_judge:+ return content, error+ return content, error, provenance++ if module is llm_judge:+ monkeypatch.setattr(module, "call_public_llm", fake_call)+ else:+ monkeypatch.setattr(module, "_ragas_goal_accuracy_enabled", lambda: False)+ monkeypatch.setattr(module, "_call_public_llm_with_provenance", fake_call)++def _assert_error_result(result: dict) -> None:assert result["score"] is Noneassert result["status"] == "error"@@ -58,6 +100,289 @@ def _assert_error_result(result: dict) -> None:assert 0 < len(result["reason"]) <= 512+ "invalid",+ [+ pytest.param("not-json", id="malformed"),+ pytest.param(+ json.dumps({"criteria": {**_CRITERIA, "ACTIONABLE": "true"}, "score": 0.6}),+ id="schema-invalid",+ ),+ ],+)+def test_accuracy_retries_invalid_response_once_and_recovers(+ judge_module,+ monkeypatch,+ invalid: str,+) -> None:+ calls: list[dict[str, Any]] = []+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _recorded_pair_script([(invalid, None), (VALID_ACCURACY_RESPONSE, None)], calls),+ )++ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ assert result["score"] == 0.6+ assert result["criteria"] == _CRITERIA+ assert [call["kwargs"]["max_tokens"] for call in calls] == [4096, 4096]+ assert "previous reply could not be parsed or validated" in calls[1]["prompt"]+++def test_accuracy_retries_non_string_reason_once_and_recovers(judge_module, monkeypatch) -> None:+ calls: list[dict[str, Any]] = []+ invalid = json.dumps({"score": 1.0, "reason": {"nested": "text"}})+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _recorded_pair_script([(invalid, None), (VALID_ACCURACY_RESPONSE, None)], calls),+ )++ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ assert len(calls) == 2+ assert result["score"] == 0.6+ assert result["reason"] == "valid retry"+ assert "previous reply could not be parsed or validated" in calls[1]["prompt"]+++def test_accuracy_invalid_twice_errors_after_exactly_one_retry(judge_module, monkeypatch) -> None:+ calls: list[dict[str, Any]] = []+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _recorded_pair_script([("not-json", None), ("still-not-json", None)], calls),+ )++ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ assert len(calls) == 2+ _assert_error_result(result)+ assert "after retry" in result["reason"]+ assert "not-json" not in result["reason"]+++def test_accuracy_initial_provider_error_does_not_retry(judge_module, monkeypatch) -> None:+ calls: list[dict[str, Any]] = []+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _recorded_pair_script([(None, "provider unavailable")], calls),+ )++ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ assert len(calls) == 1+ _assert_error_result(result)+ assert result["reason"].startswith("LLM judge error:")+++def test_accuracy_retry_provider_error_is_bounded_and_redacted(judge_module, monkeypatch) -> None:+ credential = "dummy-accuracy-retry-secret-DO-NOT-RETAIN"+ monkeypatch.setenv("OPENAI_API_KEY", credential)+ calls: list[dict[str, Any]] = []+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _recorded_pair_script(+ [+ ("not-json", None),+ (None, f"retry failed with {credential} " + ("x" * 1000)),+ ],+ calls,+ ),+ )++ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ assert len(calls) == 2+ _assert_error_result(result)+ assert result["reason"].startswith("LLM judge retry error:")+ assert credential not in result["reason"]+ assert "[REDACTED]" in result["reason"]+++ "invalid",+ [+ pytest.param("not-json", id="malformed"),+ pytest.param('{"achieved": "true", "score": 1.0}', id="schema-invalid"),+ ],+)+def test_goal_retries_invalid_response_once_and_recovers(+ judge_module,+ monkeypatch,+ invalid: str,+) -> None:+ calls: list[dict[str, Any]] = []+ _patch_goal_script(+ judge_module,+ monkeypatch,+ [+ (invalid, None, {"provider": "first-provider", "model": "first-model"}),+ (+ VALID_GOAL_RESPONSE,+ None,+ {"provider": "retry-provider", "model": "retry-model"},+ ),+ ],+ calls,+ )++ result = judge_module.judge_goal_accuracy("question", "ground truth", "agent response")++ assert result["score"] == 1.0+ assert result["user_goal"] == "complete the task"+ assert [call["kwargs"]["max_tokens"] for call in calls] == [4096, 4096]+ assert "previous reply could not be parsed or validated" in calls[1]["prompt"]+ if judge_module is eval_template:+ assert result["provider"] == "retry-provider"+ assert result["model"] == "retry-model"++[email protected]("field", ["reason", "user_goal", "end_state"])+def test_goal_retries_non_string_text_fields_once_and_recovers(+ judge_module,+ monkeypatch,+ field: str,+) -> None:+ calls: list[dict[str, Any]] = []+ invalid_payload = {+ "user_goal": "complete the task",+ "end_state": "task completed",+ "achieved": True,+ "score": 1.0,+ "reason": "valid judgment",+ }+ invalid_payload[field] = {"nested": "text"}+ _patch_goal_script(+ judge_module,+ monkeypatch,+ [+ (+ json.dumps(invalid_payload),+ None,+ {"provider": "first-provider", "model": "first-model"},+ ),+ (+ VALID_GOAL_RESPONSE,+ None,+ {"provider": "retry-provider", "model": "retry-model"},+ ),+ ],+ calls,+ )++ result = judge_module.judge_goal_accuracy("question", "ground truth", "agent response")++ assert len(calls) == 2+ assert result["score"] == 1.0+ assert result["reason"] == "valid retry"+ assert result["user_goal"] == "complete the task"+ assert result["end_state"] == "task completed"+ assert "previous reply could not be parsed or validated" in calls[1]["prompt"]+++def test_goal_invalid_twice_errors_after_exactly_one_retry(judge_module, monkeypatch) -> None:+ calls: list[dict[str, Any]] = []+ _patch_goal_script(+ judge_module,+ monkeypatch,+ [+ ("not-json", None, {"provider": "first-provider", "model": "first-model"}),+ ("still-not-json", None, {"provider": "retry-provider", "model": "retry-model"}),+ ],+ calls,+ )++ result = judge_module.judge_goal_accuracy("question", "ground truth", "agent response")++ assert len(calls) == 2+ _assert_error_result(result)+ assert "after retry" in result["reason"]+ assert "not-json" not in result["reason"]+ if judge_module is eval_template:+ assert result["provider"] == "retry-provider"+ assert result["model"] == "retry-model"+++def test_goal_initial_provider_error_does_not_retry(judge_module, monkeypatch) -> None:+ calls: list[dict[str, Any]] = []+ _patch_goal_script(+ judge_module,+ monkeypatch,+ [(None, "provider unavailable", {"provider": "first-provider", "model": "first-model"})],+ calls,+ )++ result = judge_module.judge_goal_accuracy("question", "ground truth", "agent response")++ assert len(calls) == 1+ _assert_error_result(result)+ assert result["reason"].startswith("LLM judge error:")+++def test_goal_retry_provider_error_is_bounded_redacted_and_uses_retry_provenance(+ judge_module,+ monkeypatch,+) -> None:+ credential = "dummy-goal-retry-secret-DO-NOT-RETAIN"+ monkeypatch.setenv("OPENAI_API_KEY", credential)+ calls: list[dict[str, Any]] = []+ _patch_goal_script(+ judge_module,+ monkeypatch,+ [+ ("not-json", None, {"provider": "first-provider", "model": "first-model"}),+ (+ None,+ f"retry failed with {credential} " + ("x" * 1000),+ {"provider": "retry-provider", "model": "retry-model"},+ ),+ ],+ calls,+ )++ result = judge_module.judge_goal_accuracy("question", "ground truth", "agent response")++ assert len(calls) == 2+ _assert_error_result(result)+ assert result["reason"].startswith("LLM judge retry error:")+ assert credential not in result["reason"]+ assert "[REDACTED]" in result["reason"]+ if judge_module is eval_template:+ assert result["provider"] == "retry-provider"+ assert result["model"] == "retry-model"+++ ("judge_name", "response"),+ [+ pytest.param("judge_accuracy", VALID_ACCURACY_RESPONSE, id="accuracy"),+ pytest.param("judge_goal_accuracy", VALID_GOAL_RESPONSE, id="goal-accuracy"),+ ],+)+def test_shared_structured_judge_keeps_explicit_max_tokens_override(+ monkeypatch,+ judge_name: str,+ response: str,+) -> None:+ calls: list[dict[str, Any]] = []+ monkeypatch.setattr(+ llm_judge,+ "call_public_llm",+ _recorded_pair_script([(response, None)], calls),+ )++ judge = getattr(llm_judge, judge_name)+ result = judge("question", "ground truth", "agent response", max_tokens=2048)++ assert result["score"] is not None+ assert len(calls) == 1+ assert calls[0]["kwargs"] == {"max_tokens": 2048}++def test_judge_error_reserved_fields_cannot_be_overridden(judge_module) -> None:metadata = {"score": 1.0,@@ -148,6 +473,32 @@ def test_accuracy_accepts_finite_numeric_scores(judge_module, monkeypatch, scoreassert result.get("status") != "error"+def test_accuracy_normalizes_and_bounds_reason(judge_module, monkeypatch) -> None:+ calls: list[str] = []+ content = json.dumps({"score": 1.0, "reason": " " + ("x" * 600) + " "})+ monkeypatch.setattr(judge_module, "call_public_llm", _pair_script([(content, None)], calls))++ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ assert len(calls) == 1+ assert result["reason"] == ("x" * 509) + "..."+ assert len(result["reason"]) == 512+++def test_accuracy_redacts_configured_secret_before_bounding_reason(judge_module, monkeypatch) -> None:+ credential = "SECRET-ABCDEFGHIJKLMNOPQRSTUVWXYZ-0123456789"+ prefix = "x" * 490+ monkeypatch.setenv("OPENAI_API_KEY", credential)+ content = json.dumps({"score": 1.0, "reason": prefix + credential})+ monkeypatch.setattr(judge_module, "call_public_llm", lambda *_args, **_kwargs: (content, None))++ result = judge_module.judge_accuracy("question", "ground truth", "agent response")++ assert result["reason"] == prefix + "[REDACTED]"+ assert credential not in result["reason"]+ assert "SECRET-" not in result["reason"]++@pytest.mark.parametrize("invalid_score", [pytest.param(None, id="absent"), pytest.param(True, id="boolean")])def test_accuracy_derives_score_only_from_complete_boolean_criteria(judge_module,@@ -297,6 +648,22 @@ def test_goal_requires_boolean_achieved_and_uses_only_finite_numeric_optional_scassert result.get("status") != "error"+def test_goal_normalizes_and_bounds_text_fields(judge_module, monkeypatch) -> None:+ payload = {+ "achieved": True,+ "score": 1.0,+ "reason": " " + ("r" * 600) + " ",+ "user_goal": " " + ("g" * 600) + " ",+ "end_state": " " + ("e" * 600) + " ",+ }++ result = _judge_goal_with_response(judge_module, monkeypatch, content=json.dumps(payload))++ assert result["reason"] == ("r" * 509) + "..."+ assert result["user_goal"] == ("g" * 509) + "..."+ assert result["end_state"] == ("e" * 509) + "..."++def test_goal_accepts_unique_object_before_unmatched_trailing_prose_without_retry(judge_module,monkeypatch,@@ -388,6 +755,24 @@ def test_behavior_first_provider_error_is_scoreless_without_retry(judge_module,_assert_error_result(result)+def test_behavior_retries_overflowed_numeric_metadata_once_and_recovers(judge_module, monkeypatch) -> None:+ calls: list[str] = []+ overflowed = '{"results":[{"passed":true,"reason":"ok","metadata":1e999}],"score":1.0}'+ recovered = json.dumps({"results": [{"passed": True, "reason": "ok"}], "score": 1.0, "summary": "recovered"})+ monkeypatch.setattr(+ judge_module,+ "call_public_llm",+ _pair_script([(overflowed, None), (recovered, None)], calls),+ )++ result = judge_module.judge_behavior_check("conversation", ["behavior"])++ assert len(calls) == 2+ assert result["score"] == 1.0+ assert result["reason"] == "recovered"+ assert "previous reply could not be parsed" in calls[1]++@pytest.mark.parametrize("content",[