Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

fix(tier3): compute judge scores from verdicts and save judge evidence (#303)

Narendran Raghavan80dff4c43ad6
diff --git a/.gitleaks.toml b/.gitleaks.toml
index 85dcf5b8..cbcd16fb 100644
--- a/.gitleaks.toml
+++ b/.gitleaks.toml
@@ -10,6 +10,13 @@ targetRules = ["generic-api-key"]
regexes = ['''^sk-AbCdEf1234567890$''']
paths = ['''^tests/test_tier3_(progress|result_display)\.py$''']
+[[allowlists]]
+description = "Synthetic curl basic-auth credential used by judge evidence redaction tests"
+condition = "AND"
+targetRules = ["curl-auth-user"]
+regexes = ['''^["']admin:quoted-user-password["']$''']
+paths = ['''^tests/tier3/test_judge_contracts\.py$''']
+
[[allowlists]]
description = "Synthetic NVIDIA Build token used by streaming redaction tests"
condition = "AND"
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1d6eca6d..fdca8b8f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -176,6 +176,19 @@ All notable changes to SkillEvaluator are documented in this file.
rules, and is not accepted by `validate`, so a baseline-only run cannot pass
a Tier 3 gate.
+- The Tier 3 verifier saves every LLM judge attempt to `judge_attempts.json`
+ next to the trial's `skill_evaluator_reward.json`, rewritten after each
+ attempt so it also explains a judge that failed. Each entry records the
+ metric, attempt number, method (`llm`, `ragas` or `salvage`), judge model,
+ status, and the problem with a rejected reply, plus the prompt and reply
+ with long text shortened. A `behavior_check` score rebuilt from a truncated
+ reply adds a `salvage` entry. Configured credentials, common key shapes,
+ URL userinfo, `Authorization`, API-key, token and `Cookie` header values,
+ and the values of credential flags such as `--token` and
+ `curl -u user:password` are masked. Rewards record
+ `judge_contract: skill-evaluator-judges-v2`, a reserved name that custom
+ graders cannot use.
+
- Tier 1 sets up SkillSpector 2.13.3 and Gitleaks 8.30.0 on first use from
pinned, SHA-256-verified releases, without a prompt; Semgrep is optional
unless `SKILLEVALUATOR_REQUIRE_SEMGREP=1`. `--skip <check>` on `validate` and
@@ -295,6 +308,28 @@ All notable changes to SkillEvaluator are documented in this file.
environment, so each template compiles once per process instead of once per
report.
+- Tier 3 `accuracy` grades the outcome only. The `SKILL_IDENTIFIED` criterion,
+ which capped a correct answer from the no-skill baseline at 0.8 on positive
+ cases, is replaced by `FORMAT_FOLLOWED`: the response follows any output
+ format the question explicitly requires, and when none is required it is
+ true only if the response gives an answer. A trial that ran but gave no
+ answer still scores `0.0`, as before. Skill use is still scored by
+ `skill_execution`. Every case and both arms use the same five criteria, so
+ rewards no longer record `excluded_criteria` or the
+ `applicable-accuracy-criteria/v1` policy; reports still read older rewards
+ that do. `judge_accuracy` still accepts the `skill_identification_applicable`
+ keyword but ignores it. Expect baseline accuracy to rise and accuracy lift to
+ fall for the same runs.
+
+- Tier 3 LLM judges return verdicts only, and the verifier computes every
+ score: `accuracy` is passing criteria divided by five, `goal_accuracy` is
+ `1.0` when the goal was achieved and `0.0` otherwise, and `behavior_check` is
+ the share of expected behaviors observed. A `score` field in a reply is
+ ignored, and the prompts no longer ask for one. An invalid reply is retried
+ once with the exact problem, for example `steps not judged: [2]`, instead of
+ a generic format reminder. The prompt examples are valid replies for the task
+ at hand, and the goal prompt no longer shows an empty tool-call section.
+
### Documentation
- Give the performance dashboard its own sidebar guide, retain the previous
@@ -815,6 +850,15 @@ All notable changes to SkillEvaluator are documented in this file.
killed; a second Ctrl+C skips that wait, and on Windows the task tree is
stopped immediately. Progress shows the agents that are stopping and reports
the run as interrupted instead of failed unexpectedly.
+- The prompt-based `goal_accuracy` judge, which runs whenever the RAGAS scorer
+ does not (including with the default NVIDIA Build provider), used the reply's
+ `score` field over its `achieved` verdict, so `achieved: false` with
+ `score: 1.0` scored 1.0. Other judge replies were also accepted when they
+ should have been retried: an `accuracy` reply with only a numeric total
+ (clamped into 0 to 1, so `99` scored 1.0), a `behavior_check` reply that
+ judged one step twice and skipped another, and a RAGAS goal score other than
+ `0.0` or `1.0`. Each is now rejected and retried, and a RAGAS score that is
+ not binary falls back to the prompt judge.
- A Tier 1 security scan that SkillSpector reports as partial no longer fails
with only "reports incomplete analysis (status 'partial')". The error quotes
up to three completeness exceptions as `path [phase; reason_code;
diff --git a/docs/custom-graders.mdx b/docs/custom-graders.mdx
index f7cc993a..4f114278 100644
--- a/docs/custom-graders.mdx
+++ b/docs/custom-graders.mdx
@@ -116,7 +116,7 @@ grader must not overwrite reserved standard or reward-metadata names. These
include `security`, `skill_execution`, `skill_efficiency`, `accuracy`,
`goal_accuracy`, `behavior_check`, `overall`, `metrics`, `custom_metrics`,
`custom_details`, `details`, `entry_id`, `metric_set`, `metric_set_version`,
-`evaluation_status`, `evaluation_errors`, `has_skill`, `error`,
+`evaluation_status`, `evaluation_errors`, `has_skill`, `judge_contract`, `error`,
`trajectory_detail`, and `trajectory_source`. A collision fails grading.
</Note>
diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdx
index 345170f7..365a03ba 100644
--- a/docs/tier3-live-evaluation.mdx
+++ b/docs/tier3-live-evaluation.mdx
@@ -622,14 +622,29 @@ grades each trial after the agent finishes. `security`, `skill_execution`, and
`goal_accuracy`, and `behavior_check` are LLM judges that call the configured
evaluator provider. `custom_only` runs none of them.
-Accuracy is calculated from the judge's criterion verdicts, not its numeric
-total: passing applicable criteria divided by applicable criteria. Positive
-cases score all five criteria; a positive-case reply with no criteria falls
-back to the judge's numeric score. When `should_trigger` is false or no
-`expected_skill` is declared, accuracy excludes `SKILL_IDENTIFIED` from both
-counts and scores the other four criterion verdicts. The reward details record
-the excluded criterion and `applicable-accuracy-criteria/v1` scoring policy.
-Missing criterion evidence is an error for these cases.
+The three LLM judges return verdicts only, and the verifier computes the
+scores. A `score` field in a judge reply is ignored.
+
+- `accuracy` is passing criteria divided by five. The criteria grade the
+ outcome: `ACTION_CORRECT`, `FACTUALLY_ACCURATE`, `TASK_ADDRESSED`,
+ `ACTIONABLE`, and `FORMAT_FOLLOWED` (the response follows any output format
+ the question explicitly requires; when none is required, it is true only if
+ the response gives an answer). A response that gives no answer fails all
+ five criteria and scores `0.0`, unless the expected answer is itself a
+ refusal. Every case and both arms use the same
+ five criteria. None of them asks whether the agent named or used the skill;
+ `skill_execution` scores that. A reply without all five true/false verdicts
+ is invalid.
+- `goal_accuracy` is `1.0` when the judge decides the goal was `achieved` and
+ `0.0` otherwise, the same binary scale as the RAGAS scorer.
+- `behavior_check` is the share of expected behaviors observed. The reply must
+ judge every behavior exactly once, identified by its 1-based `step` number.
+ A reply that skips, repeats, or invents a step is invalid.
+
+Rewards record `judge_contract: skill-evaluator-judges-v2`. Results written
+before this contract, whose `accuracy` included a `SKILL_IDENTIFIED` criterion,
+are not directly comparable with newer ones: a correct no-skill baseline answer
+used to be capped at `0.8` on positive cases.
### The graded trajectory
@@ -724,10 +739,25 @@ in [Judge evidence budgets](environment-variables.mdx#judge-evidence-budgets).
downgraded.
- **Reply validation.** Replies are checked against the expected fields either
way. An empty, unparseable, or invalid reply, including an empty reply from a
- reasoning model that ran out of output tokens, gets one retry with a stricter
- format reminder. `behavior_check` can also salvage complete per-behavior
- results from a truncated reply and counts the missing behaviors as not
- passed.
+ reasoning model that ran out of output tokens, gets one retry that names the
+ problem, for example `steps not judged: [2]`. `behavior_check` can also
+ salvage complete per-behavior results from a truncated reply and counts the
+ missing behaviors as not passed.
+- **Saved judge evidence.** The verifier writes every judge attempt to
+ `judge_attempts.json` next to the trial's `skill_evaluator_reward.json` as it
+ happens, so the file also explains a judge that failed. Each entry records
+ the metric, attempt number, method (`llm`, `ragas`, or `salvage`), judge
+ model, status (`accepted`, `rejected`, or `error`), and problem, along with
+ the prompt and reply. When `behavior_check` is scored from a truncated
+ reply, the LLM attempts keep their own status (`rejected`, or `error` if the
+ retry failed in transport), and an `accepted` `salvage` entry, numbered like
+ the truncated attempt, holds the rebuilt results.
+ Configured credentials, common key shapes, URL userinfo, `Authorization`,
+ API-key, token and `Cookie` header values, and credential flag values
+ (`--token`, `--password`, `curl -u user:password`) are masked, and long
+ prompts and replies are shortened. The prompts contain the task question,
+ expected answer, and agent evidence. Pass `--harbor-keep-jobs` to keep the
+ Harbor trial directories that hold this file.
- **Reasoning models.** GPT-5 and GPT-6 judges get no `temperature` and at
least 32,768 output tokens, because reasoning tokens count against the
limit; other judges get 16,384. GPT-6 judges and Claude 5.5 or later judges
@@ -750,7 +780,8 @@ in [Judge evidence budgets](environment-variables.mdx#judge-evidence-budgets).
GPT-6 reasoning model, `goal_accuracy` first tries the RAGAS
`AgentGoalAccuracyWithReference` scorer through its own client, without the
schema or retry handling above but within the same time budget. If RAGAS is
- unavailable or fails, the prompt judge runs instead.
+ unavailable, fails, or returns a score other than `0.0` or `1.0`, the prompt
+ judge runs instead.
- **Transient failures.** HTTP 408, 429, and every 5xx response except 501 and
505 are retried, including 520–524 proxy errors and 529 overload responses.
Connection errors, timeouts, and Bedrock throttling or service-unavailable
diff --git a/src/skillevaluator/tier3/eval_core/llm_judge.py b/src/skillevaluator/tier3/eval_core/llm_judge.py
index 42758683..bc63bd79 100644
--- a/src/skillevaluator/tier3/eval_core/llm_judge.py
+++ b/src/skillevaluator/tier3/eval_core/llm_judge.py
@@ -623,11 +623,16 @@ def _salvage_behavior_results(text: str) -> list[dict[str, Any]]:
STRUCTURED_JUDGE_MAX_TOKENS = MAX_COMPLETION_TOKENS
+_JUDGE_PROBLEM_CHARS = 300
-_JUDGE_RETRY_REMINDER = (
- "\n\nIMPORTANT: Your previous reply could not be parsed or validated. Respond with ONLY the "
- "minified JSON object on a single line -- no markdown fences, no prose, and keep explanations brief."
-)
+
+def _judge_retry_feedback(problem: str) -> str:
+ """Retry instruction that names what was wrong with the previous reply."""
+ return (
+ f"\n\nIMPORTANT: Your previous reply could not be parsed or validated: {problem[:_JUDGE_PROBLEM_CHARS]}. "
+ "Respond with ONLY the minified JSON object on a single line in the shape described above -- "
+ "no markdown fences, no prose, no score field, and keep explanations brief."
+ )
def _call_validated_json_judge(
@@ -635,61 +640,85 @@ def _call_validated_json_judge(
validate: Any,
call: Any,
extract: Any,
+ *,
+ salvage: Any = None,
**call_kwargs: Any,
) -> tuple[Any, str | None, dict[str, Any]]:
- """Invoke a JSON judge with one format-correction retry when payload validation fails."""
+ """Invoke a JSON judge; on an invalid reply, retry once and name the problem.
+
+ *validate* returns ``None`` for a usable reply or a short description of the
+ problem, which the retry sends back to the judge. Provider errors are not
+ retried here: the transport already retried them within its own budget.
+ *salvage* may rebuild a usable reply from text that never parsed, such as a
+ reply cut off by the output-token cap; it is tried only after both attempts.
+ """
call_kwargs.setdefault("max_tokens", STRUCTURED_JUDGE_MAX_TOKENS)
+ unparsed_replies: list[str] = []
def invoke(call_prompt: str) -> tuple[Any, str | None, dict[str, Any], str | None]:
content, error, *metadata = call(call_prompt, **call_kwargs)
provenance = metadata[0] if metadata and isinstance(metadata[0], dict) else {}
parsed = extract(content) if content else None
- validation_error = validate(parsed) if not error else None
- return parsed, error, provenance, validation_error
-
- parsed, error, provenance, validation_error = invoke(prompt)
+ problem = None
+ if not error:
+ problem = "the reply was empty" if not content else validate(parsed)
+ if content and parsed is None:
+ unparsed_replies.append(content)
+ return parsed, error, provenance, problem
+
+ parsed, error, provenance, problem = invoke(prompt)
if error:
return None, f"LLM judge error: {error}", provenance
- if validation_error is None:
+ if problem is None:
return parsed, None, provenance
- parsed, error, provenance, validation_error = invoke(prompt + _JUDGE_RETRY_REMINDER)
+ parsed, error, provenance, problem = invoke(prompt + _judge_retry_feedback(problem))
+ if not error and problem is None:
+ return parsed, None, provenance
+ if salvage is not None:
+ # Newest first: only text that never parsed can be a reply cut off mid-object.
+ for text in reversed(unparsed_replies):
+ candidate = salvage(text)
+ if candidate is not None:
+ return candidate, None, provenance
if error:
return None, f"LLM judge retry error: {error}", provenance
- if validation_error is not None:
- return None, f"{validation_error} after retry", provenance
- return parsed, None, provenance
+ return None, f"Judge response was unparseable or invalid after retry: {problem}", provenance
# ---------------------------------------------------------------------------
# Accuracy judge (5-criterion)
# ---------------------------------------------------------------------------
+# Accuracy grades the outcome only. Whether the agent found and used the skill is
+# scored by skill_execution, so a correct answer from the no-skill baseline can
+# reach 1.0 here.
+_ACCURACY_CRITERIA = (
+ ("ACTION_CORRECT", "Does the response describe or carry out the correct actions or commands for the task?"),
+ ("FACTUALLY_ACCURATE", "Are the factual claims consistent with the expected answer?"),
+ ("TASK_ADDRESSED", "Does the response directly address the user's request?"),
+ ("ACTIONABLE", "Does the response give usable results or next steps, not just an acknowledgment?"),
+ (
+ "FORMAT_FOLLOWED",
+ "If the USER QUESTION explicitly requires an output format (a template, prefix, fields, ordering, "
+ "or units), does the response follow it? When the question requires no specific format, answer true "
+ "if the response gives an answer and false if it gives none.",
+ ),
+)
+_ACCURACY_CRITERIA_KEYS = frozenset(key for key, _question in _ACCURACY_CRITERIA)
+
ACCURACY_JSON_SCHEMA: dict[str, Any] = {
"type": "object",
"properties": {
"criteria": {
"type": "object",
- "properties": {
- "SKILL_IDENTIFIED": {"type": "boolean"},
- "ACTION_CORRECT": {"type": "boolean"},
- "FACTUALLY_ACCURATE": {"type": "boolean"},
- "TASK_ADDRESSED": {"type": "boolean"},
- "ACTIONABLE": {"type": "boolean"},
- },
- "required": [
- "SKILL_IDENTIFIED",
- "ACTION_CORRECT",
- "FACTUALLY_ACCURATE",
- "TASK_ADDRESSED",
- "ACTIONABLE",
- ],
+ "properties": {key: {"type": "boolean"} for key, _question in _ACCURACY_CRITERIA},
+ "required": [key for key, _question in _ACCURACY_CRITERIA],
"additionalProperties": False,
},
- "score": {"type": "number"},
"reason": {"type": "string"},
},
- "required": ["criteria", "score", "reason"],
+ "required": ["criteria", "reason"],
"additionalProperties": False,
}
@@ -699,10 +728,9 @@ GOAL_ACCURACY_JSON_SCHEMA: dict[str, Any] = {
"user_goal": {"type": "string"},
"end_state": {"type": "string"},
"achieved": {"type": "boolean"},
- "score": {"type": "number"},
"reason": {"type": "string"},
},
- "required": ["user_goal", "end_state", "achieved", "score", "reason"],
+ "required": ["user_goal", "end_state", "achieved", "reason"],
"additionalProperties": False,
}
@@ -722,29 +750,32 @@ BEHAVIOR_CHECK_JSON_SCHEMA: dict[str, Any] = {
"additionalProperties": False,
},
},
- "score": {"type": "number"},
"summary": {"type": "string"},
},
- "required": ["results", "score", "summary"],
+ "required": ["results", "summary"],
"additionalProperties": False,
}
-ACCURACY_PROMPT = """You are an expert evaluator for AI agent responses. Evaluate by checking \
-each criterion below against the expected answer. For each criterion, determine true (satisfied) or false (not satisfied).
-
-1. SKILL_IDENTIFIED: Does the response reference or use the correct skill for the task?
-2. ACTION_CORRECT: Does the response describe or execute the correct actions/scripts?
-3. FACTUALLY_ACCURATE: Are the factual claims consistent with the expected answer?
-4. TASK_ADDRESSED: Does the response directly address the user's request?
-5. ACTIONABLE: Does the response provide actionable information (not just acknowledgment)?
+ACCURACY_PROMPT = (
+ """You are an expert evaluator for AI agent responses. Check the response against \
+the expected answer on each criterion below and decide true (satisfied) or false (not satisfied) for each.
-Compute score = count(true) / 5.
-Be lenient on exact wording but strict on factual correctness.
-
-Respond with ONLY a JSON object:
-{{"criteria": {{"SKILL_IDENTIFIED": true/false, "ACTION_CORRECT": true/false, \
-"FACTUALLY_ACCURATE": true/false, "TASK_ADDRESSED": true/false, "ACTIONABLE": true/false}}, \
-"score": <float>, "reason": "<brief summary>"}}
+"""
+ + "\n".join(f"{index}. {key}: {criterion}" for index, (key, criterion) in enumerate(_ACCURACY_CRITERIA, 1))
+ + """
+
+Judge the outcome only. Do not reward or penalize the response for naming, or not naming, \
+a tool or skill: that is scored separately.
+If the evidence holds no answer to the question (for example, the agent stopped, timed out or \
+refused), every criterion is false, unless the expected answer is itself a refusal.
+Be lenient on wording, but strict on factual correctness and on any output format the user \
+question explicitly requires.
+
+Respond with ONLY one JSON object with a "criteria" object holding exactly these five keys, each \
+true or false, and a one-sentence "reason". Do not include a score: it is computed from the criteria.
+Example of the shape (the values are only an illustration):
+{{"criteria": {{"ACTION_CORRECT": true, "FACTUALLY_ACCURATE": false, "TASK_ADDRESSED": true, "ACTIONABLE": true, \
+"FORMAT_FOLLOWED": true}}, "reason": "Ran the right command but reported the wrong version number."}}
USER QUESTION:
{question}
@@ -754,39 +785,31 @@ EXPECTED ANSWER:
SELECTED EVIDENCE (final response + produced artifacts; low-relevance steps may be omitted):
{agent_text}"""
-
-_ACCURACY_CRITERIA_KEYS = frozenset(
- {
- "SKILL_IDENTIFIED",
- "ACTION_CORRECT",
- "FACTUALLY_ACCURATE",
- "TASK_ADDRESSED",
- "ACTIONABLE",
- }
)
-def _valid_accuracy_criteria(value: Any) -> bool:
- """Return True when value is a complete 5-criterion boolean mapping."""
- return (
- isinstance(value, dict)
- and value.keys() == _ACCURACY_CRITERIA_KEYS
- and all(isinstance(item, bool) for item in value.values())
- )
-
-
def _accuracy_payload_error(parsed: Any) -> str | None:
- """Validate a parsed accuracy judge payload and return an error message if malformed."""
+ """Describe what is wrong with an accuracy reply, or return ``None`` when it is usable.
+
+ The score is computed from the five verdicts, so a ``score`` field in the
+ reply is ignored rather than trusted.
+ """
if not isinstance(parsed, dict):
- return "Judge response was not a valid JSON object"
- if "reason" in parsed and not isinstance(parsed["reason"], str):
- return "Judge response contained an invalid accuracy reason"
+ return "the reply did not contain one JSON object"
criteria = parsed.get("criteria")
- criteria_valid = _valid_accuracy_criteria(criteria)
- if "criteria" in parsed and not criteria_valid:
- return "Judge response contained invalid accuracy criteria"
- if _finite_score(parsed.get("score")) is None and not criteria_valid:
- return "Judge response contained no valid accuracy score or complete criteria"
+ if not isinstance(criteria, dict):
+ return 'the reply has no "criteria" object'
+ missing = sorted(_ACCURACY_CRITERIA_KEYS - criteria.keys())
+ if missing:
+ return f"criteria is missing {', '.join(missing)}"
+ unexpected = sorted(str(key)[:40] for key in criteria.keys() - _ACCURACY_CRITERIA_KEYS)
+ if unexpected:
+ return f"criteria has unexpected keys {', '.join(unexpected[:5])}"
+ not_boolean = sorted(key for key, value in criteria.items() if not isinstance(value, bool))
+ if not_boolean:
+ return f"criteria values must be true or false, not {', '.join(not_boolean)}"
+ if "reason" in parsed and not isinstance(parsed["reason"], str):
+ return '"reason" must be a string'
return None
@@ -795,35 +818,27 @@ def judge_accuracy(
ground_truth: str,
agent_text: str,
*,
- skill_identification_applicable: bool = True,
+ skill_identification_applicable: bool | None = None,
**kwargs: Any,
) -> dict[str, Any]:
- """Judge accuracy; exclude skill identification when no skill is expected."""
+ """Judge accuracy from five outcome verdicts; the score is computed here.
+
+ *skill_identification_applicable* is deprecated and ignored. Accuracy no
+ longer has a skill criterion, so every case and both arms use the same five
+ criteria. The keyword is still accepted so existing callers keep working.
+ """
+ del skill_identification_applicable
if not _has_judge_reference(ground_truth):
return _judge_not_applicable(_NO_GROUND_TRUTH_REASON)
- prompt = ACCURACY_PROMPT
- if not skill_identification_applicable:
- prompt = prompt.replace(
- "1. SKILL_IDENTIFIED: Does the response reference or use the correct skill for the task?",
- "1. SKILL_IDENTIFIED: Inapplicable: no skill is expected. Set false; exclude it from the score.",
- ).replace("Compute score = count(true) / 5.", "Compute score = count(true among the other four criteria) / 4.")
- # Adjust the rubric before interpolation so task/evidence text stays intact.
- prompt = prompt.format(question=question, ground_truth=ground_truth, agent_text=agent_text)
+ prompt = ACCURACY_PROMPT.format(question=question, ground_truth=ground_truth, agent_text=agent_text)
prompt = append_character_facts(prompt, question=question, ground_truth=ground_truth, agent_text=agent_text)
-
- def validate_payload(parsed: Any) -> str | None:
- error = _accuracy_payload_error(parsed)
- if not error and not skill_identification_applicable and not _valid_accuracy_criteria(parsed.get("criteria")):
- return "Accuracy requires complete criterion verdicts when skill identification is inapplicable"
- return error
-
kwargs.setdefault("response_schema", ACCURACY_JSON_SCHEMA)
kwargs.setdefault("schema_name", "accuracy_judgment")
parsed, error, _provenance = _call_validated_json_judge(
prompt,
- validate_payload,
+ _accuracy_payload_error,
call_public_llm,
_extract_json,
**kwargs,
@@ -831,73 +846,54 @@ def judge_accuracy(
if error:
return _judge_error(error)
- assert isinstance(parsed, dict)
- criteria = parsed.get("criteria")
- criteria_valid = _valid_accuracy_criteria(criteria)
-
- if criteria_valid:
- # Score only applicable verdicts; excluded criteria leave both numerator and denominator.
- excluded = set() if skill_identification_applicable else {"SKILL_IDENTIFIED"}
- applicable = [verdict for name, verdict in criteria.items() if name not in excluded]
- score = sum(verdict is True for verdict in applicable) / len(applicable)
- else:
- # Legacy score-only payloads; validate_payload rejects them when a criterion is excluded.
- assert skill_identification_applicable
- score = _finite_score(parsed.get("score"))
- assert score is not None
-
- result = {
- "score": round(score, 4),
+ criteria = {key: parsed["criteria"][key] for key, _question in _ACCURACY_CRITERIA}
+ return {
+ "score": round(sum(criteria.values()) / len(criteria), 4),
"reason": _bounded_judge_text(parsed.get("reason", "")),
- "criteria": criteria if criteria_valid else {},
+ "criteria": criteria,
}
- if not skill_identification_applicable:
- result["excluded_criteria"] = ["SKILL_IDENTIFIED"]
- result["scoring_policy"] = "applicable-accuracy-criteria/v1"
- return result
# ---------------------------------------------------------------------------
# Goal accuracy judge
# ---------------------------------------------------------------------------
-GOAL_ACCURACY_PROMPT = """You are an evaluation judge. Determine whether an AI agent achieved \
-the expected goal by analyzing the full conversation.
+GOAL_ACCURACY_PROMPT = """You are an evaluation judge. Decide whether an AI agent achieved the expected goal.
-Step 1: What was the user's goal?
-Step 2: What end state did the agent reach?
-Step 3: Compare the end state to the expected outcome.
+Step 1: State the user's goal.
+Step 2: State the end state the agent reached.
+Step 3: Decide whether that end state achieves the expected outcome.
USER REQUEST:
{question}
-EXPECTED OUTCOME (ground truth):
+EXPECTED OUTCOME:
{ground_truth}
-
-AGENT'S TOOL CALLS:
-{tool_summary}
-
+{tool_section}
END-STATE EVIDENCE:
{agent_text}
-Did the agent achieve the expected goal?
-
-Respond with ONLY a JSON object:
-{{"user_goal": "<inferred goal>", "end_state": "<what agent achieved>", \
-"achieved": true/false, "score": 1.0 or 0.0, "reason": "<brief explanation>"}}"""
+Respond with ONLY one JSON object with the keys "user_goal", "end_state", "achieved" (true or \
+false) and a one-sentence "reason". Do not include a score: it is 1.0 when achieved and 0.0 otherwise.
+Example of the shape (the values are only an illustration):
+{{"user_goal": "List the open release blockers", "end_state": "Listed two of the three blockers", \
+"achieved": false, "reason": "One blocker from the expected outcome is missing."}}"""
def _goal_payload_error(parsed: Any) -> str | None:
- """Validate a parsed goal-accuracy judge payload and return an error message if malformed."""
+ """Describe what is wrong with a goal reply, or return ``None`` when it is usable.
+
+ Goal accuracy is binary, like RAGAS AgentGoalAccuracyWithReference: the
+ score is 1.0 when ``achieved`` is true and 0.0 otherwise, so a ``score``
+ field in the reply is ignored.
+ """
if not isinstance(parsed, dict):
- return "Judge response was not a valid JSON object"
+ return "the reply did not contain one JSON object"
+ if not isinstance(parsed.get("achieved"), bool):
+ return '"achieved" must be true or false'
for field in ("reason", "user_goal", "end_state"):
if field in parsed and not isinstance(parsed[field], str):
- return f"Judge response contained an invalid {field} value"
- if not isinstance(parsed.get("achieved"), bool):
- return "Judge response contained an invalid achieved value"
- if "score" in parsed and _finite_score(parsed["score"]) is None:
- return "Judge response contained an invalid goal score"
+ return f'"{field}" must be a string'
return None
@@ -908,14 +904,15 @@ def judge_goal_accuracy(
tool_summary: str = "",
**kwargs: Any,
) -> dict[str, Any]:
- """Run the goal accuracy judge (two-step: infer goal, compare outcome)."""
+ """Run the goal accuracy judge (two-step: infer goal, compare outcome); the score is binary."""
if not _has_judge_reference(ground_truth):
return _judge_not_applicable(_NO_GROUND_TRUTH_REASON)
+ tool_section = f"\nAGENT'S TOOL CALLS:\n{tool_summary}\n" if str(tool_summary or "").strip() else ""
prompt = GOAL_ACCURACY_PROMPT.format(
question=question,
ground_truth=ground_truth,
- tool_summary=tool_summary,
+ tool_section=tool_section,
agent_text=agent_text,
)
prompt = append_character_facts(
@@ -934,18 +931,11 @@ def judge_goal_accuracy(
if error:
return _judge_error(error)
- assert isinstance(parsed, dict)
- achieved = parsed.get("achieved")
- assert isinstance(achieved, bool)
-
- score = 1.0 if achieved else 0.0
- if "score" in parsed:
- score = _finite_score(parsed["score"])
- assert score is not None
-
+ achieved = parsed["achieved"]
return {
- "score": score,
+ "score": 1.0 if achieved else 0.0,
"reason": _bounded_judge_text(parsed.get("reason", "")),
+ "achieved": achieved,
"user_goal": _bounded_judge_text(parsed.get("user_goal", "")),
"end_state": _bounded_judge_text(parsed.get("end_state", "")),
}
@@ -955,9 +945,8 @@ def judge_goal_accuracy(
# Behavior check judge
# ---------------------------------------------------------------------------
-BEHAVIOR_CHECK_PROMPT = """You are evaluating whether an AI agent followed expected behaviors \
-during a task. Analyze the full conversation and determine if each expected behavior was \
-observed.
+BEHAVIOR_CHECK_PROMPT = """You are evaluating whether an AI agent showed each expected behavior during a task. \
+Read the conversation and decide, for each numbered behavior, whether it was observed.
CONVERSATION:
{conversation}
@@ -965,11 +954,12 @@ CONVERSATION:
EXPECTED BEHAVIORS:
{behaviors}
-For each behavior, set "passed" to true (observed) or false (not observed) with a brief reason.
-
-Respond with ONLY a JSON object:
-{{"results": [{{"step": 1, "passed": true/false, "reason": "..."}}, ...], \
-"score": <float between 0.0 and 1.0>, "summary": "<brief summary>"}}"""
+Respond with ONLY one JSON object. "results" must hold exactly {expected_count} entries, one per \
+expected behavior: "step" is the behavior's number (each number from 1 to {expected_count} exactly \
+once), "passed" is true (observed) or false (not observed), and "reason" is under 15 words. Add a \
+one-sentence "summary". Do not include a score: it is computed from the results.
+Example of the shape for this task's {expected_count} behaviors (the values are only an illustration):
+{example}"""
def _slice_with_middle_marker(
@@ -1061,11 +1051,22 @@ def _compact_behavior_conversation(conversation_text: str, limit: int | None = N
# (finish_reason="length", reasoning_tokens=1024).
BEHAVIOR_JUDGE_MAX_TOKENS = STRUCTURED_JUDGE_MAX_TOKENS
-_BEHAVIOR_RETRY_REMINDER = (
- "\n\nIMPORTANT: Your previous reply could not be parsed. Respond with ONLY the "
- "minified JSON object on a single line -- no markdown fences, no prose, and "
- 'keep every "reason" under 15 words.'
-)
+
+def _behavior_prompt_example(expected_count: int) -> str:
+ """One entry per expected behavior, so the example is a valid reply if a judge copies it."""
+ return json.dumps(
+ {
+ "results": [
+ {
+ "step": step,
+ "passed": step % 2 == 1,
+ "reason": "Seen in the transcript." if step % 2 == 1 else "Not seen in the transcript.",
+ }
+ for step in range(1, expected_count + 1)
+ ],
+ "summary": "A short summary of what was and was not observed.",
+ }
+ )
def judge_behavior_check(
@@ -1073,74 +1074,50 @@ def judge_behavior_check(
expected_behaviors: list[str],
**kwargs: Any,
) -> dict[str, Any]:
- """Run the behavior check LLM judge."""
+ """Run the behavior check LLM judge; the score is the share of behaviors observed."""
if not _has_judge_reference(expected_behaviors):
return _judge_not_applicable(_NO_EXPECTED_BEHAVIOR_REASON, results=[])
+ expected_count = len(expected_behaviors)
behaviors_text = "\n".join(f"{i + 1}. {b}" for i, b in enumerate(expected_behaviors))
conversation = _compact_behavior_conversation(conversation_text)
prompt = BEHAVIOR_CHECK_PROMPT.format(
conversation=conversation,
behaviors=behaviors_text,
+ expected_count=expected_count,
+ example=_behavior_prompt_example(expected_count),
)
prompt = append_character_facts(prompt, conversation=conversation, expected_behaviors=behaviors_text)
kwargs.setdefault("max_tokens", BEHAVIOR_JUDGE_MAX_TOKENS)
kwargs.setdefault("response_schema", BEHAVIOR_CHECK_JSON_SCHEMA)
kwargs.setdefault("schema_name", "behavior_check_judgment")
- content, error = call_public_llm(prompt, **kwargs)
+ def salvage(text: str) -> dict[str, Any] | None:
+ # Complete entries from a reply cut off before its results array closed.
+ salvaged = _salvage_behavior_results(text)
+ candidate = {
+ "results": salvaged,
+ "summary": f"Salvaged {len(salvaged)}/{expected_count} behavior results from truncated judge response",
+ }
+ if not salvaged or _behavior_payload_error(candidate, expected_count, allow_partial=True):
+ return None
+ return candidate
+
+ parsed, error, _provenance = _call_validated_json_judge(
+ prompt,
+ lambda payload: _behavior_payload_error(payload, expected_count),
+ call_public_llm,
+ _extract_json,
+ salvage=salvage,
+ **kwargs,
+ )
if error:
- return _judge_error(f"LLM judge error: {error}", results=[])
-
- def _parse_judge_object(text: str | None) -> dict[str, Any] | list[Any] | None:
- """Parse a JSON object or list from judge response text."""
- return _extract_json(text) if text else None
-
- parsed = _parse_judge_object(content)
- score = _behavior_payload_score(parsed, len(expected_behaviors))
- attempts = [(content or "", parsed)]
- retry_error = None
- if score is None:
- # One retry max, with an explicit machine-readable-output reminder.
- retry_content, retry_error = call_public_llm(prompt + _BEHAVIOR_RETRY_REMINDER, **kwargs)
- if not retry_error:
- parsed = _parse_judge_object(retry_content)
- attempts.append((retry_content or "", parsed))
- score = _behavior_payload_score(parsed, len(expected_behaviors))
-
- if score is None:
- # Salvage complete entries from a truncated results array (newest first).
- for text, extracted in reversed(attempts):
- if extracted is not None:
- continue
- salvaged = _salvage_behavior_results(text)
- if salvaged:
- candidate = {
- "results": salvaged,
- "summary": (
- f"Salvaged {len(salvaged)}/{len(expected_behaviors)} behavior "
- "results from truncated judge response"
- ),
- }
- candidate_score = _behavior_payload_score(
- candidate,
- len(expected_behaviors),
- allow_partial=True,
- )
- if candidate_score is not None:
- parsed = candidate
- score = candidate_score
- break
-
- if score is None:
- if retry_error:
- return _judge_error(f"LLM judge retry error: {retry_error}", results=[])
- return _judge_error("Judge response was unparseable or invalid after retry", results=[])
-
- assert isinstance(parsed, dict)
- results = parsed["results"]
+ return _judge_error(error, results=[])
+ # A salvaged reply leaves the behaviors after the cut unjudged; they count as not observed.
+ results = sorted(parsed["results"], key=lambda result: result["step"])
+ score = sum(1 for result in results if result["passed"]) / expected_count
return {
"score": round(score, 4),
"reason": parsed.get("summary", ""),
@@ -1148,27 +1125,44 @@ def judge_behavior_check(
}
-def _behavior_payload_score(
- parsed: dict[str, Any] | list[Any] | None,
- expected_count: int,
- *,
- allow_partial: bool = False,
-) -> float | None:
+def _behavior_payload_error(parsed: Any, expected_count: int, *, allow_partial: bool = False) -> str | None:
+ """Describe what is wrong with a behavior reply, or return ``None`` when it is usable.
+
+ Each expected behavior is judged exactly once, identified by its 1-based
+ ``step`` number, with a boolean ``passed``. *allow_partial* accepts a
+ salvaged prefix that judges each of its steps once. The score is computed
+ from the verdicts, so a ``score`` field in the reply is ignored.
+ """
if not isinstance(parsed, dict):
- return None
+ return "the reply did not contain one JSON object"
results = parsed.get("results")
if not isinstance(results, list):
- return None
- if any(not isinstance(result, dict) or not isinstance(result.get("passed"), bool) for result in results):
- return None
+ return 'the reply has no "results" list'
if allow_partial:
if not results or len(results) > expected_count:
- return None
+ return f'"results" has {len(results)} entries; it needs 1 to {expected_count}'
elif len(results) != expected_count:
- return None
- if "score" in parsed and _finite_score(parsed["score"]) is None:
- return None
- denominator = expected_count if allow_partial else len(results)
- if denominator <= 0:
- return None
- return sum(1 for result in results if result["passed"]) / denominator
+ return f'"results" has {len(results)} entries; it needs exactly {expected_count}, one per behavior'
+ if any(not isinstance(result, dict) for result in results):
+ return 'every "results" entry must be an object'
+ if any(not isinstance(result.get("passed"), bool) for result in results):
+ return 'every "results" entry needs "passed" set to true or false'
+ steps = [result.get("step") for result in results]
+ if any(isinstance(step, bool) or not isinstance(step, int) for step in steps):
+ return 'every "results" entry needs an integer "step" number'
+ expected_steps = set(range(1, expected_count + 1))
+ duplicated = sorted({step for step in steps if steps.count(step) > 1})
+ unexpected = sorted(set(steps) - expected_steps)[:10]
+ missing = [] if allow_partial else sorted(expected_steps - set(steps))[:10]
+ if duplicated or missing or unexpected:
+ parts = []
+ if duplicated:
+ parts.append(f"steps judged more than once: {duplicated[:10]}")
+ if missing:
+ parts.append(f"steps not judged: {missing}")
+ if unexpected:
+ parts.append(f"steps outside 1..{expected_count}: {unexpected}")
+ return "; ".join(parts)
+ if "summary" in parsed and not isinstance(parsed["summary"], str):
+ return '"summary" must be a string'
+ return None
diff --git a/src/skillevaluator/tier3/harbor/metrics.py b/src/skillevaluator/tier3/harbor/metrics.py
index a2a6c1d5..a7bcf804 100644
--- a/src/skillevaluator/tier3/harbor/metrics.py
+++ b/src/skillevaluator/tier3/harbor/metrics.py
@@ -119,6 +119,7 @@ _RESERVED_METADATA_KEYS = {
"evaluation_errors",
"evaluation_status",
"has_skill",
+ "judge_contract",
"metric_set",
"metric_set_version",
"metrics",
diff --git a/src/skillevaluator/tier3/harbor/templates/custom_grader_runner.py b/src/skillevaluator/tier3/harbor/templates/custom_grader_runner.py
index fb073154..0511dba9 100644
--- a/src/skillevaluator/tier3/harbor/templates/custom_grader_runner.py
+++ b/src/skillevaluator/tier3/harbor/templates/custom_grader_runner.py
@@ -49,6 +49,7 @@ RESERVED = {
"evaluation_errors",
"evaluation_status",
"has_skill",
+ "judge_contract",
"metric_set",
"metric_set_version",
"metrics",
diff --git a/src/skillevaluator/tier3/harbor/templates/eval.py b/src/skillevaluator/tier3/harbor/templates/eval.py
index 7f440aaf..1e7d4e91 100644
--- a/src/skillevaluator/tier3/harbor/templates/eval.py
+++ b/src/skillevaluator/tier3/harbor/templates/eval.py
@@ -17,12 +17,15 @@ Writes:
/logs/verifier/reward.txt -- overall score (0.0-1.0)
LLM judges use the configured public provider environment variables.
-RAGAS is used for goal_accuracy and accuracy when available.
+RAGAS is used for goal_accuracy when available. Judges return verdicts only and
+the scores are computed here; every judge attempt is saved, redacted, to
+/logs/verifier/judge_attempts.json.
"""
from __future__ import annotations
import contextlib
+import hashlib
import importlib.util
import io
import ipaddress
@@ -612,6 +615,9 @@ _CREDENTIAL_ENV_VARS = (
DEFAULT_METRIC_SET = "skill-evaluator-default-v2"
+# Judges return verdicts and the verifier computes their scores. Rewards carry
+# this marker so scores from older judge prompts are not mistaken for these.
+JUDGE_CONTRACT = "skill-evaluator-judges-v2"
DEFAULT_SCORE_POLICY = "skill-evaluator-dimension-mean-v1"
DISPLAY_METRICS = [
"security",
@@ -1365,6 +1371,7 @@ _DEFAULT_MAX_DELAY = 30.0
_JUDGE_WALL_TIME_BUDGET_SEC = 180.0
_ACTIVE_JUDGE_DEADLINE: ContextVar[float | None] = ContextVar("active_judge_deadline", default=None)
_ACTIVE_JUDGE_CALLS: ContextVar[list[dict[str, Any]] | None] = ContextVar("active_judge_calls", default=None)
+_ACTIVE_JUDGE_EVIDENCE: ContextVar[list[dict[str, Any]] | None] = ContextVar("active_judge_evidence", default=None)
class EvalRetryConfig(NamedTuple):
@@ -2383,87 +2390,289 @@ def _resolve_acceptable_skills(entry, expected_skill=None):
# Keep in sync with skillevaluator.provider_config.MAX_COMPLETION_TOKENS (drift test).
STRUCTURED_JUDGE_MAX_TOKENS = 16384
+# Saved judge evidence is bounded so one long prompt cannot bloat the verifier output.
+JUDGE_EVIDENCE_PROMPT_CHARS = 16000
+JUDGE_EVIDENCE_REPLY_CHARS = 8000
+JUDGE_ATTEMPTS_FILE_NAME = "judge_attempts.json"
+_JUDGE_PROBLEM_CHARS = 300
-_JUDGE_RETRY_REMINDER = (
- "\n\nIMPORTANT: Your previous reply could not be parsed or validated. Respond with ONLY the "
- "minified JSON object on a single line -- no markdown fences, no prose, and keep explanations brief."
+
+def _judge_retry_feedback(problem):
+ """Retry instruction that names what was wrong with the previous reply."""
+ return (
+ f"\n\nIMPORTANT: Your previous reply could not be parsed or validated: {problem[:_JUDGE_PROBLEM_CHARS]}. "
+ "Respond with ONLY the minified JSON object on a single line in the shape described above -- "
+ "no markdown fences, no prose, no score field, and keep explanations brief."
+ )
+
+
+# Judge prompts quote the agent's commands and output, so saved judge evidence
+# also masks credentials that have no fixed key shape: URL userinfo
+# ("https://user:token@host"), Authorization, API-key, token and Cookie header
+# values, and the values of credential flags ("--token VALUE",
+# "curl -u user:password"). Each pattern starts at a fixed literal and scans one
+# bounded run, so long text stays linear. The header patterns accept shell
+# ("-H 'Name: value'"), JSON ('"name": "value"') and Python subscript
+# ('headers["Name"] = f"value"') forms, also with the quotes JSON-escaped
+# ('\"Name\": \"value\"'), as tool-call arguments appear in the behavior and
+# goal prompts.
+_JUDGE_EVIDENCE_URL_USERINFO_RE = re.compile(r"(://)[^\s/?#@'\"`<>]{1,256}@")
+_JUDGE_EVIDENCE_HEADER_SEPARATOR = r"\\?[\"']?\]?[ \t]{0,8}[:=][ \t]{0,8}(?:[bfru]{0,2}\\?[\"'])?"
+# Digest values hold quoted fields (username="...", response="..."), so the rest
+# of the value is masked up to a single quote, a backtick or the end of the line.
+_JUDGE_EVIDENCE_DIGEST_AUTH_HEADER_RE = re.compile(
+ r"(\bauthorization" + _JUDGE_EVIDENCE_HEADER_SEPARATOR + r"digest[ \t]{1,8})[^'`\r\n]{1,512}",
+ re.IGNORECASE,
+)
+# Other Authorization schemes vary (Bearer, Basic, ApiKey, Bot, SSWS, ...), so the
+# whole value is masked up to its closing quote or the end of the line. Bearer,
+# Basic, Token and Digest stay visible because they say which kind of credential
+# was sent, and a quoted token after them ('Bearer "token"') is masked too.
+_JUDGE_EVIDENCE_AUTH_HEADER_RE = re.compile(
+ r"(\bauthorization"
+ + _JUDGE_EVIDENCE_HEADER_SEPARATOR
+ + r"(?:(?:bearer|basic|token|digest)[ \t]{1,8}(?:\\?[\"'])?)?)"
+ r"[^\s'\"`,;\\][^'\"`\r\n\\]*",
+ re.IGNORECASE,
+)
+# X-API-Key, Api-Key, X-Auth-Token, X-Access-Token and Private-Token headers.
+_JUDGE_EVIDENCE_KEY_HEADER_RE = re.compile(
+ r"(\b(?:api-key|auth-token|access-token|private-token)" + _JUDGE_EVIDENCE_HEADER_SEPARATOR + r")"
+ r"[^\s'\"`,;\\]+",
+ re.IGNORECASE,
+)
+# A Cookie header holds "name=value" pairs joined by ";", and every pair is masked.
+_JUDGE_EVIDENCE_COOKIE_HEADER_RE = re.compile(
+ r"(\bcookie" + _JUDGE_EVIDENCE_HEADER_SEPARATOR + r")"
+ r"[^\s'\"`,;\\]+(?:;[ \t]{0,8}[^\s'\"`,;\\]+){0,64}",
+ re.IGNORECASE,
+)
+_JUDGE_EVIDENCE_CREDENTIAL_FLAG_RE = re.compile(
+ r"((?<![^\s'\"])--?(?:password|passwd|token|api-key|apikey|secret|access-token|auth-token)(?:[ \t]{1,8}|=))"
+ r"(?:\"[^\"\r\n]+\"|'[^'\r\n]+'|[^\s'\"`\\]+)",
+ re.IGNORECASE,
+)
+# "-u"/"--user" also name ordinary options (python -u, pip --user), so only a
+# user:password value is masked.
+_JUDGE_EVIDENCE_USER_FLAG_RE = re.compile(
+ r"((?<![^\s'\"])(?:-u|--user)(?:[ \t]{1,8}|=))"
+ r"(?:\"[^\"\r\n:]+:[^\"\r\n]+\"|'[^'\r\n:]+:[^'\r\n]+'|[^\s'\"`:\\]{1,256}:[^\s'\"`\\]+)"
)
-def _call_validated_json_judge(prompt, validate, call, extract, **call_kwargs):
- """Invoke a JSON judge with one format-correction retry when payload validation fails."""
+def _redact_judge_evidence(text):
+ """Mask credentials in a judge prompt, reply, or problem before it is saved."""
+ redacted = _redact_configured_credentials(_redact_evidence_text(text))
+ for pattern in _SECRET_PATTERNS:
+ redacted = pattern.sub("<redacted>", redacted)
+ redacted = _JUDGE_EVIDENCE_URL_USERINFO_RE.sub(r"\1<redacted>@", redacted)
+ for pattern in (
+ _JUDGE_EVIDENCE_DIGEST_AUTH_HEADER_RE,
+ _JUDGE_EVIDENCE_AUTH_HEADER_RE,
+ _JUDGE_EVIDENCE_KEY_HEADER_RE,
+ _JUDGE_EVIDENCE_COOKIE_HEADER_RE,
+ _JUDGE_EVIDENCE_CREDENTIAL_FLAG_RE,
+ _JUDGE_EVIDENCE_USER_FLAG_RE,
+ ):
+ redacted = pattern.sub(r"\1<redacted>", redacted)
+ return redacted
+
+
+def _verifier_output_dir():
+ """Directory that receives skill_evaluator_reward.json (see write_reward_outputs)."""
+ if (
+ SKILL_EVALUATOR_REWARD_JSON == VERIFIER_DIR / "skill_evaluator_reward.json"
+ and REWARD_JSON.parent != VERIFIER_DIR
+ ):
+ return REWARD_JSON.parent
+ return SKILL_EVALUATOR_REWARD_JSON.parent
+
+
+def _record_judge_attempt(
+ metric, attempt, *, prompt, reply, problem, method="llm", model=None, status=None, score=None
+):
+ """Save one judge attempt to judge_attempts.json while main() runs the judges.
+
+ The file is rewritten after every attempt, so it also explains a judge that
+ failed or a verifier that stopped mid-run. Saving evidence must never change
+ a score, so write failures are only logged. A ``salvage`` entry records the
+ usable part rebuilt from attempt *attempt*'s truncated reply.
+ """
+ attempts = _ACTIVE_JUDGE_EVIDENCE.get()
+ if attempts is None:
+ return
+ if status is None:
+ status = "error" if reply is None else "accepted" if problem is None else "rejected"
+ entry = {
+ "metric": metric,
+ "attempt": attempt,
+ "method": method,
+ "model": model,
+ "status": status,
+ "problem": _truncate_for_behavior(_redact_judge_evidence(problem), 1000) if problem else None,
+ }
+ if prompt is not None:
+ redacted_prompt = _redact_judge_evidence(prompt)
+ entry["prompt_sha256"] = hashlib.sha256(redacted_prompt.encode("utf-8", "replace")).hexdigest()
+ entry["prompt_chars"] = len(prompt)
+ entry["prompt"] = _truncate_for_behavior(redacted_prompt, JUDGE_EVIDENCE_PROMPT_CHARS)
+ if reply is not None:
+ entry["reply_chars"] = len(reply)
+ entry["reply"] = _truncate_for_behavior(_redact_judge_evidence(reply), JUDGE_EVIDENCE_REPLY_CHARS)
+ if score is not None:
+ entry["score"] = score
+ attempts.append(entry)
+ try:
+ path = _verifier_output_dir() / JUDGE_ATTEMPTS_FILE_NAME
+ path.parent.mkdir(parents=True, exist_ok=True)
+ tmp_path = path.with_name(path.name + ".tmp")
+ payload = {"judge_contract": JUDGE_CONTRACT, "attempts": attempts}
+ tmp_path.write_text(json.dumps(payload, indent=2), encoding="utf-8")
+ tmp_path.replace(path)
+ except TimeoutError:
+ # The required-judge deadline alarm, not a write failure.
+ raise
+ except (OSError, TypeError, ValueError) as error:
+ logger.warning("Could not save judge evidence: %s", type(error).__name__)
+
+
+def _judge_attempt_model(provenance=None, calls_before=None):
+ """Return the model that answered an attempt, or the requested model when none did."""
+ calls = _ACTIVE_JUDGE_CALLS.get()
+ if calls is not None and calls_before is not None and len(calls) > calls_before:
+ return calls[-1]["model"]
+ if provenance and provenance.get("model"):
+ return provenance["model"]
+ return _selected_judge_model()
+
+
+def _call_validated_json_judge(prompt, validate, call, extract, *, metric="judge", salvage=None, **call_kwargs):
+ """Invoke a JSON judge; on an invalid reply, retry once and name the problem.
+
+ *validate* returns ``None`` for a usable reply or a short description of the
+ problem, which the retry sends back to the judge. Provider errors are not
+ retried here: the transport already retried them within its own budget.
+ *salvage* may rebuild a usable reply from text that never parsed, such as a
+ reply cut off by the output-token cap; it is tried only after both attempts.
+ """
call_kwargs.setdefault("max_tokens", STRUCTURED_JUDGE_MAX_TOKENS)
+ unparsed_replies = []
- def invoke(call_prompt):
+ def invoke(attempt, call_prompt):
+ calls = _ACTIVE_JUDGE_CALLS.get()
+ calls_before = len(calls) if calls is not None else 0
content, error, *metadata = call(call_prompt, **call_kwargs)
provenance = metadata[0] if metadata and isinstance(metadata[0], dict) else {}
parsed = extract(content) if content else None
- validation_error = validate(parsed) if not error else None
- return parsed, error, provenance, validation_error
+ model = _judge_attempt_model(provenance, calls_before)
+ problem = None
+ if not error:
+ problem = "the reply was empty" if not content else validate(parsed)
+ if content and parsed is None:
+ unparsed_replies.append((attempt, content, model))
+ _record_judge_attempt(
+ metric,
+ attempt,
+ prompt=call_prompt,
+ reply=None if error else content or "",
+ problem=error or problem,
+ model=model,
+ )
+ return parsed, error, provenance, problem
- parsed, error, provenance, validation_error = invoke(prompt)
+ parsed, error, provenance, problem = invoke(1, prompt)
if error:
return None, f"LLM judge error: {error}", provenance
- if validation_error is None:
+ if problem is None:
return parsed, None, provenance
- parsed, error, provenance, validation_error = invoke(prompt + _JUDGE_RETRY_REMINDER)
+ parsed, error, provenance, problem = invoke(2, prompt + _judge_retry_feedback(problem))
+ if not error and problem is None:
+ return parsed, None, provenance
+ if salvage is not None:
+ # Newest first: only text that never parsed can be a reply cut off mid-object.
+ for attempt, text, model in reversed(unparsed_replies):
+ candidate = salvage(text)
+ if candidate is not None:
+ # The LLM attempts were saved with their own status (rejected,
+ # or error if the retry failed in transport); this entry shows
+ # that the score comes from the part of *attempt*'s reply that
+ # was rebuilt.
+ _record_judge_attempt(
+ metric,
+ attempt,
+ prompt=None,
+ reply=json.dumps(candidate),
+ problem=None,
+ method="salvage",
+ model=model,
+ status="accepted",
+ )
+ return candidate, None, provenance
if error:
return None, f"LLM judge retry error: {error}", provenance
- if validation_error is not None:
- return None, f"{validation_error} after retry", provenance
- return parsed, None, provenance
+ return None, f"Judge response was unparseable or invalid after retry: {problem}", provenance
# ── LLM Judge: Accuracy (5-criterion) ────────────────────────────────────────
-
-_ACCURACY_CRITERIA_KEYS = frozenset(
- {
- "SKILL_IDENTIFIED",
- "ACTION_CORRECT",
- "FACTUALLY_ACCURATE",
- "TASK_ADDRESSED",
- "ACTIONABLE",
- }
+# Accuracy grades the outcome only. Whether the agent found and used the skill is
+# scored by skill_execution, so a correct answer from the no-skill baseline can
+# reach 1.0 here.
+_ACCURACY_CRITERIA = (
+ ("ACTION_CORRECT", "Does the response describe or carry out the correct actions or commands for the task?"),
+ ("FACTUALLY_ACCURATE", "Are the factual claims consistent with the expected answer?"),
+ ("TASK_ADDRESSED", "Does the response directly address the user's request?"),
+ ("ACTIONABLE", "Does the response give usable results or next steps, not just an acknowledgment?"),
+ (
+ "FORMAT_FOLLOWED",
+ "If the USER QUESTION explicitly requires an output format (a template, prefix, fields, ordering, "
+ "or units), does the response follow it? When the question requires no specific format, answer true "
+ "if the response gives an answer and false if it gives none.",
+ ),
)
-
-
-def _valid_accuracy_criteria(value):
- """Return True when value is a complete 5-criterion boolean mapping."""
- return (
- isinstance(value, dict)
- and value.keys() == _ACCURACY_CRITERIA_KEYS
- and all(isinstance(item, bool) for item in value.values())
- )
+_ACCURACY_CRITERIA_KEYS = frozenset(key for key, _question in _ACCURACY_CRITERIA)
def _accuracy_payload_error(parsed):
- """Validate a parsed accuracy judge payload and return an error message if malformed."""
+ """Describe what is wrong with an accuracy reply, or return ``None`` when it is usable.
+
+ The score is computed from the five verdicts, so a ``score`` field in the
+ reply is ignored rather than trusted.
+ """
if not isinstance(parsed, dict):
- return "Judge response was not a valid JSON object"
- if "reason" in parsed and not isinstance(parsed["reason"], str):
- return "Judge response contained an invalid accuracy reason"
+ return "the reply did not contain one JSON object"
criteria = parsed.get("criteria")
- criteria_valid = _valid_accuracy_criteria(criteria)
- if "criteria" in parsed and not criteria_valid:
- return "Judge response contained invalid accuracy criteria"
- if _finite_score(parsed.get("score")) is None and not criteria_valid:
- return "Judge response contained no valid accuracy score or complete criteria"
+ if not isinstance(criteria, dict):
+ return 'the reply has no "criteria" object'
+ missing = sorted(_ACCURACY_CRITERIA_KEYS - criteria.keys())
+ if missing:
+ return f"criteria is missing {', '.join(missing)}"
+ unexpected = sorted(str(key)[:40] for key in criteria.keys() - _ACCURACY_CRITERIA_KEYS)
+ if unexpected:
+ return f"criteria has unexpected keys {', '.join(unexpected[:5])}"
+ not_boolean = sorted(key for key, value in criteria.items() if not isinstance(value, bool))
+ if not_boolean:
+ return f"criteria values must be true or false, not {', '.join(not_boolean)}"
+ if "reason" in parsed and not isinstance(parsed["reason"], str):
+ return '"reason" must be a string'
return None
def _goal_payload_error(parsed):
- """Validate a parsed goal-accuracy judge payload and return an error message if malformed."""
+ """Describe what is wrong with a goal reply, or return ``None`` when it is usable.
+
+ Goal accuracy is binary, like RAGAS AgentGoalAccuracyWithReference: the
+ score is 1.0 when ``achieved`` is true and 0.0 otherwise, so a ``score``
+ field in the reply is ignored.
+ """
if not isinstance(parsed, dict):
- return "Judge response was not a valid JSON object"
+ return "the reply did not contain one JSON object"
+ if not isinstance(parsed.get("achieved"), bool):
+ return '"achieved" must be true or false'
for field in ("reason", "user_goal", "end_state"):
if field in parsed and not isinstance(parsed[field], str):
- return f"Judge response contained an invalid {field} value"
- if not isinstance(parsed.get("achieved"), bool):
- return "Judge response contained an invalid achieved value"
- if "score" in parsed and _finite_score(parsed["score"]) is None:
- return "Judge response contained an invalid goal score"
+ return f'"{field}" must be a string'
return None
@@ -2472,26 +2681,13 @@ ACCURACY_JSON_SCHEMA = {
"properties": {
"criteria": {
"type": "object",
- "properties": {
- "SKILL_IDENTIFIED": {"type": "boolean"},
- "ACTION_CORRECT": {"type": "boolean"},
- "FACTUALLY_ACCURATE": {"type": "boolean"},
- "TASK_ADDRESSED": {"type": "boolean"},
- "ACTIONABLE": {"type": "boolean"},
- },
- "required": [
- "SKILL_IDENTIFIED",
- "ACTION_CORRECT",
- "FACTUALLY_ACCURATE",
- "TASK_ADDRESSED",
- "ACTIONABLE",
- ],
+ "properties": {key: {"type": "boolean"} for key, _question in _ACCURACY_CRITERIA},
+ "required": [key for key, _question in _ACCURACY_CRITERIA],
"additionalProperties": False,
},
- "score": {"type": "number"},
"reason": {"type": "string"},
},
- "required": ["criteria", "score", "reason"],
+ "required": ["criteria", "reason"],
"additionalProperties": False,
}
@@ -2501,10 +2697,9 @@ GOAL_ACCURACY_JSON_SCHEMA = {
"user_goal": {"type": "string"},
"end_state": {"type": "string"},
"achieved": {"type": "boolean"},
- "score": {"type": "number"},
"reason": {"type": "string"},
},
- "required": ["user_goal", "end_state", "achieved", "score", "reason"],
+ "required": ["user_goal", "end_state", "achieved", "reason"],
"additionalProperties": False,
}
@@ -2524,31 +2719,43 @@ BEHAVIOR_CHECK_JSON_SCHEMA = {
"additionalProperties": False,
},
},
- "score": {"type": "number"},
"summary": {"type": "string"},
},
- "required": ["results", "score", "summary"],
+ "required": ["results", "summary"],
"additionalProperties": False,
}
-def judge_accuracy(question, ground_truth, agent_text, *, skill_identification_applicable=True):
+def judge_accuracy(question, ground_truth, agent_text, *, skill_identification_applicable=None):
+ """Judge accuracy from five outcome verdicts; the score is computed here.
+
+ *skill_identification_applicable* is deprecated and ignored. Accuracy no
+ longer has a skill criterion, so every case and both arms use the same five
+ criteria. The keyword is still accepted so existing callers keep working.
+ """
+ del skill_identification_applicable
if not _has_judge_reference(ground_truth):
return _judge_not_applicable(_NO_GROUND_TRUTH_REASON)
- prompt = """You are an expert evaluator for AI agent responses. Evaluate by checking \
-each criterion below against the expected answer. For each criterion, determine true (satisfied) or false (not satisfied).
+ criteria_text = "\n".join(
+ f"{index}. {key}: {criterion}" for index, (key, criterion) in enumerate(_ACCURACY_CRITERIA, 1)
+ )
+ prompt = f"""You are an expert evaluator for AI agent responses. Check the response against \
+the expected answer on each criterion below and decide true (satisfied) or false (not satisfied) for each.
-1. SKILL_IDENTIFIED: Does the response reference or use the correct skill for the task?
-2. ACTION_CORRECT: Does the response describe or execute the correct actions/scripts?
-3. FACTUALLY_ACCURATE: Are the factual claims consistent with the expected answer?
-4. TASK_ADDRESSED: Does the response directly address the user's request?
-5. ACTIONABLE: Does the response provide actionable information (not just acknowledgment)?
+{criteria_text}
-Compute score = count(true) / 5.
-Be lenient on exact wording but strict on factual correctness.
+Judge the outcome only. Do not reward or penalize the response for naming, or not naming, \
+a tool or skill: that is scored separately.
+If the evidence holds no answer to the question (for example, the agent stopped, timed out or \
+refused), every criterion is false, unless the expected answer is itself a refusal.
+Be lenient on wording, but strict on factual correctness and on any output format the user \
+question explicitly requires.
-Respond with ONLY a JSON object:
-{{"criteria": {{"SKILL_IDENTIFIED": true, "ACTION_CORRECT": true, "FACTUALLY_ACCURATE": true, "TASK_ADDRESSED": true, "ACTIONABLE": true}}, "score": 1.0, "reason": "brief summary"}}
+Respond with ONLY one JSON object with a "criteria" object holding exactly these five keys, each \
+true or false, and a one-sentence "reason". Do not include a score: it is computed from the criteria.
+Example of the shape (the values are only an illustration):
+{{"criteria": {{"ACTION_CORRECT": true, "FACTUALLY_ACCURATE": false, "TASK_ADDRESSED": true, "ACTIONABLE": true, \
+"FORMAT_FOLLOWED": true}}, "reason": "Ran the right command but reported the wrong version number."}}
USER QUESTION:
{question}
@@ -2559,56 +2766,26 @@ EXPECTED ANSWER:
SELECTED EVIDENCE (final response + produced artifacts; low-relevance steps may be omitted):
{agent_text}"""
- if not skill_identification_applicable:
- prompt = prompt.replace(
- "1. SKILL_IDENTIFIED: Does the response reference or use the correct skill for the task?",
- "1. SKILL_IDENTIFIED: Inapplicable: no skill is expected. Set false; exclude it from the score.",
- ).replace("Compute score = count(true) / 5.", "Compute score = count(true among the other four criteria) / 4.")
- prompt = prompt.replace('"SKILL_IDENTIFIED": true', '"SKILL_IDENTIFIED": false')
- # Adjust the rubric before interpolation so task/evidence text stays intact.
- prompt = prompt.format(question=question, ground_truth=ground_truth, agent_text=agent_text)
prompt = append_character_facts(prompt, question=question, ground_truth=ground_truth, agent_text=agent_text)
- def validate_payload(parsed):
- error = _accuracy_payload_error(parsed)
- if not error and not skill_identification_applicable and not _valid_accuracy_criteria(parsed.get("criteria")):
- return "Accuracy requires complete criterion verdicts when skill identification is inapplicable"
- return error
-
parsed, error, _provenance = _call_validated_json_judge(
prompt,
- validate_payload,
+ _accuracy_payload_error,
call_public_llm,
extract_json,
+ metric="accuracy",
response_schema=ACCURACY_JSON_SCHEMA,
schema_name="accuracy_judgment",
)
if error:
return _judge_error(error)
- assert isinstance(parsed, dict)
- criteria = parsed.get("criteria")
- criteria_valid = _valid_accuracy_criteria(criteria)
-
- if criteria_valid:
- # Score only applicable verdicts; excluded criteria leave both numerator and denominator.
- excluded = set() if skill_identification_applicable else {"SKILL_IDENTIFIED"}
- applicable = [verdict for name, verdict in criteria.items() if name not in excluded]
- score = sum(verdict is True for verdict in applicable) / len(applicable)
- else:
- # Legacy score-only payloads; validate_payload rejects them when a criterion is excluded.
- assert skill_identification_applicable
- score = _finite_score(parsed.get("score"))
- assert score is not None
- result = {
- "score": round(score, 4),
+ criteria = {key: parsed["criteria"][key] for key, _question in _ACCURACY_CRITERIA}
+ return {
+ "score": round(sum(criteria.values()) / len(criteria), 4),
"reason": _bounded_judge_text(parsed.get("reason", "")),
- "criteria": criteria if criteria_valid else {},
+ "criteria": criteria,
}
- if not skill_identification_applicable:
- result["excluded_criteria"] = ["SKILL_IDENTIFIED"]
- result["scoring_policy"] = "applicable-accuracy-criteria/v1"
- return result
# ── LLM Judge: Goal Accuracy ─────────────────────────────────────────────────
@@ -2621,11 +2798,34 @@ def judge_goal_accuracy(question, ground_truth, agent_text, tool_summary=""):
if _ragas_goal_accuracy_enabled():
try:
result = _judge_goal_accuracy_ragas(question, ground_truth, agent_text, tool_summary)
- if not isinstance(result, dict) or _finite_score(result.get("score")) is None:
- raise ValueError("RAGAS returned a non-finite goal accuracy score")
- return result
+ score = result.get("score") if isinstance(result, dict) else None
+ if isinstance(score, bool) or score not in (0.0, 1.0):
+ raise ValueError("RAGAS returned a non-binary goal accuracy score")
except Exception as e:
logger.info("RAGAS not available (%s), using custom prompt", e)
+ problem = f"RAGAS failed ({type(e).__name__}: {e}); used the custom judge"
+ _record_judge_attempt(
+ "goal_accuracy",
+ 1,
+ prompt=None,
+ reply=None,
+ problem=problem,
+ method="ragas",
+ model=_judge_attempt_model(),
+ )
+ else:
+ _record_judge_attempt(
+ "goal_accuracy",
+ 1,
+ prompt=None,
+ reply=None,
+ problem=None,
+ method="ragas",
+ model=result.get("model") or _judge_attempt_model(),
+ status="accepted",
+ score=score,
+ )
+ return result
return _judge_goal_accuracy_custom(question, ground_truth, agent_text, tool_summary)
@@ -2690,11 +2890,13 @@ def _judge_goal_accuracy_ragas(question, ground_truth, agent_text, tool_summary)
loop.close()
score = float(result.value) if hasattr(result, "value") else float(result)
- if not math.isfinite(score):
- raise ValueError("RAGAS returned a non-finite goal accuracy score")
+ if score not in (0.0, 1.0):
+ # Also rejects NaN; the caller then runs the prompt judge.
+ raise ValueError("RAGAS returned a non-binary goal accuracy score")
return {
- "score": max(0.0, min(1.0, score)),
+ "score": score,
"reason": "RAGAS AgentGoalAccuracyWithReference",
+ "achieved": score == 1.0,
"method": "ragas",
"provider": "openai",
"model": _selected_judge_model(),
@@ -2702,28 +2904,28 @@ def _judge_goal_accuracy_ragas(question, ground_truth, agent_text, tool_summary)
def _judge_goal_accuracy_custom(question, ground_truth, agent_text, tool_summary):
- """Fallback: two-step custom prompt mirroring RAGAS logic."""
- prompt = f"""You are an evaluation judge. Determine whether an AI agent achieved the expected goal.
+ """Fallback: two-step custom prompt mirroring RAGAS logic, with the same binary score."""
+ tool_section = f"\nAGENT'S TOOL CALLS:\n{tool_summary}\n" if str(tool_summary or "").strip() else ""
+ prompt = f"""You are an evaluation judge. Decide whether an AI agent achieved the expected goal.
-Step 1: What was the user's goal?
-Step 2: What end state did the agent reach?
-Step 3: Compare the end state to the expected outcome.
+Step 1: State the user's goal.
+Step 2: State the end state the agent reached.
+Step 3: Decide whether that end state achieves the expected outcome.
USER REQUEST:
{question}
EXPECTED OUTCOME:
{ground_truth}
-
-AGENT'S TOOL CALLS:
-{tool_summary}
-
+{tool_section}
END-STATE EVIDENCE:
{agent_text}
-Did the agent achieve the expected goal?
-Respond with ONLY a JSON object:
-{{"user_goal": "...", "end_state": "...", "achieved": true/false, "score": 1.0, "reason": "..."}}"""
+Respond with ONLY one JSON object with the keys "user_goal", "end_state", "achieved" (true or \
+false) and a one-sentence "reason". Do not include a score: it is 1.0 when achieved and 0.0 otherwise.
+Example of the shape (the values are only an illustration):
+{{"user_goal": "List the open release blockers", "end_state": "Listed two of the three blockers", \
+"achieved": false, "reason": "One blocker from the expected outcome is missing."}}"""
prompt = append_character_facts(
prompt, question=question, ground_truth=ground_truth, tool_summary=tool_summary, agent_text=agent_text
)
@@ -2733,23 +2935,18 @@ Respond with ONLY a JSON object:
_goal_payload_error,
_call_public_llm_with_provenance,
extract_json,
+ metric="goal_accuracy",
response_schema=GOAL_ACCURACY_JSON_SCHEMA,
schema_name="goal_accuracy_judgment",
)
if error:
return _judge_error(error, **provenance)
- assert isinstance(parsed, dict)
- achieved = parsed.get("achieved")
- assert isinstance(achieved, bool)
-
- score = 1.0 if achieved else 0.0
- if "score" in parsed:
- score = _finite_score(parsed["score"])
- assert score is not None
+ achieved = parsed["achieved"]
return {
- "score": score,
+ "score": 1.0 if achieved else 0.0,
"reason": _bounded_judge_text(parsed.get("reason", "")),
+ "achieved": achieved,
"user_goal": _bounded_judge_text(parsed.get("user_goal", "")),
"end_state": _bounded_judge_text(parsed.get("end_state", "")),
"method": "custom",
@@ -2765,22 +2962,34 @@ Respond with ONLY a JSON object:
# (finish_reason="length", reasoning_tokens=1024).
BEHAVIOR_JUDGE_MAX_TOKENS = STRUCTURED_JUDGE_MAX_TOKENS
-_BEHAVIOR_RETRY_REMINDER = (
- "\n\nIMPORTANT: Your previous reply could not be parsed. Respond with ONLY the "
- "minified JSON object on a single line -- no markdown fences, no prose, and "
- 'keep every "reason" under 15 words.'
-)
+
+def _behavior_prompt_example(expected_count):
+ """One entry per expected behavior, so the example is a valid reply if a judge copies it."""
+ return json.dumps(
+ {
+ "results": [
+ {
+ "step": step,
+ "passed": step % 2 == 1,
+ "reason": "Seen in the transcript." if step % 2 == 1 else "Not seen in the transcript.",
+ }
+ for step in range(1, expected_count + 1)
+ ],
+ "summary": "A short summary of what was and was not observed.",
+ }
+ )
def judge_behavior_check(conversation_text, expected_behaviors):
if not _has_judge_reference(expected_behaviors):
return _judge_not_applicable(_NO_EXPECTED_BEHAVIOR_REASON, results=[])
+ expected_count = len(expected_behaviors)
behaviors_text = "\n".join(f"{i + 1}. {b}" for i, b in enumerate(expected_behaviors))
conversation = _compact_behavior_conversation(conversation_text)
- prompt = f"""You are evaluating whether an AI agent followed expected behaviors during a task. \
-Analyze the full conversation and determine if each expected behavior was observed.
+ prompt = f"""You are evaluating whether an AI agent showed each expected behavior during a task. \
+Read the conversation and decide, for each numbered behavior, whether it was observed.
CONVERSATION:
{conversation}
@@ -2788,71 +2997,43 @@ CONVERSATION:
EXPECTED BEHAVIORS:
{behaviors_text}
-For each behavior, set "passed" to true (observed) or false (not observed) with a brief reason.
+Respond with ONLY one JSON object. "results" must hold exactly {expected_count} entries, one per \
+expected behavior: "step" is the behavior's number (each number from 1 to {expected_count} exactly \
+once), "passed" is true (observed) or false (not observed), and "reason" is under 15 words. Add a \
+one-sentence "summary". Do not include a score: it is computed from the results.
+Example of the shape for this task's {expected_count} behaviors (the values are only an illustration):
+{_behavior_prompt_example(expected_count)}"""
-Respond with ONLY a JSON object:
-{{"results": [{{"step": 1, "passed": true, "reason": "..."}}, ...], "score": 0.67, "summary": "brief summary"}}"""
prompt = append_character_facts(prompt, conversation=conversation, expected_behaviors=behaviors_text)
- content, error = call_public_llm(
+ def salvage(text):
+ # Complete entries from a reply cut off before its results array closed.
+ salvaged = _salvage_behavior_results(text)
+ candidate = {
+ "results": salvaged,
+ "summary": f"Salvaged {len(salvaged)}/{expected_count} behavior results from truncated judge response",
+ }
+ if not salvaged or _behavior_payload_error(candidate, expected_count, allow_partial=True):
+ return None
+ return candidate
+
+ parsed, error, _provenance = _call_validated_json_judge(
prompt,
+ lambda payload: _behavior_payload_error(payload, expected_count),
+ call_public_llm,
+ extract_json,
+ metric="behavior_check",
+ salvage=salvage,
max_tokens=BEHAVIOR_JUDGE_MAX_TOKENS,
response_schema=BEHAVIOR_CHECK_JSON_SCHEMA,
schema_name="behavior_check_judgment",
)
if error:
- return _judge_error(f"LLM judge error: {error}", results=[])
-
- def _parse_judge_object(text):
- return extract_json(text) if text else None
-
- parsed = _parse_judge_object(content)
- score = _behavior_payload_score(parsed, len(expected_behaviors))
- attempts = [(content or "", parsed)]
- retry_error = None
- if score is None:
- # One retry max, with an explicit machine-readable-output reminder.
- retry_content, retry_error = call_public_llm(
- prompt + _BEHAVIOR_RETRY_REMINDER,
- max_tokens=BEHAVIOR_JUDGE_MAX_TOKENS,
- response_schema=BEHAVIOR_CHECK_JSON_SCHEMA,
- schema_name="behavior_check_judgment",
- )
- if not retry_error:
- parsed = _parse_judge_object(retry_content)
- attempts.append((retry_content or "", parsed))
- score = _behavior_payload_score(parsed, len(expected_behaviors))
-
- if score is None:
- # Salvage complete entries from a truncated results array (newest first).
- for text, extracted in reversed(attempts):
- if extracted is not None:
- continue
- salvaged = _salvage_behavior_results(text)
- if salvaged:
- candidate = {
- "results": salvaged,
- "summary": (
- f"Salvaged {len(salvaged)}/{len(expected_behaviors)} behavior "
- "results from truncated judge response"
- ),
- }
- candidate_score = _behavior_payload_score(
- candidate,
- len(expected_behaviors),
- allow_partial=True,
- )
- if candidate_score is not None:
- parsed = candidate
- score = candidate_score
- break
-
- if score is None:
- if retry_error:
- return _judge_error(f"LLM judge retry error: {retry_error}", results=[])
- return _judge_error("Judge response was unparseable or invalid after retry", results=[])
+ return _judge_error(error, results=[])
- results = parsed["results"]
+ # A salvaged reply leaves the behaviors after the cut unjudged; they count as not observed.
+ results = sorted(parsed["results"], key=lambda result: result["step"])
+ score = sum(1 for result in results if result["passed"]) / expected_count
return {
"score": round(score, 4),
"reason": parsed.get("summary", ""),
@@ -2860,25 +3041,47 @@ Respond with ONLY a JSON object:
}
-def _behavior_payload_score(parsed, expected_count, *, allow_partial=False):
+def _behavior_payload_error(parsed, expected_count, *, allow_partial=False):
+ """Describe what is wrong with a behavior reply, or return ``None`` when it is usable.
+
+ Each expected behavior is judged exactly once, identified by its 1-based
+ ``step`` number, with a boolean ``passed``. *allow_partial* accepts a
+ salvaged prefix that judges each of its steps once. The score is computed
+ from the verdicts, so a ``score`` field in the reply is ignored.
+ """
if not isinstance(parsed, dict):
- return None
+ return "the reply did not contain one JSON object"
results = parsed.get("results")
if not isinstance(results, list):
- return None
- if any(not isinstance(result, dict) or not isinstance(result.get("passed"), bool) for result in results):
- return None
+ return 'the reply has no "results" list'
if allow_partial:
if not results or len(results) > expected_count:
- return None
+ return f'"results" has {len(results)} entries; it needs 1 to {expected_count}'
elif len(results) != expected_count:
- return None
- if "score" in parsed and _finite_score(parsed["score"]) is None:
- return None
- denominator = expected_count if allow_partial else len(results)
- if denominator <= 0:
- return None
- return sum(1 for result in results if result["passed"]) / denominator
+ return f'"results" has {len(results)} entries; it needs exactly {expected_count}, one per behavior'
+ if any(not isinstance(result, dict) for result in results):
+ return 'every "results" entry must be an object'
+ if any(not isinstance(result.get("passed"), bool) for result in results):
+ return 'every "results" entry needs "passed" set to true or false'
+ steps = [result.get("step") for result in results]
+ if any(isinstance(step, bool) or not isinstance(step, int) for step in steps):
+ return 'every "results" entry needs an integer "step" number'
+ expected_steps = set(range(1, expected_count + 1))
+ duplicated = sorted({step for step in steps if steps.count(step) > 1})
+ unexpected = sorted(set(steps) - expected_steps)[:10]
+ missing = [] if allow_partial else sorted(expected_steps - set(steps))[:10]
+ if duplicated or missing or unexpected:
+ parts = []
+ if duplicated:
+ parts.append(f"steps judged more than once: {duplicated[:10]}")
+ if missing:
+ parts.append(f"steps not judged: {missing}")
+ if unexpected:
+ parts.append(f"steps outside 1..{expected_count}: {unexpected}")
+ return "; ".join(parts)
+ if "summary" in parsed and not isinstance(parsed["summary"], str):
+ return '"summary" must be a string'
+ return None
# ── Main ─────────────────────────────────────────────────────────────────────
@@ -3109,12 +3312,7 @@ def write_reward_outputs(result, overall):
sanitized_result = {key: _sanitize_error_value(value) for key, value in result.items()}
sanitized_overall = _sanitize_error_value(overall)
REWARD_JSON.parent.mkdir(parents=True, exist_ok=True)
- skill_evaluator_reward_json = SKILL_EVALUATOR_REWARD_JSON
- if (
- skill_evaluator_reward_json == VERIFIER_DIR / "skill_evaluator_reward.json"
- and REWARD_JSON.parent != VERIFIER_DIR
- ):
- skill_evaluator_reward_json = REWARD_JSON.parent / skill_evaluator_reward_json.name
+ skill_evaluator_reward_json = _verifier_output_dir() / SKILL_EVALUATOR_REWARD_JSON.name
skill_evaluator_reward_json.parent.mkdir(parents=True, exist_ok=True)
skill_evaluator_reward_json.write_text(json.dumps(sanitized_result, indent=2))
REWARD_JSON.write_text(json.dumps(_numeric_reward_payload(sanitized_result, sanitized_overall), indent=2))
@@ -3255,42 +3453,46 @@ def main():
traj, question, ground_truth=ground_truth, expected_behavior=expected_behavior
)
- # ── Eval 4: accuracy (LLM judge) ─────────────────────────────────────
- acc_result = _call_required_judge(
- "accuracy",
- judge_accuracy,
- question,
- ground_truth,
- bundles["accuracy"]["prompt_evidence"],
- skill_identification_applicable=should_trigger and bool(expected_skill),
- allow_not_applicable=not _has_judge_reference(ground_truth),
- )
- acc_score = acc_result["score"]
- details["accuracy"] = acc_result
-
- # ── Eval 5: goal_accuracy (RAGAS or custom LLM judge) ────────────────
- ga_result = _call_required_judge(
- "goal_accuracy",
- judge_goal_accuracy,
- question,
- ground_truth,
- bundles["goal_accuracy"]["prompt_evidence"],
- tool_summary="",
- allow_not_applicable=not _has_judge_reference(ground_truth),
- )
- ga_score = ga_result["score"]
- details["goal_accuracy"] = ga_result
-
- # ── Eval 6: behavior_check (LLM judge) ───────────────────────────────
- bc_result = _call_required_judge(
- "behavior_check",
- judge_behavior_check,
- bundles["behavior_check"]["prompt_evidence"],
- expected_behavior,
- allow_not_applicable=not _has_judge_reference(expected_behavior),
- )
- bc_score = bc_result["score"]
- details["behavior_check"] = bc_result
+ # Every judge attempt below is saved to judge_attempts.json as it happens.
+ judge_evidence_token = _ACTIVE_JUDGE_EVIDENCE.set([])
+ try:
+ # ── Eval 4: accuracy (LLM judge) ─────────────────────────────────
+ acc_result = _call_required_judge(
+ "accuracy",
+ judge_accuracy,
+ question,
+ ground_truth,
+ bundles["accuracy"]["prompt_evidence"],
+ allow_not_applicable=not _has_judge_reference(ground_truth),
+ )
+ acc_score = acc_result["score"]
+ details["accuracy"] = acc_result
+
+ # ── Eval 5: goal_accuracy (RAGAS or custom LLM judge) ────────────
+ ga_result = _call_required_judge(
+ "goal_accuracy",
+ judge_goal_accuracy,
+ question,
+ ground_truth,
+ bundles["goal_accuracy"]["prompt_evidence"],
+ tool_summary="",
+ allow_not_applicable=not _has_judge_reference(ground_truth),
+ )
+ ga_score = ga_result["score"]
+ details["goal_accuracy"] = ga_result
+
+ # ── Eval 6: behavior_check (LLM judge) ───────────────────────────
+ bc_result = _call_required_judge(
+ "behavior_check",
+ judge_behavior_check,
+ bundles["behavior_check"]["prompt_evidence"],
+ expected_behavior,
+ allow_not_applicable=not _has_judge_reference(expected_behavior),
+ )
+ bc_score = bc_result["score"]
+ details["behavior_check"] = bc_result
+ finally:
+ _ACTIVE_JUDGE_EVIDENCE.reset(judge_evidence_token)
# persist refs + omission metadata onto the metric details
attach_metric_evidence_refs(details, {m: bundles[m]["evidence_refs"] for m in bundles})
@@ -3308,6 +3510,7 @@ def main():
"behavior_check": bc_score,
"metric_set": DEFAULT_METRIC_SET,
"score_policy": DEFAULT_SCORE_POLICY,
+ "judge_contract": JUDGE_CONTRACT,
"entry_id": entry.get("id"),
"has_skill": entry.get("has_skill", True),
"trajectory_source": traj_meta.get("source"),
diff --git a/tests/test_accuracy_arithmetic.py b/tests/test_accuracy_arithmetic.py
index 76f26f8b..751803e7 100644
--- a/tests/test_accuracy_arithmetic.py
+++ b/tests/test_accuracy_arithmetic.py
@@ -1,7 +1,7 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
-"""Accuracy scores must implement the applicable-criterion rubric in both runtimes."""
+"""Accuracy scores must be computed from the five outcome criteria in both runtimes."""
from __future__ import annotations
@@ -38,30 +38,25 @@ def test_accuracy_derives_score_from_complete_criteria(monkeypatch, accuracy_jud
assert "error" not in result
[email protected]("skill_verdict", [True, False])
[email protected]("passed", range(5))
-def test_negative_accuracy_scores_only_applicable_criteria(monkeypatch, accuracy_judge, passed, skill_verdict):
- # Negative cases drop SKILL_IDENTIFIED from numerator and denominator; the model total stays ignored.
- applicable = sorted(llm_judge._ACCURACY_CRITERIA_KEYS - {"SKILL_IDENTIFIED"})
- criteria = {key: index < passed for index, key in enumerate(applicable)}
- criteria["SKILL_IDENTIFIED"] = skill_verdict
- payload = {"criteria": criteria, "score": 0.1, "reason": "declined correctly"}
- monkeypatch.setattr(accuracy_judge, "call_public_llm", lambda *_args, **_kwargs: (json.dumps(payload), None))
-
- result = accuracy_judge.judge_accuracy("question", "reference", "answer", skill_identification_applicable=False)
+def test_accuracy_criteria_grade_the_outcome_not_skill_use(accuracy_judge):
+ # Skill use is scored by skill_execution. With no skill criterion, a correct
+ # answer from the no-skill baseline is not capped below 1.0.
+ outcome_criteria = {"ACTION_CORRECT", "FACTUALLY_ACCURATE", "TASK_ADDRESSED", "ACTIONABLE", "FORMAT_FOLLOWED"}
+ schema = accuracy_judge.ACCURACY_JSON_SCHEMA
- assert result["score"] == pytest.approx(passed / 4)
- assert result["criteria"] == criteria
- assert result["excluded_criteria"] == ["SKILL_IDENTIFIED"]
- assert result["scoring_policy"] == "applicable-accuracy-criteria/v1"
+ assert set(accuracy_judge._ACCURACY_CRITERIA_KEYS) == outcome_criteria
+ assert set(schema["properties"]["criteria"]["required"]) == outcome_criteria
+ assert "score" not in schema["properties"]
+ assert "score" not in schema["required"]
def test_tier3_docs_describe_criterion_derived_accuracy():
docs = (Path(__file__).resolve().parents[1] / "docs" / "tier3-live-evaluation.mdx").read_text(encoding="utf-8")
normalized = " ".join(docs.split())
- assert "passing applicable criteria divided by applicable criteria" in normalized
- assert "falls back to the judge's numeric score" in normalized
+ assert "passing criteria divided by five" in normalized
+ assert "FORMAT_FOLLOWED" in normalized
+ assert "falls back to the judge's numeric score" not in normalized
assert "existing scoring contract" not in normalized
@@ -70,10 +65,10 @@ def test_tier3_docs_describe_criterion_derived_accuracy():
[
({"criteria": _criteria(3), "score": 0.6}, 0.6),
({"criteria": _criteria(3)}, 0.6),
- ({"score": 0.6}, 0.6),
+ ({"criteria": _criteria(3), "score": 1.0}, 0.6),
],
)
-def test_accuracy_preserves_consistent_and_legacy_payloads(monkeypatch, accuracy_judge, payload, expected):
+def test_accuracy_ignores_the_judge_total(monkeypatch, accuracy_judge, payload, expected):
monkeypatch.setattr(accuracy_judge, "call_public_llm", lambda *_args, **_kwargs: (json.dumps(payload), None))
assert accuracy_judge.judge_accuracy("question", "reference", "answer")["score"] == expected
diff --git a/tests/test_character_facts.py b/tests/test_character_facts.py
index 4996e65e..0423662d 100644
--- a/tests/test_character_facts.py
+++ b/tests/test_character_facts.py
@@ -106,7 +106,15 @@ def test_character_facts_use_the_same_formatting_as_prompt_placeholders(monkeypa
def test_judges_accept_non_string_fields_with_character_facts(monkeypatch, implementation, judge_name, ground_truth):
judge = llm_judge if implementation == "host" else load_harbor_eval_template()
payload = json.dumps(
- {"score": 0.0, "reason": "test verdict", "achieved": False, "user_goal": "test", "end_state": "test"}
+ {
+ "criteria": dict.fromkeys(
+ ["ACTION_CORRECT", "FACTUALLY_ACCURATE", "TASK_ADDRESSED", "ACTIONABLE", "FORMAT_FOLLOWED"], False
+ ),
+ "reason": "test verdict",
+ "achieved": False,
+ "user_goal": "test",
+ "end_state": "test",
+ }
)
mock = MagicMock(return_value=(payload, None))
monkeypatch.setattr(judge, "call_public_llm", mock)
@@ -142,9 +150,8 @@ def test_all_judges_preserve_original_evidence_and_append_facts(monkeypatch, imp
payload = json.dumps(
{
"criteria": dict.fromkeys(
- ["SKILL_IDENTIFIED", "ACTION_CORRECT", "FACTUALLY_ACCURATE", "TASK_ADDRESSED", "ACTIONABLE"], True
+ ["ACTION_CORRECT", "FACTUALLY_ACCURATE", "TASK_ADDRESSED", "ACTIONABLE", "FORMAT_FOLLOWED"], True
),
- "score": 1.0,
"reason": "test",
"achieved": True,
"user_goal": "test",
diff --git a/tests/test_ci_workflows.py b/tests/test_ci_workflows.py
index 6fee9ad0..6220f575 100644
--- a/tests/test_ci_workflows.py
+++ b/tests/test_ci_workflows.py
@@ -277,6 +277,24 @@ def test_gitleaks_single_value_allowlists_are_exactly_scoped() -> None:
}
+def test_gitleaks_curl_auth_allowlists_are_exactly_scoped() -> None:
+ # Fake `curl -u user:password` fixtures stay in history, so each allowlist
+ # must name one credential value and one test file for the curl rule only.
+ config = tomllib.loads(GITLEAKS_CONFIG.read_text(encoding="utf-8"))
+ entries = [entry for entry in config["allowlists"] if "curl-auth-user" in entry.get("targetRules", [])]
+
+ assert entries
+ for entry in entries:
+ assert set(entry) == {"description", "condition", "targetRules", "regexes", "paths"}
+ assert entry["condition"] == "AND"
+ assert entry["targetRules"] == ["curl-auth-user"]
+ [regex] = entry["regexes"]
+ [path] = entry["paths"]
+ assert regex.startswith("^") and regex.endswith("$")
+ assert ".*" not in regex
+ assert re.fullmatch(r"\^tests/[\w/]+\\\.py\$", path)
+
+
def test_gitleaks_uses_event_specific_full_history_scopes() -> None:
job = _load("security.yml")["jobs"]["gitleaks"]
checkout = next(step for step in job["steps"] if _is_checkout_step(step))
diff --git a/tests/test_llm_judge.py b/tests/test_llm_judge.py
index 7f71eec3..1c62d933 100644
--- a/tests/test_llm_judge.py
+++ b/tests/test_llm_judge.py
@@ -165,11 +165,11 @@ def test_call_public_llm_uses_production_gpt5_payload_without_temperature(
json.dumps(
{
"criteria": {
- "SKILL_IDENTIFIED": True,
"ACTION_CORRECT": True,
"FACTUALLY_ACCURATE": True,
"TASK_ADDRESSED": True,
"ACTIONABLE": True,
+ "FORMAT_FOLLOWED": True,
},
"score": 1.0,
"reason": "recovered",
@@ -342,11 +342,11 @@ def test_deceptive_openai_urls_keep_max_tokens(provider: str, request_url: str)
json.dumps(
{
"criteria": {
- "SKILL_IDENTIFIED": True,
"ACTION_CORRECT": True,
"FACTUALLY_ACCURATE": True,
"TASK_ADDRESSED": True,
"ACTIONABLE": True,
+ "FORMAT_FOLLOWED": True,
},
"score": 1.0,
"reason": "all good",
@@ -433,11 +433,11 @@ def test_structured_judges_pass_anthropic_output_config_schema(
valid_response = json.dumps(
{
"criteria": {
- "SKILL_IDENTIFIED": True,
"ACTION_CORRECT": True,
"FACTUALLY_ACCURATE": True,
"TASK_ADDRESSED": True,
"ACTIONABLE": True,
+ "FORMAT_FOLLOWED": True,
},
"score": 1.0,
"reason": "all good",
@@ -504,11 +504,11 @@ def test_client_and_template_downgrade_on_400_422_and_memoize_across_calls(
valid_response = json.dumps(
{
"criteria": {
- "SKILL_IDENTIFIED": True,
"ACTION_CORRECT": True,
"FACTUALLY_ACCURATE": True,
"TASK_ADDRESSED": True,
"ACTIONABLE": True,
+ "FORMAT_FOLLOWED": True,
},
"score": 1.0,
"reason": "downgraded ok",
@@ -641,11 +641,11 @@ def test_transient_429_preserves_schema_and_does_not_memoize_unsupported(
valid_response = json.dumps(
{
"criteria": {
- "SKILL_IDENTIFIED": True,
"ACTION_CORRECT": True,
"FACTUALLY_ACCURATE": True,
"TASK_ADDRESSED": True,
"ACTIONABLE": True,
+ "FORMAT_FOLLOWED": True,
},
"score": 1.0,
"reason": "retried 429 with schema",
@@ -689,11 +689,11 @@ def test_template_model_fallback_isolates_schema_memoization_per_model(
valid_response = json.dumps(
{
"criteria": {
- "SKILL_IDENTIFIED": True,
"ACTION_CORRECT": True,
"FACTUALLY_ACCURATE": True,
"TASK_ADDRESSED": True,
"ACTIONABLE": True,
+ "FORMAT_FOLLOWED": True,
},
"score": 1.0,
"reason": "fallback model with schema",
@@ -750,11 +750,11 @@ def test_reasoning_token_exhaustion_missing_message_retries_cleanly(
valid_response = json.dumps(
{
"criteria": {
- "SKILL_IDENTIFIED": True,
"ACTION_CORRECT": True,
"FACTUALLY_ACCURATE": True,
"TASK_ADDRESSED": True,
"ACTIONABLE": True,
+ "FORMAT_FOLLOWED": True,
},
"score": 1.0,
"reason": "recovered from missing message",
@@ -835,11 +835,11 @@ def test_anthropic_downgrades_on_400_and_memoizes_across_calls(
valid_response = json.dumps(
{
"criteria": {
- "SKILL_IDENTIFIED": True,
"ACTION_CORRECT": True,
"FACTUALLY_ACCURATE": True,
"TASK_ADDRESSED": True,
"ACTIONABLE": True,
+ "FORMAT_FOLLOWED": True,
},
"score": 1.0,
"reason": "anthropic downgraded ok",
@@ -979,11 +979,11 @@ def test_schema_builders_and_downgrade_warning_log(monkeypatch, caplog):
valid_response = json.dumps(
{
"criteria": {
- "SKILL_IDENTIFIED": True,
"ACTION_CORRECT": True,
"FACTUALLY_ACCURATE": True,
"TASK_ADDRESSED": True,
"ACTIONABLE": True,
+ "FORMAT_FOLLOWED": True,
},
"score": 1.0,
"reason": "ok",
@@ -1056,11 +1056,11 @@ def test_unrelated_400_does_not_disable_schema_flags(error_message: str) -> None
"content": json.dumps(
{
"criteria": {
- "SKILL_IDENTIFIED": True,
"ACTION_CORRECT": True,
"FACTUALLY_ACCURATE": True,
"TASK_ADDRESSED": True,
"ACTIONABLE": True,
+ "FORMAT_FOLLOWED": True,
},
"score": 1.0,
"reason": "ok",
@@ -1180,11 +1180,11 @@ def test_harbor_eval_template_unrelated_400_does_not_disable_schema(
"content": json.dumps(
{
"criteria": {
- "SKILL_IDENTIFIED": True,
"ACTION_CORRECT": True,
"FACTUALLY_ACCURATE": True,
"TASK_ADDRESSED": True,
"ACTIONABLE": True,
+ "FORMAT_FOLLOWED": True,
},
"score": 1.0,
"reason": "ok",
diff --git a/tests/tier3/test_behavior_evidence.py b/tests/tier3/test_behavior_evidence.py
index 7fa18833..fddd6f2f 100644
--- a/tests/tier3/test_behavior_evidence.py
+++ b/tests/tier3/test_behavior_evidence.py
@@ -435,21 +435,20 @@ def test_harbor_template_main_persists_evidence_refs_in_reward_json(monkeypatch,
raise RuntimeError("force custom judge")
def fake_call_public_llm(prompt, *args, **kwargs):
- if "For each criterion" in prompt:
+ if "FORMAT_FOLLOWED" in prompt:
return json.dumps(
{
"criteria": {
- "SKILL_IDENTIFIED": True,
"ACTION_CORRECT": True,
"FACTUALLY_ACCURATE": True,
"TASK_ADDRESSED": True,
"ACTIONABLE": True,
+ "FORMAT_FOLLOWED": True,
},
- "score": 1.0,
"reason": "ok",
}
), None
- if "Did the agent achieve the expected goal?" in prompt:
+ if "achieved the expected goal" in prompt:
return json.dumps({"achieved": True, "score": 1.0, "reason": "ok"}), None
return json.dumps(
{
diff --git a/tests/tier3/test_evidence_compiler.py b/tests/tier3/test_evidence_compiler.py
index 13531e9f..9592d08d 100644
--- a/tests/tier3/test_evidence_compiler.py
+++ b/tests/tier3/test_evidence_compiler.py
@@ -230,8 +230,8 @@ def test_small_accuracy_and_goal_budgets_retain_final_response_excerpt(monkeypat
captured_shared.__setitem__("last", prompt)
or (
'{"achieved": true, "score": 1.0, "reason": "ok", "criteria": '
- '{"SKILL_IDENTIFIED": true, "ACTION_CORRECT": true, "FACTUALLY_ACCURATE": true, '
- '"TASK_ADDRESSED": true, "ACTIONABLE": true}}',
+ '{"ACTION_CORRECT": true, "FACTUALLY_ACCURATE": true, '
+ '"TASK_ADDRESSED": true, "ACTIONABLE": true, "FORMAT_FOLLOWED": true}}',
None,
)
),
@@ -243,8 +243,8 @@ def test_small_accuracy_and_goal_budgets_retain_final_response_excerpt(monkeypat
captured_template.__setitem__("last", prompt)
or (
'{"achieved": true, "score": 1.0, "reason": "ok", "criteria": '
- '{"SKILL_IDENTIFIED": true, "ACTION_CORRECT": true, "FACTUALLY_ACCURATE": true, '
- '"TASK_ADDRESSED": true, "ACTIONABLE": true}}',
+ '{"ACTION_CORRECT": true, "FACTUALLY_ACCURATE": true, '
+ '"TASK_ADDRESSED": true, "ACTIONABLE": true, "FORMAT_FOLLOWED": true}}',
None,
)
),
diff --git a/tests/tier3/test_judge_contracts.py b/tests/tier3/test_judge_contracts.py
new file mode 100644
index 00000000..689933ec
--- /dev/null
+++ b/tests/tier3/test_judge_contracts.py
@@ -0,0 +1,899 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Judge reply contracts: verdicts in, scores computed in code, specific retries, saved evidence."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from types import ModuleType
+from typing import Any
+
+import pytest
+from tests.conftest import load_harbor_eval_template
+
+from skillevaluator.tier3.eval_core import llm_judge
+from skillevaluator.tier3.harbor.metrics import RESERVED_METRIC_NAMES
+
+eval_template = load_harbor_eval_template("harbor_template_eval_judge_contracts")
+
+_CRITERIA = {
+ "ACTION_CORRECT": True,
+ "FACTUALLY_ACCURATE": False,
+ "TASK_ADDRESSED": True,
+ "ACTIONABLE": False,
+ "FORMAT_FOLLOWED": True,
+}
+_ACCURACY_RESPONSE = json.dumps({"criteria": _CRITERIA, "reason": "judged from evidence"})
+_GOAL_RESPONSE = json.dumps(
+ {"user_goal": "complete the task", "end_state": "task completed", "achieved": True, "reason": "met"}
+)
+_BEHAVIOR_RESPONSE = json.dumps(
+ {
+ "results": [
+ {"step": 1, "passed": True, "reason": "ran the script"},
+ {"step": 2, "passed": False, "reason": "no report"},
+ ],
+ "summary": "one of two",
+ }
+)
+
+
[email protected](params=["shared", "template"])
+def judge_module(request: pytest.FixtureRequest) -> ModuleType:
+ return llm_judge if request.param == "shared" else eval_template
+
+
+def _script(responses: list[tuple[str | None, str | None]], calls: list[dict[str, Any]]):
+ def call(prompt: str, **kwargs: Any):
+ calls.append({"prompt": prompt, "kwargs": kwargs})
+ return responses[min(len(calls) - 1, len(responses) - 1)]
+
+ return call
+
+
+def _use_goal_script(module: ModuleType, monkeypatch: pytest.MonkeyPatch, responses, calls) -> None:
+ if module is llm_judge:
+ monkeypatch.setattr(module, "call_public_llm", _script(responses, calls))
+ return
+ scripted = _script(responses, calls)
+
+ def with_provenance(prompt: str, **kwargs: Any):
+ content, error = scripted(prompt, **kwargs)
+ return content, error, {"provider": "openai", "model": "judge-model"}
+
+ monkeypatch.setattr(module, "_ragas_goal_accuracy_enabled", lambda: False)
+ monkeypatch.setattr(module, "_call_public_llm_with_provenance", with_provenance)
+
+
+def _prompt_example(prompt: str) -> dict[str, Any]:
+ marker = "(the values are only an illustration):\n"
+ example, _end = json.JSONDecoder().raw_decode(prompt[prompt.index(marker) + len(marker) :])
+ return example
+
+
+def _capture_prompts(module: ModuleType, monkeypatch: pytest.MonkeyPatch) -> list[str]:
+ prompts: list[str] = []
+
+ def capture(prompt: str, **_kwargs: Any):
+ prompts.append(prompt)
+ return None, "stop after capturing the prompt"
+
+ monkeypatch.setattr(module, "call_public_llm", capture)
+ if module is eval_template:
+ monkeypatch.setattr(module, "_ragas_goal_accuracy_enabled", lambda: False)
+ monkeypatch.setattr(
+ module, "_call_public_llm_with_provenance", lambda prompt, **kwargs: (*capture(prompt, **kwargs), {})
+ )
+ return prompts
+
+
+# ── Prompt contracts ─────────────────────────────────────────────────────────
+
+
+def test_every_judge_prompt_example_passes_its_own_validator(judge_module, monkeypatch) -> None:
+ prompts = _capture_prompts(judge_module, monkeypatch)
+
+ judge_module.judge_accuracy("q", "gt", "evidence")
+ judge_module.judge_goal_accuracy("q", "gt", "evidence")
+ judge_module.judge_behavior_check("conversation", ["first", "second"])
+
+ accuracy, goal, behavior = (_prompt_example(prompt) for prompt in prompts)
+ assert judge_module._accuracy_payload_error(accuracy) is None
+ assert judge_module._goal_payload_error(goal) is None
+ assert judge_module._behavior_payload_error(behavior, 2) is None
+ for example in (accuracy, goal, behavior):
+ assert "score" not in example
+
+
[email protected]("behavior_count", [1, 2, 3, 5, 12])
+def test_behavior_prompt_example_is_a_valid_reply_for_any_number_of_behaviors(
+ judge_module, monkeypatch, behavior_count: int
+) -> None:
+ prompts = _capture_prompts(judge_module, monkeypatch)
+
+ judge_module.judge_behavior_check("conversation", [f"behavior {n}" for n in range(1, behavior_count + 1)])
+
+ example = _prompt_example(prompts[0])
+ assert judge_module._behavior_payload_error(example, behavior_count) is None
+ assert [entry["step"] for entry in example["results"]] == list(range(1, behavior_count + 1))
+
+
+def test_accuracy_prompt_gives_no_free_criterion_to_a_response_without_an_answer(judge_module, monkeypatch) -> None:
+ # A trial that ran but gave no answer scored 0.0 before FORMAT_FOLLOWED
+ # replaced SKILL_IDENTIFIED. Its "no format required" default must not turn
+ # into a free point for an empty response.
+ prompts = _capture_prompts(judge_module, monkeypatch)
+
+ judge_module.judge_accuracy("Which version of the tool is installed?", "1.2.3", "")
+
+ prompt = prompts[0]
+ format_line = next(line for line in prompt.splitlines() if line.startswith("5. FORMAT_FOLLOWED:"))
+ assert "Answer true when the question requires no specific format." not in format_line
+ assert "answer true if the response gives an answer and false if it gives none" in format_line
+ assert "If the evidence holds no answer to the question" in prompt
+ assert "every criterion is false" in prompt
+ # A correct refusal still scores when the expected answer is a refusal.
+ assert "unless the expected answer is itself a refusal" in prompt
+
+
+def test_goal_prompt_omits_empty_tool_section_and_keeps_real_one(judge_module, monkeypatch) -> None:
+ calls: list[dict[str, Any]] = []
+ _use_goal_script(judge_module, monkeypatch, [(_GOAL_RESPONSE, None)], calls)
+
+ judge_module.judge_goal_accuracy("question", "ground truth", "agent evidence", tool_summary="")
+ judge_module.judge_goal_accuracy("question", "ground truth", "agent evidence", tool_summary="Bash: make test")
+
+ assert "AGENT'S TOOL CALLS" not in calls[0]["prompt"]
+ assert "AGENT'S TOOL CALLS:\nBash: make test" in calls[1]["prompt"]
+
+
+def test_shared_and_template_judges_use_the_same_schemas() -> None:
+ assert eval_template.ACCURACY_JSON_SCHEMA == llm_judge.ACCURACY_JSON_SCHEMA
+ assert eval_template.GOAL_ACCURACY_JSON_SCHEMA == llm_judge.GOAL_ACCURACY_JSON_SCHEMA
+ assert eval_template.BEHAVIOR_CHECK_JSON_SCHEMA == llm_judge.BEHAVIOR_CHECK_JSON_SCHEMA
+ for schema in (
+ llm_judge.ACCURACY_JSON_SCHEMA,
+ llm_judge.GOAL_ACCURACY_JSON_SCHEMA,
+ llm_judge.BEHAVIOR_CHECK_JSON_SCHEMA,
+ ):
+ assert "score" not in schema["properties"]
+ assert "score" not in schema["required"]
+
+
+# ── Scores come from verdicts ────────────────────────────────────────────────
+
+
+def test_goal_score_follows_achieved_not_a_contradicting_score(judge_module, monkeypatch) -> None:
+ calls: list[dict[str, Any]] = []
+ reply = json.dumps({"achieved": False, "score": 1.0, "reason": "the expected file is missing"})
+ _use_goal_script(judge_module, monkeypatch, [(reply, None)], calls)
+
+ result = judge_module.judge_goal_accuracy("question", "ground truth", "agent evidence")
+
+ assert result["score"] == 0.0
+ assert result["achieved"] is False
+ assert len(calls) == 1
+
+
+ ("results", "problem"),
+ [
+ pytest.param(
+ [{"step": 1, "passed": True}, {"step": 1, "passed": True}],
+ "steps judged more than once: [1]; steps not judged: [2]",
+ id="duplicate-step",
+ ),
+ pytest.param(
+ [{"step": 1, "passed": True}, {"step": 3, "passed": True}],
+ "steps not judged: [2]; steps outside 1..2: [3]",
+ id="out-of-range-step",
+ ),
+ pytest.param(
+ [{"passed": True}, {"passed": True}],
+ 'every "results" entry needs an integer "step" number',
+ id="missing-step",
+ ),
+ pytest.param(
+ [{"step": True, "passed": True}, {"step": 2, "passed": True}],
+ 'every "results" entry needs an integer "step" number',
+ id="boolean-step",
+ ),
+ pytest.param(
+ [{"step": 1, "passed": True}],
+ '"results" has 1 entries; it needs exactly 2, one per behavior',
+ id="too-few",
+ ),
+ ],
+)
+def test_behavior_reply_must_judge_each_step_exactly_once(
+ judge_module, monkeypatch, results: list[dict[str, Any]], problem: str
+) -> None:
+ calls: list[dict[str, Any]] = []
+ invalid = json.dumps({"results": results, "summary": "all observed"})
+ monkeypatch.setattr(judge_module, "call_public_llm", _script([(invalid, None), (invalid, None)], calls))
+
+ result = judge_module.judge_behavior_check("conversation", ["b1", "b2"])
+
+ assert result["score"] is None
+ assert result["status"] == "error"
+ assert len(calls) == 2
+ assert problem in calls[1]["prompt"]
+ assert problem in result["reason"]
+
+
+def test_behavior_salvage_rejects_repeated_steps(judge_module, monkeypatch) -> None:
+ truncated = '{"results":[{"step":1,"passed":true},{"step":1,"passed":true},{"step":2,"passed":tr'
+ calls: list[dict[str, Any]] = []
+ monkeypatch.setattr(judge_module, "call_public_llm", _script([(truncated, None), (truncated, None)], calls))
+
+ result = judge_module.judge_behavior_check("conversation", ["b1", "b2", "b3"])
+
+ assert result["score"] is None
+ assert result["status"] == "error"
+
+
+# ── Specific retries ─────────────────────────────────────────────────────────
+
+
+ ("metric", "invalid", "problem"),
+ [
+ pytest.param("accuracy", "not-json", "the reply did not contain one JSON object", id="accuracy-malformed"),
+ pytest.param("accuracy", '{"score": 0.7}', 'the reply has no "criteria" object', id="accuracy-score-only"),
+ pytest.param(
+ "accuracy",
+ json.dumps({"criteria": {**_CRITERIA, "SKILL_IDENTIFIED": True}}),
+ "criteria has unexpected keys SKILL_IDENTIFIED",
+ id="accuracy-old-criterion",
+ ),
+ pytest.param(
+ "accuracy",
+ json.dumps({"criteria": {key: value for key, value in _CRITERIA.items() if key != "ACTIONABLE"}}),
+ "criteria is missing ACTIONABLE",
+ id="accuracy-missing-criterion",
+ ),
+ pytest.param("goal", '{"achieved": "true"}', '"achieved" must be true or false', id="goal-string-verdict"),
+ pytest.param("goal", '{"score": 1.0}', '"achieved" must be true or false', id="goal-score-only"),
+ pytest.param("goal", "", "the reply was empty", id="goal-empty"),
+ ],
+)
+def test_invalid_reply_is_retried_once_with_the_specific_problem(
+ judge_module, monkeypatch, metric: str, invalid: str, problem: str
+) -> None:
+ calls: list[dict[str, Any]] = []
+ valid = _ACCURACY_RESPONSE if metric == "accuracy" else _GOAL_RESPONSE
+ if metric == "accuracy":
+ monkeypatch.setattr(judge_module, "call_public_llm", _script([(invalid, None), (valid, None)], calls))
+ result = judge_module.judge_accuracy("question", "ground truth", "agent evidence")
+ else:
+ _use_goal_script(judge_module, monkeypatch, [(invalid, None), (valid, None)], calls)
+ result = judge_module.judge_goal_accuracy("question", "ground truth", "agent evidence")
+
+ assert result["score"] == (0.6 if metric == "accuracy" else 1.0)
+ assert len(calls) == 2
+ assert "previous reply could not be parsed or validated" not in calls[0]["prompt"]
+ assert f"previous reply could not be parsed or validated: {problem}" in calls[1]["prompt"]
+
+
[email protected]("applicable", [True, False])
+def test_deprecated_skill_identification_keyword_is_accepted_and_ignored(
+ judge_module, monkeypatch, applicable: bool
+) -> None:
+ calls: list[dict[str, Any]] = []
+ monkeypatch.setattr(judge_module, "call_public_llm", _script([(_ACCURACY_RESPONSE, None)], calls))
+
+ plain = judge_module.judge_accuracy("question", "ground truth", "agent evidence")
+ legacy = judge_module.judge_accuracy(
+ "question", "ground truth", "agent evidence", skill_identification_applicable=applicable
+ )
+
+ assert legacy == plain
+ assert legacy["score"] == 0.6
+ assert set(legacy["criteria"]) == set(_CRITERIA)
+ assert "excluded_criteria" not in legacy
+ assert calls[1]["prompt"] == calls[0]["prompt"]
+ assert "skill_identification_applicable" not in calls[1]["kwargs"]
+
+
+def test_required_judge_accepts_the_deprecated_accuracy_keyword(monkeypatch) -> None:
+ monkeypatch.setattr(eval_template, "call_public_llm", _script([(_ACCURACY_RESPONSE, None)], []))
+
+ result = eval_template._call_required_judge(
+ "accuracy",
+ eval_template.judge_accuracy,
+ "question",
+ "ground truth",
+ "agent evidence",
+ skill_identification_applicable=False,
+ )
+
+ assert result["score"] == 0.6
+ assert result.get("status") != "error"
+
+
+# ── Saved judge evidence and the reward contract ─────────────────────────────
+
+
+def _stage_verifier(tmp_path: Path, entry: dict[str, Any] | None = None) -> ModuleType:
+ module = load_harbor_eval_template(f"harbor_eval_judge_contracts_{tmp_path.name}")
+ agent_dir = tmp_path / "logs" / "agent"
+ verifier_dir = tmp_path / "logs" / "verifier"
+ tests_dir = tmp_path / "tests"
+ for directory in (agent_dir, verifier_dir, tests_dir):
+ directory.mkdir(parents=True)
+ module.VERIFIER_DIR = verifier_dir
+ module.ATIF_PATH = agent_dir / "trajectory.json"
+ module.ENTRY_PATH = tests_dir / "entry.json"
+ module.REWARD_JSON = verifier_dir / "reward.json"
+ module.REWARD_TXT = verifier_dir / "reward.txt"
+ module.SKILL_EVALUATOR_REWARD_JSON = verifier_dir / "skill_evaluator_reward.json"
+ module.ATIF_PATH.write_text(
+ json.dumps({"steps": [{"source": "agent", "message": "I completed the task."}]}), encoding="utf-8"
+ )
+ module.ENTRY_PATH.write_text(
+ json.dumps(
+ entry
+ or {
+ "id": "case-1",
+ "has_skill": False,
+ "question": "Complete the task",
+ "ground_truth": "Done",
+ "expected_behavior": ["ran the script", "reported the result"],
+ }
+ ),
+ encoding="utf-8",
+ )
+ module._ragas_goal_accuracy_enabled = lambda: False
+ return module
+
+
+def _routed(calls: list[str], **replies: str):
+ """Answer each judge by its prompt, so a full verifier run can use one fake."""
+
+ def call(prompt: str, **_kwargs: Any):
+ calls.append(prompt)
+ if "EXPECTED BEHAVIORS" in prompt:
+ reply = replies["behavior"]
+ elif "FORMAT_FOLLOWED" in prompt:
+ reply = replies["accuracy"]
+ else:
+ reply = replies["goal"]
+ return reply, None
+
+ return call
+
+
+def _attempts(verifier_dir: Path) -> list[dict[str, Any]]:
+ payload = json.loads((verifier_dir / "judge_attempts.json").read_text(encoding="utf-8"))
+ assert payload["judge_contract"] == eval_template.JUDGE_CONTRACT
+ return payload["attempts"]
+
+
+def test_scored_trial_records_judge_contract_and_every_attempt(tmp_path: Path) -> None:
+ verifier = _stage_verifier(tmp_path)
+ calls: list[str] = []
+ route = _routed(calls, accuracy=_ACCURACY_RESPONSE, goal=_GOAL_RESPONSE, behavior=_BEHAVIOR_RESPONSE)
+ verifier.call_public_llm = route
+ verifier._call_public_llm_with_provenance = lambda prompt, **kwargs: (
+ *route(prompt, **kwargs),
+ {"provider": "openai", "model": "goal-model"},
+ )
+
+ verifier.main()
+
+ rich = json.loads(verifier.SKILL_EVALUATOR_REWARD_JSON.read_text(encoding="utf-8"))
+ numeric = json.loads(verifier.REWARD_JSON.read_text(encoding="utf-8"))
+ assert rich["judge_contract"] == "skill-evaluator-judges-v2"
+ assert "judge_contract" not in numeric
+ assert rich["accuracy"] == numeric["accuracy"] == 0.6
+ assert rich["goal_accuracy"] == numeric["goal_accuracy"] == 1.0
+ assert rich["behavior_check"] == numeric["behavior_check"] == 0.5
+ attempts = _attempts(verifier.VERIFIER_DIR)
+ assert [(attempt["metric"], attempt["status"]) for attempt in attempts] == [
+ ("accuracy", "accepted"),
+ ("goal_accuracy", "accepted"),
+ ("behavior_check", "accepted"),
+ ]
+ assert attempts[1]["model"] == "goal-model"
+ assert attempts[0]["reply"] == _ACCURACY_RESPONSE
+ assert len(attempts[0]["prompt_sha256"]) == 64
+ assert "Complete the task" in attempts[0]["prompt"]
+
+
+def test_failed_judge_keeps_its_rejected_attempts_without_a_score(tmp_path: Path) -> None:
+ verifier = _stage_verifier(tmp_path)
+ calls: list[str] = []
+ route = _routed(calls, accuracy='{"score": 1.0}', goal=_GOAL_RESPONSE, behavior=_BEHAVIOR_RESPONSE)
+ verifier.call_public_llm = route
+ verifier._call_public_llm_with_provenance = lambda prompt, **kwargs: (*route(prompt, **kwargs), {})
+
+ with pytest.raises(SystemExit):
+ verifier.main()
+
+ rich = json.loads(verifier.SKILL_EVALUATOR_REWARD_JSON.read_text(encoding="utf-8"))
+ assert rich["accuracy"] is None
+ assert 'the reply has no "criteria" object' in rich["evaluation_errors"]["accuracy"]
+ rejected = [attempt for attempt in _attempts(verifier.VERIFIER_DIR) if attempt["metric"] == "accuracy"]
+ assert [attempt["status"] for attempt in rejected] == ["rejected", "rejected"]
+ assert rejected[1]["problem"] == 'the reply has no "criteria" object'
+ assert "previous reply could not be parsed or validated" not in rejected[0]["prompt"]
+ assert "previous reply could not be parsed or validated" in rejected[1]["prompt"]
+
+
+def test_provider_error_is_saved_as_an_error_attempt(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ verifier = _stage_verifier(tmp_path)
+ credential = "dummy-evidence-secret-DO-NOT-RETAIN"
+ monkeypatch.setenv("OPENAI_API_KEY", credential)
+ calls: list[str] = []
+ route = _routed(calls, accuracy=_ACCURACY_RESPONSE, goal=_GOAL_RESPONSE, behavior=_BEHAVIOR_RESPONSE)
+ verifier.call_public_llm = lambda prompt, **kwargs: (
+ (None, f"HTTP 503 echoed {credential}") if "FORMAT_FOLLOWED" in prompt else route(prompt, **kwargs)
+ )
+ verifier._call_public_llm_with_provenance = lambda prompt, **kwargs: (*route(prompt, **kwargs), {})
+
+ with pytest.raises(SystemExit):
+ verifier.main()
+
+ (attempt,) = [attempt for attempt in _attempts(verifier.VERIFIER_DIR) if attempt["metric"] == "accuracy"]
+ assert attempt["status"] == "error"
+ assert "HTTP 503" in attempt["problem"]
+ assert "reply" not in attempt
+ assert credential not in (verifier.VERIFIER_DIR / "judge_attempts.json").read_text(encoding="utf-8")
+
+
+def test_saved_judge_evidence_masks_credentials(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ configured = "dummy-configured-judge-credential"
+ monkeypatch.setenv("ANTHROPIC_API_KEY", configured)
+ leaked_key = "sk-" + "a" * 24
+ userinfo = "dummy-userinfo-credential"
+ verifier = _stage_verifier(
+ tmp_path,
+ {
+ "id": "case-1",
+ "question": f"Use {configured} and {leaked_key} to clone https://ci:{userinfo}@git.example.com/r.git",
+ "ground_truth": "Done",
+ },
+ )
+ calls: list[str] = []
+ accuracy_reply = json.dumps({"criteria": _CRITERIA, "reason": f"saw {configured}"})
+ route = _routed(calls, accuracy=accuracy_reply, goal=_GOAL_RESPONSE, behavior=_BEHAVIOR_RESPONSE)
+ verifier.call_public_llm = route
+ verifier._call_public_llm_with_provenance = lambda prompt, **kwargs: (*route(prompt, **kwargs), {})
+
+ verifier.main()
+
+ saved = (verifier.VERIFIER_DIR / "judge_attempts.json").read_text(encoding="utf-8")
+ assert configured not in saved
+ assert leaked_key not in saved
+ assert userinfo not in saved
+ assert "Done" in saved
+
+
+# The curl examples use repeated-letter credentials, which stay below the
+# entropy thresholds of Gitleaks' curl rules.
+ ("text", "secret", "kept"),
+ [
+ pytest.param(
+ "git clone https://deploy:[email protected]/repo.git",
+ "url-userinfo-secret",
+ "https://<redacted>@git.example.com/repo.git",
+ id="url-userinfo",
+ ),
+ pytest.param(
+ "curl -H 'Authorization: Bearer tttttttttttt' https://api.example.com",
+ "tttttttttttt",
+ "Authorization: Bearer <redacted>'",
+ id="authorization-header",
+ ),
+ pytest.param(
+ '{"headers": {"Authorization": "Basic b3BhcXVlLWJhc2ljLXNlY3JldA"}}',
+ "b3BhcXVlLWJhc2ljLXNlY3JldA",
+ '"Authorization": "Basic <redacted>"',
+ id="authorization-json",
+ ),
+ pytest.param(
+ "deploy --token flag-token-secret --dry-run", "flag-token-secret", "--token <redacted>", id="token"
+ ),
+ pytest.param("login --password=flag-password-secret", "flag-password-secret", "--password=<redacted>", id="pw"),
+ pytest.param("curl -u uuuu:pppppppppppp https://x", "pppppppppppp", "-u <redacted>", id="user-pass"),
+ pytest.param(
+ "curl -H 'X-API-Key: kkkkkkkkkkkk' https://x",
+ "kkkkkkkkkkkk",
+ "X-API-Key: <redacted>'",
+ id="api-key-header",
+ ),
+ pytest.param(
+ '{"headers": {"x-api-key": "api-key-json-secret"}}',
+ "api-key-json-secret",
+ '"x-api-key": "<redacted>"',
+ id="api-key-json",
+ ),
+ pytest.param(
+ "curl -H 'PRIVATE-TOKEN: private-token-header-secret' https://x",
+ "private-token-header-secret",
+ "PRIVATE-TOKEN: <redacted>'",
+ id="private-token-header",
+ ),
+ pytest.param(
+ "curl -H 'Cookie: theme=dark; session=cookie-session-secret' https://x",
+ "cookie-session-secret",
+ "Cookie: <redacted>'",
+ id="cookie-every-pair",
+ ),
+ pytest.param(
+ 'curl -H "Authorization: ApiKey aaaaaaaaaaaaaaaaaaaaaaaa" https://es.example',
+ "aaaaaaaaaaaaaaaaaaaaaaaa",
+ '"Authorization: <redacted>" https://es.example',
+ id="authorization-apikey-scheme",
+ ),
+ pytest.param(
+ "curl -H 'Authorization: Bot bbbbbbbbbbbbbbbbbbbbbbbb' https://x",
+ "bbbbbbbbbbbbbbbbbbbbbbbb",
+ "'Authorization: <redacted>' https://x",
+ id="authorization-bot-scheme",
+ ),
+ pytest.param(
+ "curl -H 'Authorization: SSWS ssssssssssssssssssssssss'",
+ "ssssssssssssssssssssssss",
+ "'Authorization: <redacted>'",
+ id="authorization-ssws-scheme",
+ ),
+ pytest.param(
+ 'headers["Authorization"] = "Bearer subscript-bearer-secret"',
+ "subscript-bearer-secret",
+ 'headers["Authorization"] = "Bearer <redacted>"',
+ id="authorization-subscript",
+ ),
+ pytest.param(
+ "headers['Authorization'] = f'Token fstring-token-secret'",
+ "fstring-token-secret",
+ "headers['Authorization'] = f'Token <redacted>'",
+ id="authorization-subscript-fstring",
+ ),
+ pytest.param(
+ 'headers["X-API-Key"] = "subscript-api-key-secret"',
+ "subscript-api-key-secret",
+ 'headers["X-API-Key"] = "<redacted>"',
+ id="api-key-subscript",
+ ),
+ pytest.param(
+ "headers['Cookie'] = 'session=subscript-cookie-secret'",
+ "subscript-cookie-secret",
+ "headers['Cookie'] = '<redacted>'",
+ id="cookie-subscript",
+ ),
+ pytest.param(
+ "curl -H 'Authorization: Bearer \"quoted-bearer-secret\"' https://x",
+ "quoted-bearer-secret",
+ 'Authorization: Bearer "<redacted>"',
+ id="authorization-quoted-token",
+ ),
+ pytest.param(
+ """curl -H 'Authorization: Digest username="bob", response="digest-response-secret"' https://x""",
+ "digest-response-secret",
+ "Authorization: Digest <redacted>' https://x",
+ id="authorization-digest",
+ ),
+ pytest.param(
+ 'curl -H "Authorization: Bearer ' + "L0ngOpaqueT0ken" * 60 + '" https://x',
+ "L0ngOpaqueT0ken" * 3,
+ 'Authorization: Bearer <redacted>" https://x',
+ id="authorization-value-over-256-chars",
+ ),
+ pytest.param(
+ 'curl -u "admin:quoted-user-password" https://x',
+ "quoted-user-password",
+ "curl -u <redacted> https://x",
+ id="user-flag-double-quoted",
+ ),
+ pytest.param(
+ "curl --user='admin:quoted-user-password' https://x",
+ "quoted-user-password",
+ "--user=<redacted> https://x",
+ id="user-flag-single-quoted",
+ ),
+ pytest.param(
+ "mysql --password='quoted-db-password' db",
+ "quoted-db-password",
+ "mysql --password=<redacted> db",
+ id="password-flag-single-quoted",
+ ),
+ pytest.param(
+ "tool --token 'quoted-token-value'",
+ "quoted-token-value",
+ "tool --token <redacted>",
+ id="token-flag-quoted",
+ ),
+ pytest.param(
+ 'mysql --password="' + "p" * 300 + '" db',
+ "p" * 300,
+ "mysql --password=<redacted> db",
+ id="password-flag-double-quoted-over-256-chars",
+ ),
+ pytest.param(
+ "tool --token '" + "t" * 300 + "'",
+ "t" * 300,
+ "tool --token <redacted>",
+ id="token-flag-single-quoted-over-256-chars",
+ ),
+ pytest.param(
+ 'curl -u "admin:' + "x" * 300 + '" https://x',
+ "x" * 300,
+ "curl -u <redacted> https://x",
+ id="user-flag-double-quoted-over-256-chars",
+ ),
+ pytest.param(
+ 'tool --api-key="quoted-api-key-value" run',
+ "quoted-api-key-value",
+ "tool --api-key=<redacted> run",
+ id="api-key-flag-double-quoted",
+ ),
+ ],
+)
+def test_judge_evidence_masks_credentials_without_a_key_shape(text: str, secret: str, kept: str) -> None:
+ redacted = eval_template._redact_judge_evidence(text)
+
+ assert secret not in redacted
+ assert kept in redacted
+
+
+# Behavior and goal prompts render tool-call arguments with json.dumps, so the
+# quotes around header names and values reach the judge prompt escaped.
+ ("command", "secret"),
+ [
+ pytest.param(
+ """python -c 'requests.get(u, headers={"Authorization": "ApiKey escaped-dict-apikey-secret"})'""",
+ "escaped-dict-apikey-secret",
+ id="authorization-dict",
+ ),
+ pytest.param(
+ """python -c 'h={}; h["Authorization"] = "Bearer escaped-subscript-bearer-secret"'""",
+ "escaped-subscript-bearer-secret",
+ id="authorization-subscript",
+ ),
+ pytest.param(
+ """python -c 'h={}; h["X-API-Key"]="escaped-subscript-api-key-secret"'""",
+ "escaped-subscript-api-key-secret",
+ id="api-key-subscript",
+ ),
+ pytest.param(
+ """python -c 'h={"Cookie": "session=escaped-cookie-secret"}'""",
+ "escaped-cookie-secret",
+ id="cookie-dict",
+ ),
+ pytest.param(
+ """curl -H "Authorization: Digest username=\\"bob\\", response=\\"escaped-digest-secret\\"" https://x""",
+ "escaped-digest-secret",
+ id="authorization-digest",
+ ),
+ ],
+)
+def test_judge_evidence_masks_json_escaped_tool_call_headers(command: str, secret: str) -> None:
+ trajectory = {
+ "steps": [
+ {"source": "agent", "tool_calls": [{"function_name": "Bash", "arguments": {"command": command}}]},
+ ]
+ }
+ summary = eval_template.build_conversation_summary(trajectory, "Fetch the index.")
+ assert secret in summary
+ assert '\\"' in summary
+
+ redacted = eval_template._redact_judge_evidence(summary)
+
+ assert secret not in redacted
+ assert "<redacted>" in redacted
+
+
+ "text",
+ [
+ "python -u script.py && pip install -U pkg && pip install --user pkg && git push -u origin main",
+ "judge --max-tokens 100 --token-limit 5",
+ "see https://example.com/docs?a=b@c and mailto:[email protected]",
+ "Authorization is required for this endpoint.",
+ "curl --cookie-jar cookies.txt https://x; see the api-key rotation docs",
+ ],
+)
+def test_judge_evidence_keeps_ordinary_text(text: str) -> None:
+ assert eval_template._redact_judge_evidence(text) == text
+
+
+def test_salvaged_behavior_score_is_saved_as_a_salvage_attempt(tmp_path: Path) -> None:
+ verifier = _stage_verifier(tmp_path)
+ truncated = '{"results":[{"step":1,"passed":true,"reason":"ran the script"},{"step":2,"passed":tr'
+ calls: list[str] = []
+ route = _routed(calls, accuracy=_ACCURACY_RESPONSE, goal=_GOAL_RESPONSE, behavior=truncated)
+ verifier.call_public_llm = route
+ verifier._call_public_llm_with_provenance = lambda prompt, **kwargs: (*route(prompt, **kwargs), {})
+
+ verifier.main()
+
+ rich = json.loads(verifier.SKILL_EVALUATOR_REWARD_JSON.read_text(encoding="utf-8"))
+ assert rich["behavior_check"] == 0.5
+ assert "Salvaged 1/2" in rich["details"]["behavior_check"]["reason"]
+ attempts = [attempt for attempt in _attempts(verifier.VERIFIER_DIR) if attempt["metric"] == "behavior_check"]
+ assert [(attempt["attempt"], attempt["method"], attempt["status"]) for attempt in attempts] == [
+ (1, "llm", "rejected"),
+ (2, "llm", "rejected"),
+ (2, "salvage", "accepted"),
+ ]
+ salvaged = attempts[2]
+ assert "prompt" not in salvaged
+ assert salvaged["problem"] is None
+ assert salvaged["model"] == attempts[1]["model"]
+ assert json.loads(salvaged["reply"])["results"] == [{"step": 1, "passed": True, "reason": "ran the script"}]
+
+
+def test_salvage_after_a_failed_retry_keeps_each_attempt_status(tmp_path: Path) -> None:
+ # The first reply is cut off and the retry fails in transport: the LLM
+ # attempts keep their own status, and the salvage entry is numbered like
+ # the truncated attempt it was rebuilt from.
+ verifier = _stage_verifier(tmp_path)
+ truncated = '{"results":[{"step":1,"passed":true,"reason":"ran the script"},{"step":2,"passed":tr'
+ calls: list[str] = []
+ route = _routed(calls, accuracy=_ACCURACY_RESPONSE, goal=_GOAL_RESPONSE, behavior=truncated)
+ behavior_calls: list[str] = []
+
+ def call(prompt: str, **kwargs: Any):
+ if "EXPECTED BEHAVIORS" in prompt:
+ behavior_calls.append(prompt)
+ return (truncated, None) if len(behavior_calls) == 1 else (None, "request timed out")
+ return route(prompt, **kwargs)
+
+ verifier.call_public_llm = call
+ verifier._call_public_llm_with_provenance = lambda prompt, **kwargs: (*call(prompt, **kwargs), {})
+
+ verifier.main()
+
+ rich = json.loads(verifier.SKILL_EVALUATOR_REWARD_JSON.read_text(encoding="utf-8"))
+ assert rich["behavior_check"] == 0.5
+ attempts = [attempt for attempt in _attempts(verifier.VERIFIER_DIR) if attempt["metric"] == "behavior_check"]
+ assert [(attempt["attempt"], attempt["method"], attempt["status"]) for attempt in attempts] == [
+ (1, "llm", "rejected"),
+ (2, "llm", "error"),
+ (1, "salvage", "accepted"),
+ ]
+ assert attempts[1]["problem"] == "request timed out"
+
+
+def _cut_point(limit: int) -> int:
+ # Where _truncate_for_behavior ends the kept head of a text longer than *limit*.
+ marker = "\n...[truncated]...\n"
+ return (limit - len(marker)) * 2 // 3
+
+
+def test_long_judge_evidence_is_bounded_and_redacted_before_it_is_cut(tmp_path: Path, monkeypatch) -> None:
+ verifier = _stage_verifier(tmp_path)
+ credential = "configured-cut-credential-DO-NOT-SAVE"
+ monkeypatch.setenv("ANTHROPIC_API_KEY", credential)
+ # Each credential crosses the point where the raw text would be cut, so
+ # cutting before redaction would save its first characters in clear.
+ prompt_cut = _cut_point(eval_template.JUDGE_EVIDENCE_PROMPT_CHARS)
+ reply_cut = _cut_point(eval_template.JUDGE_EVIDENCE_REPLY_CHARS)
+ prompt = "p" * (prompt_cut - 12) + credential + "p" * 50_000
+ reply = "r" * (reply_cut - 12) + credential + "r" * 50_000
+
+ token = verifier._ACTIVE_JUDGE_EVIDENCE.set([])
+ try:
+ verifier._record_judge_attempt("accuracy", 1, prompt=prompt, reply=reply, problem="too long")
+ finally:
+ verifier._ACTIVE_JUDGE_EVIDENCE.reset(token)
+
+ saved_text = (verifier.VERIFIER_DIR / "judge_attempts.json").read_text(encoding="utf-8")
+ (entry,) = _attempts(verifier.VERIFIER_DIR)
+ assert entry["prompt_chars"] == len(prompt)
+ assert entry["reply_chars"] == len(reply)
+ assert len(entry["prompt"]) <= eval_template.JUDGE_EVIDENCE_PROMPT_CHARS
+ assert len(entry["reply"]) <= eval_template.JUDGE_EVIDENCE_REPLY_CHARS
+ assert "...[truncated]..." in entry["prompt"]
+ assert "...[truncated]..." in entry["reply"]
+ assert "[REDACTED]" in entry["prompt"]
+ assert "[REDACTED]" in entry["reply"]
+ assert credential[:8] not in saved_text
+ assert credential[-8:] not in saved_text
+
+
+def test_judge_evidence_write_failure_never_changes_the_score(tmp_path: Path, monkeypatch) -> None:
+ verifier = _stage_verifier(tmp_path)
+ calls: list[str] = []
+ route = _routed(calls, accuracy=_ACCURACY_RESPONSE, goal=_GOAL_RESPONSE, behavior=_BEHAVIOR_RESPONSE)
+ verifier.call_public_llm = route
+ verifier._call_public_llm_with_provenance = lambda prompt, **kwargs: (*route(prompt, **kwargs), {})
+ verifier.JUDGE_ATTEMPTS_FILE_NAME = "missing-directory/judge_attempts.json"
+ (verifier.VERIFIER_DIR / "missing-directory").write_text("a file, not a directory", encoding="utf-8")
+
+ verifier.main()
+
+ numeric = json.loads(verifier.REWARD_JSON.read_text(encoding="utf-8"))
+ assert numeric["accuracy"] == 0.6
+ assert numeric["behavior_check"] == 0.5
+
+
+def test_judge_deadline_is_not_swallowed_by_the_evidence_writer(tmp_path: Path) -> None:
+ verifier = _stage_verifier(tmp_path)
+
+ class _DeadlineDuringWrite:
+ def __truediv__(self, _name: str):
+ raise TimeoutError("LLM judge time budget exhausted")
+
+ verifier._verifier_output_dir = _DeadlineDuringWrite
+ token = verifier._ACTIVE_JUDGE_EVIDENCE.set([])
+ try:
+ with pytest.raises(TimeoutError):
+ verifier._record_judge_attempt("accuracy", 1, prompt="p", reply="r", problem=None)
+ finally:
+ verifier._ACTIVE_JUDGE_EVIDENCE.reset(token)
+
+
+def test_judges_outside_the_verifier_run_save_no_evidence(tmp_path: Path, monkeypatch) -> None:
+ monkeypatch.setattr(eval_template, "VERIFIER_DIR", tmp_path)
+ monkeypatch.setattr(eval_template, "SKILL_EVALUATOR_REWARD_JSON", tmp_path / "skill_evaluator_reward.json")
+ monkeypatch.setattr(eval_template, "REWARD_JSON", tmp_path / "reward.json")
+ monkeypatch.setattr(eval_template, "call_public_llm", _script([(_ACCURACY_RESPONSE, None)], []))
+
+ assert eval_template.judge_accuracy("question", "ground truth", "agent evidence")["score"] == 0.6
+ assert not (tmp_path / "judge_attempts.json").exists()
+
+
[email protected]("ragas_score", [0.5, float("nan")])
+def test_non_binary_ragas_score_falls_back_to_the_prompt_judge(tmp_path: Path, ragas_score: float) -> None:
+ verifier = _stage_verifier(tmp_path)
+ verifier._ragas_goal_accuracy_enabled = lambda: True
+
+ def ragas(*_args: Any, **_kwargs: Any):
+ return {"score": ragas_score, "method": "ragas", "model": "ragas-model"}
+
+ calls: list[str] = []
+ route = _routed(calls, accuracy=_ACCURACY_RESPONSE, goal=_GOAL_RESPONSE, behavior=_BEHAVIOR_RESPONSE)
+ verifier._judge_goal_accuracy_ragas = ragas
+ verifier.call_public_llm = route
+ verifier._call_public_llm_with_provenance = lambda prompt, **kwargs: (*route(prompt, **kwargs), {})
+
+ verifier.main()
+
+ rich = json.loads(verifier.SKILL_EVALUATOR_REWARD_JSON.read_text(encoding="utf-8"))
+ assert rich["goal_accuracy"] == 1.0
+ assert rich["details"]["goal_accuracy"]["method"] == "custom"
+ goal_attempts = [attempt for attempt in _attempts(verifier.VERIFIER_DIR) if attempt["metric"] == "goal_accuracy"]
+ assert [(attempt["method"], attempt["status"]) for attempt in goal_attempts] == [
+ ("ragas", "error"),
+ ("llm", "accepted"),
+ ]
+ assert "non-binary" in goal_attempts[0]["problem"]
+
+
+def test_binary_ragas_score_is_saved_as_an_accepted_attempt(tmp_path: Path) -> None:
+ verifier = _stage_verifier(tmp_path)
+ verifier._ragas_goal_accuracy_enabled = lambda: True
+ verifier._judge_goal_accuracy_ragas = lambda *_args, **_kwargs: {
+ "score": 0.0,
+ "achieved": False,
+ "method": "ragas",
+ "model": "ragas-model",
+ }
+ calls: list[str] = []
+ route = _routed(calls, accuracy=_ACCURACY_RESPONSE, goal=_GOAL_RESPONSE, behavior=_BEHAVIOR_RESPONSE)
+ verifier.call_public_llm = route
+
+ verifier.main()
+
+ rich = json.loads(verifier.SKILL_EVALUATOR_REWARD_JSON.read_text(encoding="utf-8"))
+ assert rich["goal_accuracy"] == 0.0
+ (attempt,) = [attempt for attempt in _attempts(verifier.VERIFIER_DIR) if attempt["metric"] == "goal_accuracy"]
+ assert attempt == {
+ "metric": "goal_accuracy",
+ "attempt": 1,
+ "method": "ragas",
+ "model": "ragas-model",
+ "status": "accepted",
+ "problem": None,
+ "score": 0.0,
+ }
+
+
+def test_judge_contract_is_reserved_reward_metadata() -> None:
+ assert "judge_contract" in RESERVED_METRIC_NAMES
diff --git a/tests/tier3/test_judge_failure_artifacts.py b/tests/tier3/test_judge_failure_artifacts.py
index c4206a61..b2e7c620 100644
--- a/tests/tier3/test_judge_failure_artifacts.py
+++ b/tests/tier3/test_judge_failure_artifacts.py
@@ -118,9 +118,8 @@ def test_character_facts_keep_numeric_dataset_answers_scoreable(tmp_path, monkey
payload = json.dumps(
{
"criteria": dict.fromkeys(
- ["SKILL_IDENTIFIED", "ACTION_CORRECT", "FACTUALLY_ACCURATE", "TASK_ADDRESSED", "ACTIONABLE"], False
+ ["ACTION_CORRECT", "FACTUALLY_ACCURATE", "TASK_ADDRESSED", "ACTIONABLE", "FORMAT_FOLLOWED"], False
),
- "score": 0.0,
"reason": "test verdict",
"achieved": False,
"user_goal": "test",
@@ -460,13 +459,12 @@ def test_verifier_main_recovers_malformed_accuracy_and_goal_judges(
json.dumps(
{
"criteria": {
- "SKILL_IDENTIFIED": True,
"ACTION_CORRECT": True,
"FACTUALLY_ACCURATE": True,
"TASK_ADDRESSED": True,
"ACTIONABLE": True,
+ "FORMAT_FOLLOWED": True,
},
- "score": 1.0,
"reason": "accuracy recovered",
}
),
@@ -521,24 +519,19 @@ def test_verifier_retries_non_string_judge_text_before_collector_and_report(
goal_calls: list[str] = []
def pair_call(prompt: str, **_kwargs):
- if "SKILL_IDENTIFIED" not in prompt:
+ if "FORMAT_FOLLOWED" not in prompt:
return json.dumps({"results": [{"step": 1, "passed": True}], "score": 1.0}), None
accuracy_calls.append(prompt)
+ criteria = {
+ "ACTION_CORRECT": True,
+ "FACTUALLY_ACCURATE": True,
+ "TASK_ADDRESSED": True,
+ "ACTIONABLE": True,
+ "FORMAT_FOLLOWED": False,
+ }
if len(accuracy_calls) == 1:
- return json.dumps({"score": 1.0, "reason": {"nested": "accuracy"}}), None
- return json.dumps(
- {
- "score": 1.0,
- "reason": "accuracy recovered",
- "criteria": {
- "SKILL_IDENTIFIED": False,
- "ACTION_CORRECT": True,
- "FACTUALLY_ACCURATE": True,
- "TASK_ADDRESSED": True,
- "ACTIONABLE": True,
- },
- }
- ), None
+ return json.dumps({"criteria": criteria, "reason": {"nested": "accuracy"}}), None
+ return json.dumps({"reason": "accuracy recovered", "criteria": criteria}), None
def goal_call(prompt: str, **_kwargs):
goal_calls.append(prompt)
@@ -582,7 +575,8 @@ def test_verifier_retries_non_string_judge_text_before_collector_and_report(
assert len(accuracy_calls) == 2
assert len(goal_calls) == 2
assert collected["details"]["accuracy"]["reason"] == "accuracy recovered"
- assert collected["details"]["accuracy"]["excluded_criteria"] == ["SKILL_IDENTIFIED"]
+ assert collected["details"]["accuracy"]["score"] == 0.8
+ assert "excluded_criteria" not in collected["details"]["accuracy"]
assert collected["details"]["goal_accuracy"]["reason"] == "goal recovered"
assert all(isinstance(reason, str) for finding in findings for reason in finding["reasons"])
@@ -689,7 +683,7 @@ def test_verifier_main_keeps_accuracy_fail_closed_after_retry_exhaustion(
assert exc_info.value.code == 1
assert len(pair_calls) == 3
- accuracy_attempts = [call for call in pair_calls if "SKILL_IDENTIFIED" in call[0]]
+ accuracy_attempts = [call for call in pair_calls if "FORMAT_FOLLOWED" in call[0]]
assert len(accuracy_attempts) == 2
assert [kwargs["max_tokens"] for _, kwargs in accuracy_attempts] == [verifier.STRUCTURED_JUDGE_MAX_TOKENS] * 2
assert len(goal_calls) == 1
diff --git a/tests/tier3/test_judge_http_response_contract.py b/tests/tier3/test_judge_http_response_contract.py
index 706c88ce..0154e76c 100644
--- a/tests/tier3/test_judge_http_response_contract.py
+++ b/tests/tier3/test_judge_http_response_contract.py
@@ -21,7 +21,10 @@ def test_standalone_judge_requests_json_from_streaming_default_endpoint(monkeypa
verifier = importlib.util.module_from_spec(spec)
spec.loader.exec_module(verifier)
requests = []
- judge_reply = json.dumps({"score": 1.0, "reason": "The answer matches the expected result."})
+ criteria = ("ACTION_CORRECT", "FACTUALLY_ACCURATE", "TASK_ADDRESSED", "ACTIONABLE", "FORMAT_FOLLOWED")
+ judge_reply = json.dumps(
+ {"criteria": dict.fromkeys(criteria, True), "reason": "The answer matches the expected result."}
+ )
class StreamingDefaultEndpoint(BaseHTTPRequestHandler):
def do_POST(self):
diff --git a/tests/tier3/test_judge_parse_robustness.py b/tests/tier3/test_judge_parse_robustness.py
index db96a688..e30318f9 100644
--- a/tests/tier3/test_judge_parse_robustness.py
+++ b/tests/tier3/test_judge_parse_robustness.py
@@ -598,7 +598,12 @@ def test_template_bedrock_request_uses_model_compatible_temperature(monkeypatch,
[
pytest.param(
"judge_accuracy",
- {"score": 1.0, "reason": "recovered"},
+ {
+ "criteria": dict.fromkeys(
+ ("ACTION_CORRECT", "FACTUALLY_ACCURATE", "TASK_ADDRESSED", "ACTIONABLE", "FORMAT_FOLLOWED"), True
+ ),
+ "reason": "recovered",
+ },
id="accuracy",
),
pytest.param(
diff --git a/tests/tier3/test_judge_result_semantics.py b/tests/tier3/test_judge_result_semantics.py
index 1374fd55..715f28be 100644
--- a/tests/tier3/test_judge_result_semantics.py
+++ b/tests/tier3/test_judge_result_semantics.py
@@ -21,19 +21,20 @@ _TEMPLATE = (
Path(__file__).resolve().parents[2] / "src" / "skillevaluator" / "tier3" / "harbor" / "templates" / "eval.py"
)
_CRITERIA = {
- "SKILL_IDENTIFIED": True,
"ACTION_CORRECT": False,
"FACTUALLY_ACCURATE": True,
"TASK_ADDRESSED": True,
"ACTIONABLE": False,
+ "FORMAT_FOLLOWED": True,
}
-VALID_ACCURACY_RESPONSE = json.dumps({"criteria": _CRITERIA, "score": 0.6, "reason": "valid retry"})
+VALID_ACCURACY_RESPONSE = json.dumps({"criteria": _CRITERIA, "reason": "valid retry"})
+# Five valid criteria, so a reply built on it can fail only on what follows.
+_VALID_CRITERIA_PREFIX = '{"criteria": ' + json.dumps(_CRITERIA) + ", "
VALID_GOAL_RESPONSE = json.dumps(
{
"user_goal": "complete the task",
"end_state": "task completed",
"achieved": True,
- "score": 1.0,
"reason": "valid retry",
}
)
@@ -51,53 +52,63 @@ eval_template = _load_template_module()
@pytest.mark.parametrize("module", [llm_judge, eval_template])
[email protected]("skill_verdict", [True, False])
-def test_negative_accuracy_excludes_skill_identification(module, skill_verdict, monkeypatch):
- criteria = dict.fromkeys(_CRITERIA, True)
- criteria["SKILL_IDENTIFIED"] = skill_verdict
- criteria["ACTION_CORRECT"] = False
+def test_accuracy_grades_the_outcome_not_skill_use(module, monkeypatch):
+ # A correct no-skill or negative-case answer can reach 1.0: skill use is
+ # scored by skill_execution, not by an accuracy criterion.
calls = []
- monkeypatch.setattr(
- module,
- "call_public_llm",
- _pair_script([(json.dumps({"criteria": criteria, "score": 0.1, "reason": "case"}), None)], calls),
- )
- result = module.judge_accuracy("question", "answer", "evidence", skill_identification_applicable=False)
- assert result["score"] == 0.75
- assert result["excluded_criteria"] == ["SKILL_IDENTIFIED"]
- assert "other four criteria" in calls[0]
+ response = json.dumps({"criteria": dict.fromkeys(_CRITERIA, True), "reason": "correct"})
+ monkeypatch.setattr(module, "call_public_llm", _pair_script([(response, None)], calls))
+
+ result = module.judge_accuracy("question", "answer", "evidence")
+
+ assert result["score"] == 1.0
+ assert "SKILL_IDENTIFIED" not in calls[0]
+ assert "FORMAT_FOLLOWED" in calls[0]
+ assert "Do not reward or penalize the response for naming, or not naming" in calls[0]
+ assert "excluded_criteria" not in result
@pytest.mark.parametrize("module", [llm_judge, eval_template])
-def test_negative_accuracy_requires_criterion_evidence(module, monkeypatch):
+def test_accuracy_requires_criterion_verdicts(module, monkeypatch):
calls = []
monkeypatch.setattr(
module, "call_public_llm", _pair_script([('{"score":1,"reason":"missing verdicts"}', None)], calls)
)
- result = module.judge_accuracy("question", "answer", "evidence", skill_identification_applicable=False)
+ result = module.judge_accuracy("question", "answer", "evidence")
assert result["score"] is None
assert result["status"] == "error"
+ assert len(calls) == 2
+ assert 'the reply has no "criteria" object' in calls[1]
@pytest.mark.parametrize("module", [llm_judge, eval_template])
-def test_negative_rubric_does_not_rewrite_supplied_evidence(module, monkeypatch):
+def test_accuracy_prompt_does_not_rewrite_supplied_evidence(module, monkeypatch):
calls = []
criteria = dict.fromkeys(_CRITERIA, True)
- response = json.dumps({"criteria": criteria, "score": 1.0, "reason": "valid"})
+ response = json.dumps({"criteria": criteria, "reason": "valid"})
monkeypatch.setattr(module, "call_public_llm", _pair_script([(response, None)], calls))
- evidence = 'Evidence literal: Compute score = count(true) / 5. "SKILL_IDENTIFIED": true, "score": 0.8'
- module.judge_accuracy("question", "answer", evidence, skill_identification_applicable=False)
+ evidence = 'Evidence literal: {question} Compute score = count(true) / 5. "SKILL_IDENTIFIED": true, "score": 0.8'
+ module.judge_accuracy("question", "answer", evidence)
assert evidence in calls[0]
[email protected]("module", [llm_judge, eval_template])
-def test_report_reasons_omit_excluded_accuracy_criteria(module, monkeypatch):
- criteria = dict.fromkeys(_CRITERIA, True)
- criteria["SKILL_IDENTIFIED"] = False
- criteria["ACTION_CORRECT"] = False
- response = json.dumps({"criteria": criteria, "score": 0.6, "reason": "declined correctly"})
- monkeypatch.setattr(module, "call_public_llm", _pair_script([(response, None)], []))
- detail = module.judge_accuracy("question", "answer", "evidence", skill_identification_applicable=False)
+def test_report_reasons_still_omit_criteria_excluded_by_older_rewards():
+ # Rewards written before accuracy dropped SKILL_IDENTIFIED may still list it
+ # as excluded; reports keep honoring that.
+ criteria = {
+ "SKILL_IDENTIFIED": False,
+ "ACTION_CORRECT": False,
+ "FACTUALLY_ACCURATE": True,
+ "TASK_ADDRESSED": True,
+ "ACTIONABLE": True,
+ }
+ detail = {
+ "score": 0.75,
+ "reason": "declined correctly",
+ "criteria": criteria,
+ "excluded_criteria": ["SKILL_IDENTIFIED"],
+ "scoring_policy": "applicable-accuracy-criteria/v1",
+ }
reward = {
"entry_id": "negative-001",
**dict.fromkeys(("security", "skill_execution", "skill_efficiency", "goal_accuracy", "behavior_check"), 1.0),
@@ -493,10 +504,9 @@ def _judge_goal_with_response(
pytest.param("[]", id="list"),
pytest.param("{}", id="empty-object"),
pytest.param('prefix [{"score": 1.0}] suffix', id="prose-wrapped-list"),
- pytest.param('{"score": true}', id="boolean-score"),
- pytest.param('{"score": true, "score": 0.5}', id="duplicate-score"),
- pytest.param('{"score": NaN}', id="nan-score"),
- pytest.param('{"score": Infinity}', id="infinite-score"),
+ pytest.param(_VALID_CRITERIA_PREFIX + '"score": true, "score": 0.5}', id="duplicate-score"),
+ pytest.param(_VALID_CRITERIA_PREFIX + '"score": NaN}', id="nan-score"),
+ pytest.param(_VALID_CRITERIA_PREFIX + '"score": Infinity}', id="infinite-score"),
pytest.param('{"criteria": {"SKILL_IDENTIFIED": true}}', id="incomplete-criteria"),
pytest.param(
json.dumps({"criteria": {**_CRITERIA, "ACTIONABLE": "true"}}),
@@ -518,28 +528,51 @@ def test_accuracy_rejects_unscorable_response_shapes(judge_module, monkeypatch,
@pytest.mark.parametrize(
- ("score", "expected"),
+ "score",
[
- pytest.param(0.0, 0.0, id="genuine-zero"),
- pytest.param(-2, 0.0, id="clamp-low"),
- pytest.param(4.2, 1.0, id="clamp-high"),
- pytest.param(-(10**400), 0.0, id="clamp-huge-integer-low"),
- pytest.param(10**400, 1.0, id="clamp-huge-integer-high"),
+ pytest.param(0.0, id="zero"),
+ pytest.param(0.5, id="fraction"),
+ pytest.param(99, id="out-of-range"),
+ pytest.param(10**400, id="huge-integer"),
],
)
-def test_accuracy_accepts_finite_numeric_scores(judge_module, monkeypatch, score: float, expected: float) -> None:
+def test_accuracy_rejects_score_only_replies(judge_module, monkeypatch, score: float) -> None:
+ calls: list[str] = []
content = json.dumps({"score": score, "reason": "model judgment"})
- monkeypatch.setattr(judge_module, "call_public_llm", lambda *_args, **_kwargs: (content, None))
+ monkeypatch.setattr(judge_module, "call_public_llm", _pair_script([(content, None)], calls))
result = judge_module.judge_accuracy("question", "ground truth", "agent response")
- assert result["score"] == expected
- assert result.get("status") != "error"
+ assert len(calls) == 2
+ _assert_error_result(result)
+
+
+ "reported_score",
+ [
+ pytest.param(0.0, id="contradicting-zero"),
+ pytest.param(-2, id="below-range"),
+ pytest.param(4.2, id="above-range"),
+ pytest.param(10**400, id="huge-integer"),
+ pytest.param("0.8", id="string"),
+ pytest.param(True, id="boolean"),
+ ],
+)
+def test_accuracy_ignores_any_reported_score(judge_module, monkeypatch, reported_score) -> None:
+ calls: list[str] = []
+ content = json.dumps({"criteria": _CRITERIA, "score": reported_score, "reason": "model judgment"})
+ monkeypatch.setattr(judge_module, "call_public_llm", _pair_script([(content, None)], calls))
+
+ result = judge_module.judge_accuracy("question", "ground truth", "agent response")
+
+ assert len(calls) == 1
+ assert result["score"] == 0.6
+ assert result["criteria"] == _CRITERIA
def test_accuracy_normalizes_and_bounds_reason(judge_module, monkeypatch) -> None:
calls: list[str] = []
- content = json.dumps({"score": 1.0, "reason": " " + ("x" * 600) + " "})
+ content = json.dumps({"criteria": _CRITERIA, "reason": " " + ("x" * 600) + " "})
monkeypatch.setattr(judge_module, "call_public_llm", _pair_script([(content, None)], calls))
result = judge_module.judge_accuracy("question", "ground truth", "agent response")
@@ -553,7 +586,7 @@ def test_accuracy_redacts_configured_secret_before_bounding_reason(judge_module,
credential = "SECRET-ABCDEFGHIJKLMNOPQRSTUVWXYZ-0123456789"
prefix = "x" * 490
monkeypatch.setenv("OPENAI_API_KEY", credential)
- content = json.dumps({"score": 1.0, "reason": prefix + credential})
+ content = json.dumps({"criteria": _CRITERIA, "reason": prefix + credential})
monkeypatch.setattr(judge_module, "call_public_llm", lambda *_args, **_kwargs: (content, None))
result = judge_module.judge_accuracy("question", "ground truth", "agent response")
@@ -598,12 +631,12 @@ def test_accuracy_accepts_unique_object_after_non_json_bracket_label(judge_modul
monkeypatch.setattr(
judge_module,
"call_public_llm",
- lambda *_args, **_kwargs: ('Judge [draft]: {"score": 0.5}', None),
+ lambda *_args, **_kwargs: ("Judge [draft]: " + json.dumps({"criteria": _CRITERIA}), None),
)
result = judge_module.judge_accuracy("question", "ground truth", "agent response")
- assert result["score"] == 0.5
+ assert result["score"] == 0.6
assert result.get("status") != "error"
@@ -615,13 +648,13 @@ def test_accuracy_accepts_unique_object_before_unmatched_trailing_prose_without_
monkeypatch.setattr(
judge_module,
"call_public_llm",
- _pair_script([('{"score": 0.5}\nNote [draft', None)], calls),
+ _pair_script([(json.dumps({"criteria": _CRITERIA}) + "\nNote [draft", None)], calls),
)
result = judge_module.judge_accuracy("question", "ground truth", "agent response")
assert len(calls) == 1
- assert result["score"] == 0.5
+ assert result["score"] == 0.6
assert result.get("status") != "error"
@@ -675,8 +708,6 @@ def test_accuracy_deep_nesting_is_scoreless_instead_of_raising(judge_module, mon
pytest.param('{"achieved": 1}', id="integer-achieved"),
pytest.param('{"achieved": 1, "achieved": true}', id="duplicate-achieved"),
pytest.param('{"achieved": "true"}', id="string-achieved"),
- pytest.param('{"achieved": true, "score": true}', id="boolean-score"),
- pytest.param('{"achieved": true, "score": "0.8"}', id="string-score"),
pytest.param('{"achieved": false, "score": NaN}', id="nan-score"),
pytest.param('{"achieved": true, "score": Infinity}', id="infinite-score"),
],
@@ -693,12 +724,15 @@ def test_goal_rejects_unscorable_response_shapes(judge_module, monkeypatch, cont
pytest.param({"achieved": False, "score": 0.0}, 0.0, id="genuine-zero"),
pytest.param({"achieved": True}, 1.0, id="derive-true"),
pytest.param({"achieved": False}, 0.0, id="derive-false"),
- pytest.param({"achieved": False, "score": 0.25}, 0.25, id="finite-override"),
- pytest.param({"achieved": True, "score": -(10**400)}, 0.0, id="huge-integer-low"),
- pytest.param({"achieved": False, "score": 10**400}, 1.0, id="huge-integer-high"),
+ pytest.param({"achieved": False, "score": 1.0}, 0.0, id="contradicting-score"),
+ pytest.param({"achieved": False, "score": 0.25}, 0.0, id="fractional-score"),
+ pytest.param({"achieved": True, "score": -(10**400)}, 1.0, id="huge-integer-low"),
+ pytest.param({"achieved": False, "score": 10**400}, 0.0, id="huge-integer-high"),
+ pytest.param({"achieved": True, "score": True}, 1.0, id="boolean-score"),
+ pytest.param({"achieved": True, "score": "0.8"}, 1.0, id="string-score"),
],
)
-def test_goal_requires_boolean_achieved_and_uses_only_finite_numeric_optional_score(
+def test_goal_score_is_binary_from_achieved_and_ignores_reported_score(
judge_module,
monkeypatch,
payload: dict,
@@ -822,7 +856,7 @@ def test_behavior_first_provider_error_is_scoreless_without_retry(judge_module,
def test_behavior_retries_overflowed_numeric_metadata_once_and_recovers(judge_module, monkeypatch) -> None:
calls: list[str] = []
overflowed = '{"results":[{"passed":true,"reason":"ok","metadata":1e999}],"score":1.0}'
- recovered = json.dumps({"results": [{"passed": True, "reason": "ok"}], "score": 1.0, "summary": "recovered"})
+ recovered = json.dumps({"results": [{"step": 1, "passed": True, "reason": "ok"}], "summary": "recovered"})
monkeypatch.setattr(
judge_module,
"call_public_llm",
@@ -845,26 +879,26 @@ def test_behavior_retries_overflowed_numeric_metadata_once_and_recovers(judge_mo
pytest.param("[]", id="list"),
pytest.param("{}", id="empty-object"),
pytest.param(
- 'prefix [{"results": [{"passed": true}], "score": 1.0}] suffix',
+ 'prefix [{"results": [{"step": 1, "passed": true}], "score": 1.0}] suffix',
id="prose-wrapped-list",
),
pytest.param('{"results": "wrong", "score": 0}', id="non-list-results"),
pytest.param('{"results": [1], "score": 0}', id="non-object-result"),
- pytest.param('{"results": [{"passed": 1}], "score": 1}', id="non-boolean-passed"),
+ pytest.param('{"results": [{"step": 1, "passed": 1}], "score": 1}', id="non-boolean-passed"),
pytest.param(
- '{"results": [{"passed": 1, "passed": true}], "score": 1}',
+ '{"results": [{"step": 1, "passed": 1, "passed": true}], "score": 1}',
id="duplicate-passed",
),
pytest.param('{"results": [], "score": 0}', id="incomplete-results"),
pytest.param(
- '{"results": [{"passed": false}, {"passed": false}], "score": 0}',
+ '{"results": [{"step": 1, "passed": false}, {"step": 2, "passed": false}], "score": 0}',
id="extra-results",
),
pytest.param('{"results": [], "score": true}', id="boolean-score"),
pytest.param('{"results": [], "score": NaN}', id="nan-score"),
pytest.param('{"results": [], "score": Infinity}', id="infinite-score"),
pytest.param(
- '{"results": [{"passed": true}], "score": NaN}',
+ '{"results": [{"step": 1, "passed": true}], "score": NaN}',
id="valid-results-with-nan-score",
),
],
@@ -888,22 +922,28 @@ def test_behavior_retries_malformed_or_invalid_shapes_once_then_errors(
("payload", "expected_behaviors", "expected"),
[
pytest.param(
- {"results": [{"passed": False}], "score": 0.0, "summary": "failed"},
+ {"results": [{"step": 1, "passed": False}], "summary": "failed"},
["b1"],
0.0,
id="explicit-zero",
),
pytest.param(
- {"results": [{"passed": True}, {"passed": False}], "summary": "half"},
+ {"results": [{"step": 2, "passed": False}, {"step": 1, "passed": True}], "summary": "half"},
["b1", "b2"],
0.5,
- id="derived",
+ id="derived-out-of-order",
),
pytest.param(
- {"results": [{"passed": False}], "score": 10**400, "summary": "valid integer"},
+ {"results": [{"step": 1, "passed": False}], "score": 10**400, "summary": "ignored score"},
["b1"],
0.0,
- id="huge-integer-score",
+ id="huge-integer-score-ignored",
+ ),
+ pytest.param(
+ {"results": [{"step": 1, "passed": False}], "score": 1.0, "summary": "contradicting score"},
+ ["b1"],
+ 0.0,
+ id="contradicting-score-ignored",
),
],
)
@@ -930,7 +970,7 @@ def test_behavior_accepts_unique_object_before_unmatched_trailing_prose_without_
monkeypatch,
) -> None:
calls: list[str] = []
- content = '{"results":[{"passed":true}]}\nNote [draft'
+ content = '{"results":[{"step":1,"passed":true}]}\nNote [draft'
monkeypatch.setattr(judge_module, "call_public_llm", _pair_script([(content, None)], calls))
result = judge_module.judge_behavior_check("conversation", ["behavior"])
@@ -945,7 +985,7 @@ def test_behavior_retries_nonstandard_constant_in_metadata_then_errors(
monkeypatch,
) -> None:
calls: list[str] = []
- content = '{"results":[{"passed":true,"metadata":-Infinity}]}'
+ content = '{"results":[{"step":1,"passed":true,"metadata":-Infinity}]}'
monkeypatch.setattr(
judge_module,
"call_public_llm",
@@ -1017,7 +1057,7 @@ def test_behavior_salvage_rejects_noncompletable_current_entry(
monkeypatch,
unfinished_entry: str,
) -> None:
- malformed = '{"results":[{"passed":true},' + unfinished_entry
+ malformed = '{"results":[{"step":1,"passed":true},' + unfinished_entry
calls: list[str] = []
monkeypatch.setattr(
judge_module,
@@ -1035,9 +1075,9 @@ def test_behavior_salvage_rejects_noncompletable_current_entry(
@pytest.mark.parametrize(
"unfinished_entry",
[
- pytest.param('{"passed":false,"reason":"cut', id="unfinished-string"),
- pytest.param('{"passed":false,"reason":', id="missing-value-at-end"),
- pytest.param('{"passed":fal', id="partial-false-literal"),
+ pytest.param('{"step":2,"passed":false,"reason":"cut', id="unfinished-string"),
+ pytest.param('{"step":2,"passed":false,"reason":', id="missing-value-at-end"),
+ pytest.param('{"step":2,"passed":fal', id="partial-false-literal"),
],
)
def test_behavior_salvage_accepts_append_only_completable_current_entry(
@@ -1045,7 +1085,7 @@ def test_behavior_salvage_accepts_append_only_completable_current_entry(
monkeypatch,
unfinished_entry: str,
) -> None:
- truncated = '{"results":[{"passed":true},' + unfinished_entry
+ truncated = '{"results":[{"step":1,"passed":true},' + unfinished_entry
calls: list[str] = []
monkeypatch.setattr(
judge_module,
@@ -1057,11 +1097,11 @@ def test_behavior_salvage_accepts_append_only_completable_current_entry(
assert len(calls) == 2
assert result["score"] == 0.5
- assert result["results"] == [{"passed": True}]
+ assert result["results"] == [{"step": 1, "passed": True}]
def test_behavior_retries_ambiguous_multiple_documents_then_errors(judge_module, monkeypatch) -> None:
- content = '{"results":[{"passed":false}]} {"results":[{"passed":true}]}'
+ content = '{"results":[{"step":1,"passed":false}]} {"results":[{"step":1,"passed":true}]}'
calls: list[str] = []
monkeypatch.setattr(
judge_module,
@@ -1077,7 +1117,7 @@ def test_behavior_retries_ambiguous_multiple_documents_then_errors(judge_module,
def test_behavior_deep_nesting_is_scoreless_instead_of_raising(judge_module, monkeypatch) -> None:
- content = '{"results":[{"passed":true,"metadata":' + "[" * 10_000 + "0" + "]" * 10_000 + "}]}"
+ content = '{"results":[{"step":1,"passed":true,"metadata":' + "[" * 10_000 + "0" + "]" * 10_000 + "}]}"
calls: list[str] = []
monkeypatch.setattr(
judge_module,
@@ -1096,15 +1136,27 @@ def test_behavior_deep_nesting_is_scoreless_instead_of_raising(judge_module, mon
"malformed",
[
pytest.param(
- '{"metadata":' + "[" * 900 + "0" + "]" * 900 + ',"results":[{"passed":true},{"passed":false',
+ '{"metadata":'
+ + "[" * 900
+ + "0"
+ + "]" * 900
+ + ',"results":[{"step":1,"passed":true},{"step":2,"passed":false',
id="top-level-field",
),
pytest.param(
- '{"results":[{"passed":true,"metadata":' + "[" * 900 + "0" + "]" * 900 + '},{"passed":false',
+ '{"results":[{"step":1,"passed":true,"metadata":'
+ + "[" * 900
+ + "0"
+ + "]" * 900
+ + '},{"step":2,"passed":false',
id="complete-entry",
),
pytest.param(
- '{"results":[{"passed":true},{"passed":false,"metadata":' + "[" * 900 + "0" + "]" * 900 + ',"reason":"cut',
+ '{"results":[{"step":1,"passed":true},{"step":2,"passed":false,"metadata":'
+ + "[" * 900
+ + "0"
+ + "]" * 900
+ + ',"reason":"cut',
id="current-partial-entry",
),
],
@@ -1132,15 +1184,15 @@ def test_behavior_salvage_rejects_excessive_nesting(
"malformed",
[
pytest.param(
- '{"metadata":NaN,"results":[{"passed":true},{"passed":false',
+ '{"metadata":NaN,"results":[{"step":1,"passed":true},{"step":2,"passed":false',
id="top-level-field",
),
pytest.param(
- '{"results":[{"passed":true,"metadata":Infinity},{"passed":false',
+ '{"results":[{"step":1,"passed":true,"metadata":Infinity},{"step":2,"passed":false',
id="complete-entry",
),
pytest.param(
- '{"results":[{"passed":true},{"passed":false,"metadata":-Infinity,"reason":"cut',
+ '{"results":[{"step":1,"passed":true},{"step":2,"passed":false,"metadata":-Infinity,"reason":"cut',
id="current-partial-entry",
),
],
@@ -1167,51 +1219,53 @@ def test_behavior_salvage_rejects_nonstandard_constants(
@pytest.mark.parametrize(
"malformed",
[
- pytest.param('{"results":null,"metadata":[{"passed":true}]', id="unrelated-array"),
- pytest.param('{"metadata":{"results":[{"passed":true}]', id="nested-results"),
- pytest.param('{"results":[{"passed":true} {"passed":false', id="missing-comma"),
+ pytest.param('{"results":null,"metadata":[{"step":1,"passed":true}]', id="unrelated-array"),
+ pytest.param('{"metadata":{"results":[{"step":1,"passed":true}]', id="nested-results"),
+ pytest.param('{"results":[{"step":1,"passed":true} {"step":2,"passed":false', id="missing-comma"),
pytest.param(
- '{"results":[{"passed":true},{"passed":truX}]}',
+ '{"results":[{"step":1,"passed":true},{"step":2,"passed":truX}]}',
id="closed-invalid-token",
),
pytest.param(
- '{"results":[{"passed":true},{"passed":false,}]}',
+ '{"results":[{"step":1,"passed":true},{"step":2,"passed":false,}]}',
id="closed-trailing-comma",
),
- pytest.param('{"score":true,"results":[{"passed":true},{"passed":false', id="boolean-score"),
- pytest.param('{"score":"0.5","results":[{"passed":true},{"passed":false', id="string-score"),
- pytest.param('{"score":NaN,"results":[{"passed":true},{"passed":false', id="nan-score"),
- pytest.param('{"score":Infinity,"results":[{"passed":true},{"passed":false', id="infinite-score"),
+ pytest.param('{"score":true,"results":[{"step":1,"passed":true},{"step":2,"passed":false', id="boolean-score"),
+ pytest.param('{"score":"0.5","results":[{"step":1,"passed":true},{"step":2,"passed":false', id="string-score"),
+ pytest.param('{"score":NaN,"results":[{"step":1,"passed":true},{"step":2,"passed":false', id="nan-score"),
+ pytest.param(
+ '{"score":Infinity,"results":[{"step":1,"passed":true},{"step":2,"passed":false', id="infinite-score"
+ ),
pytest.param(
- '[{"results":[{"passed":true},{"passed":false',
+ '[{"results":[{"step":1,"passed":true},{"step":2,"passed":false',
id="array-wrapper",
),
pytest.param(
- '[] {"results":[{"passed":true},{"passed":false',
+ '[] {"results":[{"step":1,"passed":true},{"step":2,"passed":false',
id="complete-array-prefix",
),
pytest.param(
- '}] {"results":[{"passed":true},{"passed":false',
+ '}] {"results":[{"step":1,"passed":true},{"step":2,"passed":false',
id="structural-prefix",
),
pytest.param(
- '{"score":true,"score":0.5,"results":[{"passed":true},{"passed":false',
+ '{"score":true,"score":0.5,"results":[{"step":1,"passed":true},{"step":2,"passed":false',
id="duplicate-score",
),
pytest.param(
- '{"results":[{"passed":1,"passed":true},{"passed":false',
+ '{"results":[{"step":1,"passed":1,"passed":true},{"step":2,"passed":false',
id="duplicate-passed",
),
pytest.param(
- '{"results":\N{NO-BREAK SPACE}[{"passed":true},{"passed":false',
+ '{"results":\N{NO-BREAK SPACE}[{"step":1,"passed":true},{"step":2,"passed":false',
id="non-json-nbsp",
),
pytest.param(
- '{"results":\f[{"passed":true},{"passed":false',
+ '{"results":\f[{"step":1,"passed":true},{"step":2,"passed":false',
id="non-json-form-feed",
),
pytest.param(
- '{"results":\v[{"passed":true},{"passed":false',
+ '{"results":\v[{"step":1,"passed":true},{"step":2,"passed":false',
id="non-json-vertical-tab",
),
],
@@ -1236,7 +1290,7 @@ def test_behavior_salvage_rejects_invalid_top_level_prefixes(
def test_behavior_salvage_keeps_finite_top_level_score_prefix(judge_module, monkeypatch) -> None:
- truncated = '{"score":0.5,"results":[{"passed":true},{"passed":false'
+ truncated = '{"score":0.5,"results":[{"step":1,"passed":true},{"step":2,"passed":false'
calls: list[str] = []
monkeypatch.setattr(
judge_module,
@@ -1247,14 +1301,14 @@ def test_behavior_salvage_keeps_finite_top_level_score_prefix(judge_module, monk
result = judge_module.judge_behavior_check("conversation", ["b1", "b2"])
assert result["score"] == 0.5
- assert result["results"] == [{"passed": True}]
+ assert result["results"] == [{"step": 1, "passed": True}]
@pytest.mark.parametrize(
"prefix", [pytest.param("Here is the JSON:\n", id="prose"), pytest.param("```json\n", id="fence")]
)
def test_behavior_salvage_tolerates_non_structural_wrappers(judge_module, monkeypatch, prefix: str) -> None:
- truncated = f'{prefix}{{"results":[{{"passed":true}},{{"passed":false'
+ truncated = f'{prefix}{{"results":[{{"step":1,"passed":true}},{{"step":2,"passed":false'
calls: list[str] = []
monkeypatch.setattr(
judge_module,
@@ -1265,7 +1319,7 @@ def test_behavior_salvage_tolerates_non_structural_wrappers(judge_module, monkey
result = judge_module.judge_behavior_check("conversation", ["b1", "b2"])
assert result["score"] == 0.5
- assert result["results"] == [{"passed": True}]
+ assert result["results"] == [{"step": 1, "passed": True}]
def test_absent_reference_judges_are_explicitly_not_applicable(judge_module) -> None: