Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

fix(tier3): flag behavior evidence shortened by the final-response section limit (#259)

xh123637ea3f0f274fc6
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 83e17e47..1d6eca6d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -304,6 +304,19 @@ All notable changes to SkillEvaluator are documented in this file.
### Fixed
+- Tier 3 behavior-check evidence bundles now count a final answer cut by
+ the final-response section limit as truncated. The omission check
+ compared two renderings that both capped that section (800 characters
+ by default), so a long final answer could lose its middle to the cap
+ while the bundle claimed `truncated: false`. The comparison now runs
+ against a rendering without the final-response section limit. An answer
+ that still reaches the judge in full is not flagged, including one split
+ between the compact history and the end of the final-response section.
+ The `reason` names each setting that cut evidence:
+ `SKILL_EVAL_BEHAVIOR_CHECK_BUDGET`, `SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT`,
+ or both. Caps inside individual sections (write bodies, history summaries)
+ are unchanged. The evidence-budget documentation now names both causes of
+ a `behavior_check` truncation and which setting raises each.
- PR scan comments show the verdict, quality scores, first actionable fixes and
report links; full provenance and detailed findings stay in the report.
- Hosted reports use the retained site commit as the Pages deployment identity,
diff --git a/docs/reports.mdx b/docs/reports.mdx
index f3b24533..b8b40db4 100644
--- a/docs/reports.mdx
+++ b/docs/reports.mdx
@@ -366,7 +366,10 @@ completed run directly:
With standard grading, `reward.json` records `trajectory_source` (the
trajectory or agent log that was graded) and, under each LLM judge's
`details`, an `omitted` record showing whether its evidence was truncated to
- fit the [evidence budget](tier3-live-evaluation.mdx#evidence-budgets).
+ fit the [evidence budget](tier3-live-evaluation.mdx#evidence-budgets) or, for
+ `behavior_check`, the final-response section limit
+ (`SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT`). For `behavior_check`, the
+ record's `reason` names each setting that applied.
<Note>
Read commands (`view`, `compare`, `create-eval-dataset --refine`) honor the
diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdx
index 3973d573..345170f7 100644
--- a/docs/tier3-live-evaluation.mdx
+++ b/docs/tier3-live-evaluation.mdx
@@ -699,10 +699,13 @@ clipped cannot count, so on a long trajectory a behavior that appears only in
truncated history can be judged as not observed. In the results tree, each
judge's details in a trial's `reward.json` include an `omitted` record
(`count`, `truncated`, and `reason`); when `truncated` is `true` on cases that
-matter, raise that judge's budget. Larger budgets send longer judge prompts and
-use more evaluator tokens. The behavior budget never falls below the
-final-response limit plus 4,000 characters, so raising the limit leaves room
-for the tool history. Set these variables in the host environment;
+matter, raise that judge's budget. For `behavior_check`, the reason names each
+setting that cut evidence: `SKILL_EVAL_BEHAVIOR_CHECK_BUDGET`,
+`SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT`, or both. Raise each setting it
+names; raising the budget does not clear a cut made by the final-response
+limit. Larger budgets send longer judge prompts and use more evaluator
+tokens. The behavior budget never falls below the final-response limit plus
+4,000 characters, so raising the limit leaves room for the tool history. Set these variables in the host environment;
SkillEvaluator forwards them to the verifier. Defaults and accepted values are
in [Judge evidence budgets](environment-variables.mdx#judge-evidence-budgets).
diff --git a/src/skillevaluator/tier3/eval_core/atif_helpers.py b/src/skillevaluator/tier3/eval_core/atif_helpers.py
index 21c08ea2..be089464 100644
--- a/src/skillevaluator/tier3/eval_core/atif_helpers.py
+++ b/src/skillevaluator/tier3/eval_core/atif_helpers.py
@@ -971,6 +971,27 @@ def _build_verified_facts_section(facts: list[dict[str, Any]]) -> str:
return "\n".join(lines)
+def _final_answer_reaches_judge(final: str, evidence: str) -> bool:
+ """Return True when every character of the final answer appears in *evidence*.
+
+ A long answer can arrive in two pieces: the compact history repeats the
+ first 1,500 characters of a text-only final step, and the final-response
+ section keeps the end. Find the longest prefix of the answer present in
+ *evidence*; the answer is complete when the rest is present too.
+ Whitespace at the split does not count, because the history section
+ drops trailing whitespace.
+ """
+ answer = final.strip()
+ lo, hi = 0, len(answer)
+ while lo < hi:
+ mid = (lo + hi + 1) // 2
+ if answer[:mid] in evidence:
+ lo = mid
+ else:
+ hi = mid - 1
+ return answer[lo:].lstrip() in evidence
+
+
def build_metric_evidence_bundles(
traj: dict[str, Any],
question: str,
@@ -1049,15 +1070,37 @@ def build_metric_evidence_bundles(
facts_overhead = len(facts_section) + 2 if facts_section else 0
bc_budget = max(1, budgets["behavior_check"] - facts_overhead)
bc_text = build_behavior_evidence(traj, question, max_chars=bc_budget)
- bc_full = build_behavior_evidence(traj, question, max_chars=10**9)
- bc_trunc = len(bc_full) > len(bc_text)
+ # Compare against a rendering with neither limit applied. The budgeted
+ # rendering also caps the final-response section at its own limit
+ # (independent of max_chars), so comparing two budget renderings shares
+ # that cap and misses content the cap dropped from the middle of a long
+ # final answer: the bundle then reported truncated=False for evidence
+ # that was not complete.
+ bc_full = build_behavior_evidence(traj, question, max_chars=10**9, final_response_limit=10**9)
+ bc_causes: list[str] = []
+ if bc_full != bc_text:
+ # An unlimited-budget rendering with the usual final-response cap
+ # differs from the reference rendering only in that section, so it
+ # separates the two causes. The budget cut evidence when the budgeted
+ # rendering differs from it. The cap cut evidence when it changed that
+ # section and the complete final answer does not still reach the
+ # judge, whole in the compact history or split between the history
+ # and the end of that section (see _final_answer_reaches_judge). Name
+ # every cause that applied so users know which settings to raise.
+ bc_capped = build_behavior_evidence(traj, question, max_chars=10**9) if final else bc_full
+ if bc_text != bc_capped:
+ bc_causes.append("behavior evidence shortened to fit SKILL_EVAL_BEHAVIOR_CHECK_BUDGET")
+ if bc_capped != bc_full and not _final_answer_reaches_judge(final, bc_capped):
+ bc_causes.append("final response shortened to fit SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT")
+ bc_trunc = bool(bc_causes)
+ bc_reason = "; ".join(bc_causes)
bundles["behavior_check"] = {
"prompt_evidence": _prepend_facts(bc_text),
"evidence_refs": refs["behavior_check"],
"omitted": {
"count": 1 if bc_trunc else 0,
"truncated": bc_trunc,
- "reason": "lower-priority behavior history truncated to fit budget" if bc_trunc else "",
+ "reason": bc_reason,
},
"verified": facts,
}
diff --git a/tests/tier3/test_evidence_compiler.py b/tests/tier3/test_evidence_compiler.py
index 8a18b83e..13531e9f 100644
--- a/tests/tier3/test_evidence_compiler.py
+++ b/tests/tier3/test_evidence_compiler.py
@@ -368,3 +368,148 @@ def test_tool_only_trajectory_retains_latest_verification_failure(monkeypatch, i
assert len(evidence) <= {"accuracy": accuracy_budget, "goal_accuracy": goal_budget}[metric]
assert bundle["omitted"]["truncated"] is True
assert bundle["omitted"]["count"] > 0
+
+
+def test_behavior_bundle_reports_final_response_cap_loss() -> None:
+ # Regression test: a final answer longer than the behavior evidence
+ # final-response section limit loses its middle to that cap even when the
+ # whole bundle fits the total budget. The omission accounting used to
+ # compare two renderings that shared the cap, so the loss was reported
+ # as truncated=False.
+ marker = "MIGRATE_UNIQUE_COVERAGE_FIRST"
+ answer = "A" * 2000 + marker + "B" * 2000
+ traj = {"steps": [{"source": "agent", "message": answer}]}
+
+ for module in (atif_helpers, _load_harbor_template_module()):
+ bundle = module.build_metric_evidence_bundles(traj, "Explain the migration.")["behavior_check"]
+ assert marker not in bundle["prompt_evidence"]
+ assert bundle["omitted"]["truncated"] is True
+ assert bundle["omitted"]["count"] >= 1
+ assert bundle["omitted"]["reason"]
+
+
+def test_behavior_bundle_complete_evidence_is_not_reported_truncated() -> None:
+ # Control case: when nothing is dropped, the accounting must still say so.
+ traj = {"steps": [{"source": "agent", "message": "Short final answer."}]}
+
+ for module in (atif_helpers, _load_harbor_template_module()):
+ bundle = module.build_metric_evidence_bundles(traj, "What happened?")["behavior_check"]
+ assert "Short final answer." in bundle["prompt_evidence"]
+ assert bundle["omitted"]["truncated"] is False
+ assert bundle["omitted"]["count"] == 0
+ assert bundle["omitted"]["reason"] == ""
+
+
+def test_behavior_bundle_mid_length_final_answer_in_history_is_not_truncated() -> None:
+ # A final answer past the final-response section cap (784 of the default
+ # 800 characters) still reaches the judge in full when the compact
+ # history repeats the text-only final step verbatim (up to 1,500
+ # characters). The accounting must not flag that as truncated: nothing
+ # the judge reads is missing.
+ marker = "MID_ANSWER_COVERAGE_TOKEN"
+ answer = "x" * 550 + marker + "y" * 620
+ assert 800 < len(answer) < 1500
+ traj = {"steps": [{"source": "agent", "message": answer}]}
+
+ for module in (atif_helpers, _load_harbor_template_module()):
+ bundle = module.build_metric_evidence_bundles(traj, "Summarize the run.")["behavior_check"]
+ assert marker in bundle["prompt_evidence"]
+ assert bundle["omitted"]["truncated"] is False
+ assert bundle["omitted"]["count"] == 0
+ assert bundle["omitted"]["reason"] == ""
+
+
+def test_behavior_bundle_final_answer_past_history_repeat_is_truncated() -> None:
+ # Past the history repeat, the middle of the final answer is genuinely
+ # missing from the evidence, and the reason names the setting that
+ # applied so users know which limit to raise.
+ marker = "DEEP_ANSWER_COVERAGE_TOKEN"
+ answer = "A" * 2000 + marker + "B" * 2000
+ traj = {"steps": [{"source": "agent", "message": answer}]}
+
+ for module in (atif_helpers, _load_harbor_template_module()):
+ bundle = module.build_metric_evidence_bundles(traj, "Explain the migration.")["behavior_check"]
+ assert marker not in bundle["prompt_evidence"]
+ assert bundle["omitted"]["truncated"] is True
+ assert bundle["omitted"]["count"] >= 1
+ assert "SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT" in bundle["omitted"]["reason"]
+
+
+_BUDGET_REASON = "behavior evidence shortened to fit SKILL_EVAL_BEHAVIOR_CHECK_BUDGET"
+_LIMIT_REASON = "final response shortened to fit SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT"
+
+
+def _distinct_text(length: int) -> str:
+ # Numbered words keep long substrings unique, so a coverage check cannot
+ # match a repeated run by accident.
+ return " ".join(f"w{idx:04d}" for idx in range(length // 6 + 1))[:length]
+
+
[email protected]("cut_char", [" ", "-"], ids=["space-at-cut", "text-at-cut"])
[email protected](("length", "truncated"), [(1501, False), (1640, False), (1755, False), (1756, True)])
+def test_behavior_bundle_final_answer_split_across_history_and_section(
+ length: int, truncated: bool, cut_char: str
+) -> None:
+ # The compact history repeats the first 1,500 characters of a text-only
+ # final answer, and at the default limit the final-response section keeps
+ # its last 255. Together they carry an answer of up to 1,755 characters
+ # in full, although no single piece holds all of it. The history drops
+ # trailing whitespace, so cover a cut that falls on a space and one that
+ # falls on text.
+ answer = _distinct_text(length)
+ answer = answer[:1499] + cut_char + answer[1500:]
+ traj = {"steps": [{"source": "agent", "message": answer}]}
+
+ for module in (atif_helpers, _load_harbor_template_module()):
+ bundle = module.build_metric_evidence_bundles(traj, "Summarize the run.")["behavior_check"]
+ assert answer not in bundle["prompt_evidence"]
+ assert answer[:1500].rstrip() in bundle["prompt_evidence"]
+ assert bundle["omitted"]["truncated"] is truncated
+ assert bundle["omitted"]["count"] == (1 if truncated else 0)
+ assert bundle["omitted"]["reason"] == (_LIMIT_REASON if truncated else "")
+
+
+ ("env", "reason"),
+ [
+ ({}, f"{_BUDGET_REASON}; {_LIMIT_REASON}"),
+ ({"SKILL_EVAL_BEHAVIOR_CHECK_BUDGET": "100000"}, _LIMIT_REASON),
+ ({"SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT": "3000"}, _BUDGET_REASON),
+ (
+ {"SKILL_EVAL_BEHAVIOR_CHECK_BUDGET": "100000", "SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT": "3000"},
+ "",
+ ),
+ ],
+ ids=["both", "limit-only", "budget-only", "neither"],
+)
+def test_behavior_bundle_reason_names_each_setting_that_cut_evidence(
+ monkeypatch: pytest.MonkeyPatch, env: dict[str, str], reason: str
+) -> None:
+ # A long tool history overflows the default behavior budget, and a final
+ # answer past 1,755 characters also loses its middle to the default
+ # final-response limit. The reason must name every setting that applied,
+ # so raising only the budget is not mistaken for a fix.
+ for name in ("SKILL_EVAL_BEHAVIOR_CHECK_BUDGET", "SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT"):
+ monkeypatch.delenv(name, raising=False)
+ for name, value in env.items():
+ monkeypatch.setenv(name, value)
+ steps: list[dict] = []
+ for idx in range(40):
+ steps.append({
+ "source": "agent",
+ "tool_calls": [{
+ "tool_call_id": f"list-{idx}",
+ "function_name": "bash",
+ "arguments": {"command": f"ls /workspace/step-{idx:02d}"},
+ }],
+ "observation": {"results": [{"source_call_id": f"list-{idx}", "content": f"step {idx:02d} " + "z" * 292}]},
+ })
+ answer = _distinct_text(2000)
+ steps.append({"source": "agent", "message": answer})
+
+ for module in (atif_helpers, _load_harbor_template_module()):
+ bundle = module.build_metric_evidence_bundles({"steps": steps}, "Explain the migration.")["behavior_check"]
+ assert bundle["omitted"]["truncated"] is bool(reason)
+ assert bundle["omitted"]["count"] == (1 if reason else 0)
+ assert bundle["omitted"]["reason"] == reason
+ assert (answer in bundle["prompt_evidence"]) is (_LIMIT_REASON not in reason)