Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

fix(reporting): show recovered timeout attempts accurately in Tier 3 attempt details (#416)

Narendran Raghavan4a34bd3be5f5
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5335b62f..d2e5e028 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -552,6 +552,17 @@ All notable changes to SkillEvaluator are documented in this file.
info or host, so `Error: ` followed by minified JSON no longer arrives as
`Error: ...<truncated>`.
+- Tier 3 HTML **Attempt Details** now show an agent timeout that a later
+ scored attempt recovered under `--stop-on-pass`, and count it in the case's
+ attempts. In 0.8.0 a case that timed out on attempt 1, passed on attempt 2
+ and skipped attempt 3 read `1 of 2 scored` and `passed on scored attempt 2`,
+ with nothing about the timeout; it now reads `A1: timed out, recovered`,
+ `1 of 3 scored` and `passed on attempt 2`. The Result column says
+ `passed on attempt N` for every case, because N is the attempt's number, not
+ a count of scored attempts. `tier3 evaluate` also prints one line for each
+ agent with recovered timeouts. Scores, pass@k and attempt coverage are
+ unchanged.
+
- Tier 3 security grading again warns about network exfiltration, lowering
the Security score to 0.5, for an HTTPie method word after a capitalized
`HTTP`, `HTTPS`, or `Http` word in cases that 0.8.0 stopped flagging: the
diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdx
index 83f25624..5d1697d2 100644
--- a/docs/tier3-live-evaluation.mdx
+++ b/docs/tier3-live-evaluation.mdx
@@ -696,6 +696,19 @@ reports mark it publication-incomplete when cases were skipped. By contrast,
`--n-concurrent` and `--max-agents` control parallelism and elapsed time; they
do not reduce the total planned trials or model work.
+With `--stop-on-pass`, an attempt that hits Harbor's agent execution timeout
+does not end the case: the next attempt runs, and when a later attempt of the
+same case completes with a score, it recovers the timeout. The timed-out
+attempt is never scored, and its failure stays in the agent's results under
+`recovered_trial_failures`. It still used one of the case's attempts, so
+reports count it on its own: the HTML **Attempt Details** row shows it as
+`A1: timed out, recovered`, or as a count (`1 timed out, recovered`) when the
+report did not keep its attempt number, and keeps it in the attempt total. A
+case that timed out on attempt 1 and passed on attempt 2 of 3 reads
+`1 of 3 scored` and `passed on attempt 2`, and `tier3 evaluate` prints one
+line for each agent with recovered timeouts. Without `--stop-on-pass` there is
+no recovery: a timed-out attempt leaves the arm's attempt coverage incomplete.
+
### Cost and local mode
`--env-mode local` avoids managed cloud-sandbox charges by running the agent
diff --git a/src/skillevaluator/reporting/html.py b/src/skillevaluator/reporting/html.py
index 528b673a..b116bf12 100644
--- a/src/skillevaluator/reporting/html.py
+++ b/src/skillevaluator/reporting/html.py
@@ -585,6 +585,8 @@ def _sanitize_tier3_pass_condition(value: object) -> dict[str, Any]:
case = dict(raw_case)
for key in ("attempts_used", "attempts_missing", "attempts_skipped"):
case[key] = _nonnegative_count(case.get(key))
+ if "attempts_recovered" in case:
+ case["attempts_recovered"] = _nonnegative_count(case.get("attempts_recovered"))
case["best_score"] = _finite_number(case.get("best_score"), minimum=0.0, maximum=1.0)
first_pass = case.get("first_pass_attempt")
case["first_pass_attempt"] = (
diff --git a/src/skillevaluator/reporting/templates/report.html.j2 b/src/skillevaluator/reporting/templates/report.html.j2
index d42c9463..5fbc2ed2 100644
--- a/src/skillevaluator/reporting/templates/report.html.j2
+++ b/src/skillevaluator/reporting/templates/report.html.j2
@@ -3211,7 +3211,7 @@
{# Agent -> case -> condition, so each case's with/without rows sit together. #}
{% for case_id in t3_case_ids | sort %}
{% for cond_label, branch in t3_branches if case_id in branch.cases %}
- {% set _ = t3_attempt_rows.append({'agent': agent_name, 'condition': cond_label, 'case_id': case_id, 'case': branch.cases[case_id]}) %}
+ {% set _ = t3_attempt_rows.append({'agent': agent_name, 'condition': cond_label, 'arm': 'with_skill' if branch is sameas pak.with_skill else 'without_skill', 'case_id': case_id, 'case': branch.cases[case_id]}) %}
{% endfor %}
{% endfor %}
{% endfor %}
@@ -3228,6 +3228,8 @@
{% endfor %}
{% endfor %}
{% endfor %}
+ {% set t3_recovered_rows = namespace(any=false) %}
+ {% for row in t3_attempt_rows %}{% if (row.case.get('attempts_recovered') or 0) > 0 %}{% set t3_recovered_rows.any = true %}{% endif %}{% endfor %}
{% if t3_attempt_rows %}
<details class="t3-details" id="tier3-attempt-details" open>
@@ -3235,13 +3237,17 @@
<div class="t3-details-body">
<p class="t3-section-hint">
Each row is one eval case for one agent under one condition. Attempt Scores
- lists the scored attempts only; a score at or above
+ lists the scored attempts {% if t3_recovered_rows.any %}and any recovered timeouts{% else %}only{% endif %}; a score at or above
{{ "%.2f" | format(t3_pass_threshold) }} counts as passed for Pass@{{ t3_max_attempts }}
when scored goal accuracy is at least
{{ "%.2f" | format(t3_verdict_policy.get('goal_accuracy_pass_threshold', 1.0)) }}
and scored security is at least
{{ "%.2f" | format(t3_verdict_policy.get('security_pass_threshold', 1.0)) }}.
"Not scored" means no reward was found for that expected attempt; it is excluded from pass/fail.
+ {% if t3_recovered_rows.any %}"Timed out, recovered" marks an agent timeout that a later scored
+ attempt of the same case recovered; the chip shows the attempt number when the report has it,
+ and otherwise a count. It is not scored, but it used one of the case's attempts, so it counts
+ in the attempt total under Scored Attempts (the second number), not in the scored count.{% endif %}
</p>
<div class="t3-filter-bar">
<input class="t3-filter-input" type="search" placeholder="Filter by agent, case, condition, score..." oninput="tier3FilterTable(this, 't3-attempts-table')" aria-label="Filter attempt details">
@@ -3264,8 +3270,26 @@
{% set attempts_used = attempts_used_value | int if attempts_used_value is not none else (attempts | length) %}
{% set attempts_missing = attempts_missing_value | int if attempts_missing_value is not none else 0 %}
{% set attempts_skipped = attempts_skipped_value | int if attempts_skipped_value is not none else 0 %}
- {% set total_expected = attempts_used + attempts_missing + attempts_skipped %}
+ {% set attempts_recovered_value = case_data.get('attempts_recovered') %}
+ {% set attempts_recovered = attempts_recovered_value | int if attempts_recovered_value is not none else 0 %}
+ {# A recovered agent timeout is unscored but used a physical attempt: count it so the
+ denominator stays the case's attempt budget, matching the attempt numbers shown. #}
+ {% set total_expected = attempts_used + attempts_missing + attempts_skipped + attempts_recovered %}
{% if total_expected == 0 %}{% set total_expected = t3_max_attempts %}{% endif %}
+ {# Attempt numbers of recovered timeouts come from the arm's bounded recovery list;
+ when that list does not account for every recovery, show the count instead. #}
+ {% set recovered_ordinals = [] %}
+ {% if attempts_recovered > 0 %}
+ {% set row_conditions = t3_agents[row.agent].conditions %}
+ {% set row_condition = row_conditions.get(row.arm) if row_conditions is mapping else none %}
+ {% set recovered_items = row_condition.get('recovered_attempts') if row_condition is mapping else none %}
+ {% if recovered_items is sequence and recovered_items is not string and recovered_items is not mapping %}
+ {% for item in recovered_items if item is mapping and item.get('case_id') == row.case_id and item.get('attempt') is integer and item.get('attempt') > 0 %}
+ {% if item.get('attempt') not in recovered_ordinals %}{% set _ = recovered_ordinals.append(item.get('attempt')) %}{% endif %}
+ {% endfor %}
+ {% endif %}
+ {% endif %}
+ {% set recovered_ordinals = recovered_ordinals | sort if recovered_ordinals | length == attempts_recovered else [] %}
{% set best_score = case_data.get('best_score') %}
{% set passed = case_data.get('passed') %}
{% set first_pass_attempt = case_data.get('first_pass_attempt') %}
@@ -3274,12 +3298,17 @@
<td>{{ row.condition }}</td>
<td style="font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;font-size:0.82rem;">{{ row.case_id }}</td>
<td>
- {% if attempts %}
+ {% if attempts or attempts_recovered %}
<div class="t3-attempt-list">
+ {% set pending_recovered = namespace(ordinals=recovered_ordinals) %}
{% for att in attempts %}
{% set sc = att.get('score') %}
{% set att_idx_value = att.get('attempt') %}
{% set att_idx = att_idx_value | int if att_idx_value is not none else loop.index %}
+ {% for ordinal in pending_recovered.ordinals if ordinal < att_idx %}
+ <span class="t3-attempt-chip muted" title="Attempt {{ ordinal }}: agent timed out; a later scored attempt recovered it. Not scored.">A{{ ordinal }}: timed out, recovered</span>
+ {% endfor %}
+ {% set pending_recovered.ordinals = pending_recovered.ordinals | reject('lt', att_idx) | list %}
{% set attempt_verdict = t3_attempt_verdict_by_trial.get((row.agent, row.condition, att.get('trial'))) or {} %}
{% set attempt_passed = att.get('passed') and attempt_verdict.get('passed', true) %}
{% set attempt_reason = attempt_verdict.get('reason') or att.get('reason') %}
@@ -3291,12 +3320,27 @@
<span class="t3-attempt-chip {{ chip_cls }}" title="{{ attempt_reason or '' }}">A{{ att_idx }}: {{ "%.2f" | format(sc) }} {{ chip_label }}</span>
{% endif %}
{% endfor %}
+ {% for ordinal in pending_recovered.ordinals %}
+ <span class="t3-attempt-chip muted" title="Attempt {{ ordinal }}: agent timed out; a later scored attempt recovered it. Not scored.">A{{ ordinal }}: timed out, recovered</span>
+ {% endfor %}
+ {% if attempts_recovered and not recovered_ordinals %}
+ <span class="t3-attempt-chip muted" title="Agent timeouts that later scored attempts recovered. Not scored.">{{ attempts_recovered }} timed out, recovered</span>
+ {% endif %}
+ {# Published attempt rows are capped per run, so a recovered case can arrive with no
+ rows; say so instead of letting the recovered chip read as the whole history. #}
+ {% if not attempts %}
+ {% if attempts_used > 0 %}
+ <span class="t3-attempt-chip muted" title="The report keeps details for a limited number of attempts; this case's {{ attempts_used }} scored attempt{{ 's' if attempts_used != 1 else '' }} still count{{ 's' if attempts_used == 1 else '' }} but {{ 'is' if attempts_used == 1 else 'are' }} not listed.">scored attempt details omitted</span>
+ {% else %}
+ <span class="t3-attempt-chip muted">no attempts recorded</span>
+ {% endif %}
+ {% endif %}
</div>
{% else %}
<span class="t3-attempt-chip muted">no attempts recorded</span>
{% endif %}
</td>
- <td style="font-variant-numeric:tabular-nums;">{{ attempts_used }} of {{ total_expected }} scored</td>
+ <td style="font-variant-numeric:tabular-nums;"{% if attempts_recovered %} title="Includes {{ attempts_recovered }} recovered agent timeout{{ 's' if attempts_recovered != 1 else '' }}, which {{ 'are' if attempts_recovered != 1 else 'is' }} not scored"{% endif %}>{{ attempts_used }} of {{ total_expected }} scored</td>
<td>
{% if best_score is not none %}
{% set bg = 'rgba(34,197,94,0.18)' if best_score >= t3_pass_threshold else ('rgba(234,179,8,0.18)' if best_score >= 0.4 else 'rgba(239,68,68,0.18)') %}
@@ -3326,7 +3370,7 @@
{% endfor %}
{% endif %}
{% if outcome.passed and outcome.first_pass is not none %}
- <span class="t3-pill ok">passed on scored attempt {{ outcome.first_pass | int }}</span>
+ <span class="t3-pill ok">passed on attempt {{ outcome.first_pass | int }}</span>
{% elif outcome.passed %}
<span class="t3-pill ok">passed</span>
{% elif attempts_used > 0 and outcome.reason %}
diff --git a/src/skillevaluator/tier3/result_display.py b/src/skillevaluator/tier3/result_display.py
index 38fbacf6..6aca2642 100644
--- a/src/skillevaluator/tier3/result_display.py
+++ b/src/skillevaluator/tier3/result_display.py
@@ -253,6 +253,30 @@ def _attempt_coverage(data: Mapping[str, Any]) -> str:
return ""
+def _recovered_timeout_lines(agents: Mapping[str, Any]) -> list[str]:
+ """One line per agent whose agent timeouts a later scored attempt recovered."""
+ lines = []
+ for agent, raw in agents.items():
+ conditions = raw.get("conditions") if isinstance(raw, Mapping) else None
+ if not isinstance(conditions, Mapping):
+ continue
+ counts: dict[str, int] = {}
+ for variant, label in (("with_skill", "with skill"), ("without_skill", "baseline")):
+ condition = conditions.get(variant)
+ count = condition.get("recovered_attempts_total") if isinstance(condition, Mapping) else None
+ if isinstance(count, int) and not isinstance(count, bool) and count > 0:
+ counts[label] = count
+ total = sum(counts.values())
+ if total:
+ arms = ", ".join(f"{count} {label}" for label, count in counts.items())
+ subject = "the timed-out attempt is" if total == 1 else "timed-out attempts are"
+ lines.append(
+ f"{agent}: {total} agent timeout{'s' if total != 1 else ''} recovered by a later scored "
+ f"attempt ({arms}); {subject} not scored and stay{'s' if total == 1 else ''} in the run history."
+ )
+ return lines
+
+
def _concise_trial_reason(reason: str) -> str:
"""Collapse identical judge errors while retaining the affected metrics."""
prefix = "Required judge evaluation failed: "
@@ -1183,6 +1207,10 @@ def render_evaluation_result(
except Exception: # advisory panel: never break the run summary
logging.getLogger(__name__).debug("Findings report skipped", exc_info=True)
+ # A recovered timeout is neither a failure nor a score; say so instead of staying silent.
+ for line in _recovered_timeout_lines(agents) if isinstance(agents, Mapping) else []:
+ console.print(Text(safe(line), style="yellow"))
+
errors = result.get("execution_errors") or result.get("error") or []
if isinstance(errors, str):
errors = [errors]
diff --git a/tests/test_harbor_timeout_recovery.py b/tests/test_harbor_timeout_recovery.py
index 982a2864..2b7f13db 100644
--- a/tests/test_harbor_timeout_recovery.py
+++ b/tests/test_harbor_timeout_recovery.py
@@ -6,6 +6,7 @@ from __future__ import annotations
import copy
import json
+import re
from datetime import UTC, datetime, timedelta
from pathlib import Path
from uuid import uuid4
@@ -17,6 +18,12 @@ pytest.importorskip("harbor")
from harbor.models.job.result import JobResult, JobStats
from harbor.models.trial.result import TrialResult
+from skillevaluator.evaluation.tier3_report import (
+ _compact_agent_conditions,
+ agent_eval_result_from_directory,
+ render_agent_eval_html_report,
+)
+from skillevaluator.reporting import HTMLReporter
from skillevaluator.tier3.harbor.collector import (
_pass_summary,
_shared_metric_scorer,
@@ -25,6 +32,7 @@ from skillevaluator.tier3.harbor.collector import (
harbor_timeout_job_recovered,
)
from skillevaluator.tier3.harbor.runner import _merge_attempt_jobs
+from skillevaluator.tier3.result_display import render_result
@pytest.mark.parametrize(
@@ -127,8 +135,8 @@ def native_attempt(root: Path, attempt: int, *, timeout: bool = False, reward: f
return job
-def collect(root: Path, failed: Path, successful: Path, *, stop_on_pass: bool = True):
- _merge_attempt_jobs([failed, successful], root / "jobs" / "demo-opencode-with")
+def collect(root: Path, *attempt_jobs: Path, stop_on_pass: bool = True):
+ _merge_attempt_jobs(list(attempt_jobs), root / "jobs" / "demo-opencode-with")
return collect_harbor_results(
skill_name="demo",
agents=["opencode"],
@@ -364,3 +372,212 @@ def test_recovery_does_not_depend_on_directory_order(tmp_path, monkeypatch):
assert agent["overall_with_skill"] == 1.0
assert len(agent["recovered_trial_failures"]["with_skill"]) == 2
assert agent["trial_failures"]["with_skill"] == []
+
+
+def attempt_details_rows(html: str) -> list[str]:
+ """Return each Attempt Details body row as its visible text."""
+ body = html.split('id="t3-attempts-table"', 1)[1].split("<tbody>", 1)[1].split("</tbody>", 1)[0]
+ return [" ".join(re.sub(r"<[^>]+>", " ", row).split()) for row in re.findall(r"<tr>(.*?)</tr>", body, re.DOTALL)]
+
+
+def attempt_details_hint(html: str) -> str:
+ """Return the Attempt Details section hint as one line of text."""
+ section = html.split('id="tier3-attempt-details"', 1)[1]
+ return " ".join(section.split('<p class="t3-section-hint">', 1)[1].split("</p>", 1)[0].split())
+
+
+def recovery_lines(rendered: str) -> list[str]:
+ """Return the terminal lines that report recovered agent timeouts."""
+ return [line.strip() for line in rendered.splitlines() if "recovered by a later scored attempt" in line]
+
+
+def render_html(root: Path) -> str:
+ return render_agent_eval_html_report(root, root / "results", use_llm_judge=False).read_text(encoding="utf-8")
+
+
+def recovered_run_report(root: Path):
+ """Collect a timeout on attempt 1 and a pass on attempt 2 of 3, and load it for the HTML reporter."""
+ collect(root, native_attempt(root / "attempts", 1, timeout=True), native_attempt(root / "attempts", 2))
+ result = agent_eval_result_from_directory(root, root / "results", use_llm_judge=False)
+ return result, result.metadata["agent_eval"]["agents"]["opencode"]
+
+
+HINT_START = "Each row is one eval case for one agent under one condition. Attempt Scores lists the scored attempts"
+HINT_THRESHOLD = (
+ "a score at or above 0.50 counts as passed for Pass@3 when scored goal accuracy is at least 1.00 "
+ "and scored security is at least 1.00. "
+ '"Not scored" means no reward was found for that expected attempt; it is excluded from pass/fail.'
+)
+RECOVERED_HINT = (
+ f"{HINT_START} and any recovered timeouts; {HINT_THRESHOLD} "
+ '"Timed out, recovered" marks an agent timeout that a later scored attempt of the same case recovered; '
+ "the chip shows the attempt number when the report has it, and otherwise a count. "
+ "It is not scored, but it used one of the case's attempts, so it counts in "
+ "the attempt total under Scored Attempts (the second number), not in the scored count."
+)
+
+
+def test_attempt_details_show_recovered_timeout_in_the_attempt_budget(tmp_path):
+ result = collect(
+ tmp_path,
+ native_attempt(tmp_path / "attempts", 1, timeout=True),
+ native_attempt(tmp_path / "attempts", 2),
+ )
+ assert result["execution_status"] == "succeeded", result.get("execution_errors")
+
+ html = render_html(tmp_path)
+
+ # Attempt 1 timed out, attempt 2 passed, attempt 3 was skipped: three attempts, one scored.
+ assert attempt_details_rows(html) == [
+ "opencode With skill case-a A1: timed out, recovered A2: 1.00 PASS 1 of 3 scored 1.00 passed on attempt 2"
+ ]
+ assert "passed on scored attempt" not in html
+ # The hint lists the recovered chip among the attempt scores and counts it only in the second number.
+ assert attempt_details_hint(html) == RECOVERED_HINT
+ assert 'title="Includes 1 recovered agent timeout, which is not scored"' in html
+
+
+def test_attempt_details_place_each_recovered_timeout_at_its_attempt_number(tmp_path):
+ result = collect(
+ tmp_path,
+ native_attempt(tmp_path / "attempts", 1, reward=0.0),
+ native_attempt(tmp_path / "attempts", 2, timeout=True),
+ native_attempt(tmp_path / "attempts", 3),
+ )
+ assert result["execution_status"] == "succeeded", result.get("execution_errors")
+ case = result["agents"]["opencode"]["pass_at_k"]["with_skill"]["cases"]["case-a"]
+ assert (case["attempts_used"], case["attempts_recovered"], case["first_pass_attempt"]) == (2, 1, 3)
+
+ [row] = attempt_details_rows(render_html(tmp_path))
+
+ assert row == (
+ "opencode With skill case-a A1: 0.00 FAIL A2: timed out, recovered A3: 1.00 PASS "
+ "2 of 3 scored 1.00 passed on attempt 3"
+ )
+
+
+def test_attempt_details_without_recovery_only_reword_the_passing_attempt(tmp_path):
+ result = collect(
+ tmp_path,
+ native_attempt(tmp_path / "attempts", 1, reward=0.0),
+ native_attempt(tmp_path / "attempts", 2),
+ )
+ assert result["execution_status"] == "succeeded", result.get("execution_errors")
+
+ html = render_html(tmp_path)
+
+ assert attempt_details_rows(html) == [
+ "opencode With skill case-a A1: 0.00 FAIL A2: 1.00 PASS 2 of 3 scored 1.00 passed on attempt 2"
+ ]
+ assert "recovered" not in html.split('id="t3-attempts-table"', 1)[1].split("</table>", 1)[0]
+ assert attempt_details_hint(html) == f"{HINT_START} only; {HINT_THRESHOLD}"
+ assert recovery_lines(render_result(result)) == []
+
+
+def test_attempt_details_read_recovered_attempt_numbers_from_the_rows_own_arm(tmp_path):
+ result, agent = recovered_run_report(tmp_path)
+ # Give the baseline arm the same recovered case, then drop the with-skill recovery list
+ # (compacted payloads keep the per-case count but not the bounded list).
+ agent["conditions"]["without_skill"] = copy.deepcopy(agent["conditions"]["with_skill"])
+ agent["pass_at_k"]["without_skill"] = copy.deepcopy(agent["pass_at_k"]["with_skill"])
+ del agent["conditions"]["with_skill"]["recovered_attempts"]
+
+ html = HTMLReporter(report_scope="tier3").render(result)
+
+ assert attempt_details_rows(html) == [
+ "opencode With skill case-a A2: 1.00 PASS 1 timed out, recovered 1 of 3 scored 1.00 passed on attempt 2",
+ "opencode Without skill case-a A1: timed out, recovered A2: 1.00 PASS 1 of 3 scored 1.00 passed on attempt 2",
+ ]
+
+
+def test_attempt_details_hint_does_not_promise_an_attempt_number_for_a_recovered_count(tmp_path):
+ result, agent = recovered_run_report(tmp_path)
+ # Compacting a non-best agent's conditions for a large multi-agent report keeps the
+ # per-case count but drops the arm's recovery list, so the row can only show a count.
+ assert _compact_agent_conditions(agent)
+ assert "recovered_attempts" not in agent["conditions"]["with_skill"]
+
+ html = HTMLReporter(report_scope="tier3").render(result)
+
+ assert attempt_details_rows(html) == [
+ "opencode With skill case-a A2: 1.00 PASS 1 timed out, recovered 1 of 3 scored 1.00 passed on attempt 2"
+ ]
+ assert attempt_details_hint(html) == RECOVERED_HINT
+ assert "keeps its attempt number" not in html
+
+
+ ("attempts_used", "notice"),
+ [(1, "scored attempt details omitted"), (0, "no attempts recorded")],
+)
+def test_attempt_details_say_when_a_recovered_case_has_no_attempt_rows(tmp_path, attempts_used, notice):
+ result, agent = recovered_run_report(tmp_path)
+ # The collector publishes a bounded number of attempt rows per run, so a later case keeps
+ # its counts and recovery but can arrive without rows.
+ agent["pass_at_k"]["with_skill"]["cases"]["case-a"].update(attempts=[], attempts_used=attempts_used)
+
+ html = HTMLReporter(report_scope="tier3").render(result)
+
+ assert attempt_details_rows(html) == [
+ f"opencode With skill case-a A1: timed out, recovered {notice} "
+ f"{attempts_used} of {attempts_used + 2} scored 1.00 passed on attempt 2"
+ ]
+
+
[email protected]("attempts_recovered", ["1", -3, True])
+def test_attempt_details_ignore_an_invalid_recovered_count(tmp_path, attempts_recovered):
+ result, agent = recovered_run_report(tmp_path)
+ agent["pass_at_k"]["with_skill"]["cases"]["case-a"]["attempts_recovered"] = attempts_recovered
+
+ html = HTMLReporter(report_scope="tier3").render(result)
+
+ # Read like a case without recovery: no recovered chip, tooltip or hint, and the
+ # denominator is the scored and skipped attempts only.
+ assert attempt_details_rows(html) == [
+ "opencode With skill case-a A2: 1.00 PASS 1 of 2 scored 1.00 passed on attempt 2"
+ ]
+ assert attempt_details_hint(html) == f"{HINT_START} only; {HINT_THRESHOLD}"
+ assert "recovered agent timeout" not in html
+
+
+def test_tier3_evaluate_prints_the_collected_recovery_line(tmp_path):
+ result = collect(
+ tmp_path,
+ native_attempt(tmp_path / "attempts", 1, timeout=True),
+ native_attempt(tmp_path / "attempts", 2),
+ )
+
+ rendered = render_result(result)
+
+ assert recovery_lines(rendered) == [
+ "opencode: 1 agent timeout recovered by a later scored attempt (1 with skill); "
+ "the timed-out attempt is not scored and stays in the run history."
+ ]
+ assert "AgentTimeoutError" not in rendered
+
+
+def test_tier3_evaluate_prints_one_recovery_line_per_agent_with_recovered_timeouts():
+ result = {
+ "execution_status": "succeeded",
+ "agents": {
+ "both": {
+ "conditions": {
+ "with_skill": {"recovered_attempts_total": 2},
+ "without_skill": {"recovered_attempts_total": 1},
+ }
+ },
+ "baseline-only": {"conditions": {"without_skill": {"recovered_attempts_total": 1}}},
+ "none": {"conditions": {"with_skill": {"recovered_attempts_total": 0}}},
+ "boolean": {"conditions": {"with_skill": {"recovered_attempts_total": True}}},
+ "text": {"conditions": {"with_skill": {"recovered_attempts_total": "2"}}},
+ "no-conditions": {"conditions": None},
+ },
+ }
+
+ # Both arms add up; zero, boolean, text and missing counts print nothing.
+ assert recovery_lines(render_result(result)) == [
+ "both: 3 agent timeouts recovered by a later scored attempt (2 with skill, 1 baseline); "
+ "timed-out attempts are not scored and stay in the run history.",
+ "baseline-only: 1 agent timeout recovered by a later scored attempt (1 baseline); "
+ "the timed-out attempt is not scored and stays in the run history.",
+ ]