Skip to content

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

fix(tier3): keep the Rich progress frame at a fixed height (#420)

Narendran Raghavan3ef5fad69583
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0c074d43..da8b7a29 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -450,6 +450,29 @@ All notable changes to SkillEvaluator are documented in this file.
`Bedrock request failed`, like a Bedrock error that ends the judge and like
the default `legacy` policy.
+- The interactive Tier 3 progress view (`--progress rich`, or `auto` in a
+ terminal) now keeps one height for the whole run, so redraws no longer leave
+ repeated or jumbled table frames. The Provider and Run plan rows show a
+ placeholder until the plan resolves, each row is one line that ends in an
+ ellipsis when its value is too long, and stages fill a fixed number of rows,
+ at least six and one per agent, showing the most recently updated. The agent
+ models, provider and run plan are printed in full above the table whenever
+ they change, and the table left on screen when the run ends lists every
+ stage, with its full detail when the Detail column is at least 20 columns
+ wide. A stopping agent's detail now starts with the second Ctrl+C hint, so a
+ cut row still shows it. The view redraws only on progress events and
+ heartbeats, no longer also from Rich's own refresh thread. If drawing fails,
+ including a terminal write that would block, the table keeps its last frame
+ with running stages still shown as running, a line says that the run
+ continues, and no further progress is shown for that run; the final results
+ still print. When the terminal also refuses that line, or the write that
+ shows the cursor again, a line saying that the run went on is printed and
+ the cursor is shown when the run ends. Frames the terminal refused are
+ dropped then, so they are never drawn over output printed since. Before, a
+ failure on the heartbeat thread ended that thread with a traceback and left
+ the view active without updating elapsed times, and a failure on the
+ caller's thread marked the running stages as failed.
+
- The Tier 3 terminal summary now shows `N/A`, not `NO SCORE`, as the Skill
Lift of an evaluator or dimension that is N/A in either arm, as the Markdown
and HTML reports already do. Most runs with a baseline hit this: Skill
diff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdx
index 744d0c71..8b4594ce 100644
--- a/docs/cli-reference.mdx
+++ b/docs/cli-reference.mdx
@@ -589,7 +589,7 @@ Use `skillevaluator tier3 PATH` for the complete workflow with automatic missing
| `--evaluated-source-repository TEXT` | unset | Repository (`owner/name`) of the source tree being evaluated, persisted into the run's `run_config.json`. |
| `--evaluated-source-revision TEXT` | unset | Immutable revision of the evaluated source: a full Git object id, or a `sha256`/`sha384`/`sha512` digest. A short prefix is refused. |
| `--evaluator-container-revision TEXT` | unset | Digest-pinned evaluator image reference recorded beside the evaluated source. |
-| `--progress [auto\|rich\|plain\|off]` | `auto` | Progress display for the run: `rich` live view, `plain` line output, or `off`. |
+| `--progress [auto\|rich\|plain\|off]` | `auto` | Progress display for the run: `rich` live view (a fixed-height table of the run plan and recent stages, with the full plan printed above it), `plain` line output, or `off`. |
| `--progress-interval SECONDS` | mode default | Seconds between progress heartbeats, from `0.05` to `86400`: how often the `rich` view refreshes elapsed times and how often `plain` output repeats a `still running` line for active stages. Defaults to 1 second for `rich` and 60 seconds for `plain`, including `auto` with redirected or CI output. |
Without a `--model`/`--agent-model` override, agent execution uses
diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdx
index 76792cc6..20d1a758 100644
--- a/docs/tier3-live-evaluation.mdx
+++ b/docs/tier3-live-evaluation.mdx
@@ -747,6 +747,24 @@ and each uses the full `--n-concurrent` budget. For example, two agents with
`--n-concurrent 4` run up to eight trials at once, fewer when fewer cases
remain.
+In a terminal, `--progress auto` uses the `rich` view: one table of the run
+plan and its stages that keeps the same height for the whole run. Each row is
+one line, and a long value ends in an ellipsis instead of wrapping. The agent
+models, provider and run plan are also printed in full above the table
+whenever they change. The table shows the most recently updated stages in a
+fixed number of rows, at least six and at least one per agent. When the run
+ends, the table left on screen lists every stage. Its details wrap in full
+when the terminal leaves the Detail column at least 20 columns; in a narrower
+terminal each row stays on one line. The view redraws only when a stage
+changes or a heartbeat updates elapsed times. If drawing fails, including a
+terminal write that would block, the table stops updating with its stages as
+they were, a line says that the run continues, and no further progress is
+shown for that run; the final results still print. When the terminal also
+refuses that line, or the write that shows the cursor again, a line saying
+that the run went on is printed and the cursor is shown when the run ends.
+Frames the terminal refused are dropped then, so they are never drawn over
+output printed since.
+
Every `tier3 evaluate` flag, including resource overrides and Dockerfile
modes, is in the [CLI Reference](cli-reference.mdx).
diff --git a/src/skillevaluator/tier3/harbor/progress.py b/src/skillevaluator/tier3/harbor/progress.py
index 533b67a0..c6730e79 100644
--- a/src/skillevaluator/tier3/harbor/progress.py
+++ b/src/skillevaluator/tier3/harbor/progress.py
@@ -15,7 +15,7 @@ import math
import re
import sys
import threading
-from collections.abc import Mapping
+from collections.abc import Callable, Mapping
from dataclasses import dataclass
from time import monotonic
from typing import Literal, Protocol, TextIO, runtime_checkable
@@ -51,6 +51,22 @@ MIN_REFRESH_INTERVAL_SECONDS = 0.05
# One day stays below every platform's threading.TIMEOUT_MAX, beyond which the
# heartbeat thread's wait would raise OverflowError.
MAX_REFRESH_INTERVAL_SECONDS = 86400.0
+# The Rich frame keeps at least this many stage rows (more when more agents
+# run), padding with blank rows, so its height never changes mid-run. A frame
+# that grows or shrinks between redraws leaves stale or jumbled lines behind.
+_LIVE_MIN_EVENT_ROWS = 6
+_LIVE_STATE_COLUMN_WIDTH = 12
+# The rounded table's five borders and one space of padding on each side of
+# its four columns.
+_LIVE_TABLE_CHROME_WIDTH = 5 + 4 * 2
+# The final frame wraps long details only when its Detail column is at least
+# this wide. In a narrower column every detail would fold into a tall stack of
+# short pieces, so there the rows stay cut on one line as while running.
+_FINAL_DETAIL_MIN_WIDTH = 20
+_DISPLAY_STOPPED_NOTICE = "Tier 3 progress display stopped after a drawing error; the run continues."
+# Printed when the run ends instead, when the terminal refused the line above
+# or Live's stop.
+_DISPLAY_STOPPED_LATE_NOTICE = "Tier 3 progress display stopped after a drawing error; the run went on without it."
@dataclass(frozen=True, slots=True)
@@ -84,6 +100,14 @@ class Tier3RunPlan:
report_path: str | None = None
+@dataclass(frozen=True, slots=True)
+class _FailedDisplayStop:
+ """What reached the terminal when the live view stopped after a drawing failure."""
+
+ stop_refused: bool
+ notice_shown: bool
+
+
@dataclass(frozen=True, slots=True)
class ProgressEvent:
"""A structured progress transition emitted by the Tier 3 engine."""
@@ -180,6 +204,7 @@ class PlainProgressReporter:
self._plan_rendered = False
self._last_progress_at: float | None = None
self._last_events: dict[str, tuple[str, str]] = {}
+ self._failure_callback: Callable[[], None] | None = None
@property
def is_active(self) -> bool:
@@ -208,6 +233,10 @@ class PlainProgressReporter:
with self._lock:
self._secret_values.update(value for value in values if value)
+ def set_failure_callback(self, callback: Callable[[], None] | None) -> None:
+ """Notify a safety wrapper when the reporter's own heartbeat thread fails."""
+ self._failure_callback = callback
+
def emit(self, event: ProgressEvent) -> None:
with self._lock:
detail = self._event_detail(event)
@@ -235,16 +264,31 @@ class PlainProgressReporter:
self._write_line(f"[{self._elapsed()}] still running: {stages}")
def close(self) -> None:
+ self._close(after_failure=False)
+
+ def close_after_failure(self) -> None:
+ """Stop after a presentation failure without marking running stages failed.
+
+ Only the display failed. The run goes on, so a stage that is still
+ running has not been interrupted.
+ """
+ self._close(after_failure=True)
+
+ def _close(self, *, after_failure: bool) -> None:
try:
with self._lock:
- for stage in tuple(self._active):
- self.emit(
- ProgressEvent(
- stage=stage,
- state="failed",
- detail="interrupted before completion",
+ # Decided under the same lock hold that would mark the stages,
+ # so an update that arrives while a failed display stops is
+ # never reported as interrupted.
+ if not after_failure:
+ for stage in tuple(self._active):
+ self.emit(
+ ProgressEvent(
+ stage=stage,
+ state="failed",
+ detail="interrupted before completion",
+ )
)
- )
finally:
self._stop.set()
thread = self._thread
@@ -315,7 +359,28 @@ class PlainProgressReporter:
delay = max(MIN_REFRESH_INTERVAL_SECONDS, delay - (monotonic() - self._last_progress_at))
if self._stop.wait(delay):
return
- self.heartbeat()
+ try:
+ self.heartbeat()
+ except Exception:
+ # A redraw failure here must turn progress off through the
+ # safety wrapper, which stops the live display, instead of
+ # killing this thread and leaving a half-drawn frame behind.
+ # That includes a write a non-blocking terminal refuses: Rich
+ # keeps the refused frame and sends it again before the next
+ # one, so skipping it would only grow that backlog.
+ logger.debug("Tier 3 progress heartbeat failed", exc_info=True)
+ if self._stop.is_set():
+ # close() is already stopping this reporter.
+ return
+ callback = self._failure_callback
+ if callback is None:
+ self._stop.set()
+ return
+ try:
+ callback()
+ except Exception:
+ logger.debug("Tier 3 progress failure callback failed", exc_info=True)
+ return
def _safe_text(self, value: object) -> str:
return redact_progress_detail(value, secret_values=self._secret_values)
@@ -334,6 +399,22 @@ class PlainProgressReporter:
self._stop.set()
+class _FinalLiveFrame:
+ """The last Rich frame, which stays in the scrollback once Live stops.
+
+ It draws ``wrapped`` when the width it is drawn at leaves the Detail column
+ at least ``_FINAL_DETAIL_MIN_WIDTH`` wide, and ``cut`` otherwise.
+ """
+
+ def __init__(self, *, wrapped: object, cut: object, min_width: int) -> None:
+ self.wrapped = wrapped
+ self.cut = cut
+ self.min_width = min_width
+
+ def __rich_console__(self, _console, options): # rich renderable protocol
+ yield self.wrapped if options.max_width >= self.min_width else self.cut
+
+
class RichProgressReporter(PlainProgressReporter):
"""Interactive Rich Live table selected for terminal output."""
@@ -352,6 +433,20 @@ class RichProgressReporter(PlainProgressReporter):
self._live = None
self._live_plan: Tier3RunPlan | None = None
self._live_events: dict[str, tuple[ProgressEvent, str | None]] = {}
+ self._live_event_slots = _LIVE_MIN_EVENT_ROWS
+ self._printed_plan_lines: dict[str, str] = {}
+ # Set, under the same lock hold that stops Live, when a drawing failure
+ # stopped it. The run's close() finishes what the terminal refused.
+ self._failed_stop: _FailedDisplayStop | None = None
+
+ def start_progress(self) -> None:
+ with self._lock:
+ if self._started_at is None and self._failed_stop is not None:
+ # The run whose display failed was never closed. Its refused
+ # writes and its notice do not belong to this run.
+ self._failed_stop = None
+ self._drop_refused_writes()
+ super().start_progress()
def emit(self, event: ProgressEvent) -> None:
with self._lock:
@@ -367,6 +462,8 @@ class RichProgressReporter(PlainProgressReporter):
else:
self._active.pop(event.stage, None)
finished_at = self._elapsed()
+ # Move the stage to the end so the recent-event window shows it.
+ self._live_events.pop(event.stage, None)
self._live_events[event.stage] = (safe_event, finished_at)
self._refresh_live()
@@ -376,77 +473,141 @@ class RichProgressReporter(PlainProgressReporter):
if self._active:
self._refresh_live()
- def close(self) -> None:
+ def _close(self, *, after_failure: bool) -> None:
live = self._live
try:
- super().close()
+ super()._close(after_failure=after_failure)
finally:
with self._lock:
- if live is not None and live is self._live:
- try:
- live.stop()
- finally:
+ try:
+ if live is not None and live is self._live:
+ if after_failure:
+ self._stop_live_after_failure(live)
+ else:
+ try:
+ # Once Live stops, its last frame stays in the
+ # scrollback and no longer needs a fixed
+ # height, so it lists every stage in full.
+ live.update(self._build_live_table(final=True), refresh=False)
+ finally:
+ live.stop()
+ finally:
+ # Reset even when Live never started, so a failed start
+ # leaves no plan or rows behind for the next run.
+ if live is self._live:
self._live = None
- self._live_plan = None
- self._live_events.clear()
+ self._live_plan = None
+ self._live_events.clear()
+ self._live_event_slots = _LIVE_MIN_EVENT_ROWS
+ self._printed_plan_lines.clear()
+ if not after_failure:
+ failed_stop, self._failed_stop = self._failed_stop, None
+ if failed_stop is not None:
+ self._finish_failed_stop(failed_stop)
+
+ def _stop_live_after_failure(self, live) -> None:
+ """Stop the live view after a drawing failure and say that the run goes on.
+
+ The full final frame could fail the same way, so Live keeps the last
+ frame it drew. Runs under the reporter lock and records the outcome in
+ the same lock hold, so the run's close() either stops Live itself or
+ finds the record.
+ """
+ try:
+ live.stop()
+ except Exception:
+ # A terminal that refused the redraw can refuse Live's stop too,
+ # and with it the write that shows the cursor again.
+ logger.debug("Tier 3 progress display stop failed", exc_info=True)
+ self._failed_stop = _FailedDisplayStop(stop_refused=True, notice_shown=False)
+ return
+ try:
+ # Live keeps the last frame it drew, with its stages still running,
+ # so explain why it no longer updates.
+ self._write_line(_DISPLAY_STOPPED_NOTICE)
+ except Exception:
+ logger.debug("Tier 3 progress stop notice failed", exc_info=True)
+ self._failed_stop = _FailedDisplayStop(stop_refused=False, notice_shown=False)
+ else:
+ self._failed_stop = _FailedDisplayStop(stop_refused=False, notice_shown=True)
+
+ def _drop_refused_writes(self) -> str:
+ """Discard the output this thread's Rich console holds from refused writes, and return it.
+
+ Rich keeps a refused write for its thread and sends it before that
+ thread's next one. A refused frame or stop starts by moving the cursor
+ up over the frame that was on screen when drawing failed, so sent any
+ later it would erase and overwrite whatever was printed since. A
+ thread that ended took its refused writes with it.
+ """
+ try:
+ self._console.begin_capture()
+ return self._console.end_capture()
+ except Exception:
+ logger.debug("Tier 3 progress refused output could not be dropped", exc_info=True)
+ return ""
+
+ def _finish_failed_stop(self, failed_stop: _FailedDisplayStop) -> None:
+ """Show the cursor and the stop notice when the terminal refused them after a drawing failure.
+
+ Runs on the thread that closes the run, once the terminal may take
+ writes again.
+ """
+ from rich.control import Control
+
+ dropped = self._drop_refused_writes()
+ try:
+ if failed_stop.stop_refused:
+ # Live hides the cursor while it draws and shows it again when
+ # it stops, then ends the frame's last line.
+ self._console.show_cursor(True)
+ if self._console.is_terminal:
+ self._console.line()
+ elif str(Control.show_cursor(True)) in dropped:
+ # Live.start() stops Live itself when its first frame fails,
+ # so the stop above had nothing left to do. The terminal
+ # refused that earlier stop too, and its write that shows the
+ # cursor was just dropped. No frame reached the terminal, so
+ # there is no last line to end.
+ self._console.show_cursor(True)
+ if not failed_stop.notice_shown:
+ self._write_line(_DISPLAY_STOPPED_LATE_NOTICE)
+ except Exception:
+ logger.debug("Tier 3 progress terminal restore failed", exc_info=True)
def _render_plan(self, plan: Tier3RunPlan) -> None:
self._live_plan = plan
+ # The frame cuts long rows to keep its height, so each plan line is
+ # also printed in full above it whenever that line changes.
+ for label, text in self._plan_lines(plan):
+ if self._printed_plan_lines.get(label) != text:
+ self._printed_plan_lines[label] = text
+ self._write_line(f"{label}: {text}")
table = self._build_live_table()
if self._live is None:
self._live = self._live_factory(
table,
console=self._console,
- auto_refresh=True,
- refresh_per_second=8,
+ # Rich never redraws on its own timer. The frame is redrawn when
+ # the plan or a stage changes, on each heartbeat, and when other
+ # output is printed above it.
+ auto_refresh=False,
transient=False,
)
self._live.start(refresh=True)
else:
self._live.update(table, refresh=True)
- def _build_live_table(self):
- from rich import box
- from rich.spinner import Spinner
- from rich.table import Table
- from rich.text import Text
-
- plan = self._live_plan
- if plan is None:
- return Table(title="Harbor Run Configuration")
- if plan.with_skill is False:
- mode_label = "Baseline-only evaluation"
- else:
- mode_label = "Skill-lift evaluation" if plan.baseline is not False else "Agent evaluation"
- title = Text.assemble(
- ("Harbor Run Configuration", f"bold {TEXT}"),
- " · ",
- (self._safe_text(plan.skill_name), "bold"),
- " · ",
- (mode_label, MUTED),
- )
- table = Table(title=title, box=box.ROUNDED, border_style=FAINT, expand=True, show_lines=False)
- table.add_column("Stage", style="bold", no_wrap=True)
- table.add_column("State", width=12, no_wrap=True)
- table.add_column("Detail", ratio=1)
- table.add_column("Elapsed", justify="right", no_wrap=True)
-
- table.add_row("Environment", Text("configured", style=TEXT), Text(self._safe_text(plan.environment)), "")
+ def _plan_agents_text(self, plan: Tier3RunPlan) -> str:
model_map = dict(plan.agent_models)
- agents = ", ".join(
+ return ", ".join(
f"{self._safe_text(agent)}={self._safe_text(model_map[agent])}"
if agent in model_map
else self._safe_text(agent)
for agent in plan.agents
)
- table.add_row("Agents / models", Text("configured", style=TEXT), Text(agents or "none"), "")
- if plan.provider:
- table.add_row(
- "Provider",
- Text("configured", style=TEXT),
- Text(self._safe_text(plan.provider)),
- "",
- )
+
+ def _plan_values_text(self, plan: Tier3RunPlan) -> str:
values = (
("Tasks", plan.task_count),
("Cases", plan.case_count),
@@ -464,9 +625,68 @@ class RichProgressReporter(PlainProgressReporter):
None if plan.task_timeout_seconds is None else f"{plan.task_timeout_seconds:g}s",
),
)
- known = " · ".join(f"{label} {value}" for label, value in values if value is not None)
+ return " · ".join(f"{label} {value}" for label, value in values if value is not None)
+
+ def _plan_lines(self, plan: Tier3RunPlan) -> list[tuple[str, str]]:
+ lines = []
+ agents = self._plan_agents_text(plan)
+ if agents:
+ lines.append(("Agents / models", agents))
+ if plan.provider:
+ lines.append(("Provider", self._safe_text(plan.provider)))
+ known = self._plan_values_text(plan)
if known:
- table.add_row("Run plan", Text("configured", style=TEXT), Text(known), "")
+ lines.append(("Run plan", known))
+ return lines
+
+ def _build_live_table(self, *, final: bool = False):
+ from rich import box
+ from rich.cells import cell_len
+ from rich.spinner import Spinner
+ from rich.table import Table
+ from rich.text import Text
+
+ plan = self._live_plan
+ if plan is None:
+ return Table(title="Harbor Run Configuration")
+ if plan.with_skill is False:
+ mode_label = "Baseline-only evaluation"
+ else:
+ mode_label = "Skill-lift evaluation" if plan.baseline is not False else "Agent evaluation"
+ title = Text.assemble(
+ ("Harbor Run Configuration", f"bold {TEXT}"),
+ " · ",
+ (self._safe_text(plan.skill_name), "bold"),
+ " · ",
+ (mode_label, MUTED),
+ overflow="ellipsis",
+ no_wrap=True,
+ )
+ rows = [
+ ("Environment", Text("configured", style=TEXT), Text(self._safe_text(plan.environment)), ""),
+ ]
+ agents = self._plan_agents_text(plan)
+ rows.append(("Agents / models", Text("configured", style=TEXT), Text(agents or "none"), ""))
+
+ def placeholder(waiting_detail: str) -> tuple[Text, Text]:
+ # The row is always present so a plan update does not add rows. A
+ # finished run's frame stays on screen, so there it no longer says
+ # the value is still on its way.
+ if final:
+ return Text("-", style=MUTED), Text("not resolved", style=MUTED)
+ return Text("pending", style=MUTED), Text(waiting_detail)
+
+ if plan.provider:
+ provider_state, provider_detail = Text("configured", style=TEXT), Text(self._safe_text(plan.provider))
+ else:
+ provider_state, provider_detail = placeholder("resolving")
+ rows.append(("Provider", provider_state, provider_detail, ""))
+ known = self._plan_values_text(plan)
+ if known:
+ plan_state, plan_detail = Text("configured", style=TEXT), Text(known)
+ else:
+ plan_state, plan_detail = placeholder("preparing")
+ rows.append(("Run plan", plan_state, plan_detail, ""))
state_styles = {
"ready": GREEN,
"complete": GREEN,
@@ -475,7 +695,14 @@ class RichProgressReporter(PlainProgressReporter):
"delegated": AMBER,
"skipped": "dim",
}
- for stage, (event, finished_at) in self._live_events.items():
+ # Show only the most recently updated stages in a fixed number of rows.
+ # The slot count only grows (to fit every agent), so the frame never
+ # shrinks either. The final frame lists every stage, without padding.
+ self._live_event_slots = max(self._live_event_slots, len(plan.agents))
+ event_slots = self._live_event_slots
+ events = list(self._live_events.items())
+ shown_events = events if final else events[-event_slots:]
+ for stage, (event, finished_at) in shown_events:
label = (
f"Agent {stage.removeprefix('agent:')}"
if stage.startswith("agent:")
@@ -486,20 +713,60 @@ class RichProgressReporter(PlainProgressReporter):
if event.state == "running"
else Text(event.state, style=state_styles.get(event.state, "white"))
)
- table.add_row(
- Text(label),
- state,
- Text(event.detail or ""),
- Text(finished_at or self._elapsed()),
+ rows.append(
+ (
+ Text(label),
+ state,
+ Text(event.detail or ""),
+ Text(finished_at or self._elapsed()),
+ )
)
- return table
+ if not final:
+ rows.extend(("", "", "", "") for _ in range(event_slots - len(shown_events)))
+
+ def table(*, wrap_detail: bool) -> Table:
+ built = Table(title=title, box=box.ROUNDED, border_style=FAINT, expand=True, show_lines=False)
+ # One line per row: long values are cut with an ellipsis rather than
+ # wrapped, so the frame height does not depend on the text. Only the
+ # final frame may wrap its details.
+ built.add_column("Stage", style="bold", no_wrap=True, overflow="ellipsis")
+ built.add_column("State", width=_LIVE_STATE_COLUMN_WIDTH, no_wrap=True, overflow="ellipsis")
+ if wrap_detail:
+ built.add_column("Detail", ratio=1, overflow="fold")
+ else:
+ built.add_column("Detail", ratio=1, no_wrap=True, overflow="ellipsis")
+ built.add_column("Elapsed", justify="right", no_wrap=True, overflow="ellipsis")
+ for row in rows:
+ built.add_row(*row)
+ return built
+
+ if not final:
+ return table(wrap_detail=False)
+ # The final frame stays in the scrollback, so its details wrap in full
+ # where the Detail column, which gets the width the other columns
+ # leave, is wide enough.
+ stage_width = max(cell_len(str(cell)) for cell in ["Stage", *(row[0] for row in rows)])
+ elapsed_width = max(cell_len(str(cell)) for cell in ["Elapsed", *(row[3] for row in rows)])
+ return _FinalLiveFrame(
+ wrapped=table(wrap_detail=True),
+ cut=table(wrap_detail=False),
+ min_width=(
+ _LIVE_TABLE_CHROME_WIDTH
+ + stage_width
+ + _LIVE_STATE_COLUMN_WIDTH
+ + elapsed_width
+ + _FINAL_DETAIL_MIN_WIDTH
+ ),
+ )
def _refresh_live(self) -> None:
if self._live is not None:
self._live.update(self._build_live_table(), refresh=True)
def _write_line(self, line: str) -> None:
- self._console.print(line, markup=False, soft_wrap=True)
+ # Printed as is: model IDs and providers can contain ":name:", which
+ # Rich would otherwise replace with an emoji.
+ self._console.print(line, markup=False, highlight=False, emoji=False, soft_wrap=True)
class NullProgressReporter:
@@ -532,6 +799,19 @@ class SafeProgressReporter:
self._reporter = reporter
self._disabled = False
self._started = False
+ # start() holds this lock while the reporter draws a plan, so a failure
+ # reported from the reporter's heartbeat thread waits for that plan
+ # update and then stops what it drew, instead of letting it start a new
+ # live view after progress was turned off.
+ self._lock = threading.RLock()
+ # A reporter's own heartbeat thread reports failures here, so a redraw
+ # error there turns progress off just like one on the caller's thread.
+ try:
+ set_failure_callback = getattr(reporter, "set_failure_callback", None)
+ if callable(set_failure_callback):
+ set_failure_callback(self._disable)
+ except Exception:
+ logger.debug("Tier 3 progress reporter failure callback registration failed", exc_info=True)
@property
def is_active(self) -> bool:
@@ -544,8 +824,18 @@ class SafeProgressReporter:
return self._started
def start(self, plan: Tier3RunPlan) -> None:
- self._started = True
- self._call(self._reporter.start, plan)
+ with self._lock:
+ if self._disabled:
+ return
+ self._started = True
+ try:
+ self._reporter.start(plan)
+ return
+ except Exception:
+ logger.debug("Tier 3 progress reporter disabled after presentation failure", exc_info=True)
+ # Closed outside the lock: closing joins the heartbeat thread, which
+ # may be waiting for the lock to report its own failure.
+ self._disable()
def set_secret_values(self, values: list[str] | tuple[str, ...] | set[str]) -> None:
self._call(self._reporter.set_secret_values, values)
@@ -557,13 +847,16 @@ class SafeProgressReporter:
self._call(self._reporter.heartbeat)
def close(self) -> None:
+ # The lock is not held here: closing joins the heartbeat thread, which
+ # may be waiting for the lock to report a failure.
try:
self._reporter.close()
except Exception:
logger.debug("Tier 3 progress reporter cleanup failed", exc_info=True)
finally:
- self._started = False
- self._disabled = False
+ with self._lock:
+ self._started = False
+ self._disabled = False
def _call(self, callback, *args: object) -> None:
if self._disabled:
@@ -575,11 +868,23 @@ class SafeProgressReporter:
self._disable()
def _disable(self) -> None:
- if self._disabled:
- return
- self._disabled = True
+ with self._lock:
+ if self._disabled:
+ return
+ self._disabled = True
+ # A disabled reporter shows nothing, so it is no longer active.
+ self._started = False
+ # Only the display failed, so a reporter that can stop without marking
+ # its running stages failed does so.
try:
- self._reporter.close()
+ close_after_failure = getattr(self._reporter, "close_after_failure", None)
+ except Exception:
+ close_after_failure = None
+ try:
+ if callable(close_after_failure):
+ close_after_failure()
+ else:
+ self._reporter.close()
except Exception:
logger.debug("Tier 3 progress reporter cleanup failed", exc_info=True)
diff --git a/src/skillevaluator/tier3/harbor/runner.py b/src/skillevaluator/tier3/harbor/runner.py
index f920efe0..2f57157b 100644
--- a/src/skillevaluator/tier3/harbor/runner.py
+++ b/src/skillevaluator/tier3/harbor/runner.py
@@ -842,6 +842,13 @@ _HARBOR_RUN_TERMINATE_SECONDS = 0.1
# waits this long for that cleanup before it kills the process group.
_HARBOR_RUN_CANCEL_GRACE_SECONDS = 30.0
_HARBOR_RUN_CANCEL_POLL_SECONDS = 0.1
+# Progress detail for an agent whose Harbor jobs are stopping. The hint comes
+# first and is short, so the Rich progress row, which is cut to the terminal
+# width, still shows it at 80 columns next to the longest built-in stage label,
+# with columns to spare.
+_HARBOR_RUN_STOPPING_DETAIL = (
+ f"2nd Ctrl+C skips wait; stopping Harbor jobs, up to {_HARBOR_RUN_CANCEL_GRACE_SECONDS:g}s of environment cleanup"
+)
_HARBOR_RUN_REAP_SECONDS = 5.0
@@ -6582,10 +6589,7 @@ def _run_harbor_eval_impl(
ProgressEvent(
stage=f"agent:{agent}",
state="running",
- detail=(
- "stopping Harbor jobs; press Ctrl+C again to skip up to "
- f"{_HARBOR_RUN_CANCEL_GRACE_SECONDS:g}s of environment cleanup"
- ),
+ detail=_HARBOR_RUN_STOPPING_DETAIL,
)
)
wait(running, timeout=_HARBOR_RUN_CANCEL_POLL_SECONDS)
diff --git a/tests/test_tier3_progress.py b/tests/test_tier3_progress.py
index f17647a0..6a874c7c 100644
--- a/tests/test_tier3_progress.py
+++ b/tests/test_tier3_progress.py
@@ -272,6 +272,481 @@ def test_rich_reporter_owns_live_table_and_updates_stage_rows(monkeypatch: pytes
assert reporter.is_active is False
+def _render(renderable: Any, *, width: int = 80) -> str:
+ from rich.console import Console
+
+ stream = io.StringIO()
+ Console(file=stream, width=width, color_system=None).print(renderable)
+ return stream.getvalue()
+
+
+def _rendered_live_lines(reporter: Any, *, width: int = 80) -> list[str]:
+ return _render(reporter._build_live_table(), width=width).splitlines()
+
+
+def test_rich_progress_frame_height_stays_fixed_as_plan_and_events_grow() -> None:
+ progress = _progress_module()
+ reporter = progress.RichProgressReporter(stream=io.StringIO(), refresh_interval=60)
+ reporter._live_plan = progress.Tier3RunPlan(
+ skill_name="demo",
+ environment="docker",
+ agents=("codex",),
+ agent_models=(("codex", "example/model"),),
+ )
+
+ initial_lines = _rendered_live_lines(reporter)
+ reporter._live_plan = progress.Tier3RunPlan(
+ skill_name="demo",
+ environment="docker",
+ agents=("codex",),
+ agent_models=(("codex", "example/model"),),
+ provider="example-provider",
+ task_count=2,
+ case_count=1,
+ attempts=1,
+ baseline=True,
+ concurrency=4,
+ timeout_multiplier=2,
+ matrix_trials=2,
+ total_containers=2,
+ )
+ for index in range(10):
+ reporter.emit(
+ progress.ProgressEvent(
+ stage=f"stage-{index}",
+ state="complete",
+ detail=f"completed stage {index} " + "with a detail too long for one line " * 3,
+ )
+ )
+ populated_lines = _rendered_live_lines(reporter)
+
+ assert len(populated_lines) == len(initial_lines)
+ assert "example-provider" in "\n".join(populated_lines)
+ assert "…" in "\n".join(populated_lines)
+
+
+def test_rich_progress_title_does_not_change_frame_height_in_a_narrow_terminal() -> None:
+ progress = _progress_module()
+ reporter = progress.RichProgressReporter(stream=io.StringIO(), refresh_interval=60)
+ reporter._live_plan = progress.Tier3RunPlan(
+ skill_name="a-skill-with-a-very-long-name",
+ environment="docker",
+ agents=tuple(f"agent-{index}" for index in range(8)),
+ )
+
+ initial_height = len(_rendered_live_lines(reporter, width=20))
+ reporter._live_plan = progress.Tier3RunPlan(
+ skill_name="a-skill-with-a-very-long-name",
+ environment="docker",
+ agents=("agent-0",),
+ provider="example-provider",
+ task_count=2,
+ case_count=1,
+ attempts=1,
+ baseline=True,
+ concurrency=4,
+ timeout_multiplier=2,
+ total_containers=2,
+ )
+ for index in range(10):
+ reporter.emit(
+ progress.ProgressEvent(
+ stage=f"stage-{index}",
+ state="complete",
+ detail=f"completed stage {index}",
+ )
+ )
+
+ assert len(_rendered_live_lines(reporter, width=20)) == initial_height
+
+
+def test_rich_progress_frame_does_not_shrink_when_the_agent_set_shrinks() -> None:
+ progress = _progress_module()
+ reporter = progress.RichProgressReporter(stream=io.StringIO(), refresh_interval=60)
+
+ reporter._live_plan = progress.Tier3RunPlan(
+ skill_name="demo",
+ environment="docker",
+ agents=tuple(f"agent-{index}" for index in range(8)),
+ )
+ initial_height = len(_rendered_live_lines(reporter))
+ reporter._live_plan = progress.Tier3RunPlan(skill_name="demo", environment="docker", agents=("agent-0",))
+
+ assert len(_rendered_live_lines(reporter)) == initial_height
+
+
+def test_rich_progress_uses_manual_refresh_and_recent_event_window(monkeypatch: pytest.MonkeyPatch) -> None:
+ import rich.live
+
+ progress = _progress_module()
+ _FakeLive.instances.clear()
+ monkeypatch.setattr(rich.live, "Live", _FakeLive)
+ reporter = progress.RichProgressReporter(stream=_TTYBuffer(), refresh_interval=60)
+ reporter.start(_plan(progress))
+
+ for index in range(7):
+ reporter.emit(
+ progress.ProgressEvent(
+ stage=f"stage-{index}",
+ state="complete",
+ detail=f"completed stage {index}",
+ )
+ )
+ reporter.emit(progress.ProgressEvent(stage="stage-0", state="running", detail="active again"))
+
+ live = _FakeLive.instances[0]
+ text = _render(live.renderables[-1])
+ # Rich's own refresh thread would redraw outside the reporter lock.
+ assert live.kwargs["auto_refresh"] is False
+ assert "refresh_per_second" not in live.kwargs
+ # The six most recently updated stages stay visible; the oldest drops out.
+ assert "active again" in text
+ assert all(f"completed stage {index}" in text for index in range(2, 7))
+ assert "completed stage 1" not in text
+ reporter.close()
+
+
+def test_rich_progress_resets_state_when_live_initialization_fails() -> None:
+ progress = _progress_module()
+ reporter = progress.RichProgressReporter(stream=io.StringIO(), refresh_interval=60)
+
+ def _fail_live_initialization(*_args, **_kwargs):
+ raise RuntimeError("terminal initialization failed")
+
+ reporter._live_factory = _fail_live_initialization
+ plan = progress.Tier3RunPlan(
+ skill_name="demo",
+ environment="docker",
+ agents=tuple(f"agent-{index}" for index in range(8)),
+ )
+ with pytest.raises(RuntimeError, match="terminal initialization failed"):
+ reporter.start(plan)
+ reporter.close()
+
+ assert reporter._live is None
+ assert reporter._live_plan is None
+ assert reporter._live_events == {}
+ assert reporter._live_event_slots == progress._LIVE_MIN_EVENT_ROWS
+ # A stale record of printed plan lines would hide the next run's plan.
+ assert reporter._printed_plan_lines == {}
+
+
+def test_rich_progress_prints_the_full_plan_above_the_fixed_frame(monkeypatch: pytest.MonkeyPatch) -> None:
+ import rich.live
+
+ progress = _progress_module()
+ _FakeLive.instances.clear()
+ monkeypatch.setattr(rich.live, "Live", _FakeLive)
+ monkeypatch.setenv("COLUMNS", "80")
+ output = _TTYBuffer()
+ reporter = progress.RichProgressReporter(stream=output, refresh_interval=60)
+ agents = ("claude-code", "codex")
+ options = {"attempts": 3, "baseline": True, "with_skill": True, "concurrency": 4, "max_agents": 2}
+ reporter.start(
+ progress.Tier3RunPlan(
+ skill_name="pdf-processing",
+ environment="docker",
+ agents=agents,
+ timeout_multiplier=1.5,
+ **options,
+ )
+ )
+ resolved = progress.Tier3RunPlan(
+ skill_name="pdf-processing",
+ environment="docker",
+ agents=agents,
+ agent_models=(("claude-code", "anthropic/claude-sonnet-4-5"), ("codex", "openai/gpt-5.1-codex")),
+ provider="nv_build",
+ task_count=3,
+ case_count=12,
+ timeout_multiplier=1.5,
+ matrix_trials=72,
+ preflight_trials=2,
+ total_containers=74,
+ task_timeout_seconds=1350,
+ **options,
+ )
+ reporter.start(resolved)
+ # An unchanged plan update prints nothing again.
+ reporter.start(resolved)
+
+ live = _FakeLive.instances[0]
+ frame_heights = {len(_render(renderable).splitlines()) for renderable in live.renderables}
+ printed = output.getvalue().splitlines()
+ reporter.close()
+
+ assert printed == [
+ "Agents / models: claude-code, codex",
+ "Run plan: Attempts 3 · With skill yes · Baseline yes · Concurrency 4 · Max agents 2 · Timeout 1.5x",
+ "Agents / models: claude-code=anthropic/claude-sonnet-4-5, codex=openai/gpt-5.1-codex",
+ "Provider: nv_build",
+ "Run plan: Tasks 3 · Cases 12 · Attempts 3 · With skill yes · Baseline yes · Concurrency 4 · "
+ "Max agents 2 · Timeout 1.5x · Matrix trials 72 · Preflight trials 2 · Containers 74 · "
+ "Max scaled top-level agent timeout 1350s",
+ ]
+ # The frame below keeps one height at 80 columns while the plan resolves.
+ assert len(frame_heights) == 1
+
+
+def test_rich_progress_prints_colon_delimited_plan_values_verbatim(monkeypatch: pytest.MonkeyPatch) -> None:
+ import rich.live
+
+ progress = _progress_module()
+ _FakeLive.instances.clear()
+ monkeypatch.setattr(rich.live, "Live", _FakeLive)
+ output = _TTYBuffer()
+ reporter = progress.RichProgressReporter(stream=output, refresh_interval=60)
+ # Rich would read ":llama:" and ":rocket:" as emoji codes.
+ reporter.start(
+ progress.Tier3RunPlan(
+ skill_name="demo",
+ environment="docker",
+ agents=("opencode",),
+ agent_models=(("opencode", "ollama:llama:3.1-8b"),),
+ provider="custom:rocket:proxy",
+ )
+ )
+ printed = output.getvalue().splitlines()
+ reporter.close()
+
+ assert printed == [
+ "Agents / models: opencode=ollama:llama:3.1-8b",
+ "Provider: custom:rocket:proxy",
+ ]
+
+
+def test_rich_final_frame_marks_a_plan_that_never_resolved(monkeypatch: pytest.MonkeyPatch) -> None:
+ import rich.live
+
+ progress = _progress_module()
+ _FakeLive.instances.clear()
+ monkeypatch.setattr(rich.live, "Live", _FakeLive)
+ reporter = progress.RichProgressReporter(stream=_TTYBuffer(), refresh_interval=60)
+ # The CLI's first plan has no provider or counts; then configuration fails.
+ reporter.start(progress.Tier3RunPlan(skill_name="demo", environment="docker", agents=("codex",)))
+ reporter.emit(
+ progress.ProgressEvent(
+ stage="configuration",
+ state="failed",
+ detail="A public LLM provider is required for live evaluation",
+ )
+ )
+ live = _FakeLive.instances[0]
+ running_frame = _render(live.renderables[-1], width=180)
+ reporter.close()
+ final_frame = _render(live.renderables[-1], width=180)
+
+ assert "resolving" in running_frame
+ assert "preparing" in running_frame
+ provider_row = next(line for line in final_frame.splitlines() if "Provider" in line)
+ plan_row = next(line for line in final_frame.splitlines() if "Run plan" in line)
+ assert "not resolved" in provider_row
+ assert "not resolved" in plan_row
+ assert not any(word in final_frame for word in ("pending", "resolving", "preparing"))
+ assert "A public LLM provider is required for live evaluation" in final_frame
+ assert live.stop_calls == 1
+
+
+def test_rich_final_frame_lists_every_stage(monkeypatch: pytest.MonkeyPatch) -> None:
+ import rich.live
+
+ progress = _progress_module()
+ _FakeLive.instances.clear()
+ monkeypatch.setattr(rich.live, "Live", _FakeLive)
+ reporter = progress.RichProgressReporter(stream=_TTYBuffer(), refresh_interval=60)
+ reporter.start(_plan(progress))
+ reporter.emit(progress.ProgressEvent(stage="docker-images", state="skipped", detail="local environment selected"))
+ for index in range(7):
+ reporter.emit(
+ progress.ProgressEvent(
+ stage=f"agent:agent-{index}",
+ state="complete",
+ detail=f"agent {index} finished",
+ )
+ )
+ reporter.emit(progress.ProgressEvent(stage="run-finished", state="complete", detail="Tier 3 evaluation finished"))
+ live = _FakeLive.instances[0]
+ running_frame = _render(live.renderables[-1], width=180)
+ reporter.close()
+ final_frame = _render(live.renderables[-1], width=180)
+
+ assert "local environment selected" not in running_frame
+ assert "agent 0 finished" not in running_frame
+ # Once the run is over the frame stays in the scrollback, so it keeps every outcome.
+ assert "local environment selected" in final_frame
+ assert all(f"agent {index} finished" in final_frame for index in range(7))
+ assert "Tier 3 evaluation finished" in final_frame
+ assert live.stop_calls == 1
+
+
+def _row_detail(frame: str, stage_label: str) -> str:
+ """Return a row's Detail cell, joining the lines it wraps onto."""
+ detail: list[str] = []
+ for line in frame.splitlines():
+ cells = [cell.strip() for cell in line.split("│")[1:-1]]
+ if len(cells) != 4:
+ continue
+ # A wrapped row continues on lines whose Stage cell is empty.
+ if cells[0] == stage_label or (detail and not cells[0]):
+ detail.append(cells[2])
+ elif detail:
+ break
+ return " ".join(part for part in detail if part)
+
+
+def test_rich_final_frame_wraps_details_the_running_frame_cuts(monkeypatch: pytest.MonkeyPatch) -> None:
+ import rich.live
+
+ progress = _progress_module()
+ _FakeLive.instances.clear()
+ monkeypatch.setattr(rich.live, "Live", _FakeLive)
+ reporter = progress.RichProgressReporter(stream=_TTYBuffer(), refresh_interval=60)
+ reporter.start(_plan(progress))
+ reason = "one or more Harbor jobs failed; inspect the retained artifacts and the trial logs"
+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="failed", detail=reason))
+ live = _FakeLive.instances[0]
+ running_frame = _render(live.renderables[-1])
+ reporter.close()
+ final_frame = _render(live.renderables[-1])
+
+ # At 80 columns the running frame cuts the reason to keep one line per row.
+ assert _row_detail(running_frame, "Agent codex") != reason
+ assert "…" in _row_detail(running_frame, "Agent codex")
+ # The final frame no longer needs a fixed height, so it shows the reason in full.
+ assert _row_detail(final_frame, "Agent codex") == reason
+
+
+def test_rich_final_frame_keeps_rows_on_one_line_where_the_detail_column_is_narrow(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ import rich.live
+
+ progress = _progress_module()
+ _FakeLive.instances.clear()
+ monkeypatch.setattr(rich.live, "Live", _FakeLive)
+ reporter = progress.RichProgressReporter(stream=_TTYBuffer(), refresh_interval=60)
+ reporter.start(_plan(progress))
+ reporter.emit(
+ progress.ProgressEvent(
+ stage="agent-runtime-preflight",
+ state="skipped",
+ detail="disabled by default; enable with --agent-runtime-preflight",
+ )
+ )
+ reason = "one or more Harbor jobs failed; inspect the retained artifacts and the trial logs"
+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="failed", detail=reason))
+ live = _FakeLive.instances[0]
+ reporter.close()
+ final = live.renderables[-1]
+
+ # Stage (23 characters for Agent Runtime Preflight), State and Elapsed leave
+ # the Detail column 20 characters wide at 76 columns, so the details wrap.
+ wide = _render(final, width=76)
+ assert _row_detail(wide, "Agent codex") == reason
+ assert max(len(line.split("│")[3]) - 2 for line in wide.splitlines() if line.count("│") == 5) == 20
+ # One column narrower, folding every detail into a tall stack of short
+ # pieces would leave the frame unreadable, so each row stays on one line.
+ for width in (75, 60):
+ narrow = _render(final, width=width)
+ rows = [line for line in narrow.splitlines() if line.count("│") == 5]
+ # The header, four plan rows and two stages.
+ assert len(rows) == 7, (width, narrow)
+ assert _row_detail(narrow, "Agent codex").endswith("…")
+ assert all(len(line) == width for line in narrow.splitlines() if line.strip())
+
+
+def _stopping_frame_reporter(progress: Any, output: io.StringIO, agents: tuple[str, ...], stopping_detail: str) -> Any:
+ """Return a Rich reporter whose stage sequence matches a run stopped by Ctrl+C."""
+ reporter = progress.RichProgressReporter(stream=output, refresh_interval=60)
+ plan = progress.Tier3RunPlan(
+ skill_name="pdf-processing",
+ environment="docker",
+ agents=agents,
+ agent_models=tuple((agent, f"provider/{agent}-large-model") for agent in agents),
+ provider="nv_build",
+ task_count=3,
+ case_count=12,
+ attempts=3,
+ baseline=True,
+ with_skill=True,
+ concurrency=4,
+ max_agents=2,
+ timeout_multiplier=1.5,
+ matrix_trials=72,
+ preflight_trials=0,
+ total_containers=72,
+ task_timeout_seconds=1350,
+ )
+ reporter.start(plan)
+ for stage in (
+ "configuration",
+ "model-resolution",
+ "environment-preflight",
+ "credential-validation",
+ "with-skill-tasks",
+ "baseline-tasks",
+ "case-selection",
+ "docker-images",
+ ):
+ reporter.emit(progress.ProgressEvent(stage=stage, state="complete", detail="done"))
+ # The longest built-in stage label stays in the recent-stage window.
+ reporter.emit(
+ progress.ProgressEvent(
+ stage="agent-runtime-preflight",
+ state="skipped",
+ detail="disabled by default; enable with --agent-runtime-preflight",
+ )
+ )
+ for agent in agents:
+ reporter.emit(progress.ProgressEvent(stage=f"agent:{agent}", state="running", detail="with-skill + baseline"))
+ for agent in agents:
+ reporter.emit(progress.ProgressEvent(stage=f"agent:{agent}", state="running", detail=stopping_detail))
+ return reporter
+
+
[email protected]("agents", [("claude-code", "codex"), ("mini-swe-agent",)])
+def test_rich_progress_at_80_columns_shows_the_ctrl_c_hint_and_the_full_plan(
+ monkeypatch: pytest.MonkeyPatch,
+ agents: tuple[str, ...],
+) -> None:
+ import rich.live
+
+ from skillevaluator.tier3.harbor import runner
+
+ progress = _progress_module()
+ _FakeLive.instances.clear()
+ monkeypatch.setattr(rich.live, "Live", _FakeLive)
+ monkeypatch.setenv("COLUMNS", "80")
+ stream = _TTYBuffer()
+ reporter = _stopping_frame_reporter(progress, stream, agents, runner._HARBOR_RUN_STOPPING_DETAIL)
+ output = stream.getvalue()
+ table = _FakeLive.instances[0].renderables[-1]
+ reporter.close()
+
+ # The frame cuts each row to the terminal width; the second Ctrl+C hint
+ # comes first in the detail and is short, so it is still readable. It
+ # still fits two columns narrower: a legacy Windows console uses one
+ # column less, and one is to spare.
+ hint = runner._HARBOR_RUN_STOPPING_DETAIL.split(";")[0]
+ assert "Ctrl+C" in hint
+ for width in (80, 78):
+ frame = _render(table, width=width)
+ agent_rows = [line for line in frame.splitlines() if line.startswith("│ Agent ") and "running" in line]
+ assert len(agent_rows) == len(agents)
+ assert all(hint in row for row in agent_rows), (width, agent_rows)
+ # Every plan value is printed in full above the frame.
+ assert all(f"{agent}=provider/{agent}-large-model" in output for agent in agents)
+ for value in (
+ "Concurrency 4",
+ "Max agents 2",
+ "Timeout 1.5x",
+ "Matrix trials 72",
+ "Containers 72",
+ "Max scaled top-level agent timeout 1350s",
+ ):
+ assert value in output
+
+
def test_rich_reporter_immediately_starts_one_live_box_and_keeps_stage_updates_inside_it(
monkeypatch: pytest.MonkeyPatch,
) -> None:
@@ -294,7 +769,13 @@ def test_rich_reporter_immediately_starts_one_live_box_and_keeps_stage_updates_i
Console(file=initial, force_terminal=False, width=180).print(live.renderables[0])
assert "Harbor Run Configuration" in initial.getvalue()
assert "demo-skill" in initial.getvalue()
- assert output.getvalue() == ""
+ # Only the full plan lines are printed above the box; no stage line is.
+ plan_lines = (
+ "Agents / models: codex=gpt-5, opencode=nvidia/llama\n"
+ "Provider: openai\n"
+ "Run plan: Tasks 2 · Cases 2 · Attempts 3 · Baseline yes · Concurrency 4 · Max agents 2 · Timeout 1.5x\n"
+ )
+ assert output.getvalue() == plan_lines
reporter.emit(progress.ProgressEvent(stage="agent-runtime-preflight", state="running"))
@@ -302,7 +783,7 @@ def test_rich_reporter_immediately_starts_one_live_box_and_keeps_stage_updates_i
Console(file=updated, force_terminal=False, width=180).print(live.renderables[-1])
assert "Agent Runtime Preflight" in updated.getvalue()
assert "running" in updated.getvalue()
- assert output.getvalue() == ""
+ assert output.getvalue() == plan_lines
reporter.close()
@@ -555,16 +1036,623 @@ def test_safe_reporter_disables_broken_presentation_without_raising() -> None:
reporter.set_secret_values(["secret"])
reporter.start(_plan(progress))
+ # A disabled reporter shows nothing, so starting it must not mark it active.
+ assert reporter.is_active is False
reporter.emit(progress.ProgressEvent(stage="configuration", state="running"))
reporter.heartbeat()
reporter.close()
assert broken.close_calls >= 1
reporter.start(_plan(progress))
+ assert reporter.is_active is False
reporter.close()
assert broken.start_calls == 1
+def test_safe_reporter_becomes_inactive_after_post_start_failure() -> None:
+ progress = _progress_module()
+
+ class FailingReporter:
+ active = False
+
+ @property
+ def is_active(self) -> bool:
+ return self.active
+
+ def start(self, _plan) -> None:
+ self.active = True
+
+ def set_secret_values(self, _values) -> None:
+ pass
+
+ def emit(self, _event) -> None:
+ raise RuntimeError("redraw failed")
+
+ def heartbeat(self) -> None:
+ pass
+
+ def close(self) -> None:
+ self.active = False
+
+ reporter = progress.safe_progress_reporter(FailingReporter())
+ reporter.start(_plan(progress))
+ assert reporter.is_active is True
+
+ reporter.emit(progress.ProgressEvent(stage="configuration", state="ready"))
+
+ assert reporter.is_active is False
+ reporter.close()
+
+
+def test_safe_reporter_disables_after_heartbeat_redraw_failure() -> None:
+ progress = _progress_module()
+ heartbeat_failed = threading.Event()
+ failed_on: list[threading.Thread] = []
+
+ class FailingLive:
+ def __init__(self, table, **_kwargs) -> None:
+ self.renderables = [table]
+ self.update_calls = 0
+ self.stop_calls = 0
+
+ def start(self, *, refresh: bool = False) -> None:
+ assert refresh
+
+ def update(self, table, *, refresh: bool = False) -> None:
+ assert refresh
+ self.renderables.append(table)
+ self.update_calls += 1
+ if self.update_calls == 2:
+ failed_on.append(threading.current_thread())
+ heartbeat_failed.set()
+ raise RuntimeError("heartbeat redraw failed")
+
+ def stop(self) -> None:
+ self.stop_calls += 1
+
+ output = io.StringIO()
+ rich_reporter = progress.RichProgressReporter(stream=output, refresh_interval=0.05)
+ rich_reporter._live_factory = FailingLive
+ reporter = progress.safe_progress_reporter(rich_reporter)
+ reporter.start(_plan(progress))
+ heartbeat = rich_reporter._thread
+ live = rich_reporter._live
+ assert heartbeat is not None and live is not None
+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="evaluating"))
+
+ assert heartbeat_failed.wait(timeout=10)
+ # The redraw sets the event before it raises, so the heartbeat thread may still
+ # be turning the reporter off. That thread ends right after, so wait for it.
+ heartbeat.join(timeout=10)
+ assert not heartbeat.is_alive()
+ assert failed_on == [heartbeat]
+ assert reporter.is_active is False
+ assert live.stop_calls == 1
+ assert rich_reporter._live is None
+ # Only the display failed: the agent is never drawn as failed or interrupted,
+ # and a line under the frozen frame says the run goes on.
+ frames = [_render(table, width=180) for table in live.renderables]
+ assert not any("failed" in frame or "interrupted" in frame for frame in frames)
+ assert "running" in frames[-1]
+ assert "Tier 3 progress display stopped after a drawing error; the run continues." in output.getvalue()
+ updates_before = live.update_calls
+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="complete"))
+ assert live.update_calls == updates_before
+ reporter.close()
+
+
+_STOP_NOTICE = "Tier 3 progress display stopped after a drawing error; the run continues."
+_LATE_STOP_NOTICE = "Tier 3 progress display stopped after a drawing error; the run went on without it."
+
+
+def _assert_cursor_shown_and_one_stop_notice(output: str, notice: str) -> None:
+ # Live hid the cursor while it drew; it is shown again after the last hide.
+ assert "\x1b[?25l" in output
+ assert output.rfind("\x1b[?25h") > output.rfind("\x1b[?25l")
+ assert output.count("Tier 3 progress display stopped after a drawing error") == 1
+ # The notice is a line of its own below the last frame, even when the
+ # terminal refused the stop that ends the frame's last line.
+ visible = output.replace("\x1b[?25h", "").replace("\x1b[?25l", "")
+ assert f"\n{notice}\n" in visible
+
+
+ "refusal",
+ ["once", "always", "every-thread-for-a-while", "notice-only"],
+ ids=["refused-once", "refused-always", "every-thread-refused-for-a-while", "notice-refused"],
+)
+def test_heartbeat_write_that_would_block_turns_progress_off(
+ monkeypatch: pytest.MonkeyPatch,
+ refusal: str,
+) -> None:
+ from rich.console import Console
+
+ progress = _progress_module()
+ monkeypatch.setenv("TERM", "xterm-256color")
+ uncaught: list[BaseException | None] = []
+ monkeypatch.setattr(threading, "excepthook", lambda args: uncaught.append(args.exc_value))
+ refused = threading.Event()
+ heartbeat_writes: list[int] = []
+ # The heartbeat thread's first write is the refused redraw, its second
+ # Live's stop and its third the notice under the last frame.
+ refused_heartbeat_writes = {"once": {1}, "notice-only": {1, 3}}.get(refusal, set())
+
+ class NonBlockingTerminal(_TTYBuffer):
+ """A terminal another process made non-blocking: it refuses some writes."""
+
+ # While set, the terminal's buffer is full and it refuses every
+ # thread's writes.
+ blocked = False
+
+ def write(self, text: str) -> int:
+ if self.blocked:
+ refused.set()
+ raise BlockingIOError(errno.EAGAIN, "write could not complete without blocking", 0)
+ if threading.current_thread().name == "tier3-progress-heartbeat":
+ heartbeat_writes.append(len(text))
+ if refusal == "always" or len(heartbeat_writes) in refused_heartbeat_writes:
+ refused.set()
+ raise BlockingIOError(errno.EAGAIN, "write could not complete without blocking", 0)
+ return super().write(text)
+
+ stream = NonBlockingTerminal()
+ rich_reporter = progress.RichProgressReporter(stream=stream, refresh_interval=0.05)
+ # A real Rich Live, which keeps a refused frame in its buffer and sends it
+ # again with the next one.
+ rich_reporter._console = Console(
+ file=stream,
+ force_terminal=True,
+ legacy_windows=False,
+ width=120,
+ color_system=None,
+ )
+ reporter = progress.safe_progress_reporter(rich_reporter)
+ try:
+ reporter.start(_plan(progress))
+ heartbeat = rich_reporter._thread
+ assert heartbeat is not None
+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="evaluating"))
+ stream.blocked = refusal == "every-thread-for-a-while"
+
+ assert refused.wait(timeout=10)
+ # A refused write turns progress off, so the heartbeat thread ends
+ # instead of resending a growing backlog of frames.
+ heartbeat.join(timeout=10)
+ assert not heartbeat.is_alive()
+ # The terminal takes writes again before the run ends.
+ stream.blocked = False
+ assert reporter.is_active is False
+ assert rich_reporter._live is None
+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="complete", detail="agent finished"))
+ output = stream.getvalue()
+ assert "agent finished" not in output
+ # Only the display failed, so no stage is reported as failed.
+ assert "failed" not in output
+ assert "interrupted" not in output
+ if refusal == "once":
+ # The terminal took the stop, so the frame left behind and the line
+ # under it say that the agent is still running.
+ assert "running" in output
+ assert _STOP_NOTICE in output
+ finally:
+ reporter.close()
+ assert uncaught == []
+ # A stop or notice the terminal refused on the heartbeat thread is lost
+ # with that thread, so closing the run shows the cursor and the notice.
+ _assert_cursor_shown_and_one_stop_notice(
+ stream.getvalue(),
+ _STOP_NOTICE if refusal == "once" else _LATE_STOP_NOTICE,
+ )
+
+
+ "refused_writes",
+ [{1, 2}, {1, 3}],
+ ids=["update-and-stop-refused", "update-and-notice-refused"],
+)
+def test_callers_write_that_would_block_is_finished_when_progress_closes(
+ monkeypatch: pytest.MonkeyPatch,
+ refused_writes: set[int],
+) -> None:
+ from rich.console import Console
+
+ progress = _progress_module()
+ monkeypatch.setenv("TERM", "xterm-256color")
+ caller = threading.current_thread()
+ caller_writes: list[int] = []
+
+ class NonBlockingTerminal(_TTYBuffer):
+ """Refuses the caller's update, then its stop of Live or the notice under the last frame."""
+
+ armed = False
+
+ def write(self, text: str) -> int:
+ if self.armed and threading.current_thread() is caller:
+ caller_writes.append(len(text))
+ if len(caller_writes) in refused_writes:
+ raise BlockingIOError(errno.EAGAIN, "write could not complete without blocking", 0)
+ return super().write(text)
+
+ stream = NonBlockingTerminal()
+ # No heartbeat during the test, so every write comes from this thread.
+ rich_reporter = progress.RichProgressReporter(stream=stream, refresh_interval=60)
+ rich_reporter._console = Console(
+ file=stream,
+ force_terminal=True,
+ legacy_windows=False,
+ width=120,
+ color_system=None,
+ )
+ reporter = progress.safe_progress_reporter(rich_reporter)
+ try:
+ reporter.start(_plan(progress))
+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="evaluating"))
+ stream.armed = True
+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="still evaluating"))
+ assert reporter.is_active is False
+ assert rich_reporter._live is None
+ stream.armed = False
+ # Once progress is off, other output (a logged warning, say) goes
+ # straight to the terminal below the last frame.
+ for number in range(1, 4):
+ stream.write(f"warning {number} printed after the display stopped\n")
+ printed_before_close = len(stream.getvalue())
+ finally:
+ reporter.close()
+ output = stream.getvalue()
+ assert "failed" not in output
+ assert "interrupted" not in output
+ closing_output = output[printed_before_close:]
+ # Rich keeps this thread's refused writes and would send them with its
+ # next one. They move the cursor up over the frame that was on screen
+ # when drawing failed, so close() drops them instead of erasing the
+ # warnings, and prints the late notice in place of a refused one.
+ assert "\x1b[1A" not in closing_output
+ assert "\x1b[2K" not in closing_output
+ assert closing_output.endswith(f"{_LATE_STOP_NOTICE}\n")
+ if 2 in refused_writes:
+ assert "\x1b[?25h" in closing_output
+ _assert_cursor_shown_and_one_stop_notice(output, _LATE_STOP_NOTICE)
+
+
+def test_run_end_close_during_a_refused_heartbeat_stop_shows_the_cursor(monkeypatch: pytest.MonkeyPatch) -> None:
+ from rich.console import Console
+
+ progress = _progress_module()
+ monkeypatch.setenv("TERM", "xterm-256color")
+ uncaught: list[BaseException | None] = []
+ monkeypatch.setattr(threading, "excepthook", lambda args: uncaught.append(args.exc_value))
+ lock = _InterleavingLock()
+ closers: list[threading.Thread] = []
+
+ def close_the_run() -> None:
+ # The run ends while the heartbeat thread is still turning a failed
+ # display off: its stop was refused and it has let go of the lock.
+ closer = threading.Thread(target=reporter.close, name="run-end-close")
+ closers.append(closer)
+ closer.start()
+ closer.join(timeout=10)
+
+ class NonBlockingTerminal(_TTYBuffer):
+ """Refuses every heartbeat-thread write while ``refusing`` is set."""
+
+ refusing = True
+
+ def write(self, text: str) -> int:
+ if self.refusing and threading.current_thread().name == "tier3-progress-heartbeat":
+ if "\x1b[?25h" in text:
+ # Live's stop, which shows the cursor again.
+ lock.arm(close_the_run)
+ raise BlockingIOError(errno.EAGAIN, "write could not complete without blocking", 0)
+ return super().write(text)
+
+ stream = NonBlockingTerminal()
+ rich_reporter = progress.RichProgressReporter(stream=stream, refresh_interval=0.05)
+ rich_reporter._lock = lock
+ rich_reporter._console = Console(
+ file=stream,
+ force_terminal=True,
+ legacy_windows=False,
+ width=120,
+ color_system=None,
+ )
+ reporter = progress.safe_progress_reporter(rich_reporter)
+ reporter.start(_plan(progress))
+ heartbeat = rich_reporter._thread
+ assert heartbeat is not None
+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="evaluating"))
+
+ heartbeat.join(timeout=10)
+ assert not heartbeat.is_alive()
+ assert len(closers) == 1
+ assert not closers[0].is_alive()
+ stream.refusing = False
+ first_run = stream.getvalue()
+ _assert_cursor_shown_and_one_stop_notice(first_run, _LATE_STOP_NOTICE)
+ # Nothing of the failed run is left for the next one.
+ reporter.start(_plan(progress))
+ reporter.close()
+ assert "display stopped" not in stream.getvalue()[len(first_run) :]
+ assert uncaught == []
+
+
+def test_a_new_run_without_close_carries_nothing_from_a_refused_stop(monkeypatch: pytest.MonkeyPatch) -> None:
+ from rich.console import Console
+
+ progress = _progress_module()
+ monkeypatch.setenv("TERM", "xterm-256color")
+
+ class NonBlockingTerminal(_TTYBuffer):
+ """Refuses every write while ``refusing`` is set."""
+
+ refusing = False
+
+ def write(self, text: str) -> int:
+ if self.refusing:
+ raise BlockingIOError(errno.EAGAIN, "write could not complete without blocking", 0)
+ return super().write(text)
+
+ stream = NonBlockingTerminal()
+ rich_reporter = progress.RichProgressReporter(stream=stream, refresh_interval=60)
+ rich_reporter._console = Console(
+ file=stream,
+ force_terminal=True,
+ legacy_windows=False,
+ width=120,
+ color_system=None,
+ )
+ reporter = progress.safe_progress_reporter(rich_reporter)
+ reporter.start(_plan(progress))
+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="evaluating"))
+ stream.refusing = True
+ # The update and Live's stop are both refused.
+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="still evaluating"))
+ assert reporter.is_active is False
+ stream.refusing = False
+ first_run = stream.getvalue()
+
+ # The reporter is used again without the close() that ends the failed run.
+ rich_reporter.start(_plan(progress))
+ rich_reporter.emit(progress.ProgressEvent(stage="agent:codex", state="complete", detail="agent finished"))
+ rich_reporter.close()
+
+ second_run = stream.getvalue()[len(first_run) :]
+ assert "agent finished" in second_run
+ assert "still evaluating" not in second_run
+ assert "display stopped" not in second_run
+ assert second_run.rfind("\x1b[?25h") > second_run.rfind("\x1b[?25l")
+
+
[email protected]("notice_refused", [True, False], ids=["notice-refused", "notice-accepted"])
+def test_refused_first_frame_and_stop_leave_the_cursor_shown(
+ monkeypatch: pytest.MonkeyPatch,
+ notice_refused: bool,
+) -> None:
+ from rich.console import Console
+ from rich.live import Live
+
+ progress = _progress_module()
+ monkeypatch.setenv("TERM", "xterm-256color")
+
+ class NonBlockingTerminal(_TTYBuffer):
+ """Takes Live's hide-cursor write, then refuses the first frame and Live's own stop."""
+
+ armed = False
+
+ def write(self, text: str) -> int:
+ refused = self.armed and "\x1b[?25l" not in text
+ if refused and (notice_refused or _STOP_NOTICE not in text):
+ raise BlockingIOError(errno.EAGAIN, "write could not complete without blocking", 0)
+ return super().write(text)
+
+ stream = NonBlockingTerminal()
+ rich_reporter = progress.RichProgressReporter(stream=stream, refresh_interval=60)
+ rich_reporter._console = Console(
+ file=stream,
+ force_terminal=True,
+ legacy_windows=False,
+ width=120,
+ color_system=None,
+ )
+
+ def live_on_a_refusing_terminal(*args: Any, **kwargs: Any) -> Any:
+ # The plan lines above the frame are already printed.
+ stream.armed = True
+ return Live(*args, **kwargs)
+
+ rich_reporter._live_factory = live_on_a_refusing_terminal
+ reporter = progress.safe_progress_reporter(rich_reporter)
+ try:
+ # Live.start() stops Live itself when its first frame fails, then
+ # raises. The terminal refuses that stop as well, so the cursor Live
+ # hid stays hidden, and Live is already stopped when progress turns off.
+ reporter.start(_plan(progress))
+ assert reporter.is_active is False
+ assert rich_reporter._live is None
+ stream.armed = False
+ printed_before_close = len(stream.getvalue())
+ finally:
+ reporter.close()
+ output = stream.getvalue()
+ _assert_cursor_shown_and_one_stop_notice(output, _LATE_STOP_NOTICE if notice_refused else _STOP_NOTICE)
+ # Nothing the terminal refused is sent when the run ends.
+ assert "\x1b[1A" not in output[printed_before_close:]
+ if notice_refused:
+ # No frame reached the terminal, so the late notice follows the plan
+ # lines without a blank line.
+ assert "Harbor Run Configuration" not in output
+ assert "\n\n" not in output.replace("\x1b[?25h", "").replace("\x1b[?25l", "")
+ else:
+ # The notice went out with the frame and the stop the terminal had
+ # refused just before, so the frame is on screen above it.
+ assert "Harbor Run Configuration" in output
+
+
+class _InterleavingLock:
+ """A reentrant lock that runs a hook once, when the arming thread next releases it fully."""
+
+ def __init__(self) -> None:
+ self._lock = threading.RLock()
+ self._depth = 0
+ self._hook: tuple[threading.Thread, Any] | None = None
+
+ def arm(self, hook: Any) -> None:
+ self._hook = (threading.current_thread(), hook)
+
+ def __enter__(self) -> _InterleavingLock:
+ self._lock.acquire()
+ self._depth += 1
+ return self
+
+ def __exit__(self, *exc_info: object) -> None:
+ self._depth -= 1
+ released = self._depth == 0
+ self._lock.release()
+ armed = self._hook
+ if released and armed is not None and armed[0] is threading.current_thread():
+ self._hook = None
+ armed[1]()
+
+
+def test_an_update_that_races_a_drawing_failure_is_not_drawn_as_failed(monkeypatch: pytest.MonkeyPatch) -> None:
+ import rich.live
+
+ progress = _progress_module()
+ _FakeLive.instances.clear()
+ monkeypatch.setattr(rich.live, "Live", _FakeLive)
+ rich_reporter = progress.RichProgressReporter(stream=_TTYBuffer(), refresh_interval=60)
+ lock = _InterleavingLock()
+ rich_reporter._lock = lock
+ reporter = progress.safe_progress_reporter(rich_reporter)
+ reporter.start(_plan(progress))
+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="attempt 1 of 3"))
+
+ def late_agent_update() -> None:
+ # The caller's next agent update passed the wrapper's check before the
+ # failure turned progress off, and reaches the reporter while it stops.
+ update = threading.Thread(
+ target=rich_reporter.emit,
+ args=(progress.ProgressEvent(stage="agent:codex", state="running", detail="attempt 2 of 3"),),
+ )
+ update.start()
+ update.join(timeout=10)
+
+ lock.arm(late_agent_update)
+ # What the heartbeat thread's failure callback runs.
+ reporter._disable()
+ live = _FakeLive.instances[0]
+ frames = [_render(renderable, width=180) for renderable in live.renderables]
+ reporter.close()
+
+ # Only the display failed: the agent is never drawn as failed, and the
+ # frame left on screen shows the update as still running.
+ assert not any("failed" in frame or "interrupted" in frame for frame in frames)
+ assert "attempt 2 of 3" in frames[-1]
+ assert live.stop_calls == 1
+
+
+def test_heartbeat_failure_during_a_plan_update_stops_what_the_update_drew(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ import rich.live
+
+ progress = _progress_module()
+ _FakeLive.instances.clear()
+ monkeypatch.setattr(rich.live, "Live", _FakeLive)
+
+ class RacingReporter(progress.RichProgressReporter):
+ report_failure: Any = None
+ failure_thread: threading.Thread | None = None
+ race = False
+
+ def set_failure_callback(self, callback) -> None:
+ super().set_failure_callback(callback)
+ self.report_failure = callback
+
+ def start(self, plan) -> None:
+ if self.race:
+ # The heartbeat thread reports a redraw failure while this plan
+ # update is in flight.
+ self.failure_thread = threading.Thread(target=self.report_failure)
+ self.failure_thread.start()
+ self.failure_thread.join(timeout=0.2)
+ super().start(plan)
+
+ rich_reporter = RacingReporter(stream=_TTYBuffer(), refresh_interval=60)
+ reporter = progress.safe_progress_reporter(rich_reporter)
+ reporter.start(_plan(progress))
+ rich_reporter.race = True
+ reporter.start(_plan(progress))
+ failure_thread = rich_reporter.failure_thread
+ assert failure_thread is not None
+ failure_thread.join(timeout=10)
+
+ assert not failure_thread.is_alive()
+ # The failure waited for the update, then stopped the frame it drew.
+ assert reporter.is_active is False
+ assert rich_reporter.is_active is False
+ assert [live.stop_calls for live in _FakeLive.instances] == [1]
+ reporter.close()
+
+
+def test_safe_reporter_wraps_a_reporter_whose_callback_registration_fails() -> None:
+ progress = _progress_module()
+
+ class RefusingReporter(progress.NullProgressReporter):
+ def __init__(self) -> None:
+ self.events: list[Any] = []
+
+ def set_failure_callback(self, _callback) -> None:
+ raise RuntimeError("failure callbacks are not supported")
+
+ def emit(self, event) -> None:
+ self.events.append(event)
+
+ class StrictProxy:
+ """Forwards the reporter protocol and refuses any other attribute."""
+
+ def __init__(self) -> None:
+ self.target = RefusingReporter()
+
+ def __getattr__(self, name: str) -> Any:
+ if name in {"is_active", "start", "set_secret_values", "emit", "heartbeat", "close"}:
+ return getattr(self.target, name)
+ raise RuntimeError(f"unsupported attribute: {name}")
+
+ refusing = RefusingReporter()
+ proxy = StrictProxy()
+
+ for target, events in ((refusing, refusing.events), (proxy, proxy.target.events)):
+ reporter = progress.safe_progress_reporter(target)
+ reporter.start(_plan(progress))
+ reporter.emit(progress.ProgressEvent(stage="configuration", state="running"))
+ reporter.close()
+ assert [event.stage for event in events] == ["configuration"]
+
+
+def test_heartbeat_failure_without_a_safety_wrapper_stops_the_thread(monkeypatch: pytest.MonkeyPatch) -> None:
+ progress = _progress_module()
+ uncaught: list[BaseException | None] = []
+ monkeypatch.setattr(threading, "excepthook", lambda args: uncaught.append(args.exc_value))
+
+ class FailingHeartbeatReporter(progress.PlainProgressReporter):
+ def heartbeat(self) -> None:
+ raise RuntimeError("heartbeat failed")
+
+ reporter = FailingHeartbeatReporter(stream=io.StringIO(), refresh_interval=0.05)
+ reporter.start(_plan(progress))
+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running"))
+ heartbeat = reporter._thread
+ assert heartbeat is not None
+
+ heartbeat.join(timeout=10)
+
+ assert not heartbeat.is_alive()
+ assert reporter.is_active is False
+ assert uncaught == []
+ reporter.close()
+
+
def test_safe_reporter_tracks_lifecycle_when_delegate_has_no_active_state() -> None:
progress = _progress_module()
reporter = progress.safe_progress_reporter(progress.NullProgressReporter())
@@ -3102,7 +4190,9 @@ def test_second_interrupt_forces_running_agents_to_stop(
assert reporter.interrupts == 2
events = [(event.stage, event.state, event.detail) for event in reporter.events]
stopping = [detail for stage, state, detail in events if stage == "agent:codex" and state == "running"]
- assert stopping[-1].startswith("stopping Harbor jobs; press Ctrl+C again")
+ assert stopping[-1] == runner._HARBOR_RUN_STOPPING_DETAIL
+ assert stopping[-1].startswith("2nd Ctrl+C skips wait; stopping Harbor jobs")
+ assert stopping[-1].endswith("up to 30s of environment cleanup")
assert ("agent:codex", "failed", "interrupted before completion") in events
assert events[-1] == ("run-finished", "failed", "Tier 3 evaluation interrupted")