Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
Python agent-evaluationagent-pluginsagent-securityagent-skillsagentic-aibenchmarkci-templatesclaude-codecodexevaluateevaluationharnesspluginplugin-evaluationsecurity-scannerskill-evalskill-evaluationskill-evaluatorskill-gap-analysisskills
fix(tier3): say progress stopped when a plan line is refused, and keep the running spinner moving (#548)
Narendran Raghavane466e83c8e00
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 28e623c6..3584017b 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -600,6 +600,27 @@ All notable changes to SkillEvaluator are documented in this file.ownership key, for example from containers that each start with a freshhome directory. The refusal itself is unchanged, and `docs/reports.mdx`explains it.+- When the terminal refuses one of the plan lines printed before the+ interactive Tier 3 progress view's table first appears, such as a write that+ would block, a line now says that the run continues, as when drawing the+ table fails. When the terminal still refuses writes then, the line saying+ that the run went on is printed instead, when the run ends, so only one of+ the two appears. When the terminal takes the writes before that line and+ refuses only the line itself, a stream that keeps what the terminal refused,+ as standard error does, sends that line when the run ends in place of the+ later one. Before, progress turned off with no notice, and the refused line+ was printed again with the next run's plan. After a refused table update,+ standard error printed both lines when the terminal then refused the line+ saying that the run continues. The view also no longer keeps its own copy of+ output the terminal still refuses when the run ends, such as the final+ table, and sends it with the next run on the same reporter, where it could+ erase the lines printed between the runs. A stream that keeps what the+ terminal refused, as standard error does, can still send it later, and a+ final table sent then can still erase those lines. A running stage's spinner+ in that view now moves on one frame with each redraw, at least once a second+ by default, instead of staying on its first frame. These follow the 0.9.0+ Fixed entry that starts "The interactive Tier 3 progress view+ (`--progress rich`, or `auto` in a terminal) now keeps one height".## 0.9.0 - 2026-11-13diff --git a/docs/tier3-live-evaluation.mdx b/docs/tier3-live-evaluation.mdxindex 54789eee..f24d0915 100644--- a/docs/tier3-live-evaluation.mdx+++ b/docs/tier3-live-evaluation.mdx@@ -1014,9 +1014,19 @@ terminal write that would block, the table stops updating with its stages asthey were, a line says that the run continues, and no further progress isshown for that run; the final results still print. When the terminal alsorefuses that line, or the write that shows the cursor again, a line saying-that the run went on is printed and the cursor is shown when the run ends.-Frames the terminal refused are dropped then, so they are never drawn over-output printed since.+that the run went on is printed and the cursor is shown when the run ends. If+the terminal refused only that line, a stream that keeps what the terminal+refused, as standard error does, sends that line then instead of the later+one, although after a failed first frame both can appear. If the terminal+still refuses writes when the run ends, such a stream sends the later line and+the write that shows the cursor once the terminal takes writes again, but only+if its buffer has room for them. A buffer already full of a refused table can+drop that line and leave the cursor hidden. When any run ends, whether or not+drawing failed, the view drops its own copy of output the terminal still+refuses, such as a refused final table, so it does not draw that output over+lines printed since. A stream that keeps what the terminal refused, as+standard error does, can still send it later, and a table sent then can still+erase those lines.Every `tier3 evaluate` flag, including resource overrides and Dockerfilemodes, is in the [CLI Reference](/skills/skillevaluator/cli-reference).diff --git a/src/skillevaluator/tier3/harbor/progress.py b/src/skillevaluator/tier3/harbor/progress.pyindex c6730e79..3dea4abf 100644--- a/src/skillevaluator/tier3/harbor/progress.py+++ b/src/skillevaluator/tier3/harbor/progress.py@@ -10,6 +10,7 @@ or discarded by an API caller.from __future__ import annotations+import ioimport loggingimport mathimport re@@ -67,6 +68,11 @@ _DISPLAY_STOPPED_NOTICE = "Tier 3 progress display stopped after a drawing error# Printed when the run ends instead, when the terminal refused the line above# or Live's stop._DISPLAY_STOPPED_LATE_NOTICE = "Tier 3 progress display stopped after a drawing error; the run went on without it."+# Frames of a running stage's spinner, as in Rich's "dots". The view redraws+# only on progress events and heartbeats, so a running stage moves on one frame+# with each redraw. Rich's own spinner picks its frame from the clock, so it+# would stand still whenever redraws come a whole number of its cycles apart.+_RUNNING_FRAMES = "\u280b\u2819\u2839\u2838\u283c\u2834\u2826\u2827\u2807\u280f"@dataclass(frozen=True, slots=True)@@ -106,6 +112,9 @@ class _FailedDisplayStop:stop_refused: boolnotice_shown: bool+ # The terminal refused the notice, but the stream kept it to send later,+ # as standard error's buffer does.+ notice_kept: bool = False@dataclass(frozen=True, slots=True)@@ -433,6 +442,9 @@ class RichProgressReporter(PlainProgressReporter):self._live = Noneself._live_plan: Tier3RunPlan | None = Noneself._live_events: dict[str, tuple[ProgressEvent, str | None]] = {}+ # How many times each running stage has been drawn since it started+ # running, which picks its spinner frame.+ self._live_spinner_steps: dict[str, int] = {}self._live_event_slots = _LIVE_MIN_EVENT_ROWSself._printed_plan_lines: dict[str, str] = {}# Set, under the same lock hold that stops Live, when a drawing failure@@ -461,6 +473,7 @@ class RichProgressReporter(PlainProgressReporter):finished_at = Noneelse:self._active.pop(event.stage, None)+ self._live_spinner_steps.pop(event.stage, None)finished_at = self._elapsed()# Move the stage to the end so the recent-event window shows it.self._live_events.pop(event.stage, None)@@ -491,6 +504,22 @@ class RichProgressReporter(PlainProgressReporter):live.update(self._build_live_table(final=True), refresh=False)finally:live.stop()+ elif (+ after_failure+ and self._live is None+ and self._live_plan is not None+ and self._failed_stop is None+ ):+ # Drawing failed before Live started, for example on a+ # plan line printed above where the frame goes. There+ # is no frame to stop, but progress is off for the run+ # all the same, so the notice is due. A stream that+ # refuses a write can still have taken it, as standard+ # error's buffer does, and then sends it with the+ # notice. Rich's copy of the refused line is dropped+ # first so the line is not printed twice.+ self._drop_refused_writes()+ self._stop_live_after_failure(None)finally:# Reset even when Live never started, so a failed start# leaves no plan or rows behind for the next run.@@ -498,39 +527,79 @@ class RichProgressReporter(PlainProgressReporter):self._live = Noneself._live_plan = Noneself._live_events.clear()+ self._live_spinner_steps.clear()self._live_event_slots = _LIVE_MIN_EVENT_ROWSself._printed_plan_lines.clear()if not after_failure:failed_stop, self._failed_stop = self._failed_stop, Noneif failed_stop is not None:self._finish_failed_stop(failed_stop)+ # Whatever the terminal still refuses as the run ends,+ # such as the final frame and Live's stop or the late+ # notice, belongs to no later run on this reporter.+ self._drop_refused_writes()def _stop_live_after_failure(self, live) -> None:"""Stop the live view after a drawing failure and say that the run goes on.The full final frame could fail the same way, so Live keeps the last- frame it drew. Runs under the reporter lock and records the outcome in- the same lock hold, so the run's close() either stops Live itself or- finds the record.+ frame it drew. ``live`` is ``None`` when drawing failed before Live+ started, and then only the notice is printed. Runs under the reporter+ lock and records the outcome in the same lock hold, so the run's+ close() either stops Live itself or finds the record."""+ # The console holds no refused output once Live stops here, and the+ # caller drops it before Live started. Live that stopped itself when+ # its first frame failed leaves that frame and stop held when the+ # terminal refused that stop too. They then go out with the notice,+ # and a stream's buffer may keep only part of them. Nothing here+ # tells whether that stop went out, so after a failed first frame+ # the run's close() writes the late notice either way, although a+ # stream that took the stop and kept the notice sends both.+ notice_alone = live is None or bool(getattr(live, "is_started", False))+ if live is not None:+ try:+ live.stop()+ except Exception:+ # A terminal that refused the redraw can refuse Live's stop too,+ # and with it the write that shows the cursor again.+ logger.debug("Tier 3 progress display stop failed", exc_info=True)+ self._failed_stop = _FailedDisplayStop(stop_refused=True, notice_shown=False)+ returntry:- live.stop()+ # A stream can keep what the terminal refused, as standard error's+ # buffer does, and send it later. It would then also keep a refused+ # notice and send it before the late one, so while the terminal+ # still refuses what the stream holds, the notice waits for the+ # run's close().+ self._console.file.flush()except Exception:- # A terminal that refused the redraw can refuse Live's stop too,- # and with it the write that shows the cursor again.- logger.debug("Tier 3 progress display stop failed", exc_info=True)- self._failed_stop = _FailedDisplayStop(stop_refused=True, notice_shown=False)+ logger.debug("Tier 3 progress stop notice failed", exc_info=True)+ self._failed_stop = _FailedDisplayStop(stop_refused=False, notice_shown=False)returntry:# Live keeps the last frame it drew, with its stages still running,# so explain why it no longer updates.self._write_line(_DISPLAY_STOPPED_NOTICE)- except Exception:+ except Exception as error:logger.debug("Tier 3 progress stop notice failed", exc_info=True)- self._failed_stop = _FailedDisplayStop(stop_refused=False, notice_shown=False)+ # The flush above emptied the stream's buffer, so a buffer that+ # keeps refused writes now holds the notice and sends it later.+ # The run's close() then sends it instead of the late one. A+ # buffer smaller than the notice would keep only part of it, but+ # standard error's buffer is many times the notice's size.+ notice_kept = notice_alone and isinstance(error, BlockingIOError) and self._stream_keeps_refused_writes()+ self._failed_stop = _FailedDisplayStop(stop_refused=False, notice_shown=False, notice_kept=notice_kept)else:self._failed_stop = _FailedDisplayStop(stop_refused=False, notice_shown=True)+ def _stream_keeps_refused_writes(self) -> bool:+ """Whether the console's stream keeps a write the terminal refused, as standard error's buffer does."""+ try:+ return isinstance(getattr(self._console.file, "buffer", None), (io.BufferedWriter, io.BufferedRandom))+ except Exception:+ return False+def _drop_refused_writes(self) -> str:"""Discard the output this thread's Rich console holds from refused writes, and return it.@@ -556,12 +625,19 @@ class RichProgressReporter(PlainProgressReporter):from rich.control import Controldropped = self._drop_refused_writes()+ # Whether the frame's last line is ended before the late notice.+ line_ended = Truetry:if failed_stop.stop_refused:# Live hides the cursor while it draws and shows it again when# it stops, then ends the frame's last line.+ line_ended = not self._console.is_terminalself._console.show_cursor(True)if self._console.is_terminal:+ # The console keeps a line the terminal refuses and sends+ # it before its next write, so the line ends before the+ # notice either way.+ line_ended = Trueself._console.line()elif str(Control.show_cursor(True)) in dropped:# Live.start() stops Live itself when its first frame fails,@@ -570,10 +646,35 @@ class RichProgressReporter(PlainProgressReporter):# cursor was just dropped. No frame reached the terminal, so# there is no last line to end.self._console.show_cursor(True)- if not failed_stop.notice_shown:- self._write_line(_DISPLAY_STOPPED_LATE_NOTICE)except Exception:+ # The run's close() then drops what the terminal still refuses.logger.debug("Tier 3 progress terminal restore failed", exc_info=True)+ if failed_stop.notice_shown:+ return+ if failed_stop.notice_kept:+ try:+ # The stream kept the notice the terminal refused. It goes out+ # now, or with the stream's next write if the terminal still+ # refuses it, and a late notice would only repeat it.+ self._console.file.flush()+ except Exception:+ logger.debug("Tier 3 progress kept stop notice could not be sent", exc_info=True)+ return+ try:+ # Written even when the terminal refused the cursor just now: a+ # stream that keeps what the terminal refused, as standard error+ # does, can send the notice with it later if its buffer has room.+ # A refused cursor write leaves the line above unwritten, and the+ # stream may have kept only the start of the refused stop, without+ # its newline, so the notice then starts a new line of its own.+ # Where the stream kept the whole stop, a blank line comes first,+ # as it does when the terminal takes the cursor write.+ if line_ended:+ self._write_line(_DISPLAY_STOPPED_LATE_NOTICE)+ else:+ self._write_line(f"\n{_DISPLAY_STOPPED_LATE_NOTICE}")+ except Exception:+ logger.debug("Tier 3 progress late stop notice failed", exc_info=True)def _render_plan(self, plan: Tier3RunPlan) -> None:self._live_plan = plan@@ -642,7 +743,6 @@ class RichProgressReporter(PlainProgressReporter):def _build_live_table(self, *, final: bool = False):from rich import boxfrom rich.cells import cell_len- from rich.spinner import Spinnerfrom rich.table import Tablefrom rich.text import Text@@ -708,11 +808,12 @@ class RichProgressReporter(PlainProgressReporter):if stage.startswith("agent:")else stage.replace("-", " ").title())- state = (- Spinner("dots", text=Text("running", style=TEXT))- if event.state == "running"- else Text(event.state, style=state_styles.get(event.state, "white"))- )+ if event.state == "running":+ step = self._live_spinner_steps.get(stage, 0)+ self._live_spinner_steps[stage] = step + 1+ state = Text.assemble(_RUNNING_FRAMES[step % len(_RUNNING_FRAMES)], " ", ("running", TEXT))+ else:+ state = Text(event.state, style=state_styles.get(event.state, "white"))rows.append((Text(label),diff --git a/tests/test_tier3_progress.py b/tests/test_tier3_progress.pyindex 6a874c7c..3782c975 100644--- a/tests/test_tier3_progress.py+++ b/tests/test_tier3_progress.py@@ -22,6 +22,15 @@ from typing import Any, ClassVarimport pytest[email protected](autouse=True)+def _rich_detects_interactive_terminals(monkeypatch: pytest.MonkeyPatch) -> None:+ # Rich reads TTY_INTERACTIVE when it builds a console, and+ # TTY_INTERACTIVE=0 turns off the in-place redraws that the tests with a+ # forced terminal drive. Clear it so they draw the same frames wherever+ # they run.+ monkeypatch.delenv("TTY_INTERACTIVE", raising=False)++class _TTYBuffer(io.StringIO):def isatty(self) -> bool:return True@@ -406,6 +415,45 @@ def test_rich_progress_uses_manual_refresh_and_recent_event_window(monkeypatch:reporter.close()+def test_rich_progress_running_spinner_moves_on_with_each_redraw(monkeypatch: pytest.MonkeyPatch) -> None:+ import re++ import rich.live+ from rich.spinner import Spinner++ progress = _progress_module()+ _FakeLive.instances.clear()+ monkeypatch.setattr(rich.live, "Live", _FakeLive)+ reporter = progress.RichProgressReporter(stream=_TTYBuffer(), refresh_interval=60)+ reporter.start(_plan(progress))+ live = _FakeLive.instances[0]+ frames = Spinner("dots").frames++ def latest_frame() -> int:+ rendered = _render(live.renderables[-1], width=120)+ glyph = re.search(r"Agent codex\s*│ (\S) running", rendered)+ assert glyph is not None, rendered+ return frames.index(glyph.group(1))++ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="evaluating"))+ assert latest_frame() == 0+ # Rich no longer redraws on its own timer, so the spinner moves on one+ # frame with each heartbeat or update that redraws the view, however far+ # apart they come.+ reporter.heartbeat()+ assert latest_frame() == 1+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="attempt 2 of 3"))+ assert latest_frame() == 2+ for _ in range(len(frames)):+ reporter.heartbeat()+ assert latest_frame() == 2+ # A stage that runs again after it finished starts from the first frame.+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="complete", detail="attempt 2 of 3"))+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="attempt 3 of 3"))+ assert latest_frame() == 0+ reporter.close()++def test_rich_progress_resets_state_when_live_initialization_fails() -> None:progress = _progress_module()reporter = progress.RichProgressReporter(stream=io.StringIO(), refresh_interval=60)@@ -1488,6 +1536,638 @@ def test_refused_first_frame_and_stop_leave_the_cursor_shown(assert "Harbor Run Configuration" in output+ ("refused_writes", "notice"),+ [+ ({1}, _STOP_NOTICE),+ ({2}, _STOP_NOTICE),+ ({3}, _STOP_NOTICE),+ ({1, 2}, _LATE_STOP_NOTICE),+ ],+ ids=["agents-line-refused", "provider-line-refused", "run-plan-line-refused", "agents-line-and-notice-refused"],+)+def test_a_refused_plan_line_before_the_frame_turns_progress_off_with_a_notice(+ monkeypatch: pytest.MonkeyPatch,+ refused_writes: set[int],+ notice: str,+) -> None:+ from rich.console import Console++ progress = _progress_module()+ monkeypatch.setenv("TERM", "xterm-256color")+ writes: list[str] = []++ class NonBlockingTerminal(_TTYBuffer):+ """Refuses some of the first writes, which print the plan lines before Live starts."""++ def write(self, text: str) -> int:+ # Dropping the console's held output also writes an empty string,+ # which carries nothing to refuse.+ if text:+ writes.append(text)+ if len(writes) in refused_writes:+ raise BlockingIOError(errno.EAGAIN, "write could not complete without blocking", 0)+ return super().write(text)++ stream = NonBlockingTerminal()+ rich_reporter = progress.RichProgressReporter(stream=stream, refresh_interval=60)+ rich_reporter._console = Console(+ file=stream,+ force_terminal=True,+ legacy_windows=False,+ width=120,+ color_system=None,+ )+ reporter = progress.safe_progress_reporter(rich_reporter)+ reporter.start(_plan(progress))+ # The plan line was refused before Live existed, so there is no frame to stop.+ assert reporter.is_active is False+ assert rich_reporter._live is None+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="evaluating"))+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="complete", detail="agent finished"))+ reporter.close()+ first_run = stream.getvalue()++ # Progress is off for the run, and a line of its own says so: on time when+ # the terminal took it, or when the run ends.+ assert "Harbor Run Configuration" not in first_run+ assert "agent finished" not in first_run+ assert first_run.count("Tier 3 progress display stopped after a drawing error") == 1+ assert f"\n{first_run}".endswith(f"\n{notice}\n")+ assert first_run.count("Agents / models:") <= 1++ # The refused plan line is not sent again with the next run's plan.+ reporter.start(_plan(progress))+ reporter.close()+ second_run = stream.getvalue()[len(first_run) :]+ assert second_run.count("Agents / models:") == 1+ assert "display stopped" not in second_run+++ "refused_from",+ ["plan-line", "update"],+ ids=["refused-from-the-plan-line", "refused-from-an-update"],+)+def test_writes_refused_until_the_run_closes_are_not_sent_with_the_next_run(+ monkeypatch: pytest.MonkeyPatch,+ refused_from: str,+) -> None:+ from rich.console import Console++ progress = _progress_module()+ monkeypatch.setenv("TERM", "xterm-256color")++ class NonBlockingTerminal(_TTYBuffer):+ """Refuses every write while ``refusing`` is set, and keeps nothing it refused."""++ refusing = False++ def write(self, text: str) -> int:+ if self.refusing and text:+ raise BlockingIOError(errno.EAGAIN, "write could not complete without blocking", 0)+ return super().write(text)++ stream = NonBlockingTerminal()+ rich_reporter = progress.RichProgressReporter(stream=stream, refresh_interval=60)+ rich_reporter._console = Console(+ file=stream,+ force_terminal=True,+ legacy_windows=False,+ width=120,+ color_system=None,+ )+ reporter = progress.safe_progress_reporter(rich_reporter)+ stream.refusing = refused_from == "plan-line"+ reporter.start(_plan(progress))+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="evaluating"))+ stream.refusing = True+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="still evaluating"))+ assert reporter.is_active is False+ # The terminal still refuses what close() writes: the late notice, and+ # after a refused frame the write that shows the cursor again.+ reporter.close()+ stream.refusing = False+ first_run = stream.getvalue()++ reporter.start(_plan(progress))+ reporter.close()+ second_run = stream.getvalue()[len(first_run) :]+ # Nothing the terminal refused during the failed run comes out first.+ assert second_run.startswith("Agents / models:")+ assert second_run.count("Agents / models:") == 1+ assert "display stopped" not in second_run+ assert "still evaluating" not in second_run++[email protected]("stage_state", ["complete", "running"], ids=["stage-finished", "stage-still-running"])+def test_a_final_frame_refused_when_the_run_closes_is_not_sent_with_the_next_run(+ monkeypatch: pytest.MonkeyPatch,+ stage_state: str,+) -> None:+ from rich.console import Console++ progress = _progress_module()+ monkeypatch.setenv("TERM", "xterm-256color")++ class NonBlockingTerminal(_TTYBuffer):+ """Refuses every write while ``refusing`` is set, and keeps nothing it refused."""++ refusing = False++ def write(self, text: str) -> int:+ if self.refusing and text:+ raise BlockingIOError(errno.EAGAIN, "write could not complete without blocking", 0)+ return super().write(text)++ stream = NonBlockingTerminal()+ rich_reporter = progress.RichProgressReporter(stream=stream, refresh_interval=60)+ rich_reporter._console = Console(+ file=stream,+ force_terminal=True,+ legacy_windows=False,+ width=120,+ color_system=None,+ )+ reporter = progress.safe_progress_reporter(rich_reporter)+ reporter.start(_plan(progress))+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="evaluating"))+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state=stage_state, detail="agent finished"))+ # Nothing failed while the run went on. The terminal refuses only what the+ # run's close() writes: the final frame and Live's stop.+ assert reporter.is_active is True+ stream.refusing = True+ reporter.close()+ stream.refusing = False+ # The caller prints its results below the frame.+ stream.write("Result: pass\n")+ stream.write("Report: report.html\n")+ first_run = stream.getvalue()++ reporter.start(_plan(progress))+ reporter.close()+ second_run = stream.getvalue()[len(first_run) :]+ # The refused final frame starts by moving the cursor up over the frame on+ # screen. Sent first in the next run, it would erase the result lines and+ # draw the old frame over them.+ assert second_run.startswith("Agents / models:")+ assert second_run.count("Agents / models:") == 1+ assert "agent finished" not in second_run+ assert "interrupted before completion" not in second_run+++class _WouldBlockTerminal(io.RawIOBase):+ """A terminal file another process made non-blocking: some writes would block and take nothing."""++ def __init__(self, refused_writes: set[int]) -> None:+ super().__init__()+ self.refused_writes = refused_writes+ self.calls = 0+ self.received = bytearray()++ def writable(self) -> bool:+ return True++ def isatty(self) -> bool:+ return True++ def write(self, data: Any) -> int | None:+ self.calls += 1+ if self.calls in self.refused_writes:+ # What a raw write returns for EAGAIN.+ return None+ self.received += bytes(data)+ return len(data)+++ ("refused_writes", "plan_line", "notice"),+ [+ ({1}, "Agents / models:", _STOP_NOTICE),+ ({2}, "Provider:", _STOP_NOTICE),+ ({3}, "Run plan:", _STOP_NOTICE),+ # The terminal takes the refused line when it is flushed, then+ # refuses the notice. The stream keeps the notice and sends it when+ # the run ends, in place of the late one.+ ({1, 3}, "Agents / models:", _STOP_NOTICE),+ # The terminal stays full until start() returns: it refuses the plan+ # line and every flush after it, so the notice waits for close().+ ({1, 2, 3}, "Agents / models:", _LATE_STOP_NOTICE),+ ],+ ids=[+ "agents-line-refused",+ "provider-line-refused",+ "run-plan-line-refused",+ "agents-line-and-notice-refused",+ "terminal-full-during-start",+ ],+)+def test_a_plan_line_refused_by_a_buffered_terminal_stream_is_printed_once(+ monkeypatch: pytest.MonkeyPatch,+ refused_writes: set[int],+ plan_line: str,+ notice: str,+) -> None:+ from rich.console import Console++ progress = _progress_module()+ monkeypatch.setenv("TERM", "xterm-256color")+ terminal = _WouldBlockTerminal(refused_writes)+ # Shaped like the standard error stream the CLI passes. Its buffer keeps a+ # line the terminal refused and sends it with the next write.+ stream = io.TextIOWrapper(io.BufferedWriter(terminal), encoding="utf-8", newline="\n", line_buffering=True)+ rich_reporter = progress.RichProgressReporter(stream=stream, refresh_interval=60)+ rich_reporter._console = Console(+ file=stream,+ force_terminal=True,+ legacy_windows=False,+ width=120,+ color_system=None,+ )+ reporter = progress.safe_progress_reporter(rich_reporter)+ reporter.start(_plan(progress))+ assert reporter.is_active is False+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="evaluating"))+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="complete", detail="agent finished"))+ reporter.close()+ # The terminal takes whatever the stream still holds once the run is over.+ terminal.refused_writes = set()+ stream.flush()+ output = terminal.received.decode("utf-8")++ # The refused line reaches the terminal once, from the stream's buffer,+ # and one stop notice ends the run's output. The stream would also keep a+ # notice the terminal refused and send it before the late one, so the+ # notice is not written while the terminal still refuses the line, and a+ # notice the stream kept is not followed by the late one.+ assert output.count(plan_line) == 1+ assert output.count("Tier 3 progress display stopped after a drawing error") == 1+ assert f"\n{output}".endswith(f"\n{notice}\n")+ assert "Harbor Run Configuration" not in output+ assert "agent finished" not in output+++ "full_at_close", [False, True], ids=["terminal-takes-writes-at-close", "terminal-full-at-close"]+)+def test_a_stop_notice_kept_by_a_buffered_terminal_stream_is_not_repeated_when_the_run_closes(+ monkeypatch: pytest.MonkeyPatch,+ full_at_close: bool,+) -> None:+ from rich.console import Console++ progress = _progress_module()+ monkeypatch.setenv("TERM", "xterm-256color")++ class Terminal(_WouldBlockTerminal):+ """Refuses the first write after it is armed, and the first write that carries the stop notice."""++ armed = False+ full = False+ notice_refused = False++ def write(self, data: Any) -> int | None:+ if self.full:+ return None+ if self.armed:+ self.armed = False+ return None+ if not self.notice_refused and _STOP_NOTICE.encode("utf-8") in bytes(data):+ self.notice_refused = True+ return None+ return super().write(data)++ terminal = Terminal(set())+ # Shaped like standard error: its buffer keeps what the terminal refused.+ stream = io.TextIOWrapper(io.BufferedWriter(terminal), encoding="utf-8", newline="\n", line_buffering=True)+ rich_reporter = progress.RichProgressReporter(stream=stream, refresh_interval=60)+ rich_reporter._console = Console(+ file=stream,+ force_terminal=True,+ legacy_windows=False,+ width=120,+ color_system=None,+ )+ reporter = progress.safe_progress_reporter(rich_reporter)+ reporter.start(_plan(progress))+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="evaluating"))+ # The terminal refuses the update and takes Live's stop, then refuses the+ # notice under the last frame. The stream keeps that notice.+ terminal.armed = True+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="still evaluating"))+ assert reporter.is_active is False+ assert terminal.notice_refused+ terminal.full = full_at_close+ reporter.close()+ # The terminal takes whatever the stream still holds once the run is over.+ terminal.full = False+ stream.flush()+ # How a refused frame splits across the stream's writes depends on the+ # Python version's buffer, so the bytes need not be valid UTF-8.+ output = terminal.received.decode("utf-8", errors="replace")++ # The kept notice reaches the terminal, when the run ends or with the+ # stream's next write, and no late notice repeats it.+ _assert_cursor_shown_and_one_stop_notice(output, _STOP_NOTICE)+ assert output.endswith(f"{_STOP_NOTICE}\n")+++def test_a_stop_notice_refused_after_a_refused_first_frame_is_not_taken_as_kept(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ from rich.console import Console+ from rich.live import Live++ progress = _progress_module()+ monkeypatch.setenv("TERM", "xterm-256color")++ class Terminal(_WouldBlockTerminal):+ """Refuses every write while ``refusing`` is set, and the first write that carries the stop notice."""++ refusing = False+ notice_refused = False++ def write(self, data: Any) -> int | None:+ if self.refusing:+ return None+ if not self.notice_refused and _STOP_NOTICE.encode("utf-8") in bytes(data):+ self.notice_refused = True+ return None+ return super().write(data)++ terminal = Terminal(set())+ # Shaped like standard error, with a buffer smaller than the first frame:+ # it keeps only the start of a refused frame, whatever the Python+ # version's default buffer size.+ stream = io.TextIOWrapper(+ io.BufferedWriter(terminal, buffer_size=1024), encoding="utf-8", newline="\n", line_buffering=True+ )++ class LiveOnAFullTerminal(Live):+ def start(self, refresh: bool = False) -> None:+ try:+ super().start(refresh=refresh)+ finally:+ # The terminal takes writes again once Live gave up.+ terminal.refusing = False++ def refresh(self) -> None:+ # The plan lines and Live's hide-cursor write went out. The+ # terminal refuses the first frame and the stop that Live.start()+ # runs when that frame fails, so Rich's console holds both.+ terminal.refusing = True+ super().refresh()++ rich_reporter = progress.RichProgressReporter(stream=stream, refresh_interval=60)+ rich_reporter._console = Console(+ file=stream,+ force_terminal=True,+ legacy_windows=False,+ width=120,+ color_system=None,+ )+ rich_reporter._live_factory = LiveOnAFullTerminal+ reporter = progress.safe_progress_reporter(rich_reporter)+ reporter.start(_plan(progress))+ # The notice went out after the held frame and stop, in one write larger+ # than the stream's buffer, and the terminal refused it. The stream kept+ # only the start of the frame.+ assert reporter.is_active is False+ assert terminal.notice_refused+ reporter.close()+ stream.flush()+ output = terminal.received.decode("utf-8", errors="replace")++ # The notice was lost with the rest of that write, so the late notice+ # takes its place, after the write that shows the cursor again.+ assert output.count("Tier 3 progress display stopped after a drawing error") == 1+ assert output.endswith(f"{_LATE_STOP_NOTICE}\n")+ assert output.rfind("\x1b[?25h") > output.rfind("\x1b[?25l")++[email protected]("refused_from", ["first-frame", "update"], ids=["first-frame-refused", "update-refused"])+def test_late_notice_is_written_when_the_terminal_refuses_the_cursor_as_the_run_closes(+ monkeypatch: pytest.MonkeyPatch,+ refused_from: str,+) -> None:+ from rich.console import Console+ from rich.live import Live++ progress = _progress_module()+ monkeypatch.setenv("TERM", "xterm-256color")++ class Terminal(_WouldBlockTerminal):+ """Refuses every write while ``refusing`` is set."""++ refusing = False++ def write(self, data: Any) -> int | None:+ if self.refusing:+ return None+ return super().write(data)++ terminal = Terminal(set())+ # Shaped like standard error: its buffer keeps what the terminal refused.+ stream = io.TextIOWrapper(io.BufferedWriter(terminal), encoding="utf-8", newline="\n", line_buffering=True)++ class LiveOnAFillingTerminal(Live):+ def refresh(self) -> None:+ # The plan lines and Live's hide-cursor write went out; the+ # terminal fills up from the first frame on.+ if refused_from == "first-frame":+ terminal.refusing = True+ super().refresh()++ rich_reporter = progress.RichProgressReporter(stream=stream, refresh_interval=60)+ rich_reporter._console = Console(+ file=stream,+ force_terminal=True,+ legacy_windows=False,+ width=120,+ color_system=None,+ )+ rich_reporter._live_factory = LiveOnAFillingTerminal+ reporter = progress.safe_progress_reporter(rich_reporter)+ reporter.start(_plan(progress))+ if refused_from == "update":+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="evaluating"))+ terminal.refusing = True+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="still evaluating"))+ assert reporter.is_active is False+ # The terminal takes writes again while the run goes on, and a warning+ # printed then sends what the stream kept.+ terminal.refusing = False+ stream.write("warning printed after the display stopped\n")+ # It is full again when the run ends, so close() cannot show the cursor.+ terminal.refusing = True+ reporter.close()+ terminal.refusing = False+ stream.flush()+ # The stream's buffer can keep only the start of a refused frame and send+ # it with the next write, cut inside a box-drawing character. How much it+ # keeps depends on the Python version's buffer, so the bytes need not be+ # valid UTF-8.+ output = terminal.received.decode("utf-8", errors="replace")++ # The warning emptied the stream's buffer, so it keeps the late notice+ # with the refused show-cursor write and sends both once the terminal+ # takes writes again.+ _assert_cursor_shown_and_one_stop_notice(output, _LATE_STOP_NOTICE)+ assert output.endswith(f"{_LATE_STOP_NOTICE}\n")+++def test_late_notice_starts_a_line_when_the_stream_sent_a_cut_refused_frame(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ from rich.console import Console++ progress = _progress_module()+ monkeypatch.setenv("TERM", "xterm-256color")+ show_cursor = b"\x1b[?25h"++ class Terminal(_WouldBlockTerminal):+ """Refuses every write while ``refusing`` is set, and one show-cursor write on its own once armed."""++ refusing = False+ cursor_armed = False+ sent_before_cursor = b""++ def write(self, data: Any) -> int | None:+ if self.refusing:+ return None+ if self.cursor_armed and bytes(data) == show_cursor:+ self.cursor_armed = False+ self.sent_before_cursor = bytes(self.received)+ return None+ return super().write(data)++ terminal = Terminal(set())+ # Shaped like standard error, with a buffer smaller than a frame: it keeps+ # only the start of a refused frame, whatever the Python version's default+ # buffer size.+ stream = io.TextIOWrapper(+ io.BufferedWriter(terminal, buffer_size=1024), encoding="utf-8", newline="\n", line_buffering=True+ )+ rich_reporter = progress.RichProgressReporter(stream=stream, refresh_interval=60)+ rich_reporter._console = Console(+ file=stream,+ force_terminal=True,+ legacy_windows=False,+ width=120,+ color_system=None,+ )+ reporter = progress.safe_progress_reporter(rich_reporter)+ reporter.start(_plan(progress))+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="evaluating"))+ # The terminal refuses the update and Live's stop. The stream keeps the+ # start of the update frame and has no room left for the stop.+ terminal.refusing = True+ reporter.emit(progress.ProgressEvent(stage="agent:codex", state="running", detail="still evaluating"))+ assert reporter.is_active is False+ # Nothing else is printed before the run ends. The terminal then takes+ # the part of the frame the stream kept and refuses the write that shows+ # the cursor again, so the line that ends the frame is not written.+ terminal.refusing = False+ terminal.cursor_armed = True+ reporter.close()+ stream.flush()+ assert terminal.cursor_armed is False+ # What reached the terminal before that write ends inside the frame.+ assert terminal.sent_before_cursor.endswith(b"\n") is False+ output = terminal.received.decode("utf-8", errors="replace")++ # The late notice starts a line of its own instead of joining the cut+ # frame's last line.+ _assert_cursor_shown_and_one_stop_notice(output, _LATE_STOP_NOTICE)+ assert output.endswith(f"\n{_LATE_STOP_NOTICE}\n")+++def test_changelog_says_a_stream_can_still_send_refused_output_later() -> None:+ # close() drops only the view's own copy of what the terminal refused. A+ # stream shaped like standard error keeps its copy and sends it with its+ # next write, and no public API drops it. The entry is found by its+ # opening words so that a release heading above it can change.+ lead = (+ "- When the terminal refuses one of the plan lines printed before the "+ "interactive Tier 3 progress view's table first appears"+ )+ changelog = (Path(__file__).resolve().parents[1] / "CHANGELOG.md").read_text(encoding="utf-8")+ # Read each bullet with its indented continuation lines, so the entry is+ # found wherever it sits in its subsection.+ bullets: list[list[str]] = []+ for line in changelog.splitlines():+ if line.startswith("- "):+ bullets.append([line])+ elif bullets and line.startswith(" ") and line.strip():+ bullets[-1].append(line)+ elif not line.strip() and bullets and bullets[-1]:+ bullets.append([])+ blocks = [" ".join(" ".join(bullet).split()) for bullet in bullets if bullet]+ entries = [block for block in blocks if block.startswith(lead)]++ assert len(entries) == 1+ # One blank line, not two, separates the entry from what follows it.+ raw_entry_start = changelog.index("- When the terminal refuses one of the plan lines printed before the\n")+ raw_entry_end = changelog.index("\n\n", raw_entry_start)+ assert not changelog.startswith("\n\n\n", raw_entry_end)+ entry = entries[0]+ assert "no longer keeps its own copy of output the terminal still refuses when the run ends" in entry+ assert (+ "A stream that keeps what the terminal refused, as standard error does, can still send it later, "+ "and a final table sent then can still erase those lines." in entry+ )+ # A notice line the stream kept goes out when the run ends, in place of+ # the late one, as the buffered-stream tests above check.+ assert (+ "When the terminal takes the writes before that line and refuses only the line itself, a stream that "+ "keeps what the terminal refused, as standard error does, sends that line when the run ends in place of "+ "the later one." in entry+ )+++def test_live_evaluation_docs_say_a_stream_can_still_send_refused_output_later() -> None:+ # The guide's description of the rich view states the same limit as the+ # CHANGELOG entry above, instead of saying refused frames can never be+ # drawn over later output. The paragraph is found by its opening words.+ # The view drops its copy when every run ends, so the sentence is not+ # tied to the drawing failure described just before it: after a failure+ # there is no final table.+ lead = "In a terminal, `--progress auto` uses the `rich` view"+ guide = (Path(__file__).resolve().parents[1] / "docs" / "tier3-live-evaluation.mdx").read_text(encoding="utf-8")+ blocks = [" ".join(block.split()) for block in guide.split("\n\n")]+ paragraphs = [block for block in blocks if block.startswith(lead)]++ assert len(paragraphs) == 1+ paragraph = paragraphs[0]+ assert "never drawn over" not in paragraph+ assert "The view then drops" not in paragraph+ assert (+ "When any run ends, whether or not drawing failed, the view drops its own copy of output the terminal "+ "still refuses, such as a refused final table, so it does not draw that output over lines printed since."+ in paragraph+ )+ assert (+ "A stream that keeps what the terminal refused, as standard error does, can still send it later, "+ "and a table sent then can still erase those lines." in paragraph+ )+ # Standard error keeps a refused notice line and sends it when the run+ # ends, so the late line does not follow it, except after a failed first+ # frame.+ assert (+ "If the terminal refused only that line, a stream that keeps what the terminal refused, as standard "+ "error does, sends that line then instead of the later one, although after a failed first frame both "+ "can appear." in paragraph+ )+ # The late line and the cursor write are kept only while the stream's+ # buffer has room: one already full of a refused frame takes neither, and+ # the terminal never gets them.+ assert (+ "If the terminal still refuses writes when the run ends, such a stream sends the later line and the "+ "write that shows the cursor once the terminal takes writes again, but only if its buffer has room for "+ "them. A buffer already full of a refused table can drop that line and leave the cursor hidden." in paragraph+ )++class _InterleavingLock:"""A reentrant lock that runs a hook once, when the arming thread next releases it fully."""