A self-improving RLM agent for coding workflows and long-running autonomous tasks.
Rust
feat(ci): add UI interaction benchmarks to Prime Agent performance CI (ENG-6043) (#2211)
Seth Karten7a049d50c61e
diff --git a/scripts/benchmarks/README.md b/scripts/benchmarks/README.mdindex c2bca0e7d3..31c648cb1f 100644--- a/scripts/benchmarks/README.md+++ b/scripts/benchmarks/README.md@@ -65,7 +65,7 @@ and other missing tools. Their setup time and disk usage are outside the installThe controller, PTY harness, artifact server, and build user are excluded. RSS can double-countshared pages.-Startup and memory use 10 trials per revision. Installation uses three; sizes are measured once.+Startup and memory use 10 trials per revision. Installation uses three; sizes are measured once. UI interactions use 3 trials per revision (`ui_trials`).Compiled revisions provision pinned Bun tooling and build their Linux x64 archive during untimedsetup. The installer selects its normal default from the available artifacts; the harness does notforce Node or compiled mode. Loopback downloads use the installer's explicit test exception, while@@ -133,6 +133,56 @@ size, and 1% for disk footprint. A same-revision calibration on separate sandboxSmaller signed differences remain visible. Revisit these floors after collecting more control runs. This is a practical noise filter, not a statistical significance test. Inspectraw trials before acting on small changes; sandbox scheduling and filesystem caches still introduce noise.+### UI interactions++A separate probe drives the interactive TUI over a PTY with the same `pexpect`/`pyte` harness the+startup metrics use, against a deterministic on-disk fixture set. Every UI trial regenerates the+sessions directory, session artifacts, and spawn ledger from scratch, so trials cannot inherit state.++The fixture set is fixed and mirrors a long-lived install: 194 top-level sessions (3 large —+one of them a very large ~40 MB transcript — 150 medium, 40 small fan-out children of the chain+root, and the root itself) plus a chain of 6 nested subagent sessions. Large sessions hold 1,999+user/assistant/toolResult triples (~5 MB of JSONL each), the very large one 15,999 (~40 MB),+medium sessions 119, and each subagent 399. Content is synthetic but shaped like real transcripts+(thinking, tool calls, tool output). The subagent chain and the fan-out are linked through the+same spawn-ledger records the daemon replays (~46 edges), so roster hydration, catalog scanning,+and saved-session paths are exercised at a scale where regressions are visible. No inference is+called at any point.++Each trial stops all benchmark-user processes first, then measures:++- **Resume large session (cold):** `prime-agent --resume <id>` from process spawn until the resumed+ transcript tail is rendered and the editor echoes a typed marker. This is the "switch into a large+ session" path including daemon and session-worker startup.+- **Switch into large session:** in a running TUI, `/resume <id>` keystroke to rendered tail plus+ echoed marker. Measures the in-place session switch with a ~40 MB transcript.+- **Open agents view from a session:** the left-arrow keystroke to the rendered agents-view splash.+- **Full agents roster, many sessions:** after the agents-view splash is rendered, until the+ roster lists the fixture's saved sessions and its count stops growing. Saved sessions stream in, so this isolates+ catalog scanning, spawn-ledger replay, and hydration across 200 session files.+- **Open another session from agents view:** typing the target's unique session-id prefix,+ right-arrow to open, until the target transcript renders and echoes. This is the full+ "session → agents view → another session" round trip with many sessions on disk.+- **Open chain parent from agents view:** back to the agents view, search the subagent chain's+ root session by id prefix, right-arrow to open. The parent becomes live.+- **Open subagent session at depth 6:** back to the agents view, then drill into the live chain:+ filter by the deepest subagent's id prefix, clear the search (expansion needs an empty query),+ expand the parent, refilter past the fan-out siblings, and repeat clear-expand-step for each of+ the six levels until the deepest subagent row is selected, then right-arrow to open. This is the+ exact manual flow the product requires today, so the metric includes its real cost.+- **CPU per interaction:** summed `utime+stime` across the whole benchmark-user process tree+ (TUI client, daemon, session workers, Python kernels) between two instants around each interaction.+- **UI memory after interactions:** summed RSS across the same tree after the scenario settles.+ Per-process snapshots with PSS and CPU are saved as raw artifacts, as for idle memory.++Readiness is never a spinner or a status line: a trial only counts when the target session's+final transcript line is visible and the editor echoes a marker, so half-rendered states fail+loudly instead of measuring fast. The roster wait likewise only settles once the fixture's saved+sessions are listed, so a stalled or empty catalog scan fails the trial instead of recording a+fast sample. Raw results also record PTY bytes per interaction, a proxy for+how much the renderer redraws. UI trials default to 3 per revision (`ui_trials` in `config.json`)+to bound sandbox cost; the scenario takes roughly 60–90 s per trial.+## Lifecycle and costsThe main workflow posts a single marked comment and updates it in place. A new push replaces the@@ -147,7 +197,7 @@ SHAs, environment details, installed dependency inventory, transcripts, raw obseand estimated sandbox costs. The publisher validates schema, identity, finite numbers, and reportsize and escapes sandbox-provided text.-The controller stops scheduling work at 20 minutes or its $1 estimated budget target. Every sandbox+The controller stops scheduling work at 25 minutes or its $1 estimated budget target. Every sandboxhas a 30-minute TTL. Teardown runs in `finally`; the independent completion workflow deletes anyremaining sandboxes with the exact repository/run/attempt labels, including after cancellation.Cleanup enumerates all pages before deleting so pagination cannot skip a sandbox.@@ -162,7 +212,8 @@ Log-collection and deferred sandbox-cleanup problems appear as operational warnimark complete measurements as partial; missing or failed measurements still fail the command.At the configured list rates, two sandboxes cost about $0.01/minute together. A 10-minute run costs-about $0.10 in sandbox compute. There are no inference charges. Full sandbox lifetimes are recorded,+about $0.10 in sandbox compute. There are no inference charges. UI-interaction trials add roughly+3–6 minutes per run at the default 3 trials per revision. Full sandbox lifetimes are recorded,including provisioning, setup, building, measurements, and cleanup. The $1 target is an estimate,not a hard billing cap; scheduling and deletion latency can increase the final bill.diff --git a/scripts/benchmarks/config.json b/scripts/benchmarks/config.jsonindex f0a6962115..3e2e5a519d 100644--- a/scripts/benchmarks/config.json+++ b/scripts/benchmarks/config.json@@ -7,10 +7,11 @@"trials": 10,"install_trials": 3,"debounce_seconds": 2,- "timeout_seconds": 1200,+ "timeout_seconds": 1500,"ttl_minutes": 30,"budget_usd": 1,"cpu_usd_per_hour": 0.05,"memory_usd_per_gb_hour": 0.01,- "disk_usd_per_gb_hour": 0.001+ "disk_usd_per_gb_hour": 0.001,+ "ui_trials": 3}diff --git a/scripts/benchmarks/controller.py b/scripts/benchmarks/controller.pyindex 62de2276bc..8077175281 100644--- a/scripts/benchmarks/controller.py+++ b/scripts/benchmarks/controller.py@@ -26,7 +26,7 @@ from schema import (REMOTE = "/opt/prime-benchmark"OWNER_LABEL = "prime-agent-benchmarks-v1"-FILES = ("schema.py", "terminal.py", "kernel.py", "worker.py", "pyproject.toml", "uv.lock")+FILES = ("schema.py", "terminal.py", "kernel.py", "worker.py", "ui.py", "pyproject.toml", "uv.lock")def elapsed_seconds(start: datetime) -> float:@@ -279,10 +279,11 @@ class Controller:("install", self.report.config.install_trials),("measure", self.report.config.trials),("runtime", self.report.config.trials),+ ("ui", self.report.config.ui_trials),):blocked: set[str] = set()failures: dict[str, tuple[str, int]] = {}- if phase in ("measure", "runtime"):+ if phase in ("measure", "runtime", "ui"):for role in ready:side = self.report.main if role == "main" else self.report.pr_headif not any(diff --git a/scripts/benchmarks/report.py b/scripts/benchmarks/report.pyindex c70a9b2813..4de03c49bf 100644--- a/scripts/benchmarks/report.py+++ b/scripts/benchmarks/report.py@@ -7,7 +7,7 @@ from collections import Counterfrom dataclasses import dataclassfrom typing import Literal-from schema import Metric, Observation, Report+from schema import UI_METRIC_KEYS, Metric, Observation, ReportMARKER = "<!-- prime-agent-benchmark:v1 -->"PERFORMANCE_NOISE_FLOOR = 0.2@@ -107,6 +107,43 @@ RUNTIME_METRICS = (PERFORMANCE_NOISE_FLOOR,),)+UI_METRICS = (+ Definition("resume_large", "Resume large session (cold)", 1000, "ms", 0.1, PERFORMANCE_NOISE_FLOOR),+ Definition("resume_large_cpu", "CPU, resume large session", 1000, "ms", 0.05, PERFORMANCE_NOISE_FLOOR),+ Definition("switch_large", "Switch into large session", 1000, "ms", 0.05, PERFORMANCE_NOISE_FLOOR),+ Definition(+ "switch_large_cpu", "CPU, switch into large session", 1000, "ms", 0.02, PERFORMANCE_NOISE_FLOOR+ ),+ Definition("agents_view", "Open agents view from a session", 1000, "ms", 0.02, PERFORMANCE_NOISE_FLOOR),+ Definition("agents_view_cpu", "CPU, open agents view", 1000, "ms", 0.01, PERFORMANCE_NOISE_FLOOR),+ Definition("agents_roster", "Full agents roster, many sessions", 1, "s", 0.1, PERFORMANCE_NOISE_FLOOR),+ Definition("agents_roster_cpu", "CPU, full agents roster", 1, "s", 0.05, PERFORMANCE_NOISE_FLOOR),+ Definition(+ "agents_open", "Open another session from agents view", 1000, "ms", 0.1, PERFORMANCE_NOISE_FLOOR+ ),+ Definition("agents_open_cpu", "CPU, open from agents view", 1000, "ms", 0.05, PERFORMANCE_NOISE_FLOOR),+ Definition(+ "subagent_open",+ "Open subagent session at depth 6",+ 1000,+ "ms",+ 0.1,+ PERFORMANCE_NOISE_FLOOR,+ ),+ Definition(+ "subagent_open_cpu", "CPU, open subagent at depth 6", 1000, "ms", 0.05, PERFORMANCE_NOISE_FLOOR+ ),+ Definition(+ "parent_open",+ "Open chain parent from agents view",+ 1000,+ "ms",+ 0.1,+ PERFORMANCE_NOISE_FLOOR,+ ),+ Definition("parent_open_cpu", "CPU, open chain parent", 1000, "ms", 0.05, PERFORMANCE_NOISE_FLOOR),+ Definition("ui_rss", "UI memory after interactions", 1e-6, "MB", 10485760, PERFORMANCE_NOISE_FLOOR),+)def escape(text: str) -> str:@@ -190,10 +227,12 @@ def comparison(def comparisons(report: Report) -> dict[Metric, Comparison]:results = {}- for definition in (*METRICS, *RUNTIME_METRICS):+ for definition in (*METRICS, *RUNTIME_METRICS, *UI_METRICS):expected = report.config.install_trials if definition.key == "install" else report.config.trialsif definition.key in ("bundle", "disk"):expected = 1+ if definition.key in UI_METRIC_KEYS:+ expected = report.config.ui_trialsresults[definition.key] = comparison(definition,report.main.metrics.get(definition.key, []),@@ -272,7 +311,7 @@ def render(report: Report) -> str:"| --- | ---: | ---: | ---: |",])- for definition in (*METRICS, *RUNTIME_METRICS):+ for definition in (*METRICS, *RUNTIME_METRICS, *UI_METRICS):if definition == RUNTIME_METRICS[0]:lines.extend([@@ -283,6 +322,16 @@ def render(report: Report) -> str:"| --- | ---: | ---: | ---: |",])+ if definition == UI_METRICS[0]:+ lines.extend(+ [+ "",+ "**UI interactions**",+ "",+ "| Metric | Main | This PR | Change |",+ "| --- | ---: | ---: | ---: |",+ ]+ )result = results[definition.key]lines.append(f"| {definition.title} | {result.main} | {result.head} | {result.change} |")compute = sum(s.estimated_usd for s in report.sandboxes)@@ -319,6 +368,15 @@ def render(report: Report) -> str:"State fixture: a 10,000-row × 8-column integer DataFrame and a 10,000-integer list.","Restore runs in a fresh kernel, including pandas imports; kernel startup is excluded.","Kernel RSS covers the isolated Python process; loaded RSS follows the pandas workload.",+ "UI trials use a fresh fixture set: 194 top-level sessions including one ~40 MB transcript,",+ "40 ledger fan-out children, and a 6-deep subagent chain (~46 spawn edges).",+ "Large fixtures hold 1,999 complete triples (~5 MB JSONL); medium 119; subagents 399 each.",+ "Interactions: cold --resume of a large session, warm /resume switch, left-arrow to agents view,",+ "roster settle with many saved sessions, search-and-open of another large session, and opening",+ "the chain parent, then a clear-expand-step drill into the live chain to depth 6.",+ "Readiness is the rendered transcript tail plus a confirmed editor echo.",+ "CPU metrics sum utime+stime across the whole benchmark-user process tree per interaction.",+ "UI memory sums RSS after the interactions; PTY byte counts are in the raw results.","Costs estimate full sandbox lifetimes at configured rates, including setup and build.",f"Budget target: ${report.config.budget_usd:g}; not a billing cap. ""Performance changes are informational.",@@ -330,7 +388,7 @@ def render(report: Report) -> str:"| --- | ---: | ---: | ---: | ---: |",])- for definition in (*METRICS, *RUNTIME_METRICS):+ for definition in (*METRICS, *RUNTIME_METRICS, *UI_METRICS):left = report.main.metrics.get(definition.key, [])right = report.pr_head.metrics.get(definition.key, [])lines.append(diff --git a/scripts/benchmarks/schema.py b/scripts/benchmarks/schema.pyindex 475e80cf67..18393bdcbe 100644--- a/scripts/benchmarks/schema.py+++ b/scripts/benchmarks/schema.py@@ -27,7 +27,41 @@ Metric = Literal["restore","kernel_rss","loaded_rss",+ "resume_large",+ "resume_large_cpu",+ "switch_large",+ "switch_large_cpu",+ "agents_view",+ "agents_view_cpu",+ "agents_roster",+ "agents_roster_cpu",+ "agents_open",+ "agents_open_cpu",+ "subagent_open",+ "subagent_open_cpu",+ "parent_open",+ "parent_open_cpu",+ "ui_rss",]+UI_METRIC_KEYS = frozenset(+ {+ "resume_large",+ "resume_large_cpu",+ "switch_large",+ "switch_large_cpu",+ "agents_view",+ "agents_view_cpu",+ "agents_roster",+ "agents_roster_cpu",+ "agents_open",+ "agents_open_cpu",+ "subagent_open",+ "subagent_open_cpu",+ "parent_open",+ "parent_open_cpu",+ "ui_rss",+ }+)NonNegative = Annotated[float, Field(ge=0, le=1e15, allow_inf_nan=False)]ROOT = Path(__file__).resolve().parentUV_VERSION = "0.12.9"@@ -46,6 +80,7 @@ class Config(StrictModel):trials: Annotated[int, Field(ge=3, le=50)]install_trials: Annotated[int, Field(ge=1, le=10)]failure_limit: Annotated[int, Field(ge=1, le=10)] = 2+ ui_trials: Annotated[int, Field(ge=1, le=50)] = 3debounce_seconds: Annotated[int, Field(ge=0, le=300)]timeout_seconds: Annotated[int, Field(ge=60, le=3600)]ttl_minutes: Annotated[int, Field(ge=1, le=120)]@@ -89,6 +124,7 @@ class ProcessMemory(StrictModel):name: Annotated[str, Field(max_length=80)]rss: NonNegativepss: NonNegative | None = None+ cpu: NonNegative = 0.0class Side(StrictModel):@@ -155,6 +191,8 @@ class Report(StrictModel):expected = self.config.install_trials if metric == "install" else self.config.trialsif metric in ("bundle", "disk"):expected = 1+ if metric in UI_METRIC_KEYS:+ expected = self.config.ui_trialsif (len(samples) > expectedor len({s.trial for s in samples}) != len(samples)@@ -181,6 +219,23 @@ PHASE_METRICS: dict[str, tuple[Metric, ...]] = {"kernel_rss","loaded_rss",),+ "ui": (+ "resume_large",+ "resume_large_cpu",+ "switch_large",+ "switch_large_cpu",+ "agents_view",+ "agents_view_cpu",+ "agents_roster",+ "agents_roster_cpu",+ "agents_open",+ "agents_open_cpu",+ "subagent_open",+ "subagent_open_cpu",+ "parent_open",+ "parent_open_cpu",+ "ui_rss",+ ),}@@ -193,10 +248,18 @@ def trial_errors(side: Side, phase: str, trial: int) -> list[str]:return errors or [f"{metric}: measurement missing" for metric in metrics if not samples[metric]]-def side_complete(side: Side, trials: int, installs: int) -> bool:+def side_complete(side: Side, trials: int, installs: int, ui_trials: int) -> bool:for metrics in PHASE_METRICS.values():for metric in metrics:- count = 1 if metric in ("bundle", "disk") else installs if metric == "install" else trials+ count = (+ 1+ if metric in ("bundle", "disk")+ else installs+ if metric == "install"+ else ui_trials+ if metric in UI_METRIC_KEYS+ else trials+ )samples = side.metrics.get(metric, [])if ({sample.trial for sample in samples} != set(range(count))@@ -209,7 +272,7 @@ def side_complete(side: Side, trials: int, installs: int) -> bool:def report_complete(report: Report) -> bool:return not report.errors and all(- side_complete(side, report.config.trials, report.config.install_trials)+ side_complete(side, report.config.trials, report.config.install_trials, report.config.ui_trials)for side in (report.main, report.pr_head))diff --git a/scripts/benchmarks/tests/test_benchmarks.py b/scripts/benchmarks/tests/test_benchmarks.pyindex 3cdb8fa5d8..ea5c3aab2d 100644--- a/scripts/benchmarks/tests/test_benchmarks.py+++ b/scripts/benchmarks/tests/test_benchmarks.py@@ -18,8 +18,9 @@ from pydantic import ValidationErrorfrom cli import completed_report, main, validate_completion, workflow_sourcefrom controller import Canceled, Controller, cleanup, labelsfrom github import TITLE, GitHub-from report import MARKER, METRICS, RUNTIME_METRICS, comparison, render+from report import MARKER, METRICS, RUNTIME_METRICS, UI_METRICS, comparison, renderfrom schema import (+ UI_METRIC_KEYS,Config,Observation,ProcessMemory,@@ -186,11 +187,11 @@ class ReportTests(unittest.TestCase):report.pr_head.metrics["bundle"] = []text = render(report)self.assertIn(- "**Overall: 1 regressed · 1 improved · 13 no clear change · 1 incomplete · 1 unavailable.**",+ "**Overall: 1 regressed · 1 improved · 13 no clear change · 1 incomplete · 16 unavailable.**",text,)tables = text.split("<details>")[0]- self.assertEqual(tables.count("| Metric | Main | This PR | Change |"), 2)+ self.assertEqual(tables.count("| Metric | Main | This PR | Change |"), 3)for row in tables.splitlines():if row.startswith("|"):self.assertEqual(len(row.strip("|").split("|")), 4)@@ -711,14 +712,18 @@ class MeasurementTests(unittest.TestCase):def test_completion_requires_every_metric(self):side = Side(sha=SHA)- for definition in (*METRICS, *RUNTIME_METRICS):- count = 1 if definition.key in ("bundle", "disk") else (3 if definition.key == "install" else 10)+ for definition in (*METRICS, *RUNTIME_METRICS, *UI_METRICS):+ count = (+ 1+ if definition.key in ("bundle", "disk")+ else (3 if definition.key == "install" else (2 if definition.key in UI_METRIC_KEYS else 10))+ )side.metrics[definition.key] = observations(*([1] * count))- self.assertTrue(side_complete(side, 10, 3))- for definition in (*METRICS, *RUNTIME_METRICS):+ self.assertTrue(side_complete(side, 10, 3, 2))+ for definition in (*METRICS, *RUNTIME_METRICS, *UI_METRICS):incomplete = side.model_copy(deep=True)incomplete.metrics[definition.key][-1] = Observation(trial=0, error="failed")- self.assertFalse(side_complete(incomplete, 10, 3))+ self.assertFalse(side_complete(incomplete, 10, 3, 2))if __name__ == "__main__":diff --git a/scripts/benchmarks/tests/test_failure_reporting.py b/scripts/benchmarks/tests/test_failure_reporting.pyindex 3dd5347b05..86f5f58542 100644--- a/scripts/benchmarks/tests/test_failure_reporting.py+++ b/scripts/benchmarks/tests/test_failure_reporting.py@@ -116,8 +116,11 @@ class FailureBudgetTests(unittest.TestCase):),)self.assertEqual(len([c for c in calls if c[:2] == ("pr", "install")]), 2)- self.assertFalse(any(c[0] == "pr" and c[1] in ("measure", "runtime") for c in calls))+ self.assertFalse(any(c[0] == "pr" and c[1] in ("measure", "runtime", "ui") for c in calls))+ self.assertTrue(any(c[:2] == ("main", "ui") for c in calls))self.assertNotIn("cold", controller.report.pr_head.metrics)+ self.assertNotIn("resume_large", controller.report.pr_head.metrics)+ self.assertTrue(any("pr ui: skipped" in e for e in controller.report.errors))self.assertTrue(any("first installation" in e for e in controller.report.errors))@patch.dict(os.environ, {"PRIME_SANDBOX_API_KEY": "fake"})diff --git a/scripts/benchmarks/tests/test_ui.py b/scripts/benchmarks/tests/test_ui.pynew file mode 100644index 0000000000..7da5206100--- /dev/null+++ b/scripts/benchmarks/tests/test_ui.py@@ -0,0 +1,349 @@+from __future__ import annotations++import hashlib+import json+import os+import tempfile+import unittest+from datetime import UTC, datetime+from pathlib import Path+from typing import get_args+from unittest.mock import patch++import ui+from report import UI_METRICS, render+from schema import UI_METRIC_KEYS, Config, Metric, Observation, ProcessMemory, Report, Request, Side+from ui import (+ expand_subagents,+ expected_roster_inactive,+ fixture_spec,+ session_id,+ session_name,+ spawn_ledger_path,+ tail_marker,+ total_cpu,+ ui_measure,+ wait_for_roster,+ write_fixtures,+)+++def small_fixture(**overrides):+ """Fixture generation with tiny counts so tests stay fast."""+ constants = {+ "MEDIUM_COUNT": 3,+ "MEDIUM_MESSAGES": 2,+ "LARGE_COUNT": 2,+ "LARGE_MESSAGES": 3,+ "HUGE_MESSAGES": 4,+ "FANOUT_COUNT": 2,+ "FANOUT_MESSAGES": 2,+ "SUBAGENT_DEPTH": 4,+ "SUBAGENT_MESSAGES": 2,+ }+ constants.update(overrides)+ return [patch(f"ui.{name}", value) for name, value in constants.items()]+++def ui_request() -> Request:+ """A minimal valid request for ui_measure entry checks."""+ return Request(+ repository="PrimeIntellect-ai/prime-agent",+ source_repository="PrimeIntellect-ai/prime-agent",+ sha="a" * 40,+ harness_sha="a" * 40,+ pr=42,+ run_id=100,+ attempt=1,+ role="main",+ config=Config.load(),+ )+++class FixtureTests(unittest.TestCase):+ def setUp(self):+ temporary = tempfile.TemporaryDirectory()+ self.addCleanup(temporary.cleanup)+ self.root = Path(temporary.name)+ self.agent_dir = self.root / "agent"+ self.workspace = self.root / "workspace"+ self.workspace.mkdir()+ self.spec: object = None++ def small_fixture(self, **overrides) -> None:+ for patcher in small_fixture(**overrides):+ patcher.start()+ self.addCleanup(patcher.stop)+ self.spec = fixture_spec()++ def write(self):+ write_fixtures(self.agent_dir, self.workspace, self.spec)+ return self.agent_dir / "sessions"++ def test_ids_and_names_are_deterministic(self):+ self.assertEqual(session_id("large", 1), session_id("large", 1))+ self.assertEqual(session_id("sub", 6), fixture_spec().subagents[5])+ self.assertNotEqual(session_id("large", 1), session_id("large", 2))+ self.assertEqual(session_name("large", 1), "ui-bench-large-01")+ self.assertEqual(tail_marker("sub", 6), "ui-bench-tail-sub-06")++ def test_write_fixtures_regenerates_identical_files_and_clears_stale_state(self):+ self.small_fixture()+ sessions = self.write()+ first = {path.name: path.read_bytes() for path in sorted(self.agent_dir.rglob("*")) if path.is_file()}+ stale = sessions / "stale.jsonl"+ stale.write_text("stale")+ (self.agent_dir / "session-artifacts").mkdir(parents=True)+ (self.agent_dir / "session-artifacts" / "leftover.json").write_text("{}")+ (self.agent_dir / "session-leases").mkdir(parents=True)+ write_fixtures(self.agent_dir, self.workspace, self.spec)+ second = {+ path.name: path.read_bytes() for path in sorted(self.agent_dir.rglob("*")) if path.is_file()+ }+ self.assertEqual(first, second)+ self.assertFalse(stale.exists())+ self.assertFalse((self.agent_dir / "session-artifacts" / "leftover.json").exists())+ expected = ui.LARGE_COUNT + ui.MEDIUM_COUNT + 1 + ui.SUBAGENT_DEPTH + ui.FANOUT_COUNT+ self.assertEqual(len(list(sessions.glob("*.jsonl"))), expected)++ def test_session_file_shape_and_tree_linking(self):+ self.small_fixture(LARGE_MESSAGES=4)+ sessions = self.write()+ identifier = self.spec.large[0]+ entries = [json.loads(line) for line in (sessions / f"{identifier}.jsonl").read_text().splitlines()]+ header = entries[0]+ self.assertEqual(header["type"], "session")+ self.assertEqual(header["version"], 3)+ self.assertEqual(header["id"], identifier)+ self.assertEqual(header["cwd"], str(self.workspace))+ self.assertEqual(header["rlmDepth"], 0)+ names = [entry for entry in entries if entry.get("type") == "session_info"]+ self.assertEqual(names[0]["name"], session_name("large", 1))+ previous = None+ for entry in entries[1:]:+ self.assertEqual(entry["parentId"], previous)+ previous = entry["id"]+ assistants = [entry for entry in entries if entry.get("message", {}).get("role") == "assistant"]+ self.assertTrue(any(tail_marker("large", 1) in json.dumps(entry) for entry in assistants[-1:]))+ self.assertGreater(len(entries), ui.LARGE_MESSAGES)++ def test_subagent_chain_depths_and_spawn_ledger(self):+ self.small_fixture()+ sessions = self.write()+ ledger = spawn_ledger_path(self.agent_dir, sessions)+ expected_digest = hashlib.sha256(str(Path(os.path.realpath(sessions))).encode()).hexdigest()[:16]+ self.assertEqual(ledger.name, f"{expected_digest}.jsonl")+ records = [json.loads(line) for line in ledger.read_text().splitlines()]+ self.assertEqual(records[0]["op"], "meta")+ self.assertEqual(records[0]["sessionsDir"], str(Path(os.path.realpath(sessions))))+ spawns = records[1:]+ self.assertEqual(len(spawns), len(self.spec.subagents) + len(self.spec.fanout))+ chain = [record for record in spawns if record["name"].startswith("ui-bench-sub-")]+ self.assertEqual(len(chain), len(self.spec.subagents))+ fanout = [record for record in spawns if record["name"].startswith("ui-bench-fan-")]+ self.assertEqual(len(fanout), len(self.spec.fanout))+ parent = str(sessions / f"{self.spec.root}.jsonl")+ for record in fanout:+ self.assertEqual(record["depth"], 1)+ self.assertEqual(record["parent"], parent)+ for depth, (record, identifier) in enumerate(zip(chain, self.spec.subagents, strict=True), start=1):+ self.assertEqual(record["op"], "spawn")+ self.assertEqual(record["v"], 1)+ self.assertEqual(record["depth"], depth)+ self.assertEqual(record["parent"], parent)+ self.assertEqual(record["child"], str(sessions / f"{identifier}.jsonl"))+ self.assertEqual(record["name"], session_name("sub", depth))+ parent = record["child"]+ header = json.loads((sessions / f"{identifier}.jsonl").read_text().splitlines()[0])+ self.assertEqual(header["rlmDepth"], depth)+++class ProbeLogicTests(unittest.TestCase):+ def test_total_cpu_sums_process_tree(self):+ processes = [+ ProcessMemory(pid=1, name="daemon", rss=10, cpu=1.25),+ ProcessMemory(pid=2, name="worker", rss=20, cpu=0.75),+ ProcessMemory(pid=3, name="kernel", rss=30, cpu=2.0),+ ]+ self.assertEqual(total_cpu(processes), 4.0)+ self.assertEqual(total_cpu([]), 0.0)++ def test_wait_for_roster_requires_a_settled_count(self):+ frames = [+ "agents 0 running, 1 idle, 5 inactive",+ "agents 0 running, 1 idle, 40 inactive",+ "agents 0 running, 1 idle, 70 inactive",+ "agents 0 running, 1 idle, 70 inactive",+ "agents 0 running, 1 idle, 70 inactive",+ "agents 0 running, 1 idle, 70 inactive",+ ]+ terminal = FakeTerminal(frames)+ with patch("time.perf_counter", side_effect=[float(i) for i in range(200)]):+ seconds = wait_for_roster(terminal, minimum=70, settle=3.0, timeout=90)+ self.assertGreaterEqual(seconds, 0.0)+ self.assertIn("70 inactive", terminal.display.text())++ with self.assertRaises(TimeoutError):+ wait_for_roster(+ FakeTerminal(["agents 0 running, 1 idle, 1 inactive"] * 8),+ minimum=70,+ settle=3.0,+ timeout=0.05,+ )++ def test_wait_for_roster_never_settles_below_the_expected_fixture_count(self):+ # A splash stuck at an empty or partial roster must not count as hydrated,+ # even when its count is unchanged for the whole settle window.+ terminal = FakeTerminal(["agents 0 running, 1 idle, 5 inactive"] * 8)+ with patch("time.perf_counter", side_effect=[float(i) for i in range(400)]):+ with self.assertRaises(TimeoutError):+ wait_for_roster(terminal, minimum=70, settle=3.0, timeout=90)++ def test_expected_roster_inactive_counts_top_level_fixtures_minus_the_live_target(self):+ self.assertEqual(expected_roster_inactive(), ui.LARGE_COUNT + ui.MEDIUM_COUNT)++ def test_ui_trial_requires_a_successful_first_installation(self):+ side = Side(sha="a" * 40)+ with self.assertRaisesRegex(RuntimeError, "first installation"):+ ui_measure(ui_request(), side, 0, results=Path("/unused"), homes=Path("/unused"), user="bench")++ def test_expand_subagents_requires_a_newly_expanded_row(self):+ # An ancestor already shows "▾", so only a growing count proves the selected row expanded;+ # the first expand keystroke is eaten and the retry lands.+ frames = [+ "▾ ui-bench-sub-00",+ "▾ ui-bench-sub-00",+ "▾ ui-bench-sub-00\n▾ ui-bench-sub-01",+ ]+ terminal = FakeTerminal(frames)+ expand_subagents(terminal)+ self.assertEqual(terminal.sent, [ui.EXPAND_ARROW, ui.EXPAND_ARROW])++ def test_expand_subagents_times_out_when_nothing_expands(self):+ frames = ["▸ ui-bench-sub-00"] * 8+ terminal = FakeTerminal(frames)+ with self.assertRaises(TimeoutError):+ expand_subagents(terminal)+ self.assertEqual(terminal.sent, [ui.EXPAND_ARROW] * 4)+++class FakeDisplay:+ def __init__(self, text: str = ""):+ self._text = text++ def text(self) -> str:+ return self._text++ def set(self, text: str) -> None:+ self._text = text+++class FakeChild:+ """Records sent keystrokes the way pexpect.spawn.send would receive them."""++ def __init__(self, sent: list[str]):+ self.sent = sent++ def send(self, data: str) -> None:+ self.sent.append(data)+++class FakeTerminal:+ """Scripted terminal: until() and settle() advance the frames one at a time."""++ def __init__(self, frames: list[str]):+ self.display = FakeDisplay(frames[0] if frames else "")+ self.frames = list(frames[1:])+ self.sent: list[str] = []+ self.child = FakeChild(self.sent)+ self.bytes = 0++ def pump(self) -> bool:+ if not self.frames:+ return False+ self.display.set(self.frames.pop(0))+ self.bytes += len(self.display.text())+ return True++ def until(self, predicate, timeout: float) -> None: # type: ignore[no-untyped-def]+ while not predicate(self.display):+ if not self.pump():+ raise TimeoutError("Timed out waiting for the expected terminal state")++ def settle(self, seconds: float) -> None:+ self.pump()+++class SchemaTests(unittest.TestCase):+ def test_ui_metric_keys_match_report_definitions_and_metric_literal(self):+ self.assertEqual(UI_METRIC_KEYS, {definition.key for definition in UI_METRICS})+ for definition in UI_METRICS:+ self.assertIn(definition.key, get_args(Metric))++ def test_config_accepts_and_defaults_ui_trials(self):+ config = Config.load()+ self.assertEqual(config.ui_trials, 3)+ raw = json.loads(Path(__file__).resolve().parents[1].joinpath("config.json").read_text())+ self.assertIn("ui_trials", raw)++ def test_report_validator_uses_ui_trial_counts(self):+ started = datetime(2026, 9, 9, tzinfo=UTC)+ config = Config.load()+ report = Report(+ repository="PrimeIntellect-ai/prime-agent",+ head_repository="PrimeIntellect-ai/prime-agent",+ pr=42,+ run_id=100,+ harness_sha="a" * 40,+ base_sha="a" * 40,+ head_sha="b" * 40,+ started_at=started,+ config=config,+ main=Side(sha="a" * 40),+ pr_head=Side(sha="b" * 40),+ )+ for side in (report.main, report.pr_head):+ for definition in UI_METRICS:+ side.metrics[definition.key] = [+ Observation(trial=trial, value=1.0) for trial in range(config.ui_trials)+ ]+ report.model_validate(report.model_dump())++ overflowing = report.pr_head.model_copy(deep=True)+ overflowing.metrics["ui_rss"].append(Observation(trial=config.ui_trials, value=1.0))+ with self.assertRaises(ValueError):+ Report.model_validate({**report.model_dump(), "pr_head": overflowing.model_dump()})++ def test_render_includes_ui_section(self):+ started = datetime(2026, 9, 9, tzinfo=UTC)+ config = Config.load()+ report = Report(+ repository="PrimeIntellect-ai/prime-agent",+ head_repository="PrimeIntellect-ai/prime-agent",+ pr=42,+ run_id=100,+ harness_sha="a" * 40,+ base_sha="a" * 40,+ head_sha="b" * 40,+ started_at=started,+ config=config,+ main=Side(sha="a" * 40),+ pr_head=Side(sha="b" * 40),+ )+ for definition in UI_METRICS:+ report.main.metrics[definition.key] = [+ Observation(trial=trial, value=2.0) for trial in range(config.ui_trials)+ ]+ report.pr_head.metrics[definition.key] = [+ Observation(trial=trial, value=2.0) for trial in range(config.ui_trials)+ ]+ report.status = "completed"+ text = render(report)+ self.assertIn("**UI interactions**", text)+ self.assertIn("Resume large session (cold)", text)+ self.assertIn("Open subagent session at depth 6", text)+ self.assertIn("UI memory after interactions", text)+++if __name__ == "__main__":+ unittest.main()diff --git a/scripts/benchmarks/ui.py b/scripts/benchmarks/ui.pynew file mode 100644index 0000000000..eae63a3cb8--- /dev/null+++ b/scripts/benchmarks/ui.py@@ -0,0 +1,633 @@+from __future__ import annotations++import hashlib+import json+import os+import pwd+import re+import shutil+import time+import uuid+from dataclasses import dataclass+from datetime import UTC, datetime+from pathlib import Path++from schema import ProcessMemory, Request, Side+from terminal import Terminal++# Fixture shape: "many sessions, many large sessions" plus a deep subagent chain.+# Scale mirrors real long-lived installs: hundreds of saved sessions, one very+# large transcript, and a spawn ledger with enough edges to stress hydration.+MEDIUM_COUNT = 150+MEDIUM_MESSAGES = 120+LARGE_COUNT = 3+LARGE_MESSAGES = 2000+HUGE_MESSAGES = 16000+FANOUT_COUNT = 40+FANOUT_MESSAGES = 20+SUBAGENT_DEPTH = 6+SUBAGENT_MESSAGES = 400++BASE_TIMESTAMP_MS = 946684800000 # 2000-01-01T00:00:00Z; fixed so fixtures are deterministic+TOOL_OUTPUT_LINES = 60++AGENTS_VIEW_HINT = "type to search sessions"+SEARCH_PLACEHOLDER = "Search sessions"+ROSTER_COUNT = re.compile(r"agents\s+(\d+) running, (\d+) idle, (\d+) inactive")+READY_MARKER = "benchready"+LEFT_ARROW = "\x1b[D"+RIGHT_ARROW = "\x1b[C"+DOWN_ARROW = "\x1b[B"+EXPAND_ARROW = "\x1b[1;3C"+BACKSPACE = "\x7f"+ENTER = "\r"+FIXTURE_NAMESPACE = uuid.UUID("b0d5f4a1-6d64-4bf8-9d7a-2f4f0a9c1e10")+++def session_id(kind: str, index: int) -> str:+ return str(uuid.uuid5(FIXTURE_NAMESPACE, f"prime-agent-ui-bench/{kind}/{index}"))+++def session_name(kind: str, index: int) -> str:+ return f"ui-bench-{kind}-{index:02d}"+++def tail_marker(kind: str, index: int) -> str:+ return f"ui-bench-tail-{kind}-{index:02d}"+++@dataclass(frozen=True)+class FixtureSpec:+ """Session ids the probe addresses; every id is derived deterministically."""++ large: list[str]+ medium: list[str]+ root: str+ subagents: list[str]+ fanout: list[str]++ @property+ def resume_id(self) -> str:+ """Cold-resume target; never touched by the warm scenario."""+ return self.large[1]++ @property+ def switch_id(self) -> str:+ """Warm in-session /resume target; the very large transcript."""+ return self.large[2]++ @property+ def open_id(self) -> str:+ """Search-and-open target from the agents view."""+ return self.large[0]+++def _iso(ms: int) -> str:+ return datetime.fromtimestamp(ms / 1000, tz=UTC).isoformat(timespec="milliseconds").replace("+00:00", "Z")+++def _session_lines(+ name: str,+ identifier: str,+ count: int,+ cwd: Path,+ *,+ depth: int,+ tail: str | None,+) -> list[dict]:+ """One deterministic session transcript: header, name, model, then a realistic message mix."""+ timestamp = BASE_TIMESTAMP_MS+ parent: str | None = None+ lines: list[dict] = []+ prefix = identifier.replace("-", "")[:8]++ def link(entry_id: str) -> dict:+ nonlocal parent, timestamp+ timestamp += 1000+ entry = {"id": entry_id, "parentId": parent, "timestamp": _iso(timestamp)}+ parent = entry_id+ return entry++ lines.append(+ {+ "type": "session",+ "version": 3,+ "id": identifier,+ "timestamp": _iso(timestamp),+ "cwd": str(cwd),+ "rlmDepth": depth,+ }+ )+ entry = link(f"{prefix}n")+ entry.update({"type": "session_info", "name": name})+ lines.append(entry)+ entry = link(f"{prefix}m")+ entry.update({"type": "model_change", "provider": "prime-inference", "modelId": "internal/glm-5.3-fast"})+ lines.append(entry)+ usage = {+ "input": 100,+ "output": 50,+ "cacheRead": 0,+ "cacheWrite": 0,+ "totalTokens": 150,+ "cost": {"input": 0.0001, "output": 0.0001, "cacheRead": 0, "cacheWrite": 0, "total": 0.0002},+ }+ for index in range(count):+ final = index == count - 1+ entry = link(f"{prefix}u{index}")+ entry.update(+ {+ "type": "message",+ "message": {+ "role": "user",+ "timestamp": 0,+ "content": (+ f"Please continue task {name} step {index}: review the module, run the checks, "+ "and summarize findings for the migration notes."+ ),+ },+ }+ )+ lines.append(entry)+ text = (+ f"Working on {name} step {index}. The parser handles nested records correctly; the next edit "+ "keeps the schema stable while trimming the duplicated branch."+ )+ if final and tail:+ text = f"Finished all steps. {tail}"+ content = [+ {+ "type": "thinking",+ "thinking": (+ f"Step {index} for {name}: check the invariants, then update the affected call "+ "sites before running the full suite again to confirm no behavior changed."+ ),+ },+ {"type": "text", "text": text},+ ]+ if not final:+ content.append(+ {"type": "toolCall", "id": f"call-{index}", "name": "ipython", "arguments": {"code": "pass"}}+ )+ entry = link(f"{prefix}a{index}")+ entry.update(+ {+ "type": "message",+ "message": {+ "role": "assistant",+ "timestamp": 0,+ "content": content,+ "api": "responses",+ "provider": "prime-inference",+ "model": "internal/glm-5.3-fast",+ "usage": usage,+ "stopReason": "stop" if final else "toolUse",+ },+ }+ )+ lines.append(entry)+ if not final:+ entry = link(f"{prefix}t{index}")+ entry.update(+ {+ "type": "message",+ "message": {+ "role": "toolResult",+ "toolCallId": f"call-{index}",+ "toolName": "ipython",+ "content": [{"type": "text", "text": "all checks passed\n" * TOOL_OUTPUT_LINES}],+ "isError": False,+ "timestamp": 0,+ },+ }+ )+ lines.append(entry)+ return lines+++def spawn_ledger_path(agent_dir: Path, sessions_dir: Path) -> Path:+ canonical = Path(os.path.realpath(sessions_dir))+ digest = hashlib.sha256(str(canonical).encode()).hexdigest()[:16]+ return agent_dir / "rlm-ledger" / f"{digest}.jsonl"+++def write_fixtures(agent_dir: Path, workspace: Path, spec: FixtureSpec, *, uid: int | None = None) -> None:+ """Regenerate every session file and the spawn ledger from scratch for one trial.++ The harness runs as root; pass the benchmark user's ids so the spawned CLI can+ read and append its own sessions.+ """+ sessions_dir = agent_dir / "sessions"+ for directory in (sessions_dir, agent_dir / "session-artifacts", agent_dir / "session-leases"):+ shutil.rmtree(directory, ignore_errors=True)+ sessions_dir.mkdir(parents=True, exist_ok=True)++ def write(identifier: str, name: str, count: int, *, depth: int = 0, tail: str | None = None) -> Path:+ path = sessions_dir / f"{identifier}.jsonl"+ path.write_text(+ "".join(+ json.dumps(line, separators=(",", ":")) + "\n"+ for line in _session_lines(name, identifier, count, workspace, depth=depth, tail=tail)+ )+ )+ return path++ for index in range(1, LARGE_COUNT + 1):+ identifier = spec.large[index - 1]+ messages = HUGE_MESSAGES if index == 3 else LARGE_MESSAGES+ write(identifier, session_name("large", index), messages, tail=tail_marker("large", index))+ for index in range(1, MEDIUM_COUNT + 1):+ identifier = spec.medium[index - 1]+ write(identifier, session_name("medium", index), MEDIUM_MESSAGES)+ parent = write(spec.root, session_name("root", 0), SUBAGENT_MESSAGES, tail=tail_marker("root", 0))+ edges = []+ for fanout in range(1, FANOUT_COUNT + 1):+ identifier = spec.fanout[fanout - 1]+ path = write(identifier, session_name("fan", fanout), FANOUT_MESSAGES, depth=1)+ edges.append(+ {+ "childId": identifier,+ "parent": str(parent),+ "child": str(path),+ "depth": 1,+ "name": session_name("fan", fanout),+ }+ )+ for depth in range(1, SUBAGENT_DEPTH + 1):+ identifier = spec.subagents[depth - 1]+ path = write(+ identifier,+ session_name("sub", depth),+ SUBAGENT_MESSAGES,+ depth=depth,+ tail=tail_marker("sub", depth),+ )+ edges.append(+ {+ "childId": identifier,+ "parent": str(parent),+ "child": str(path),+ "depth": depth,+ "name": session_name("sub", depth),+ }+ )+ parent = path+ ledger = spawn_ledger_path(agent_dir, sessions_dir)+ ledger.parent.mkdir(parents=True, exist_ok=True)+ records = [+ {+ "v": 1,+ "op": "meta",+ "at": _iso(BASE_TIMESTAMP_MS),+ "sessionsDir": str(Path(os.path.realpath(sessions_dir))),+ }+ ]+ records.extend(+ {"v": 1, "op": "spawn", "at": _iso(BASE_TIMESTAMP_MS + edge["depth"]), **edge} for edge in edges+ )+ ledger.write_text("".join(json.dumps(record, separators=(",", ":")) + "\n" for record in records))+ if uid is not None:+ gid = pwd.getpwuid(uid).pw_gid+ os.chown(agent_dir, uid, gid)+ for path in agent_dir.rglob("*"):+ os.chown(path, uid, gid)+++def fixture_spec() -> FixtureSpec:+ return FixtureSpec(+ large=[session_id("large", index) for index in range(1, LARGE_COUNT + 1)],+ medium=[session_id("medium", index) for index in range(1, MEDIUM_COUNT + 1)],+ root=session_id("root", 0),+ subagents=[session_id("sub", depth) for depth in range(1, SUBAGENT_DEPTH + 1)],+ fanout=[session_id("fan", index) for index in range(1, FANOUT_COUNT + 1)],+ )+++def process_stats(uid: int) -> list[ProcessMemory]:+ """Snapshot every live process of one user with RSS, PSS, and cumulative CPU time."""+ stats: list[ProcessMemory] = []+ try:+ clock_ticks = os.sysconf("SC_CLK_TCK")+ except (ValueError, AttributeError, OSError):+ return stats+ for status_path in Path("/proc").glob("[0-9]*/status"):+ try:+ fields = dict(line.split(":", 1) for line in status_path.read_text().splitlines() if ":" in line)+ if int(fields["Uid"].split()[0]) != uid or fields["State"].strip().startswith("Z"):+ continue+ stat_fields = (status_path.parent / "stat").read_text().rsplit(")", 1)[1].split()+ cpu_seconds = (int(stat_fields[11]) + int(stat_fields[12])) / clock_ticks+ pss = None+ try:+ for line in (status_path.parent / "smaps_rollup").read_text().splitlines():+ if line.startswith("Pss:"):+ pss = int(line.split()[1]) * 1024+ except (PermissionError, FileNotFoundError, ProcessLookupError):+ pass+ stats.append(+ ProcessMemory(+ pid=int(status_path.parent.name),+ name=fields["Name"].strip(),+ rss=int(fields.get("VmRSS", "0").split()[0]) * 1024,+ pss=pss,+ cpu=cpu_seconds,+ )+ )+ except (PermissionError, FileNotFoundError, ProcessLookupError, ValueError, IndexError):+ continue+ return stats+++def total_cpu(stats: list[ProcessMemory]) -> float:+ return sum(process.cpu or 0.0 for process in stats)+++def quiet(terminal: Terminal, seconds: float = 0.6, cap: float = 8.0) -> None:+ """Pump output until the terminal stays silent, so the editor can accept input again."""+ started = time.perf_counter()+ last_output = time.perf_counter()+ while time.perf_counter() - started < cap:+ if terminal.pump():+ last_output = time.perf_counter()+ elif time.perf_counter() - last_output >= seconds:+ return+++def input_ready(terminal: Terminal, marker: str, *, timeout: float = 120) -> None:+ """Wait for a rendered marker, then confirm the editor echoes, retrying past input loss.++ Keystrokes sent while a freshly opened session is still mounting can be dropped,+ so the echo is retried with a quiet window instead of failing the trial.+ """+ terminal.until(lambda display: marker in display.text(), timeout)+ for _ in range(5):+ quiet(terminal, 0.8)+ terminal.child.send(READY_MARKER)+ try:+ terminal.until(lambda display: READY_MARKER in display.text(), 6)+ except TimeoutError:+ terminal.child.send(BACKSPACE * (len(READY_MARKER) + 2))+ continue+ terminal.child.send(BACKSPACE * len(READY_MARKER))+ terminal.until(lambda display: READY_MARKER not in display.text(), 10)+ return+ raise TimeoutError("Editor did not echo the readiness marker")+++def clear_search(terminal: Terminal, *, attempts: int = 5) -> None:+ """Empty the agents-view search box; keystrokes can be eaten by re-entry renders."""+ for _ in range(attempts):+ terminal.child.send(BACKSPACE * 24)+ try:+ terminal.until(lambda display: SEARCH_PLACEHOLDER in display.text(), 5)+ return+ except TimeoutError:+ continue+ raise TimeoutError("Agents-view search box did not clear")+++def type_query(terminal: Terminal, query: str, *, attempts: int = 3) -> None:+ """Type a search query and verify it echoed before filtering on it."""+ for _ in range(attempts):+ clear_search(terminal)+ terminal.child.send(query)+ try:+ terminal.until(lambda display: query in display.text(), 8)+ return+ except TimeoutError:+ continue+ raise TimeoutError("Agents-view search query did not echo")+++def expand_subagents(terminal: Terminal, *, attempts: int = 4) -> None:+ """Expand the selected row's subagent list; the toggle retries to survive eaten keystrokes."""+ expanded = terminal.display.text().count("▾")+ for _ in range(attempts):+ terminal.child.send(EXPAND_ARROW)+ terminal.settle(1.0)+ if terminal.display.text().count("▾") > expanded:+ return+ raise TimeoutError("Selected agent subagents did not expand")+++def roster_inactive(display) -> int | None: # type: ignore[no-untyped-def]+ for line in display.text().splitlines():+ match = ROSTER_COUNT.search(line)+ if match:+ return int(match.group(3))+ return None+++def expected_roster_inactive() -> int:+ """Saved top-level fixture sessions a hydrated roster must list.++ Large, medium, and the chain-root fixtures are top-level roster rows, but+ the warm scenario holds the switch target live; nested fan-out and+ subagent-chain files stay out of the top-level count either way.+ """+ top_level = LARGE_COUNT + MEDIUM_COUNT + 1+ return top_level - 1+++def wait_for_roster(terminal: Terminal, *, minimum: int, settle: float = 3.0, timeout: float = 90) -> float:+ """Wait until the roster lists `minimum` inactive sessions and stops growing.++ Saved sessions stream in, but an empty or stalled roster must not count+ as settled just because its count stopped moving.+ """+ started = time.perf_counter()+ last_count: int | None = None+ last_change = time.perf_counter()+ while time.perf_counter() - started < timeout:+ terminal.settle(0.4)+ count = roster_inactive(terminal.display)+ if (+ count is not None+ and count == last_count+ and count >= minimum+ and time.perf_counter() - last_change >= settle+ ):+ return time.perf_counter() - started+ if count != last_count:+ last_count = count+ last_change = time.perf_counter()+ raise TimeoutError("Agents-view roster did not settle")+++def ui_measure(request: Request, side: Side, trial: int, *, results: Path, homes: Path, user: str) -> None:+ """One UI-interaction trial: fresh fixtures, a cold resume, then the warm navigation scenario."""+ from worker import clean_error, environment, record, stop_processes++ if not any(s.value is not None and s.trial == 0 for s in side.metrics.get("install", [])):+ raise RuntimeError("The first installation must succeed before interactive measurements")+ home = homes / user+ workspace = home / "workspace"+ agent_dir = home / ".prime/agent"+ uid = pwd.getpwnam(user).pw_uid+ spec = fixture_spec()+ details: dict[str, dict] = {}+ metric = "resume_large"++ def note(name: str, *, seconds: float, cpu: float, pty_bytes: int) -> None:+ details[name] = {"seconds": round(seconds, 4), "cpu": round(cpu, 4), "pty_bytes": pty_bytes}++ def cpu_total() -> float:+ return total_cpu(process_stats(uid))++ stop_processes(user)+ write_fixtures(agent_dir, workspace, spec, uid=uid)+ env = environment(user)+ runuser = "/usr/sbin/runuser"+ try:+ # Cold resume of a large session from the CLI, from process spawn to usable editor.+ cpu_start = cpu_total()+ terminal = Terminal(+ [runuser, "-u", user, "--", "prime-agent", "--resume", spec.resume_id],+ workspace,+ env,+ results / f"ui-resume-{trial}",+ )+ try:+ metric = "resume_large"+ bytes_start = terminal.bytes+ input_ready(terminal, tail_marker("large", 2))+ elapsed = time.perf_counter() - terminal.started+ cpu = cpu_total() - cpu_start+ record(side, "resume_large", trial, elapsed) # type: ignore[arg-type]+ record(side, "resume_large_cpu", trial, cpu) # type: ignore[arg-type]+ note("resume_large", seconds=elapsed, cpu=cpu, pty_bytes=terminal.bytes - bytes_start)+ finally:+ terminal.close()+ stop_processes(user)++ # Warm navigation scenario in a single TUI process.+ metric = "switch_large"+ terminal = Terminal(+ [runuser, "-u", user, "--", "prime-agent"], workspace, env, results / f"ui-scenario-{trial}"+ )+ try:+ terminal.ready()++ # Warm switch into a different large session from the running editor.+ started = time.perf_counter()+ cpu_start = cpu_total()+ bytes_start = terminal.bytes+ terminal.child.send(f"/resume {spec.switch_id}{ENTER}")+ input_ready(terminal, tail_marker("large", 3))+ elapsed = time.perf_counter() - started+ cpu = cpu_total() - cpu_start+ record(side, "switch_large", trial, elapsed) # type: ignore[arg-type]+ record(side, "switch_large_cpu", trial, cpu) # type: ignore[arg-type]+ note("switch_large", seconds=elapsed, cpu=cpu, pty_bytes=terminal.bytes - bytes_start)++ # Session -> agents view: keystroke to rendered roster splash.+ metric = "agents_view"+ started = time.perf_counter()+ cpu_start = cpu_total()+ bytes_start = terminal.bytes+ terminal.child.send(LEFT_ARROW)+ terminal.until(lambda display: AGENTS_VIEW_HINT in display.text(), 60)+ elapsed = time.perf_counter() - started+ cpu = cpu_total() - cpu_start+ record(side, "agents_view", trial, elapsed) # type: ignore[arg-type]+ record(side, "agents_view_cpu", trial, cpu) # type: ignore[arg-type]+ note("agents_view", seconds=elapsed, cpu=cpu, pty_bytes=terminal.bytes - bytes_start)++ # Full roster with many sessions: saved sessions and ledger children stream in.+ metric = "agents_roster"+ cpu_start = cpu_total()+ bytes_start = terminal.bytes+ roster_seconds = wait_for_roster(terminal, minimum=expected_roster_inactive())+ cpu = cpu_total() - cpu_start+ record(side, "agents_roster", trial, roster_seconds) # type: ignore[arg-type]+ record(side, "agents_roster_cpu", trial, cpu) # type: ignore[arg-type]+ note("agents_roster", seconds=roster_seconds, cpu=cpu, pty_bytes=terminal.bytes - bytes_start)++ # Agents view -> another large session, found by its unique id prefix.+ metric = "agents_open"+ started = time.perf_counter()+ cpu_start = cpu_total()+ bytes_start = terminal.bytes+ type_query(terminal, spec.open_id[:8])+ terminal.until(lambda display: session_name("large", 1) in display.text(), 60)+ terminal.settle(0.8)+ terminal.child.send(RIGHT_ARROW)+ input_ready(terminal, tail_marker("large", 1))+ elapsed = time.perf_counter() - started+ cpu = cpu_total() - cpu_start+ record(side, "agents_open", trial, elapsed) # type: ignore[arg-type]+ record(side, "agents_open_cpu", trial, cpu) # type: ignore[arg-type]+ note("agents_open", seconds=elapsed, cpu=cpu, pty_bytes=terminal.bytes - bytes_start)++ # Chain parent: open the root so the deepest subagent opens against a live parent.+ metric = "parent_open"+ started = time.perf_counter()+ cpu_start = cpu_total()+ bytes_start = terminal.bytes+ terminal.child.send(LEFT_ARROW)+ terminal.until(lambda display: AGENTS_VIEW_HINT in display.text(), 60)+ terminal.settle(1.0)+ type_query(terminal, spec.root[:8])+ terminal.until(lambda display: session_name("root", 0) in display.text(), 60)+ terminal.settle(0.5)+ terminal.child.send(RIGHT_ARROW)+ input_ready(terminal, tail_marker("root", 0))+ elapsed = time.perf_counter() - started+ cpu = cpu_total() - cpu_start+ record(side, "parent_open", trial, elapsed) # type: ignore[arg-type]+ record(side, "parent_open_cpu", trial, cpu) # type: ignore[arg-type]+ note("parent_open", seconds=elapsed, cpu=cpu, pty_bytes=terminal.bytes - bytes_start)++ # Deepest subagent session: search, drill in, open at depth SUBAGENT_DEPTH+ # while its parent session is live, matching real subagent workflows.+ metric = "subagent_open"+ started = time.perf_counter()+ cpu_start = cpu_total()+ bytes_start = terminal.bytes+ terminal.child.send(LEFT_ARROW)+ terminal.until(lambda display: AGENTS_VIEW_HINT in display.text(), 60)+ terminal.settle(1.0)+ query = spec.subagents[-1][:8]+ type_query(terminal, query)+ terminal.until(lambda display: session_name("root", 0) in display.text(), 60)+ clear_search(terminal)+ expand_subagents(terminal)+ # The search box must be empty to expand; refilter to the chain so the+ # first fan-out sibling cannot steal the down-arrow, then drill a level+ # at a time until the deepest subagent row is selected.+ type_query(terminal, query)+ terminal.settle(0.5)+ terminal.child.send(DOWN_ARROW)+ terminal.settle(0.8)+ for _ in range(SUBAGENT_DEPTH - 1):+ clear_search(terminal)+ expand_subagents(terminal)+ terminal.settle(0.8)+ terminal.child.send(DOWN_ARROW)+ terminal.settle(0.8)+ terminal.child.send(RIGHT_ARROW)+ input_ready(terminal, tail_marker("sub", SUBAGENT_DEPTH))+ elapsed = time.perf_counter() - started+ cpu = cpu_total() - cpu_start+ record(side, "subagent_open", trial, elapsed) # type: ignore[arg-type]+ record(side, "subagent_open_cpu", trial, cpu) # type: ignore[arg-type]+ note("subagent_open", seconds=elapsed, cpu=cpu, pty_bytes=terminal.bytes - bytes_start)++ # Whole-tree memory after the interactions.+ metric = "ui_rss"+ terminal.settle(1)+ processes = process_stats(uid)+ if not processes:+ raise RuntimeError("No owned processes found for the UI memory measurement")+ record(side, "ui_rss", trial, sum(process.rss for process in processes)) # type: ignore[arg-type]+ side.processes = processes+ (results / f"ui-memory-{trial}.json").write_text(+ json.dumps([process.model_dump() for process in processes], indent=2) + "\n"+ )+ finally:+ terminal.close()+ except Exception as error:+ record(side, metric, trial, error=clean_error(error)) # type: ignore[arg-type]+ finally:+ stop_processes(user)+ (results / f"ui-{trial}.json").write_text(json.dumps(details, indent=2, sort_keys=True) + "\n")diff --git a/scripts/benchmarks/worker.py b/scripts/benchmarks/worker.pyindex d345afae48..93f60b23b1 100644--- a/scripts/benchmarks/worker.py+++ b/scripts/benchmarks/worker.py@@ -547,7 +547,7 @@ def runtime(side: Side, trial: int) -> None:def main() -> None:parser = argparse.ArgumentParser()- parser.add_argument("phase", choices=("prepare", "install", "measure", "runtime"))+ parser.add_argument("phase", choices=("prepare", "install", "measure", "runtime", "ui"))parser.add_argument("--trial", type=int, default=0)args = parser.parse_args()RESULTS.mkdir(exist_ok=True)@@ -569,6 +569,17 @@ def main() -> None:install(request, result.side, args.trial)elif args.phase == "measure":measure(request, result.side, args.trial)+ elif args.phase == "ui":+ from ui import ui_measure++ ui_measure(+ request,+ result.side,+ args.trial,+ results=RESULTS,+ homes=HOMES,+ user="benchmark1",+ )else:runtime(result.side, args.trial)except Exception as error: