Skip to content

A self-improving RLM agent for coding workflows and long-running autonomous tasks.

Rust

feat(ci): add UI interaction benchmarks to Prime Agent performance CI (ENG-6043) (#2211)

Seth Karten7a049d50c61e
diff --git a/scripts/benchmarks/README.md b/scripts/benchmarks/README.md
index c2bca0e7d3..31c648cb1f 100644
--- a/scripts/benchmarks/README.md
+++ b/scripts/benchmarks/README.md
@@ -65,7 +65,7 @@ and other missing tools. Their setup time and disk usage are outside the install
The controller, PTY harness, artifact server, and build user are excluded. RSS can double-count
shared pages.
-Startup and memory use 10 trials per revision. Installation uses three; sizes are measured once.
+Startup and memory use 10 trials per revision. Installation uses three; sizes are measured once. UI interactions use 3 trials per revision (`ui_trials`).
Compiled revisions provision pinned Bun tooling and build their Linux x64 archive during untimed
setup. The installer selects its normal default from the available artifacts; the harness does not
force Node or compiled mode. Loopback downloads use the installer's explicit test exception, while
@@ -133,6 +133,56 @@ size, and 1% for disk footprint. A same-revision calibration on separate sandbox
Smaller signed differences remain visible. Revisit these floors after collecting more control runs. This is a practical noise filter, not a statistical significance test. Inspect
raw trials before acting on small changes; sandbox scheduling and filesystem caches still introduce noise.
+### UI interactions
+
+A separate probe drives the interactive TUI over a PTY with the same `pexpect`/`pyte` harness the
+startup metrics use, against a deterministic on-disk fixture set. Every UI trial regenerates the
+sessions directory, session artifacts, and spawn ledger from scratch, so trials cannot inherit state.
+
+The fixture set is fixed and mirrors a long-lived install: 194 top-level sessions (3 large —
+one of them a very large ~40 MB transcript — 150 medium, 40 small fan-out children of the chain
+root, and the root itself) plus a chain of 6 nested subagent sessions. Large sessions hold 1,999
+user/assistant/toolResult triples (~5 MB of JSONL each), the very large one 15,999 (~40 MB),
+medium sessions 119, and each subagent 399. Content is synthetic but shaped like real transcripts
+(thinking, tool calls, tool output). The subagent chain and the fan-out are linked through the
+same spawn-ledger records the daemon replays (~46 edges), so roster hydration, catalog scanning,
+and saved-session paths are exercised at a scale where regressions are visible. No inference is
+called at any point.
+
+Each trial stops all benchmark-user processes first, then measures:
+
+- **Resume large session (cold):** `prime-agent --resume <id>` from process spawn until the resumed
+ transcript tail is rendered and the editor echoes a typed marker. This is the "switch into a large
+ session" path including daemon and session-worker startup.
+- **Switch into large session:** in a running TUI, `/resume <id>` keystroke to rendered tail plus
+ echoed marker. Measures the in-place session switch with a ~40 MB transcript.
+- **Open agents view from a session:** the left-arrow keystroke to the rendered agents-view splash.
+- **Full agents roster, many sessions:** after the agents-view splash is rendered, until the
+ roster lists the fixture's saved sessions and its count stops growing. Saved sessions stream in, so this isolates
+ catalog scanning, spawn-ledger replay, and hydration across 200 session files.
+- **Open another session from agents view:** typing the target's unique session-id prefix,
+ right-arrow to open, until the target transcript renders and echoes. This is the full
+ "session → agents view → another session" round trip with many sessions on disk.
+- **Open chain parent from agents view:** back to the agents view, search the subagent chain's
+ root session by id prefix, right-arrow to open. The parent becomes live.
+- **Open subagent session at depth 6:** back to the agents view, then drill into the live chain:
+ filter by the deepest subagent's id prefix, clear the search (expansion needs an empty query),
+ expand the parent, refilter past the fan-out siblings, and repeat clear-expand-step for each of
+ the six levels until the deepest subagent row is selected, then right-arrow to open. This is the
+ exact manual flow the product requires today, so the metric includes its real cost.
+- **CPU per interaction:** summed `utime+stime` across the whole benchmark-user process tree
+ (TUI client, daemon, session workers, Python kernels) between two instants around each interaction.
+- **UI memory after interactions:** summed RSS across the same tree after the scenario settles.
+ Per-process snapshots with PSS and CPU are saved as raw artifacts, as for idle memory.
+
+Readiness is never a spinner or a status line: a trial only counts when the target session's
+final transcript line is visible and the editor echoes a marker, so half-rendered states fail
+loudly instead of measuring fast. The roster wait likewise only settles once the fixture's saved
+sessions are listed, so a stalled or empty catalog scan fails the trial instead of recording a
+fast sample. Raw results also record PTY bytes per interaction, a proxy for
+how much the renderer redraws. UI trials default to 3 per revision (`ui_trials` in `config.json`)
+to bound sandbox cost; the scenario takes roughly 60–90 s per trial.
+
## Lifecycle and costs
The main workflow posts a single marked comment and updates it in place. A new push replaces the
@@ -147,7 +197,7 @@ SHAs, environment details, installed dependency inventory, transcripts, raw obse
and estimated sandbox costs. The publisher validates schema, identity, finite numbers, and report
size and escapes sandbox-provided text.
-The controller stops scheduling work at 20 minutes or its $1 estimated budget target. Every sandbox
+The controller stops scheduling work at 25 minutes or its $1 estimated budget target. Every sandbox
has a 30-minute TTL. Teardown runs in `finally`; the independent completion workflow deletes any
remaining sandboxes with the exact repository/run/attempt labels, including after cancellation.
Cleanup enumerates all pages before deleting so pagination cannot skip a sandbox.
@@ -162,7 +212,8 @@ Log-collection and deferred sandbox-cleanup problems appear as operational warni
mark complete measurements as partial; missing or failed measurements still fail the command.
At the configured list rates, two sandboxes cost about $0.01/minute together. A 10-minute run costs
-about $0.10 in sandbox compute. There are no inference charges. Full sandbox lifetimes are recorded,
+about $0.10 in sandbox compute. There are no inference charges. UI-interaction trials add roughly
+3–6 minutes per run at the default 3 trials per revision. Full sandbox lifetimes are recorded,
including provisioning, setup, building, measurements, and cleanup. The $1 target is an estimate,
not a hard billing cap; scheduling and deletion latency can increase the final bill.
diff --git a/scripts/benchmarks/config.json b/scripts/benchmarks/config.json
index f0a6962115..3e2e5a519d 100644
--- a/scripts/benchmarks/config.json
+++ b/scripts/benchmarks/config.json
@@ -7,10 +7,11 @@
"trials": 10,
"install_trials": 3,
"debounce_seconds": 2,
- "timeout_seconds": 1200,
+ "timeout_seconds": 1500,
"ttl_minutes": 30,
"budget_usd": 1,
"cpu_usd_per_hour": 0.05,
"memory_usd_per_gb_hour": 0.01,
- "disk_usd_per_gb_hour": 0.001
+ "disk_usd_per_gb_hour": 0.001,
+ "ui_trials": 3
}
diff --git a/scripts/benchmarks/controller.py b/scripts/benchmarks/controller.py
index 62de2276bc..8077175281 100644
--- a/scripts/benchmarks/controller.py
+++ b/scripts/benchmarks/controller.py
@@ -26,7 +26,7 @@ from schema import (
REMOTE = "/opt/prime-benchmark"
OWNER_LABEL = "prime-agent-benchmarks-v1"
-FILES = ("schema.py", "terminal.py", "kernel.py", "worker.py", "pyproject.toml", "uv.lock")
+FILES = ("schema.py", "terminal.py", "kernel.py", "worker.py", "ui.py", "pyproject.toml", "uv.lock")
def elapsed_seconds(start: datetime) -> float:
@@ -279,10 +279,11 @@ class Controller:
("install", self.report.config.install_trials),
("measure", self.report.config.trials),
("runtime", self.report.config.trials),
+ ("ui", self.report.config.ui_trials),
):
blocked: set[str] = set()
failures: dict[str, tuple[str, int]] = {}
- if phase in ("measure", "runtime"):
+ if phase in ("measure", "runtime", "ui"):
for role in ready:
side = self.report.main if role == "main" else self.report.pr_head
if not any(
diff --git a/scripts/benchmarks/report.py b/scripts/benchmarks/report.py
index c70a9b2813..4de03c49bf 100644
--- a/scripts/benchmarks/report.py
+++ b/scripts/benchmarks/report.py
@@ -7,7 +7,7 @@ from collections import Counter
from dataclasses import dataclass
from typing import Literal
-from schema import Metric, Observation, Report
+from schema import UI_METRIC_KEYS, Metric, Observation, Report
MARKER = "<!-- prime-agent-benchmark:v1 -->"
PERFORMANCE_NOISE_FLOOR = 0.2
@@ -107,6 +107,43 @@ RUNTIME_METRICS = (
PERFORMANCE_NOISE_FLOOR,
),
)
+UI_METRICS = (
+ Definition("resume_large", "Resume large session (cold)", 1000, "ms", 0.1, PERFORMANCE_NOISE_FLOOR),
+ Definition("resume_large_cpu", "CPU, resume large session", 1000, "ms", 0.05, PERFORMANCE_NOISE_FLOOR),
+ Definition("switch_large", "Switch into large session", 1000, "ms", 0.05, PERFORMANCE_NOISE_FLOOR),
+ Definition(
+ "switch_large_cpu", "CPU, switch into large session", 1000, "ms", 0.02, PERFORMANCE_NOISE_FLOOR
+ ),
+ Definition("agents_view", "Open agents view from a session", 1000, "ms", 0.02, PERFORMANCE_NOISE_FLOOR),
+ Definition("agents_view_cpu", "CPU, open agents view", 1000, "ms", 0.01, PERFORMANCE_NOISE_FLOOR),
+ Definition("agents_roster", "Full agents roster, many sessions", 1, "s", 0.1, PERFORMANCE_NOISE_FLOOR),
+ Definition("agents_roster_cpu", "CPU, full agents roster", 1, "s", 0.05, PERFORMANCE_NOISE_FLOOR),
+ Definition(
+ "agents_open", "Open another session from agents view", 1000, "ms", 0.1, PERFORMANCE_NOISE_FLOOR
+ ),
+ Definition("agents_open_cpu", "CPU, open from agents view", 1000, "ms", 0.05, PERFORMANCE_NOISE_FLOOR),
+ Definition(
+ "subagent_open",
+ "Open subagent session at depth 6",
+ 1000,
+ "ms",
+ 0.1,
+ PERFORMANCE_NOISE_FLOOR,
+ ),
+ Definition(
+ "subagent_open_cpu", "CPU, open subagent at depth 6", 1000, "ms", 0.05, PERFORMANCE_NOISE_FLOOR
+ ),
+ Definition(
+ "parent_open",
+ "Open chain parent from agents view",
+ 1000,
+ "ms",
+ 0.1,
+ PERFORMANCE_NOISE_FLOOR,
+ ),
+ Definition("parent_open_cpu", "CPU, open chain parent", 1000, "ms", 0.05, PERFORMANCE_NOISE_FLOOR),
+ Definition("ui_rss", "UI memory after interactions", 1e-6, "MB", 10485760, PERFORMANCE_NOISE_FLOOR),
+)
def escape(text: str) -> str:
@@ -190,10 +227,12 @@ def comparison(
def comparisons(report: Report) -> dict[Metric, Comparison]:
results = {}
- for definition in (*METRICS, *RUNTIME_METRICS):
+ for definition in (*METRICS, *RUNTIME_METRICS, *UI_METRICS):
expected = report.config.install_trials if definition.key == "install" else report.config.trials
if definition.key in ("bundle", "disk"):
expected = 1
+ if definition.key in UI_METRIC_KEYS:
+ expected = report.config.ui_trials
results[definition.key] = comparison(
definition,
report.main.metrics.get(definition.key, []),
@@ -272,7 +311,7 @@ def render(report: Report) -> str:
"| --- | ---: | ---: | ---: |",
]
)
- for definition in (*METRICS, *RUNTIME_METRICS):
+ for definition in (*METRICS, *RUNTIME_METRICS, *UI_METRICS):
if definition == RUNTIME_METRICS[0]:
lines.extend(
[
@@ -283,6 +322,16 @@ def render(report: Report) -> str:
"| --- | ---: | ---: | ---: |",
]
)
+ if definition == UI_METRICS[0]:
+ lines.extend(
+ [
+ "",
+ "**UI interactions**",
+ "",
+ "| Metric | Main | This PR | Change |",
+ "| --- | ---: | ---: | ---: |",
+ ]
+ )
result = results[definition.key]
lines.append(f"| {definition.title} | {result.main} | {result.head} | {result.change} |")
compute = sum(s.estimated_usd for s in report.sandboxes)
@@ -319,6 +368,15 @@ def render(report: Report) -> str:
"State fixture: a 10,000-row × 8-column integer DataFrame and a 10,000-integer list.",
"Restore runs in a fresh kernel, including pandas imports; kernel startup is excluded.",
"Kernel RSS covers the isolated Python process; loaded RSS follows the pandas workload.",
+ "UI trials use a fresh fixture set: 194 top-level sessions including one ~40 MB transcript,",
+ "40 ledger fan-out children, and a 6-deep subagent chain (~46 spawn edges).",
+ "Large fixtures hold 1,999 complete triples (~5 MB JSONL); medium 119; subagents 399 each.",
+ "Interactions: cold --resume of a large session, warm /resume switch, left-arrow to agents view,",
+ "roster settle with many saved sessions, search-and-open of another large session, and opening",
+ "the chain parent, then a clear-expand-step drill into the live chain to depth 6.",
+ "Readiness is the rendered transcript tail plus a confirmed editor echo.",
+ "CPU metrics sum utime+stime across the whole benchmark-user process tree per interaction.",
+ "UI memory sums RSS after the interactions; PTY byte counts are in the raw results.",
"Costs estimate full sandbox lifetimes at configured rates, including setup and build.",
f"Budget target: ${report.config.budget_usd:g}; not a billing cap. "
"Performance changes are informational.",
@@ -330,7 +388,7 @@ def render(report: Report) -> str:
"| --- | ---: | ---: | ---: | ---: |",
]
)
- for definition in (*METRICS, *RUNTIME_METRICS):
+ for definition in (*METRICS, *RUNTIME_METRICS, *UI_METRICS):
left = report.main.metrics.get(definition.key, [])
right = report.pr_head.metrics.get(definition.key, [])
lines.append(
diff --git a/scripts/benchmarks/schema.py b/scripts/benchmarks/schema.py
index 475e80cf67..18393bdcbe 100644
--- a/scripts/benchmarks/schema.py
+++ b/scripts/benchmarks/schema.py
@@ -27,7 +27,41 @@ Metric = Literal[
"restore",
"kernel_rss",
"loaded_rss",
+ "resume_large",
+ "resume_large_cpu",
+ "switch_large",
+ "switch_large_cpu",
+ "agents_view",
+ "agents_view_cpu",
+ "agents_roster",
+ "agents_roster_cpu",
+ "agents_open",
+ "agents_open_cpu",
+ "subagent_open",
+ "subagent_open_cpu",
+ "parent_open",
+ "parent_open_cpu",
+ "ui_rss",
]
+UI_METRIC_KEYS = frozenset(
+ {
+ "resume_large",
+ "resume_large_cpu",
+ "switch_large",
+ "switch_large_cpu",
+ "agents_view",
+ "agents_view_cpu",
+ "agents_roster",
+ "agents_roster_cpu",
+ "agents_open",
+ "agents_open_cpu",
+ "subagent_open",
+ "subagent_open_cpu",
+ "parent_open",
+ "parent_open_cpu",
+ "ui_rss",
+ }
+)
NonNegative = Annotated[float, Field(ge=0, le=1e15, allow_inf_nan=False)]
ROOT = Path(__file__).resolve().parent
UV_VERSION = "0.12.9"
@@ -46,6 +80,7 @@ class Config(StrictModel):
trials: Annotated[int, Field(ge=3, le=50)]
install_trials: Annotated[int, Field(ge=1, le=10)]
failure_limit: Annotated[int, Field(ge=1, le=10)] = 2
+ ui_trials: Annotated[int, Field(ge=1, le=50)] = 3
debounce_seconds: Annotated[int, Field(ge=0, le=300)]
timeout_seconds: Annotated[int, Field(ge=60, le=3600)]
ttl_minutes: Annotated[int, Field(ge=1, le=120)]
@@ -89,6 +124,7 @@ class ProcessMemory(StrictModel):
name: Annotated[str, Field(max_length=80)]
rss: NonNegative
pss: NonNegative | None = None
+ cpu: NonNegative = 0.0
class Side(StrictModel):
@@ -155,6 +191,8 @@ class Report(StrictModel):
expected = self.config.install_trials if metric == "install" else self.config.trials
if metric in ("bundle", "disk"):
expected = 1
+ if metric in UI_METRIC_KEYS:
+ expected = self.config.ui_trials
if (
len(samples) > expected
or len({s.trial for s in samples}) != len(samples)
@@ -181,6 +219,23 @@ PHASE_METRICS: dict[str, tuple[Metric, ...]] = {
"kernel_rss",
"loaded_rss",
),
+ "ui": (
+ "resume_large",
+ "resume_large_cpu",
+ "switch_large",
+ "switch_large_cpu",
+ "agents_view",
+ "agents_view_cpu",
+ "agents_roster",
+ "agents_roster_cpu",
+ "agents_open",
+ "agents_open_cpu",
+ "subagent_open",
+ "subagent_open_cpu",
+ "parent_open",
+ "parent_open_cpu",
+ "ui_rss",
+ ),
}
@@ -193,10 +248,18 @@ def trial_errors(side: Side, phase: str, trial: int) -> list[str]:
return errors or [f"{metric}: measurement missing" for metric in metrics if not samples[metric]]
-def side_complete(side: Side, trials: int, installs: int) -> bool:
+def side_complete(side: Side, trials: int, installs: int, ui_trials: int) -> bool:
for metrics in PHASE_METRICS.values():
for metric in metrics:
- count = 1 if metric in ("bundle", "disk") else installs if metric == "install" else trials
+ count = (
+ 1
+ if metric in ("bundle", "disk")
+ else installs
+ if metric == "install"
+ else ui_trials
+ if metric in UI_METRIC_KEYS
+ else trials
+ )
samples = side.metrics.get(metric, [])
if (
{sample.trial for sample in samples} != set(range(count))
@@ -209,7 +272,7 @@ def side_complete(side: Side, trials: int, installs: int) -> bool:
def report_complete(report: Report) -> bool:
return not report.errors and all(
- side_complete(side, report.config.trials, report.config.install_trials)
+ side_complete(side, report.config.trials, report.config.install_trials, report.config.ui_trials)
for side in (report.main, report.pr_head)
)
diff --git a/scripts/benchmarks/tests/test_benchmarks.py b/scripts/benchmarks/tests/test_benchmarks.py
index 3cdb8fa5d8..ea5c3aab2d 100644
--- a/scripts/benchmarks/tests/test_benchmarks.py
+++ b/scripts/benchmarks/tests/test_benchmarks.py
@@ -18,8 +18,9 @@ from pydantic import ValidationError
from cli import completed_report, main, validate_completion, workflow_source
from controller import Canceled, Controller, cleanup, labels
from github import TITLE, GitHub
-from report import MARKER, METRICS, RUNTIME_METRICS, comparison, render
+from report import MARKER, METRICS, RUNTIME_METRICS, UI_METRICS, comparison, render
from schema import (
+ UI_METRIC_KEYS,
Config,
Observation,
ProcessMemory,
@@ -186,11 +187,11 @@ class ReportTests(unittest.TestCase):
report.pr_head.metrics["bundle"] = []
text = render(report)
self.assertIn(
- "**Overall: 1 regressed · 1 improved · 13 no clear change · 1 incomplete · 1 unavailable.**",
+ "**Overall: 1 regressed · 1 improved · 13 no clear change · 1 incomplete · 16 unavailable.**",
text,
)
tables = text.split("<details>")[0]
- self.assertEqual(tables.count("| Metric | Main | This PR | Change |"), 2)
+ self.assertEqual(tables.count("| Metric | Main | This PR | Change |"), 3)
for row in tables.splitlines():
if row.startswith("|"):
self.assertEqual(len(row.strip("|").split("|")), 4)
@@ -711,14 +712,18 @@ class MeasurementTests(unittest.TestCase):
def test_completion_requires_every_metric(self):
side = Side(sha=SHA)
- for definition in (*METRICS, *RUNTIME_METRICS):
- count = 1 if definition.key in ("bundle", "disk") else (3 if definition.key == "install" else 10)
+ for definition in (*METRICS, *RUNTIME_METRICS, *UI_METRICS):
+ count = (
+ 1
+ if definition.key in ("bundle", "disk")
+ else (3 if definition.key == "install" else (2 if definition.key in UI_METRIC_KEYS else 10))
+ )
side.metrics[definition.key] = observations(*([1] * count))
- self.assertTrue(side_complete(side, 10, 3))
- for definition in (*METRICS, *RUNTIME_METRICS):
+ self.assertTrue(side_complete(side, 10, 3, 2))
+ for definition in (*METRICS, *RUNTIME_METRICS, *UI_METRICS):
incomplete = side.model_copy(deep=True)
incomplete.metrics[definition.key][-1] = Observation(trial=0, error="failed")
- self.assertFalse(side_complete(incomplete, 10, 3))
+ self.assertFalse(side_complete(incomplete, 10, 3, 2))
if __name__ == "__main__":
diff --git a/scripts/benchmarks/tests/test_failure_reporting.py b/scripts/benchmarks/tests/test_failure_reporting.py
index 3dd5347b05..86f5f58542 100644
--- a/scripts/benchmarks/tests/test_failure_reporting.py
+++ b/scripts/benchmarks/tests/test_failure_reporting.py
@@ -116,8 +116,11 @@ class FailureBudgetTests(unittest.TestCase):
),
)
self.assertEqual(len([c for c in calls if c[:2] == ("pr", "install")]), 2)
- self.assertFalse(any(c[0] == "pr" and c[1] in ("measure", "runtime") for c in calls))
+ self.assertFalse(any(c[0] == "pr" and c[1] in ("measure", "runtime", "ui") for c in calls))
+ self.assertTrue(any(c[:2] == ("main", "ui") for c in calls))
self.assertNotIn("cold", controller.report.pr_head.metrics)
+ self.assertNotIn("resume_large", controller.report.pr_head.metrics)
+ self.assertTrue(any("pr ui: skipped" in e for e in controller.report.errors))
self.assertTrue(any("first installation" in e for e in controller.report.errors))
@patch.dict(os.environ, {"PRIME_SANDBOX_API_KEY": "fake"})
diff --git a/scripts/benchmarks/tests/test_ui.py b/scripts/benchmarks/tests/test_ui.py
new file mode 100644
index 0000000000..7da5206100
--- /dev/null
+++ b/scripts/benchmarks/tests/test_ui.py
@@ -0,0 +1,349 @@
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import tempfile
+import unittest
+from datetime import UTC, datetime
+from pathlib import Path
+from typing import get_args
+from unittest.mock import patch
+
+import ui
+from report import UI_METRICS, render
+from schema import UI_METRIC_KEYS, Config, Metric, Observation, ProcessMemory, Report, Request, Side
+from ui import (
+ expand_subagents,
+ expected_roster_inactive,
+ fixture_spec,
+ session_id,
+ session_name,
+ spawn_ledger_path,
+ tail_marker,
+ total_cpu,
+ ui_measure,
+ wait_for_roster,
+ write_fixtures,
+)
+
+
+def small_fixture(**overrides):
+ """Fixture generation with tiny counts so tests stay fast."""
+ constants = {
+ "MEDIUM_COUNT": 3,
+ "MEDIUM_MESSAGES": 2,
+ "LARGE_COUNT": 2,
+ "LARGE_MESSAGES": 3,
+ "HUGE_MESSAGES": 4,
+ "FANOUT_COUNT": 2,
+ "FANOUT_MESSAGES": 2,
+ "SUBAGENT_DEPTH": 4,
+ "SUBAGENT_MESSAGES": 2,
+ }
+ constants.update(overrides)
+ return [patch(f"ui.{name}", value) for name, value in constants.items()]
+
+
+def ui_request() -> Request:
+ """A minimal valid request for ui_measure entry checks."""
+ return Request(
+ repository="PrimeIntellect-ai/prime-agent",
+ source_repository="PrimeIntellect-ai/prime-agent",
+ sha="a" * 40,
+ harness_sha="a" * 40,
+ pr=42,
+ run_id=100,
+ attempt=1,
+ role="main",
+ config=Config.load(),
+ )
+
+
+class FixtureTests(unittest.TestCase):
+ def setUp(self):
+ temporary = tempfile.TemporaryDirectory()
+ self.addCleanup(temporary.cleanup)
+ self.root = Path(temporary.name)
+ self.agent_dir = self.root / "agent"
+ self.workspace = self.root / "workspace"
+ self.workspace.mkdir()
+ self.spec: object = None
+
+ def small_fixture(self, **overrides) -> None:
+ for patcher in small_fixture(**overrides):
+ patcher.start()
+ self.addCleanup(patcher.stop)
+ self.spec = fixture_spec()
+
+ def write(self):
+ write_fixtures(self.agent_dir, self.workspace, self.spec)
+ return self.agent_dir / "sessions"
+
+ def test_ids_and_names_are_deterministic(self):
+ self.assertEqual(session_id("large", 1), session_id("large", 1))
+ self.assertEqual(session_id("sub", 6), fixture_spec().subagents[5])
+ self.assertNotEqual(session_id("large", 1), session_id("large", 2))
+ self.assertEqual(session_name("large", 1), "ui-bench-large-01")
+ self.assertEqual(tail_marker("sub", 6), "ui-bench-tail-sub-06")
+
+ def test_write_fixtures_regenerates_identical_files_and_clears_stale_state(self):
+ self.small_fixture()
+ sessions = self.write()
+ first = {path.name: path.read_bytes() for path in sorted(self.agent_dir.rglob("*")) if path.is_file()}
+ stale = sessions / "stale.jsonl"
+ stale.write_text("stale")
+ (self.agent_dir / "session-artifacts").mkdir(parents=True)
+ (self.agent_dir / "session-artifacts" / "leftover.json").write_text("{}")
+ (self.agent_dir / "session-leases").mkdir(parents=True)
+ write_fixtures(self.agent_dir, self.workspace, self.spec)
+ second = {
+ path.name: path.read_bytes() for path in sorted(self.agent_dir.rglob("*")) if path.is_file()
+ }
+ self.assertEqual(first, second)
+ self.assertFalse(stale.exists())
+ self.assertFalse((self.agent_dir / "session-artifacts" / "leftover.json").exists())
+ expected = ui.LARGE_COUNT + ui.MEDIUM_COUNT + 1 + ui.SUBAGENT_DEPTH + ui.FANOUT_COUNT
+ self.assertEqual(len(list(sessions.glob("*.jsonl"))), expected)
+
+ def test_session_file_shape_and_tree_linking(self):
+ self.small_fixture(LARGE_MESSAGES=4)
+ sessions = self.write()
+ identifier = self.spec.large[0]
+ entries = [json.loads(line) for line in (sessions / f"{identifier}.jsonl").read_text().splitlines()]
+ header = entries[0]
+ self.assertEqual(header["type"], "session")
+ self.assertEqual(header["version"], 3)
+ self.assertEqual(header["id"], identifier)
+ self.assertEqual(header["cwd"], str(self.workspace))
+ self.assertEqual(header["rlmDepth"], 0)
+ names = [entry for entry in entries if entry.get("type") == "session_info"]
+ self.assertEqual(names[0]["name"], session_name("large", 1))
+ previous = None
+ for entry in entries[1:]:
+ self.assertEqual(entry["parentId"], previous)
+ previous = entry["id"]
+ assistants = [entry for entry in entries if entry.get("message", {}).get("role") == "assistant"]
+ self.assertTrue(any(tail_marker("large", 1) in json.dumps(entry) for entry in assistants[-1:]))
+ self.assertGreater(len(entries), ui.LARGE_MESSAGES)
+
+ def test_subagent_chain_depths_and_spawn_ledger(self):
+ self.small_fixture()
+ sessions = self.write()
+ ledger = spawn_ledger_path(self.agent_dir, sessions)
+ expected_digest = hashlib.sha256(str(Path(os.path.realpath(sessions))).encode()).hexdigest()[:16]
+ self.assertEqual(ledger.name, f"{expected_digest}.jsonl")
+ records = [json.loads(line) for line in ledger.read_text().splitlines()]
+ self.assertEqual(records[0]["op"], "meta")
+ self.assertEqual(records[0]["sessionsDir"], str(Path(os.path.realpath(sessions))))
+ spawns = records[1:]
+ self.assertEqual(len(spawns), len(self.spec.subagents) + len(self.spec.fanout))
+ chain = [record for record in spawns if record["name"].startswith("ui-bench-sub-")]
+ self.assertEqual(len(chain), len(self.spec.subagents))
+ fanout = [record for record in spawns if record["name"].startswith("ui-bench-fan-")]
+ self.assertEqual(len(fanout), len(self.spec.fanout))
+ parent = str(sessions / f"{self.spec.root}.jsonl")
+ for record in fanout:
+ self.assertEqual(record["depth"], 1)
+ self.assertEqual(record["parent"], parent)
+ for depth, (record, identifier) in enumerate(zip(chain, self.spec.subagents, strict=True), start=1):
+ self.assertEqual(record["op"], "spawn")
+ self.assertEqual(record["v"], 1)
+ self.assertEqual(record["depth"], depth)
+ self.assertEqual(record["parent"], parent)
+ self.assertEqual(record["child"], str(sessions / f"{identifier}.jsonl"))
+ self.assertEqual(record["name"], session_name("sub", depth))
+ parent = record["child"]
+ header = json.loads((sessions / f"{identifier}.jsonl").read_text().splitlines()[0])
+ self.assertEqual(header["rlmDepth"], depth)
+
+
+class ProbeLogicTests(unittest.TestCase):
+ def test_total_cpu_sums_process_tree(self):
+ processes = [
+ ProcessMemory(pid=1, name="daemon", rss=10, cpu=1.25),
+ ProcessMemory(pid=2, name="worker", rss=20, cpu=0.75),
+ ProcessMemory(pid=3, name="kernel", rss=30, cpu=2.0),
+ ]
+ self.assertEqual(total_cpu(processes), 4.0)
+ self.assertEqual(total_cpu([]), 0.0)
+
+ def test_wait_for_roster_requires_a_settled_count(self):
+ frames = [
+ "agents 0 running, 1 idle, 5 inactive",
+ "agents 0 running, 1 idle, 40 inactive",
+ "agents 0 running, 1 idle, 70 inactive",
+ "agents 0 running, 1 idle, 70 inactive",
+ "agents 0 running, 1 idle, 70 inactive",
+ "agents 0 running, 1 idle, 70 inactive",
+ ]
+ terminal = FakeTerminal(frames)
+ with patch("time.perf_counter", side_effect=[float(i) for i in range(200)]):
+ seconds = wait_for_roster(terminal, minimum=70, settle=3.0, timeout=90)
+ self.assertGreaterEqual(seconds, 0.0)
+ self.assertIn("70 inactive", terminal.display.text())
+
+ with self.assertRaises(TimeoutError):
+ wait_for_roster(
+ FakeTerminal(["agents 0 running, 1 idle, 1 inactive"] * 8),
+ minimum=70,
+ settle=3.0,
+ timeout=0.05,
+ )
+
+ def test_wait_for_roster_never_settles_below_the_expected_fixture_count(self):
+ # A splash stuck at an empty or partial roster must not count as hydrated,
+ # even when its count is unchanged for the whole settle window.
+ terminal = FakeTerminal(["agents 0 running, 1 idle, 5 inactive"] * 8)
+ with patch("time.perf_counter", side_effect=[float(i) for i in range(400)]):
+ with self.assertRaises(TimeoutError):
+ wait_for_roster(terminal, minimum=70, settle=3.0, timeout=90)
+
+ def test_expected_roster_inactive_counts_top_level_fixtures_minus_the_live_target(self):
+ self.assertEqual(expected_roster_inactive(), ui.LARGE_COUNT + ui.MEDIUM_COUNT)
+
+ def test_ui_trial_requires_a_successful_first_installation(self):
+ side = Side(sha="a" * 40)
+ with self.assertRaisesRegex(RuntimeError, "first installation"):
+ ui_measure(ui_request(), side, 0, results=Path("/unused"), homes=Path("/unused"), user="bench")
+
+ def test_expand_subagents_requires_a_newly_expanded_row(self):
+ # An ancestor already shows "▾", so only a growing count proves the selected row expanded;
+ # the first expand keystroke is eaten and the retry lands.
+ frames = [
+ "▾ ui-bench-sub-00",
+ "▾ ui-bench-sub-00",
+ "▾ ui-bench-sub-00\n▾ ui-bench-sub-01",
+ ]
+ terminal = FakeTerminal(frames)
+ expand_subagents(terminal)
+ self.assertEqual(terminal.sent, [ui.EXPAND_ARROW, ui.EXPAND_ARROW])
+
+ def test_expand_subagents_times_out_when_nothing_expands(self):
+ frames = ["▸ ui-bench-sub-00"] * 8
+ terminal = FakeTerminal(frames)
+ with self.assertRaises(TimeoutError):
+ expand_subagents(terminal)
+ self.assertEqual(terminal.sent, [ui.EXPAND_ARROW] * 4)
+
+
+class FakeDisplay:
+ def __init__(self, text: str = ""):
+ self._text = text
+
+ def text(self) -> str:
+ return self._text
+
+ def set(self, text: str) -> None:
+ self._text = text
+
+
+class FakeChild:
+ """Records sent keystrokes the way pexpect.spawn.send would receive them."""
+
+ def __init__(self, sent: list[str]):
+ self.sent = sent
+
+ def send(self, data: str) -> None:
+ self.sent.append(data)
+
+
+class FakeTerminal:
+ """Scripted terminal: until() and settle() advance the frames one at a time."""
+
+ def __init__(self, frames: list[str]):
+ self.display = FakeDisplay(frames[0] if frames else "")
+ self.frames = list(frames[1:])
+ self.sent: list[str] = []
+ self.child = FakeChild(self.sent)
+ self.bytes = 0
+
+ def pump(self) -> bool:
+ if not self.frames:
+ return False
+ self.display.set(self.frames.pop(0))
+ self.bytes += len(self.display.text())
+ return True
+
+ def until(self, predicate, timeout: float) -> None: # type: ignore[no-untyped-def]
+ while not predicate(self.display):
+ if not self.pump():
+ raise TimeoutError("Timed out waiting for the expected terminal state")
+
+ def settle(self, seconds: float) -> None:
+ self.pump()
+
+
+class SchemaTests(unittest.TestCase):
+ def test_ui_metric_keys_match_report_definitions_and_metric_literal(self):
+ self.assertEqual(UI_METRIC_KEYS, {definition.key for definition in UI_METRICS})
+ for definition in UI_METRICS:
+ self.assertIn(definition.key, get_args(Metric))
+
+ def test_config_accepts_and_defaults_ui_trials(self):
+ config = Config.load()
+ self.assertEqual(config.ui_trials, 3)
+ raw = json.loads(Path(__file__).resolve().parents[1].joinpath("config.json").read_text())
+ self.assertIn("ui_trials", raw)
+
+ def test_report_validator_uses_ui_trial_counts(self):
+ started = datetime(2026, 9, 9, tzinfo=UTC)
+ config = Config.load()
+ report = Report(
+ repository="PrimeIntellect-ai/prime-agent",
+ head_repository="PrimeIntellect-ai/prime-agent",
+ pr=42,
+ run_id=100,
+ harness_sha="a" * 40,
+ base_sha="a" * 40,
+ head_sha="b" * 40,
+ started_at=started,
+ config=config,
+ main=Side(sha="a" * 40),
+ pr_head=Side(sha="b" * 40),
+ )
+ for side in (report.main, report.pr_head):
+ for definition in UI_METRICS:
+ side.metrics[definition.key] = [
+ Observation(trial=trial, value=1.0) for trial in range(config.ui_trials)
+ ]
+ report.model_validate(report.model_dump())
+
+ overflowing = report.pr_head.model_copy(deep=True)
+ overflowing.metrics["ui_rss"].append(Observation(trial=config.ui_trials, value=1.0))
+ with self.assertRaises(ValueError):
+ Report.model_validate({**report.model_dump(), "pr_head": overflowing.model_dump()})
+
+ def test_render_includes_ui_section(self):
+ started = datetime(2026, 9, 9, tzinfo=UTC)
+ config = Config.load()
+ report = Report(
+ repository="PrimeIntellect-ai/prime-agent",
+ head_repository="PrimeIntellect-ai/prime-agent",
+ pr=42,
+ run_id=100,
+ harness_sha="a" * 40,
+ base_sha="a" * 40,
+ head_sha="b" * 40,
+ started_at=started,
+ config=config,
+ main=Side(sha="a" * 40),
+ pr_head=Side(sha="b" * 40),
+ )
+ for definition in UI_METRICS:
+ report.main.metrics[definition.key] = [
+ Observation(trial=trial, value=2.0) for trial in range(config.ui_trials)
+ ]
+ report.pr_head.metrics[definition.key] = [
+ Observation(trial=trial, value=2.0) for trial in range(config.ui_trials)
+ ]
+ report.status = "completed"
+ text = render(report)
+ self.assertIn("**UI interactions**", text)
+ self.assertIn("Resume large session (cold)", text)
+ self.assertIn("Open subagent session at depth 6", text)
+ self.assertIn("UI memory after interactions", text)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/scripts/benchmarks/ui.py b/scripts/benchmarks/ui.py
new file mode 100644
index 0000000000..eae63a3cb8
--- /dev/null
+++ b/scripts/benchmarks/ui.py
@@ -0,0 +1,633 @@
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import pwd
+import re
+import shutil
+import time
+import uuid
+from dataclasses import dataclass
+from datetime import UTC, datetime
+from pathlib import Path
+
+from schema import ProcessMemory, Request, Side
+from terminal import Terminal
+
+# Fixture shape: "many sessions, many large sessions" plus a deep subagent chain.
+# Scale mirrors real long-lived installs: hundreds of saved sessions, one very
+# large transcript, and a spawn ledger with enough edges to stress hydration.
+MEDIUM_COUNT = 150
+MEDIUM_MESSAGES = 120
+LARGE_COUNT = 3
+LARGE_MESSAGES = 2000
+HUGE_MESSAGES = 16000
+FANOUT_COUNT = 40
+FANOUT_MESSAGES = 20
+SUBAGENT_DEPTH = 6
+SUBAGENT_MESSAGES = 400
+
+BASE_TIMESTAMP_MS = 946684800000 # 2000-01-01T00:00:00Z; fixed so fixtures are deterministic
+TOOL_OUTPUT_LINES = 60
+
+AGENTS_VIEW_HINT = "type to search sessions"
+SEARCH_PLACEHOLDER = "Search sessions"
+ROSTER_COUNT = re.compile(r"agents\s+(\d+) running, (\d+) idle, (\d+) inactive")
+READY_MARKER = "benchready"
+LEFT_ARROW = "\x1b[D"
+RIGHT_ARROW = "\x1b[C"
+DOWN_ARROW = "\x1b[B"
+EXPAND_ARROW = "\x1b[1;3C"
+BACKSPACE = "\x7f"
+ENTER = "\r"
+FIXTURE_NAMESPACE = uuid.UUID("b0d5f4a1-6d64-4bf8-9d7a-2f4f0a9c1e10")
+
+
+def session_id(kind: str, index: int) -> str:
+ return str(uuid.uuid5(FIXTURE_NAMESPACE, f"prime-agent-ui-bench/{kind}/{index}"))
+
+
+def session_name(kind: str, index: int) -> str:
+ return f"ui-bench-{kind}-{index:02d}"
+
+
+def tail_marker(kind: str, index: int) -> str:
+ return f"ui-bench-tail-{kind}-{index:02d}"
+
+
+@dataclass(frozen=True)
+class FixtureSpec:
+ """Session ids the probe addresses; every id is derived deterministically."""
+
+ large: list[str]
+ medium: list[str]
+ root: str
+ subagents: list[str]
+ fanout: list[str]
+
+ @property
+ def resume_id(self) -> str:
+ """Cold-resume target; never touched by the warm scenario."""
+ return self.large[1]
+
+ @property
+ def switch_id(self) -> str:
+ """Warm in-session /resume target; the very large transcript."""
+ return self.large[2]
+
+ @property
+ def open_id(self) -> str:
+ """Search-and-open target from the agents view."""
+ return self.large[0]
+
+
+def _iso(ms: int) -> str:
+ return datetime.fromtimestamp(ms / 1000, tz=UTC).isoformat(timespec="milliseconds").replace("+00:00", "Z")
+
+
+def _session_lines(
+ name: str,
+ identifier: str,
+ count: int,
+ cwd: Path,
+ *,
+ depth: int,
+ tail: str | None,
+) -> list[dict]:
+ """One deterministic session transcript: header, name, model, then a realistic message mix."""
+ timestamp = BASE_TIMESTAMP_MS
+ parent: str | None = None
+ lines: list[dict] = []
+ prefix = identifier.replace("-", "")[:8]
+
+ def link(entry_id: str) -> dict:
+ nonlocal parent, timestamp
+ timestamp += 1000
+ entry = {"id": entry_id, "parentId": parent, "timestamp": _iso(timestamp)}
+ parent = entry_id
+ return entry
+
+ lines.append(
+ {
+ "type": "session",
+ "version": 3,
+ "id": identifier,
+ "timestamp": _iso(timestamp),
+ "cwd": str(cwd),
+ "rlmDepth": depth,
+ }
+ )
+ entry = link(f"{prefix}n")
+ entry.update({"type": "session_info", "name": name})
+ lines.append(entry)
+ entry = link(f"{prefix}m")
+ entry.update({"type": "model_change", "provider": "prime-inference", "modelId": "internal/glm-5.3-fast"})
+ lines.append(entry)
+ usage = {
+ "input": 100,
+ "output": 50,
+ "cacheRead": 0,
+ "cacheWrite": 0,
+ "totalTokens": 150,
+ "cost": {"input": 0.0001, "output": 0.0001, "cacheRead": 0, "cacheWrite": 0, "total": 0.0002},
+ }
+ for index in range(count):
+ final = index == count - 1
+ entry = link(f"{prefix}u{index}")
+ entry.update(
+ {
+ "type": "message",
+ "message": {
+ "role": "user",
+ "timestamp": 0,
+ "content": (
+ f"Please continue task {name} step {index}: review the module, run the checks, "
+ "and summarize findings for the migration notes."
+ ),
+ },
+ }
+ )
+ lines.append(entry)
+ text = (
+ f"Working on {name} step {index}. The parser handles nested records correctly; the next edit "
+ "keeps the schema stable while trimming the duplicated branch."
+ )
+ if final and tail:
+ text = f"Finished all steps. {tail}"
+ content = [
+ {
+ "type": "thinking",
+ "thinking": (
+ f"Step {index} for {name}: check the invariants, then update the affected call "
+ "sites before running the full suite again to confirm no behavior changed."
+ ),
+ },
+ {"type": "text", "text": text},
+ ]
+ if not final:
+ content.append(
+ {"type": "toolCall", "id": f"call-{index}", "name": "ipython", "arguments": {"code": "pass"}}
+ )
+ entry = link(f"{prefix}a{index}")
+ entry.update(
+ {
+ "type": "message",
+ "message": {
+ "role": "assistant",
+ "timestamp": 0,
+ "content": content,
+ "api": "responses",
+ "provider": "prime-inference",
+ "model": "internal/glm-5.3-fast",
+ "usage": usage,
+ "stopReason": "stop" if final else "toolUse",
+ },
+ }
+ )
+ lines.append(entry)
+ if not final:
+ entry = link(f"{prefix}t{index}")
+ entry.update(
+ {
+ "type": "message",
+ "message": {
+ "role": "toolResult",
+ "toolCallId": f"call-{index}",
+ "toolName": "ipython",
+ "content": [{"type": "text", "text": "all checks passed\n" * TOOL_OUTPUT_LINES}],
+ "isError": False,
+ "timestamp": 0,
+ },
+ }
+ )
+ lines.append(entry)
+ return lines
+
+
+def spawn_ledger_path(agent_dir: Path, sessions_dir: Path) -> Path:
+ canonical = Path(os.path.realpath(sessions_dir))
+ digest = hashlib.sha256(str(canonical).encode()).hexdigest()[:16]
+ return agent_dir / "rlm-ledger" / f"{digest}.jsonl"
+
+
+def write_fixtures(agent_dir: Path, workspace: Path, spec: FixtureSpec, *, uid: int | None = None) -> None:
+ """Regenerate every session file and the spawn ledger from scratch for one trial.
+
+ The harness runs as root; pass the benchmark user's ids so the spawned CLI can
+ read and append its own sessions.
+ """
+ sessions_dir = agent_dir / "sessions"
+ for directory in (sessions_dir, agent_dir / "session-artifacts", agent_dir / "session-leases"):
+ shutil.rmtree(directory, ignore_errors=True)
+ sessions_dir.mkdir(parents=True, exist_ok=True)
+
+ def write(identifier: str, name: str, count: int, *, depth: int = 0, tail: str | None = None) -> Path:
+ path = sessions_dir / f"{identifier}.jsonl"
+ path.write_text(
+ "".join(
+ json.dumps(line, separators=(",", ":")) + "\n"
+ for line in _session_lines(name, identifier, count, workspace, depth=depth, tail=tail)
+ )
+ )
+ return path
+
+ for index in range(1, LARGE_COUNT + 1):
+ identifier = spec.large[index - 1]
+ messages = HUGE_MESSAGES if index == 3 else LARGE_MESSAGES
+ write(identifier, session_name("large", index), messages, tail=tail_marker("large", index))
+ for index in range(1, MEDIUM_COUNT + 1):
+ identifier = spec.medium[index - 1]
+ write(identifier, session_name("medium", index), MEDIUM_MESSAGES)
+ parent = write(spec.root, session_name("root", 0), SUBAGENT_MESSAGES, tail=tail_marker("root", 0))
+ edges = []
+ for fanout in range(1, FANOUT_COUNT + 1):
+ identifier = spec.fanout[fanout - 1]
+ path = write(identifier, session_name("fan", fanout), FANOUT_MESSAGES, depth=1)
+ edges.append(
+ {
+ "childId": identifier,
+ "parent": str(parent),
+ "child": str(path),
+ "depth": 1,
+ "name": session_name("fan", fanout),
+ }
+ )
+ for depth in range(1, SUBAGENT_DEPTH + 1):
+ identifier = spec.subagents[depth - 1]
+ path = write(
+ identifier,
+ session_name("sub", depth),
+ SUBAGENT_MESSAGES,
+ depth=depth,
+ tail=tail_marker("sub", depth),
+ )
+ edges.append(
+ {
+ "childId": identifier,
+ "parent": str(parent),
+ "child": str(path),
+ "depth": depth,
+ "name": session_name("sub", depth),
+ }
+ )
+ parent = path
+ ledger = spawn_ledger_path(agent_dir, sessions_dir)
+ ledger.parent.mkdir(parents=True, exist_ok=True)
+ records = [
+ {
+ "v": 1,
+ "op": "meta",
+ "at": _iso(BASE_TIMESTAMP_MS),
+ "sessionsDir": str(Path(os.path.realpath(sessions_dir))),
+ }
+ ]
+ records.extend(
+ {"v": 1, "op": "spawn", "at": _iso(BASE_TIMESTAMP_MS + edge["depth"]), **edge} for edge in edges
+ )
+ ledger.write_text("".join(json.dumps(record, separators=(",", ":")) + "\n" for record in records))
+ if uid is not None:
+ gid = pwd.getpwuid(uid).pw_gid
+ os.chown(agent_dir, uid, gid)
+ for path in agent_dir.rglob("*"):
+ os.chown(path, uid, gid)
+
+
+def fixture_spec() -> FixtureSpec:
+ return FixtureSpec(
+ large=[session_id("large", index) for index in range(1, LARGE_COUNT + 1)],
+ medium=[session_id("medium", index) for index in range(1, MEDIUM_COUNT + 1)],
+ root=session_id("root", 0),
+ subagents=[session_id("sub", depth) for depth in range(1, SUBAGENT_DEPTH + 1)],
+ fanout=[session_id("fan", index) for index in range(1, FANOUT_COUNT + 1)],
+ )
+
+
+def process_stats(uid: int) -> list[ProcessMemory]:
+ """Snapshot every live process of one user with RSS, PSS, and cumulative CPU time."""
+ stats: list[ProcessMemory] = []
+ try:
+ clock_ticks = os.sysconf("SC_CLK_TCK")
+ except (ValueError, AttributeError, OSError):
+ return stats
+ for status_path in Path("/proc").glob("[0-9]*/status"):
+ try:
+ fields = dict(line.split(":", 1) for line in status_path.read_text().splitlines() if ":" in line)
+ if int(fields["Uid"].split()[0]) != uid or fields["State"].strip().startswith("Z"):
+ continue
+ stat_fields = (status_path.parent / "stat").read_text().rsplit(")", 1)[1].split()
+ cpu_seconds = (int(stat_fields[11]) + int(stat_fields[12])) / clock_ticks
+ pss = None
+ try:
+ for line in (status_path.parent / "smaps_rollup").read_text().splitlines():
+ if line.startswith("Pss:"):
+ pss = int(line.split()[1]) * 1024
+ except (PermissionError, FileNotFoundError, ProcessLookupError):
+ pass
+ stats.append(
+ ProcessMemory(
+ pid=int(status_path.parent.name),
+ name=fields["Name"].strip(),
+ rss=int(fields.get("VmRSS", "0").split()[0]) * 1024,
+ pss=pss,
+ cpu=cpu_seconds,
+ )
+ )
+ except (PermissionError, FileNotFoundError, ProcessLookupError, ValueError, IndexError):
+ continue
+ return stats
+
+
+def total_cpu(stats: list[ProcessMemory]) -> float:
+ return sum(process.cpu or 0.0 for process in stats)
+
+
+def quiet(terminal: Terminal, seconds: float = 0.6, cap: float = 8.0) -> None:
+ """Pump output until the terminal stays silent, so the editor can accept input again."""
+ started = time.perf_counter()
+ last_output = time.perf_counter()
+ while time.perf_counter() - started < cap:
+ if terminal.pump():
+ last_output = time.perf_counter()
+ elif time.perf_counter() - last_output >= seconds:
+ return
+
+
+def input_ready(terminal: Terminal, marker: str, *, timeout: float = 120) -> None:
+ """Wait for a rendered marker, then confirm the editor echoes, retrying past input loss.
+
+ Keystrokes sent while a freshly opened session is still mounting can be dropped,
+ so the echo is retried with a quiet window instead of failing the trial.
+ """
+ terminal.until(lambda display: marker in display.text(), timeout)
+ for _ in range(5):
+ quiet(terminal, 0.8)
+ terminal.child.send(READY_MARKER)
+ try:
+ terminal.until(lambda display: READY_MARKER in display.text(), 6)
+ except TimeoutError:
+ terminal.child.send(BACKSPACE * (len(READY_MARKER) + 2))
+ continue
+ terminal.child.send(BACKSPACE * len(READY_MARKER))
+ terminal.until(lambda display: READY_MARKER not in display.text(), 10)
+ return
+ raise TimeoutError("Editor did not echo the readiness marker")
+
+
+def clear_search(terminal: Terminal, *, attempts: int = 5) -> None:
+ """Empty the agents-view search box; keystrokes can be eaten by re-entry renders."""
+ for _ in range(attempts):
+ terminal.child.send(BACKSPACE * 24)
+ try:
+ terminal.until(lambda display: SEARCH_PLACEHOLDER in display.text(), 5)
+ return
+ except TimeoutError:
+ continue
+ raise TimeoutError("Agents-view search box did not clear")
+
+
+def type_query(terminal: Terminal, query: str, *, attempts: int = 3) -> None:
+ """Type a search query and verify it echoed before filtering on it."""
+ for _ in range(attempts):
+ clear_search(terminal)
+ terminal.child.send(query)
+ try:
+ terminal.until(lambda display: query in display.text(), 8)
+ return
+ except TimeoutError:
+ continue
+ raise TimeoutError("Agents-view search query did not echo")
+
+
+def expand_subagents(terminal: Terminal, *, attempts: int = 4) -> None:
+ """Expand the selected row's subagent list; the toggle retries to survive eaten keystrokes."""
+ expanded = terminal.display.text().count("▾")
+ for _ in range(attempts):
+ terminal.child.send(EXPAND_ARROW)
+ terminal.settle(1.0)
+ if terminal.display.text().count("▾") > expanded:
+ return
+ raise TimeoutError("Selected agent subagents did not expand")
+
+
+def roster_inactive(display) -> int | None: # type: ignore[no-untyped-def]
+ for line in display.text().splitlines():
+ match = ROSTER_COUNT.search(line)
+ if match:
+ return int(match.group(3))
+ return None
+
+
+def expected_roster_inactive() -> int:
+ """Saved top-level fixture sessions a hydrated roster must list.
+
+ Large, medium, and the chain-root fixtures are top-level roster rows, but
+ the warm scenario holds the switch target live; nested fan-out and
+ subagent-chain files stay out of the top-level count either way.
+ """
+ top_level = LARGE_COUNT + MEDIUM_COUNT + 1
+ return top_level - 1
+
+
+def wait_for_roster(terminal: Terminal, *, minimum: int, settle: float = 3.0, timeout: float = 90) -> float:
+ """Wait until the roster lists `minimum` inactive sessions and stops growing.
+
+ Saved sessions stream in, but an empty or stalled roster must not count
+ as settled just because its count stopped moving.
+ """
+ started = time.perf_counter()
+ last_count: int | None = None
+ last_change = time.perf_counter()
+ while time.perf_counter() - started < timeout:
+ terminal.settle(0.4)
+ count = roster_inactive(terminal.display)
+ if (
+ count is not None
+ and count == last_count
+ and count >= minimum
+ and time.perf_counter() - last_change >= settle
+ ):
+ return time.perf_counter() - started
+ if count != last_count:
+ last_count = count
+ last_change = time.perf_counter()
+ raise TimeoutError("Agents-view roster did not settle")
+
+
+def ui_measure(request: Request, side: Side, trial: int, *, results: Path, homes: Path, user: str) -> None:
+ """One UI-interaction trial: fresh fixtures, a cold resume, then the warm navigation scenario."""
+ from worker import clean_error, environment, record, stop_processes
+
+ if not any(s.value is not None and s.trial == 0 for s in side.metrics.get("install", [])):
+ raise RuntimeError("The first installation must succeed before interactive measurements")
+ home = homes / user
+ workspace = home / "workspace"
+ agent_dir = home / ".prime/agent"
+ uid = pwd.getpwnam(user).pw_uid
+ spec = fixture_spec()
+ details: dict[str, dict] = {}
+ metric = "resume_large"
+
+ def note(name: str, *, seconds: float, cpu: float, pty_bytes: int) -> None:
+ details[name] = {"seconds": round(seconds, 4), "cpu": round(cpu, 4), "pty_bytes": pty_bytes}
+
+ def cpu_total() -> float:
+ return total_cpu(process_stats(uid))
+
+ stop_processes(user)
+ write_fixtures(agent_dir, workspace, spec, uid=uid)
+ env = environment(user)
+ runuser = "/usr/sbin/runuser"
+ try:
+ # Cold resume of a large session from the CLI, from process spawn to usable editor.
+ cpu_start = cpu_total()
+ terminal = Terminal(
+ [runuser, "-u", user, "--", "prime-agent", "--resume", spec.resume_id],
+ workspace,
+ env,
+ results / f"ui-resume-{trial}",
+ )
+ try:
+ metric = "resume_large"
+ bytes_start = terminal.bytes
+ input_ready(terminal, tail_marker("large", 2))
+ elapsed = time.perf_counter() - terminal.started
+ cpu = cpu_total() - cpu_start
+ record(side, "resume_large", trial, elapsed) # type: ignore[arg-type]
+ record(side, "resume_large_cpu", trial, cpu) # type: ignore[arg-type]
+ note("resume_large", seconds=elapsed, cpu=cpu, pty_bytes=terminal.bytes - bytes_start)
+ finally:
+ terminal.close()
+ stop_processes(user)
+
+ # Warm navigation scenario in a single TUI process.
+ metric = "switch_large"
+ terminal = Terminal(
+ [runuser, "-u", user, "--", "prime-agent"], workspace, env, results / f"ui-scenario-{trial}"
+ )
+ try:
+ terminal.ready()
+
+ # Warm switch into a different large session from the running editor.
+ started = time.perf_counter()
+ cpu_start = cpu_total()
+ bytes_start = terminal.bytes
+ terminal.child.send(f"/resume {spec.switch_id}{ENTER}")
+ input_ready(terminal, tail_marker("large", 3))
+ elapsed = time.perf_counter() - started
+ cpu = cpu_total() - cpu_start
+ record(side, "switch_large", trial, elapsed) # type: ignore[arg-type]
+ record(side, "switch_large_cpu", trial, cpu) # type: ignore[arg-type]
+ note("switch_large", seconds=elapsed, cpu=cpu, pty_bytes=terminal.bytes - bytes_start)
+
+ # Session -> agents view: keystroke to rendered roster splash.
+ metric = "agents_view"
+ started = time.perf_counter()
+ cpu_start = cpu_total()
+ bytes_start = terminal.bytes
+ terminal.child.send(LEFT_ARROW)
+ terminal.until(lambda display: AGENTS_VIEW_HINT in display.text(), 60)
+ elapsed = time.perf_counter() - started
+ cpu = cpu_total() - cpu_start
+ record(side, "agents_view", trial, elapsed) # type: ignore[arg-type]
+ record(side, "agents_view_cpu", trial, cpu) # type: ignore[arg-type]
+ note("agents_view", seconds=elapsed, cpu=cpu, pty_bytes=terminal.bytes - bytes_start)
+
+ # Full roster with many sessions: saved sessions and ledger children stream in.
+ metric = "agents_roster"
+ cpu_start = cpu_total()
+ bytes_start = terminal.bytes
+ roster_seconds = wait_for_roster(terminal, minimum=expected_roster_inactive())
+ cpu = cpu_total() - cpu_start
+ record(side, "agents_roster", trial, roster_seconds) # type: ignore[arg-type]
+ record(side, "agents_roster_cpu", trial, cpu) # type: ignore[arg-type]
+ note("agents_roster", seconds=roster_seconds, cpu=cpu, pty_bytes=terminal.bytes - bytes_start)
+
+ # Agents view -> another large session, found by its unique id prefix.
+ metric = "agents_open"
+ started = time.perf_counter()
+ cpu_start = cpu_total()
+ bytes_start = terminal.bytes
+ type_query(terminal, spec.open_id[:8])
+ terminal.until(lambda display: session_name("large", 1) in display.text(), 60)
+ terminal.settle(0.8)
+ terminal.child.send(RIGHT_ARROW)
+ input_ready(terminal, tail_marker("large", 1))
+ elapsed = time.perf_counter() - started
+ cpu = cpu_total() - cpu_start
+ record(side, "agents_open", trial, elapsed) # type: ignore[arg-type]
+ record(side, "agents_open_cpu", trial, cpu) # type: ignore[arg-type]
+ note("agents_open", seconds=elapsed, cpu=cpu, pty_bytes=terminal.bytes - bytes_start)
+
+ # Chain parent: open the root so the deepest subagent opens against a live parent.
+ metric = "parent_open"
+ started = time.perf_counter()
+ cpu_start = cpu_total()
+ bytes_start = terminal.bytes
+ terminal.child.send(LEFT_ARROW)
+ terminal.until(lambda display: AGENTS_VIEW_HINT in display.text(), 60)
+ terminal.settle(1.0)
+ type_query(terminal, spec.root[:8])
+ terminal.until(lambda display: session_name("root", 0) in display.text(), 60)
+ terminal.settle(0.5)
+ terminal.child.send(RIGHT_ARROW)
+ input_ready(terminal, tail_marker("root", 0))
+ elapsed = time.perf_counter() - started
+ cpu = cpu_total() - cpu_start
+ record(side, "parent_open", trial, elapsed) # type: ignore[arg-type]
+ record(side, "parent_open_cpu", trial, cpu) # type: ignore[arg-type]
+ note("parent_open", seconds=elapsed, cpu=cpu, pty_bytes=terminal.bytes - bytes_start)
+
+ # Deepest subagent session: search, drill in, open at depth SUBAGENT_DEPTH
+ # while its parent session is live, matching real subagent workflows.
+ metric = "subagent_open"
+ started = time.perf_counter()
+ cpu_start = cpu_total()
+ bytes_start = terminal.bytes
+ terminal.child.send(LEFT_ARROW)
+ terminal.until(lambda display: AGENTS_VIEW_HINT in display.text(), 60)
+ terminal.settle(1.0)
+ query = spec.subagents[-1][:8]
+ type_query(terminal, query)
+ terminal.until(lambda display: session_name("root", 0) in display.text(), 60)
+ clear_search(terminal)
+ expand_subagents(terminal)
+ # The search box must be empty to expand; refilter to the chain so the
+ # first fan-out sibling cannot steal the down-arrow, then drill a level
+ # at a time until the deepest subagent row is selected.
+ type_query(terminal, query)
+ terminal.settle(0.5)
+ terminal.child.send(DOWN_ARROW)
+ terminal.settle(0.8)
+ for _ in range(SUBAGENT_DEPTH - 1):
+ clear_search(terminal)
+ expand_subagents(terminal)
+ terminal.settle(0.8)
+ terminal.child.send(DOWN_ARROW)
+ terminal.settle(0.8)
+ terminal.child.send(RIGHT_ARROW)
+ input_ready(terminal, tail_marker("sub", SUBAGENT_DEPTH))
+ elapsed = time.perf_counter() - started
+ cpu = cpu_total() - cpu_start
+ record(side, "subagent_open", trial, elapsed) # type: ignore[arg-type]
+ record(side, "subagent_open_cpu", trial, cpu) # type: ignore[arg-type]
+ note("subagent_open", seconds=elapsed, cpu=cpu, pty_bytes=terminal.bytes - bytes_start)
+
+ # Whole-tree memory after the interactions.
+ metric = "ui_rss"
+ terminal.settle(1)
+ processes = process_stats(uid)
+ if not processes:
+ raise RuntimeError("No owned processes found for the UI memory measurement")
+ record(side, "ui_rss", trial, sum(process.rss for process in processes)) # type: ignore[arg-type]
+ side.processes = processes
+ (results / f"ui-memory-{trial}.json").write_text(
+ json.dumps([process.model_dump() for process in processes], indent=2) + "\n"
+ )
+ finally:
+ terminal.close()
+ except Exception as error:
+ record(side, metric, trial, error=clean_error(error)) # type: ignore[arg-type]
+ finally:
+ stop_processes(user)
+ (results / f"ui-{trial}.json").write_text(json.dumps(details, indent=2, sort_keys=True) + "\n")
diff --git a/scripts/benchmarks/worker.py b/scripts/benchmarks/worker.py
index d345afae48..93f60b23b1 100644
--- a/scripts/benchmarks/worker.py
+++ b/scripts/benchmarks/worker.py
@@ -547,7 +547,7 @@ def runtime(side: Side, trial: int) -> None:
def main() -> None:
parser = argparse.ArgumentParser()
- parser.add_argument("phase", choices=("prepare", "install", "measure", "runtime"))
+ parser.add_argument("phase", choices=("prepare", "install", "measure", "runtime", "ui"))
parser.add_argument("--trial", type=int, default=0)
args = parser.parse_args()
RESULTS.mkdir(exist_ok=True)
@@ -569,6 +569,17 @@ def main() -> None:
install(request, result.side, args.trial)
elif args.phase == "measure":
measure(request, result.side, args.trial)
+ elif args.phase == "ui":
+ from ui import ui_measure
+
+ ui_measure(
+ request,
+ result.side,
+ args.trial,
+ results=RESULTS,
+ homes=HOMES,
+ user="benchmark1",
+ )
else:
runtime(result.side, args.trial)
except Exception as error: