Skip to content

A self-improving RLM agent for coding workflows and long-running autonomous tasks.

Rust

pa-perf: two-mark agents view, transcript paging/top, paint-gap gauge, idle cost (#3723)

Sebastian Müllerc971d48e3c0c
diff --git a/crates/pa-perf/Cargo.toml b/crates/pa-perf/Cargo.toml
index 7f1ed12b61..cec2ccdd2b 100644
--- a/crates/pa-perf/Cargo.toml
+++ b/crates/pa-perf/Cargo.toml
@@ -22,7 +22,8 @@ vt100 = "0.15"
# pty + poll for the bench harness: the same nix version the workspace
# already resolves for pa-core/pa-cli.
[target.'cfg(unix)'.dependencies]
-nix = { version = "0.29", features = ["term", "signal", "process", "fs", "poll"] }
+# `feature` provides sysconf(CLK_TCK) for idle-cost's /proc tick counts.
+nix = { version = "0.29", features = ["term", "signal", "process", "fs", "poll", "feature"] }
[dependencies.pa-types]
path = "../pa-types"
diff --git a/crates/pa-perf/README.md b/crates/pa-perf/README.md
index f6442e40d6..ab4b1e252d 100644
--- a/crates/pa-perf/README.md
+++ b/crates/pa-perf/README.md
@@ -119,6 +119,40 @@ the liveness deadline only fails broken runs):
| `open-session` | cold | `--resume <huge session>` -> the huge session's tail marker (the final user row; its reply is one line) |
| `session-switch` | cold | enter on the uniquely-filtered switch-b row -> B's transcript marker (the final user row; its reply is one line) |
| `at-completion` | cold | the `@` keystroke -> the first completion menu row |
+| `transcript-paging` | cold | `--resume <huge session>`, then 20 PageUps: each keystroke -> the first frame whose topmost block ordinal is above the previous frame's (one sample per page) |
+| `transcript-top` | cold | `--resume <huge session>`, then the `tui.viewport.top` key -> the first frame showing the session's first block (ordinal 0); the head is never in the attach tail, so this is a daemon page fetch |
+| `idle-cost` | cold | owned daemon + TUI, ready, then a 30 s idle window: CPU ms (utime+stime) of the daemon, its descendants, and the TUI, plus their context switches (medium only, Linux only) |
+
+First paint: `agents-view` and `agents-view-warm` also record the first
+usable paint of the same run (`first_paint`: the first frame whose roster
+lists a loaded row); the median stays the settled mark the baseline keys
+mean. `cold-start`/`cold-ready` (and `warm-*`) are already a
+first-paint/ready pair from one launch. `open-session` and `session-switch`
+have no separate first paint: their marker is the first usable frame.
+
+Block ordinals: the corpus prefixes every generated text block of the huge
+session with `ord<n>`, `n` counting up from 0 in transcript order. The 20
+PageUps stay inside the attach tail (the daemon serves the last 512
+messages), so `transcript-paging` measures the TUI's own scroll and render;
+`transcript-top` measures the daemon's head-page fetch (`get_messages`).
+
+Stall gauge (`worst_gap`): each run's longest no-output stretch inside its
+measured windows (trigger -> mark). The report prints its p95 and max; it
+never gates.
+
+idle-cost walks `/proc` from the owned pids down
+(`/proc/<pid>/task/*/children`, never by name) and sums CPU ticks (`stat`)
+and context switches (`task/*/status`) over the processes alive at the
+window's end; a process that exits inside the window is not counted, and
+the run fails if the daemon or the TUI exits. It runs once per suite on the
+medium leg: `pa-perf run` skips it elsewhere and on non-Linux, and
+`bench --scenario idle-cost` errors on non-Linux. CPU resolution is one
+clock tick (10 ms at 100 Hz).
+
+Screen needles see only completed frames: the chat surface brackets its
+paints in synchronized output (mode 2026), and the terminal the bench
+presents as (ghostty) shows such a frame only at its end. The agents view
+paints unbracketed, so its needles see bytes as they arrive.
Mechanics (the repo's pty-e2e discipline): the TUI child runs in its own
session with NO controlling tty (`setsid` pre-exec; vendored crossterm
@@ -139,9 +173,12 @@ pid/name-based kill.
pa-perf report --baseline crates/pa-perf/baseline.json --results results/*.json [--markdown OUT]
```
-Prints the aggregate table (scenario/profile, median, p95, baseline, delta)
-and exits nonzero ONLY when a non-report-only median exceeds twice the
-baseline. Noisy scenarios (>20% spread across 3 CI runs) are marked
+Prints the aggregate table (scenario/profile, median, p95, first-paint
+median, worst gap p95 / max, baseline, delta), then one line per idle-cost
+result (CPU ms and context switches), and exits nonzero ONLY when a
+non-report-only median exceeds twice the baseline. The first-paint and
+worst-gap columns never gate; scenarios without a baseline entry print as
+`(no baseline)`. Noisy scenarios (>20% spread across 3 CI runs) are marked
`report_only` in the baseline: their deltas always print, the gate never
fails. `baseline.json` is committed from CI numbers, not laptop/sandbox
ones.
diff --git a/crates/pa-perf/src/bench/mod.rs b/crates/pa-perf/src/bench/mod.rs
index 95d74d364f..7a7ebf6c9f 100644
--- a/crates/pa-perf/src/bench/mod.rs
+++ b/crates/pa-perf/src/bench/mod.rs
@@ -34,6 +34,15 @@ pub enum Scenario {
SessionSwitch,
/// The `@` keystroke -> the first completion menu row.
AtCompletion,
+ /// `--resume <huge session>`, then each PageUp -> the first frame scrolled
+ /// up a page (one sample per page).
+ TranscriptPaging,
+ /// `--resume <huge session>`, then the top key -> the session's first
+ /// block (a daemon page fetch).
+ TranscriptTop,
+ /// CPU ms of the owned daemon, its descendants, and the TUI over an idle
+ /// window after ready. Linux-only (/proc).
+ IdleCost,
}
/// Every bench knob for one scenario over one corpus.
@@ -86,6 +95,26 @@ pub struct BenchResult {
pub needle: String,
pub binary: String,
pub corpus: BenchCorpus,
+ /// The first usable paint of the same runs (the median above stays the
+ /// settled mark, the baseline's meaning).
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub first_paint: Option<Series>,
+ /// Each run's longest no-output gap inside its measured windows (a
+ /// flag, never gated).
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub worst_gap: Option<Series>,
+ /// idle-cost: each run's context switches over the idle window.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub wakeups: Option<Vec<u64>>,
+}
+
+/// A secondary per-run series (ms) with its summary statistics.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct Series {
+ pub samples_ms: Vec<f64>,
+ pub median_ms: f64,
+ pub p95_ms: f64,
+ pub max_ms: f64,
}
/// The corpus summary the result records (baseline keys use scenario/profile).
diff --git a/crates/pa-perf/src/bench/pty.rs b/crates/pa-perf/src/bench/pty.rs
index d5dfbab810..8d76f88d19 100644
--- a/crates/pa-perf/src/bench/pty.rs
+++ b/crates/pa-perf/src/bench/pty.rs
@@ -21,6 +21,12 @@ pub(crate) const KITTY_ANSWER: &[u8] = b"\x1b[?7u\x1b[?62;c";
/// The idle poll slice between empty reads (poll(2) drives the wakeups).
const IDLE_POLL: Duration = Duration::from_millis(25);
+/// The synchronized-output bracket (mode 2026): the terminal the bench
+/// presents as (ghostty) shows only completed frames, so screen predicates
+/// must too.
+const SYNC_BEGIN: &[u8] = b"\x1b[?2026h";
+const SYNC_END: &[u8] = b"\x1b[?2026l";
+
/// One product child on a pty, plus the collected master stream and the
/// decoded screen (a vt100 model): byte needles cover literal paints; the
/// screen model covers content that only exists as cell diffs (the agents
@@ -36,6 +42,10 @@ pub(crate) struct PtyChild {
kitty_answered: bool,
/// Set only after the exited child's final PTY bytes have drained.
exited: Option<std::process::ExitStatus>,
+ /// When each read chunk arrived (the stall gauge's no-output gaps).
+ arrivals: Vec<Instant>,
+ /// Inside an unfinished synchronized-output frame.
+ in_frame: bool,
}
impl PtyChild {
@@ -84,6 +94,8 @@ impl PtyChild {
screen: vt100::Parser::new(rows, cols, 0),
kitty_answered: false,
exited: None,
+ arrivals: Vec::new(),
+ in_frame: false,
})
}
@@ -194,9 +206,17 @@ impl PtyChild {
/// Include the previous read's overlap when a terminal query is split.
fn receive(&mut self, chunk: &[u8]) -> anyhow::Result<()> {
+ self.arrivals.push(Instant::now());
let from = self.output.len().saturating_sub(KITTY_QUERY.len() - 1);
+ let frame_from = self.output.len().saturating_sub(SYNC_BEGIN.len() - 1);
self.output.extend_from_slice(chunk);
self.screen.process(chunk);
+ let tail = &self.output[frame_from..];
+ match (rfind(tail, SYNC_BEGIN), rfind(tail, SYNC_END)) {
+ (Some(begin), end) if end.is_none_or(|end| begin > end) => self.in_frame = true,
+ (_, Some(_)) => self.in_frame = false,
+ _ => {}
+ }
if !self.kitty_answered && find(&self.output[from..], KITTY_QUERY).is_some() {
self.write(KITTY_ANSWER)?;
self.kitty_answered = true;
@@ -234,7 +254,7 @@ impl PtyChild {
settled: impl Fn(&str) -> bool,
) -> anyhow::Result<Instant> {
loop {
- if settled(&self.screen_text()) {
+ if !self.in_frame && settled(&self.screen_text()) {
return Ok(Instant::now());
}
if let Some(status) = self.exited {
@@ -254,10 +274,34 @@ impl PtyChild {
}
/// The decoded screen's text (rows joined; trailing blanks trimmed).
- fn screen_text(&mut self) -> String {
+ pub(crate) fn screen_text(&self) -> String {
self.screen.screen().contents().trim_end().to_string()
}
+ /// Keep reading (and answering) the child's output until `deadline`: a
+ /// measured window must not stall the child on a full pty.
+ ///
+ /// # Errors
+ ///
+ /// Errors when the child exits inside the window.
+ #[cfg(target_os = "linux")]
+ pub(crate) fn drain_until(&mut self, deadline: Instant) -> anyhow::Result<()> {
+ loop {
+ self.pump(Some(deadline))?;
+ if let Some(status) = self.exited {
+ anyhow::bail!("child exited with {status} inside the measured window");
+ }
+ if Instant::now() >= deadline {
+ return Ok(());
+ }
+ }
+ }
+
+ /// The longest no-output stretch from `from` until now.
+ pub(crate) fn longest_gap_since(&self, from: Instant) -> Duration {
+ longest_gap(&self.arrivals, from, Instant::now())
+ }
+
/// The child's pid (teardown signals this pid only).
pub(crate) fn pid(&self) -> i32 {
self.child.id() as i32
@@ -306,6 +350,27 @@ fn find(haystack: &[u8], needle: &[u8]) -> Option<usize> {
haystack.windows(needle.len()).position(|w| w == needle)
}
+fn rfind(haystack: &[u8], needle: &[u8]) -> Option<usize> {
+ haystack.windows(needle.len()).rposition(|w| w == needle)
+}
+
+/// The longest stretch of `from..=to` in which no chunk arrived; the window
+/// edges bound the first and last stretch (`arrivals` is ascending).
+fn longest_gap(arrivals: &[Instant], from: Instant, to: Instant) -> Duration {
+ let first = arrivals.partition_point(|at| *at < from);
+ let mut previous = from;
+ let mut longest = Duration::ZERO;
+ for &at in arrivals[first..]
+ .iter()
+ .take_while(|at| **at <= to)
+ .chain(std::iter::once(&to))
+ {
+ longest = longest.max(at.saturating_duration_since(previous));
+ previous = at;
+ }
+ longest
+}
+
#[cfg(test)]
mod tests {
use super::*;
@@ -335,6 +400,49 @@ mod tests {
}
}
+ #[test]
+ fn screen_predicates_see_only_completed_frames_at_every_read_boundary() {
+ let dir = tempfile::tempdir().expect("temp dir");
+ let frame = [SYNC_BEGIN, b"painted".as_slice(), SYNC_END].concat();
+ for split in 1..frame.len() {
+ let mut child = PtyChild::spawn(
+ std::path::Path::new("/bin/cat"),
+ &[],
+ &[],
+ dir.path(),
+ 120,
+ 36,
+ )
+ .expect("spawn mock terminal peer");
+ child.receive(&frame[..split]).expect("first chunk");
+ assert_eq!(
+ child.in_frame,
+ split >= SYNC_BEGIN.len(),
+ "split at {split}"
+ );
+ child.receive(&frame[split..]).expect("second chunk");
+ assert!(!child.in_frame, "the frame end was missed at {split}");
+ }
+ }
+
+ #[test]
+ fn the_longest_gap_is_bounded_by_the_window_edges() {
+ let start = Instant::now();
+ let at = |ms| start + Duration::from_millis(ms);
+ let arrivals = [at(0), at(5), at(40), at(45), at(200)];
+ // Arrivals before the window never shorten its leading stretch.
+ assert_eq!(
+ longest_gap(&arrivals, at(10), at(42)),
+ Duration::from_millis(30)
+ );
+ // The trailing stretch runs to the window end, not the next arrival.
+ assert_eq!(
+ longest_gap(&arrivals, at(10), at(100)),
+ Duration::from_millis(55)
+ );
+ assert_eq!(longest_gap(&[], at(0), at(7)), Duration::from_millis(7));
+ }
+
#[test]
fn exited_children_fail_missing_needles_without_waiting_for_liveness_timeout() {
let directory = tempfile::tempdir().expect("mock directory");
@@ -367,6 +475,26 @@ mod tests {
.contains("exit status: 7"));
}
+ #[cfg(target_os = "linux")]
+ #[test]
+ fn a_child_exit_seen_by_the_last_drain_pump_fails_the_window() {
+ let directory = tempfile::tempdir().expect("mock directory");
+ let mut child = PtyChild::spawn(
+ std::path::Path::new("/bin/sh"),
+ &["-c".into(), "exit 7".into()],
+ &[],
+ directory.path(),
+ 120,
+ 36,
+ )
+ .expect("spawn exiting mock");
+ child.child.wait().expect("the mock exits");
+ let error = child
+ .drain_until(Instant::now())
+ .expect_err("a dead child fails the window");
+ assert!(error.to_string().contains("exit status: 7"), "{error}");
+ }
+
#[test]
fn final_output_is_drained_before_reporting_child_exit() {
let directory = tempfile::tempdir().expect("mock directory");
diff --git a/crates/pa-perf/src/bench/runner.rs b/crates/pa-perf/src/bench/runner.rs
index 419154aa69..311415962c 100644
--- a/crates/pa-perf/src/bench/runner.rs
+++ b/crates/pa-perf/src/bench/runner.rs
@@ -8,8 +8,8 @@ use std::time::{Duration, Instant};
use anyhow::Context;
-use crate::bench::{BenchCorpus, BenchOptions, BenchResult, RunOptions, Scenario};
-use crate::corpus::{self, CacheMode, CorpusOptions, Manifest};
+use crate::bench::{BenchCorpus, BenchOptions, BenchResult, RunOptions, Scenario, Series};
+use crate::corpus::{self, CacheMode, CorpusOptions, Manifest, Profile};
use super::pty::PtyChild;
@@ -24,6 +24,53 @@ const FIRST_PAINT_NEEDLE: &[u8] = "▗▄▄█▀".as_bytes();
/// "0 (0%)".
const READY_NEEDLE: &str = "0 (0%)";
+const PAGE_UP: &[u8] = b"\x1b[5~";
+/// Few enough pages to stay inside the attach tail (TUI-local scrolling).
+const PAGES: usize = 20;
+/// The product's default `tui.viewport.top` key (shift+alt+up).
+const VIEWPORT_TOP: &[u8] = b"\x1b[1;4A";
+
+const IDLE_WINDOW: Duration = Duration::from_secs(30);
+
+/// One run's measurements (transcript-paging settles once per page).
+#[derive(Debug, Default)]
+struct Measured {
+ settled: Vec<f64>,
+ first_paint: Option<f64>,
+ worst_gap: Option<f64>,
+ wakeups: Option<u64>,
+}
+
+impl Measured {
+ fn single(mark: Mark) -> Self {
+ Self {
+ settled: vec![mark.ms],
+ worst_gap: Some(mark.gap_ms),
+ ..Self::default()
+ }
+ }
+}
+
+/// A trigger -> mark interval and its longest no-output gap.
+#[derive(Debug, Clone, Copy)]
+struct Mark {
+ ms: f64,
+ gap_ms: f64,
+}
+
+impl Mark {
+ fn since(child: &PtyChild, started: Instant) -> Self {
+ Self {
+ ms: millis(started.elapsed()),
+ gap_ms: millis(child.longest_gap_since(started)),
+ }
+ }
+}
+
+fn millis(duration: Duration) -> f64 {
+ duration.as_secs_f64() * 1000.0
+}
+
/// One bench run's anchored paths.
struct RunCtx {
binary: PathBuf,
@@ -98,10 +145,10 @@ pub(super) fn run(options: &BenchOptions) -> anyhow::Result<BenchResult> {
&options.corpus,
Duration::from_secs(options.liveness_secs),
)?;
- let samples = sample_runs(&ctx, options.runs, |index| {
- scenario_once(&ctx, options.scenario, index)
+ let runs = sample_runs(&ctx, options.runs, |index| {
+ Ok(launch(&ctx, options.scenario, index)?.0)
})?;
- let result = build_result(&ctx, options.scenario, samples);
+ let result = build_result(&ctx, options.scenario, runs);
write_result(&result, &options.json)?;
Ok(result)
}
@@ -112,9 +159,9 @@ pub(super) fn run(options: &BenchOptions) -> anyhow::Result<BenchResult> {
fn sample_runs(
ctx: &RunCtx,
runs: usize,
- mut scenario: impl FnMut(usize) -> anyhow::Result<f64>,
-) -> anyhow::Result<Vec<f64>> {
- let mut samples = Vec::with_capacity(runs + 1);
+ mut scenario: impl FnMut(usize) -> anyhow::Result<Measured>,
+) -> anyhow::Result<Vec<Measured>> {
+ let mut measured = Vec::with_capacity(runs);
let total = runs + 1;
for index in 0..total {
restore(ctx)?;
@@ -124,44 +171,41 @@ fn sample_runs(
outcome?;
continue;
}
- let sample = outcome.context("a sampled run failed")?;
- samples.push(sample);
+ let run = outcome.context("a sampled run failed")?;
println!(
- "run {index}/{runs}: {sample:.1}ms (wall {:.1}s)",
+ "run {index}/{runs}: {:.1?}ms (wall {:.1}s)",
+ run.settled,
started.elapsed().as_secs_f32()
);
+ measured.push(run);
}
- Ok(samples)
-}
-
-/// One scenario pass; returns the sample in ms.
-fn scenario_once(ctx: &RunCtx, scenario: Scenario, run: usize) -> anyhow::Result<f64> {
- let (sample, _) = launch(ctx, scenario, run)?;
- Ok(sample)
+ Ok(measured)
}
/// The ONE launch path: the pre-started daemon for warm-* scenarios, the
/// scenario's own resume target + first needle, the post-launch mechanics,
/// and the daemon teardown on every path. Both `run()` (single scenario)
/// and `run_suite()` call this.
-fn launch(ctx: &RunCtx, scenario: Scenario, run: usize) -> anyhow::Result<(f64, Launch)> {
+fn launch(ctx: &RunCtx, scenario: Scenario, run: usize) -> anyhow::Result<(Measured, Launch)> {
let socket = run_socket(ctx, run);
std::fs::create_dir_all(socket.parent().expect("a socket dir"))?;
let mut daemon = if matches!(
scenario,
- Scenario::WarmStart | Scenario::WarmReady | Scenario::AgentsViewWarm
+ Scenario::WarmStart | Scenario::WarmReady | Scenario::AgentsViewWarm | Scenario::IdleCost
) {
// AgentsViewWarm pre-starts so both TUIs share the daemon the first
// populate warms; the single-scenario path cannot express that (its
// second launch would spawn a second daemon), so it runs the first
- // populate only and the sample is the suite's warm number.
+ // populate only and the sample is the suite's warm number. IdleCost
+ // pre-starts so the daemon is an owned pid whose /proc it can read.
Some(start_daemon(ctx, &socket)?)
} else {
None
};
- let outcome: anyhow::Result<(f64, Launch)> = (|| {
+ let daemon_pid = daemon.as_ref().map(|daemon| daemon.child.id() as i32);
+ let outcome: anyhow::Result<(Measured, Launch)> = (|| {
let mut launched = spawn_for_scenario(ctx, &socket, scenario, run)?;
- let sample = launch_scenario(ctx, &mut launched, scenario)?;
+ let sample = launch_scenario(ctx, &mut launched, scenario, daemon_pid)?;
// The measured client must disconnect before daemon shutdown.
// Keeping it alive while waiting for daemon exit can hold server
// connection draining open or start the client's reconnect path.
@@ -229,7 +273,9 @@ pub(super) fn run_suite(options: &RunOptions) -> anyhow::Result<Vec<BenchResult>
let cold_ctx = RunCtx::load(&options.binary, &options.out.join("corpus-cold"), liveness)?;
let warm_ctx = RunCtx::load(&options.binary, &options.out.join("corpus-warm"), liveness)?;
- let mut samples: BTreeMap<Scenario, Vec<f64>> = BTreeMap::new();
+ let mut samples: BTreeMap<Scenario, Vec<Measured>> = BTreeMap::new();
+ // Each scenario's share of the suite's wall time (the CI budget).
+ let mut wall: BTreeMap<Scenario, Duration> = BTreeMap::new();
let total = options.runs + 1;
for index in 0..total {
let started = Instant::now();
@@ -238,51 +284,84 @@ pub(super) fn run_suite(options: &RunOptions) -> anyhow::Result<Vec<BenchResult>
// per launch; the cold-start/cold-ready (and warm-*) pairs share
// one launch each.
restore(&cold_ctx)?;
- let (start_sample, launched) = launch(&cold_ctx, Scenario::ColdStart, index)?;
- let ready_sample = launched
- .ready_ms
- .context("the cold launch never attached")?;
- push(&mut samples, index, Scenario::ColdStart, start_sample);
- push(&mut samples, index, Scenario::ColdReady, ready_sample);
+ let (start, launched) = launch(&cold_ctx, Scenario::ColdStart, index)?;
+ let ready = launched.ready.context("the cold launch never attached")?;
+ push(&mut samples, index, Scenario::ColdStart, start);
+ push(
+ &mut samples,
+ index,
+ Scenario::ColdReady,
+ Measured::single(ready),
+ );
for scenario in [
Scenario::AgentsView,
Scenario::AgentsViewWarm,
Scenario::OpenSession,
Scenario::SessionSwitch,
Scenario::AtCompletion,
+ Scenario::TranscriptPaging,
+ Scenario::TranscriptTop,
] {
+ let launched = Instant::now();
restore(&cold_ctx)?;
- let (sample, _) = launch(&cold_ctx, scenario, index)?;
- push(&mut samples, index, scenario, sample);
+ let (measured, _) = launch(&cold_ctx, scenario, index)?;
+ push(&mut samples, index, scenario, measured);
+ *wall.entry(scenario).or_default() += launched.elapsed();
}
restore(&warm_ctx)?;
- let (start_sample, launched) = launch(&warm_ctx, Scenario::WarmStart, index)?;
- let ready_sample = launched
- .ready_ms
- .context("the warm launch never attached")?;
- push(&mut samples, index, Scenario::WarmStart, start_sample);
- push(&mut samples, index, Scenario::WarmReady, ready_sample);
+ let (start, launched) = launch(&warm_ctx, Scenario::WarmStart, index)?;
+ let ready = launched.ready.context("the warm launch never attached")?;
+ push(&mut samples, index, Scenario::WarmStart, start);
+ push(
+ &mut samples,
+ index,
+ Scenario::WarmReady,
+ Measured::single(ready),
+ );
println!(
"run {index}/{} complete ({:.1}s)",
options.runs,
started.elapsed().as_secs_f32()
);
}
+ // One idle window per suite on one profile: an idle ready session's
+ // cost is per process, not per corpus size, so one leg pays the window.
+ if options.profile == Profile::Medium {
+ if cfg!(target_os = "linux") {
+ let launched = Instant::now();
+ restore(&cold_ctx)?;
+ let (measured, _) = launch(&cold_ctx, Scenario::IdleCost, total)?;
+ samples.insert(Scenario::IdleCost, vec![measured]);
+ wall.insert(Scenario::IdleCost, launched.elapsed());
+ } else {
+ println!("idle-cost skipped: it reads /proc (Linux-only)");
+ }
+ }
+ for (scenario, spent) in &wall {
+ println!(
+ "wall {}: {:.1}s",
+ crate::report::key_of(*scenario),
+ spent.as_secs_f32()
+ );
+ }
let results_dir = options.out.join("results");
std::fs::create_dir_all(&results_dir)?;
let mut results = Vec::new();
- for (scenario, mut samples) in samples {
- samples.sort_by(f64::total_cmp);
- if samples.len() != options.runs {
+ for (scenario, runs) in samples {
+ let expected = if scenario == Scenario::IdleCost {
+ 1
+ } else {
+ options.runs
+ };
+ if runs.len() != expected {
anyhow::bail!(
- "{scenario:?} collected {} samples, expected {}",
- samples.len(),
- options.runs
+ "{scenario:?} collected {} runs, expected {expected}",
+ runs.len(),
);
}
let ctx = corpus_for(scenario, &cold_ctx, &warm_ctx);
- let result = build_result(ctx, scenario, samples);
+ let result = build_result(ctx, scenario, runs);
let name = crate::report::key_of(scenario);
// Per-profile unique names: the CI legs merge their artifacts into
// one results dir, and same-named files would overwrite each other.
@@ -309,14 +388,22 @@ fn corpus_for<'a>(scenario: Scenario, cold: &'a RunCtx, warm: &'a RunCtx) -> &'a
| Scenario::AgentsViewWarm
| Scenario::OpenSession
| Scenario::SessionSwitch
- | Scenario::AtCompletion => cold,
+ | Scenario::AtCompletion
+ | Scenario::TranscriptPaging
+ | Scenario::TranscriptTop
+ | Scenario::IdleCost => cold,
}
}
-/// Push a sample unless this is the discarded warm-up run.
-fn push(samples: &mut BTreeMap<Scenario, Vec<f64>>, index: usize, scenario: Scenario, sample: f64) {
+/// Push a run unless this is the discarded warm-up run.
+fn push(
+ samples: &mut BTreeMap<Scenario, Vec<Measured>>,
+ index: usize,
+ scenario: Scenario,
+ measured: Measured,
+) {
if index > 0 {
- samples.entry(scenario).or_default().push(sample);
+ samples.entry(scenario).or_default().push(measured);
}
}
@@ -328,7 +415,7 @@ fn spawn_for_scenario(
run: usize,
) -> anyhow::Result<Launch> {
let (resume, first_needle) = match scenario {
- Scenario::OpenSession => (
+ Scenario::OpenSession | Scenario::TranscriptPaging | Scenario::TranscriptTop => (
Some(ctx.manifest.markers.huge.file.clone()),
ctx.manifest.markers.huge.needle.as_bytes().to_vec(),
),
@@ -345,7 +432,11 @@ fn spawn_for_scenario(
let mut launch = Launch::spawn(ctx, socket, resume.as_deref(), &first_needle, dump)?;
if matches!(
scenario,
- Scenario::ColdStart | Scenario::WarmStart | Scenario::ColdReady | Scenario::WarmReady
+ Scenario::ColdStart
+ | Scenario::WarmStart
+ | Scenario::ColdReady
+ | Scenario::WarmReady
+ | Scenario::IdleCost
) {
launch.wait_ready()?;
}
@@ -354,16 +445,28 @@ fn spawn_for_scenario(
/// The scenario mechanics after the launch needle; a completed launch
/// never dumps its stream.
-fn launch_scenario(ctx: &RunCtx, launch: &mut Launch, scenario: Scenario) -> anyhow::Result<f64> {
- let sample = post_launch(ctx, launch, scenario)?;
+fn launch_scenario(
+ ctx: &RunCtx,
+ launch: &mut Launch,
+ scenario: Scenario,
+ daemon_pid: Option<i32>,
+) -> anyhow::Result<Measured> {
+ let measured = post_launch(ctx, launch, scenario, daemon_pid)?;
launch.completed = true;
- Ok(sample)
+ Ok(measured)
}
/// The scenario mechanics after the launch needle.
-fn post_launch(ctx: &RunCtx, launch: &mut Launch, scenario: Scenario) -> anyhow::Result<f64> {
+fn post_launch(
+ ctx: &RunCtx,
+ launch: &mut Launch,
+ scenario: Scenario,
+ daemon_pid: Option<i32>,
+) -> anyhow::Result<Measured> {
match scenario {
- Scenario::ColdStart | Scenario::WarmStart | Scenario::OpenSession => Ok(launch.first_ms),
+ Scenario::ColdStart | Scenario::WarmStart | Scenario::OpenSession => {
+ Ok(Measured::single(launch.first))
+ }
Scenario::AgentsView => agents_view_populate(ctx, &mut launch.child),
Scenario::AgentsViewWarm => {
// The first populate warms the daemon's in-process scan cache;
@@ -375,13 +478,21 @@ fn post_launch(ctx: &RunCtx, launch: &mut Launch, scenario: Scenario) -> anyhow:
.join("failed-streams")
.join("agents-view-warm-second.bin");
let mut second = Launch::spawn(ctx, &socket, None, FIRST_PAINT_NEEDLE, dump)?;
- let sample = agents_view_populate(ctx, &mut second.child)?;
+ let measured = agents_view_populate(ctx, &mut second.child)?;
second.completed = true;
- Ok(sample)
+ Ok(measured)
}
Scenario::SessionSwitch => switch_to_target_b(ctx, &mut launch.child),
Scenario::AtCompletion => at_completion(ctx, &mut launch.child),
- Scenario::ColdReady | Scenario::WarmReady => Ok(launch.ready_ms.unwrap_or(launch.first_ms)),
+ Scenario::TranscriptPaging => transcript_paging(ctx, &mut launch.child),
+ Scenario::TranscriptTop => transcript_top(ctx, &mut launch.child),
+ Scenario::IdleCost => idle::measure(
+ &mut launch.child,
+ daemon_pid.expect("launch pre-starts the idle-cost daemon"),
+ ),
+ Scenario::ColdReady | Scenario::WarmReady => {
+ Ok(Measured::single(launch.ready.unwrap_or(launch.first)))
+ }
}
}
@@ -390,8 +501,8 @@ fn post_launch(ctx: &RunCtx, launch: &mut Launch, scenario: Scenario) -> anyhow:
struct Launch {
child: PtyChild,
socket: PathBuf,
- first_ms: f64,
- ready_ms: Option<f64>,
+ first: Mark,
+ ready: Option<Mark>,
spawned: Instant,
liveness: Duration,
/// Where the raw pty stream lands when the launch does NOT complete (the
@@ -434,10 +545,13 @@ impl Launch {
"the first paint"
};
let mut launch = Self {
- first_ms: 0.0,
+ first: Mark {
+ ms: 0.0,
+ gap_ms: 0.0,
+ },
child,
socket: socket.to_path_buf(),
- ready_ms: None,
+ ready: None,
spawned: started,
liveness: ctx.liveness,
stream_dump: Some(stream_dump),
@@ -449,7 +563,7 @@ impl Launch {
what,
started + ctx.liveness,
)?;
- launch.first_ms = started.elapsed().as_secs_f64() * 1000.0;
+ launch.first = Mark::since(&launch.child, started);
Ok(launch)
}
@@ -461,7 +575,7 @@ impl Launch {
"the attach pill",
self.spawned + self.liveness,
)?;
- self.ready_ms = Some(self.spawned.elapsed().as_secs_f64() * 1000.0);
+ self.ready = Some(Mark::since(&self.child, self.spawned));
Ok(())
}
}
@@ -485,22 +599,28 @@ impl Drop for Launch {
/// cell diffs, so the byte stream cannot see it — the screen model does).
/// The count includes the TUI's own fresh session (rendered as one more
/// inactive row).
-fn agents_view_populate(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<f64> {
+fn agents_view_populate(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<Measured> {
let started = Instant::now();
child.write(b"\x1b[D")?;
+ child.wait_screen("the first roster row", started + ctx.liveness, |screen| {
+ roster_counts(screen).is_some_and(|(loaded, _)| loaded >= 1)
+ })?;
+ let first_paint = millis(started.elapsed());
wait_roster(ctx, child, ctx.manifest.markers.roster_inactive + 1)?;
- Ok(started.elapsed().as_secs_f64() * 1000.0)
+ Ok(Measured {
+ first_paint: Some(first_paint),
+ ..Measured::single(Mark::since(child, started))
+ })
}
-/// The agents view opens on its first catalog page: the `Inactive (N)`
-/// heading counts the loaded rows and the `M older sessions not loaded`
-/// footer the rest, so the page is settled once both account for every
-/// expected session. An unpaged roster has no footer.
-fn roster_settled(screen: &str, expected: usize) -> bool {
+/// The agents view's roster accounting: the `Inactive (N)` heading counts
+/// the loaded rows and the `M older sessions not loaded` footer the rest
+/// (an unpaged roster has no footer). `None` while no roster is on screen.
+fn roster_counts(screen: &str) -> Option<(usize, usize)> {
let loaded = screen
.split_once("Inactive (")
.and_then(|(_, rest)| rest.split_once(')'))
- .and_then(|(number, _)| number.parse::<usize>().ok());
+ .and_then(|(number, _)| number.parse::<usize>().ok())?;
let unloaded = screen
.lines()
.find_map(|line| {
@@ -508,7 +628,13 @@ fn roster_settled(screen: &str, expected: usize) -> bool {
number.parse::<usize>().ok()
})
.unwrap_or(0);
- loaded.is_some_and(|loaded| loaded + unloaded == expected)
+ Some((loaded, unloaded))
+}
+
+/// The first catalog page is settled once the loaded and not-yet-loaded
+/// counts account for every expected session.
+fn roster_settled(screen: &str, expected: usize) -> bool {
+ roster_counts(screen).is_some_and(|(loaded, unloaded)| loaded + unloaded == expected)
}
fn wait_roster(ctx: &RunCtx, child: &mut PtyChild, expected: usize) -> anyhow::Result<Instant> {
@@ -522,8 +648,10 @@ fn wait_roster(ctx: &RunCtx, child: &mut PtyChild, expected: usize) -> anyhow::R
/// From the opened session A: open the agents view, uniquely filter to
/// switch-b, and time enter -> B's transcript marker. The `zq` token in
/// switch-b's name matches no other row (no generated text carries the
-/// letter z), so Enter always drills THE row.
-fn switch_to_target_b(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<f64> {
+/// letter z), so Enter always drills THE row. (No first-paint mark: the
+/// agents view stays up until B's first chat frame, which already carries
+/// the marker.)
+fn switch_to_target_b(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<Measured> {
child.write(b"\x1b[D")?;
// Wait the settled roster before typing: keys racing the scan flood
// were the CI-runner failure (the query echoed nothing). The resumed
@@ -548,14 +676,14 @@ fn switch_to_target_b(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<f64>
"the switch-b transcript",
Instant::now() + ctx.liveness,
)?;
- Ok(started.elapsed().as_secs_f64() * 1000.0)
+ Ok(Measured::single(Mark::since(child, started)))
}
/// The `@` completion: the trigger is the `@` keystroke alone; the needle
/// is the first painted completion menu row (the menu lists the project
/// tree's entries — `src/` is a top-level dir of the generated project
/// that the pre-menu screen never contains).
-fn at_completion(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<f64> {
+fn at_completion(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<Measured> {
let started = Instant::now();
child.write(b"@")?;
// The menu lists the project tree's entries — `src/` is a top-level
@@ -566,7 +694,72 @@ fn at_completion(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<f64> {
"the completion menu row",
Instant::now() + ctx.liveness,
)?;
- Ok(started.elapsed().as_secs_f64() * 1000.0)
+ Ok(Measured::single(Mark::since(child, started)))
+}
+
+/// From the resumed tail: PAGES PageUps, each timed to the first frame whose
+/// topmost block ordinal is above the previous frame's.
+fn transcript_paging(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<Measured> {
+ wait_ordinal_frame(ctx, child)?;
+ let first = topmost_ordinal(&child.screen_text());
+ let mut pages = Vec::with_capacity(PAGES);
+ for page in 1..=PAGES {
+ let before = topmost_ordinal(&child.screen_text())
+ .with_context(|| format!("no block ordinal on screen before page {page}"))?;
+ let started = Instant::now();
+ child.write(PAGE_UP)?;
+ child.wait_screen(
+ &format!("page {page} above block ordinal {before}"),
+ started + ctx.liveness,
+ |screen| topmost_ordinal(screen).is_some_and(|top| top < before),
+ )?;
+ pages.push(Mark::since(child, started));
+ }
+ println!(
+ "transcript-paging: topmost block ordinal {first:?} -> {:?}",
+ topmost_ordinal(&child.screen_text())
+ );
+ Ok(Measured {
+ settled: pages.iter().map(|page| page.ms).collect(),
+ worst_gap: pages.iter().map(|page| page.gap_ms).reduce(f64::max),
+ ..Measured::default()
+ })
+}
+
+/// The top key -> the first frame showing block ordinal 0. The head is never
+/// in the attach tail, so the TUI fetches it from the daemon.
+fn transcript_top(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<Measured> {
+ wait_ordinal_frame(ctx, child)?;
+ let started = Instant::now();
+ child.write(VIEWPORT_TOP)?;
+ child.wait_screen(
+ "the session's first block",
+ started + ctx.liveness,
+ |screen| topmost_ordinal(screen) == Some(0),
+ )?;
+ Ok(Measured::single(Mark::since(child, started)))
+}
+
+fn wait_ordinal_frame(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<Instant> {
+ child.wait_screen(
+ "an ordinal-tagged transcript frame",
+ Instant::now() + ctx.liveness,
+ |screen| topmost_ordinal(screen).is_some(),
+ )
+}
+
+/// The smallest block ordinal on screen is the topmost block.
+fn topmost_ordinal(screen: &str) -> Option<u64> {
+ screen
+ .match_indices(corpus::ORDINAL_PREFIX)
+ .filter_map(|(at, prefix)| {
+ let digits = &screen[at + prefix.len()..];
+ let end = digits
+ .find(|c: char| !c.is_ascii_digit())
+ .unwrap_or(digits.len());
+ digits[..end].parse().ok()
+ })
+ .min()
}
/// The per-run isolated socket path.
@@ -735,14 +928,16 @@ fn restore(ctx: &RunCtx) -> anyhow::Result<()> {
Ok(())
}
-/// The result object for one scenario's samples.
-fn build_result(ctx: &RunCtx, scenario: Scenario, samples: Vec<f64>) -> BenchResult {
- let (median_ms, p95_ms) = stats(&samples);
+/// The result object for one scenario's runs (settled marks pooled).
+fn build_result(ctx: &RunCtx, scenario: Scenario, runs: Vec<Measured>) -> BenchResult {
+ let mut samples: Vec<f64> = runs.iter().flat_map(|run| run.settled.clone()).collect();
+ samples.sort_by(f64::total_cmp);
+ let (median_ms, p95_ms, _) = stats(&samples);
BenchResult {
schema: 1,
scenario,
profile: crate::report::profile_key(ctx.manifest.profile),
- runs: samples.len(),
+ runs: runs.len(),
median_ms,
p95_ms,
samples_ms: samples,
@@ -755,11 +950,27 @@ fn build_result(ctx: &RunCtx, scenario: Scenario, samples: Vec<f64>) -> BenchRes
child_sessions: ctx.manifest.counts.child_sessions,
archive: ctx.manifest.counts.archive,
},
+ first_paint: series(runs.iter().map(|run| run.first_paint).collect()),
+ worst_gap: series(runs.iter().map(|run| run.worst_gap).collect()),
+ wakeups: runs.iter().map(|run| run.wakeups).collect(),
}
}
-/// Median + p95 over the samples.
-fn stats(samples: &[f64]) -> (f64, f64) {
+/// The series when every run measured it.
+fn series(samples: Option<Vec<f64>>) -> Option<Series> {
+ let mut samples = samples?;
+ samples.sort_by(f64::total_cmp);
+ let (median_ms, p95_ms, max_ms) = stats(&samples);
+ Some(Series {
+ samples_ms: samples,
+ median_ms,
+ p95_ms,
+ max_ms,
+ })
+}
+
+/// Median, p95, and max over the samples.
+fn stats(samples: &[f64]) -> (f64, f64, f64) {
let mut sorted = samples.to_vec();
sorted.sort_by(f64::total_cmp);
let median = if sorted.len() % 2 == 1 {
@@ -768,7 +979,11 @@ fn stats(samples: &[f64]) -> (f64, f64) {
(sorted[sorted.len() / 2 - 1] + sorted[sorted.len() / 2]) / 2.0
};
let p95_index = ((sorted.len() as f64 - 1.0) * 0.95).ceil() as usize;
- (median, sorted[p95_index.min(sorted.len() - 1)])
+ (
+ median,
+ sorted[p95_index.min(sorted.len() - 1)],
+ sorted[sorted.len() - 1],
+ )
}
fn needle_of(scenario: Scenario, ctx: &RunCtx) -> String {
@@ -794,6 +1009,17 @@ fn needle_of(scenario: Scenario, ctx: &RunCtx) -> String {
ctx.manifest.markers.switch_b.needle
),
Scenario::AtCompletion => "the first @ completion menu row".to_string(),
+ Scenario::TranscriptPaging => format!(
+ "per PageUp ({PAGES} per run) on the huge session: the first frame whose topmost block ordinal is above the previous frame's"
+ ),
+ Scenario::TranscriptTop => {
+ "the top key on the huge session -> the first frame showing its first block (ordinal 0)"
+ .to_string()
+ }
+ Scenario::IdleCost => format!(
+ "CPU ms of the owned daemon, its descendants, and the TUI over a {}s idle window after ready",
+ IDLE_WINDOW.as_secs()
+ ),
}
}
@@ -802,6 +1028,171 @@ fn write_result(result: &BenchResult, path: &Path) -> anyhow::Result<()> {
std::fs::write(path, json).with_context(|| format!("write {}", path.display()))
}
+/// idle-cost: processes are found from the owned pids down (/proc
+/// children), never by name.
+#[cfg(target_os = "linux")]
+mod idle {
+ use std::collections::BTreeMap;
+ use std::time::Instant;
+
+ use anyhow::Context;
+
+ use super::{Measured, PtyChild, IDLE_WINDOW};
+
+ /// utime+stime (clock ticks) and context switches summed over threads.
+ #[derive(Debug, Clone, Copy, Default)]
+ struct Usage {
+ ticks: u64,
+ switches: u64,
+ }
+
+ pub(super) fn measure(child: &mut PtyChild, daemon: i32) -> anyhow::Result<Measured> {
+ let roots = [daemon, child.pid()];
+ let before = snapshot(&roots);
+ child.drain_until(Instant::now() + IDLE_WINDOW)?;
+ let after = snapshot(&roots);
+ anyhow::ensure!(
+ after.contains_key(&daemon),
+ "the daemon exited inside the idle window"
+ );
+ // A process that exited inside the window is gone from `after`, so
+ // its share is not counted.
+ let delta = after.iter().fold(Usage::default(), |sum, (pid, end)| {
+ let start = before.get(pid).copied().unwrap_or_default();
+ Usage {
+ ticks: sum.ticks + end.ticks.saturating_sub(start.ticks),
+ switches: sum.switches + end.switches.saturating_sub(start.switches),
+ }
+ });
+ let ticks_per_second = nix::unistd::sysconf(nix::unistd::SysconfVar::CLK_TCK)?
+ .context("CLK_TCK is defined on Linux")?;
+ let cpu_ms = delta.ticks as f64 * 1000.0 / ticks_per_second as f64;
+ println!(
+ "idle-cost: {cpu_ms:.0}ms CPU, {} context switches, {} processes",
+ delta.switches,
+ after.len()
+ );
+ Ok(Measured {
+ settled: vec![cpu_ms],
+ wakeups: Some(delta.switches),
+ ..Measured::default()
+ })
+ }
+
+ /// Every live process under `roots` (inclusive) with its counters.
+ fn snapshot(roots: &[i32]) -> BTreeMap<i32, Usage> {
+ let mut usage = BTreeMap::new();
+ let mut stack = roots.to_vec();
+ while let Some(pid) = stack.pop() {
+ let Some(counters) = read_usage(pid) else {
+ continue;
+ };
+ usage.insert(pid, counters);
+ stack.extend(children(pid));
+ }
+ usage
+ }
+
+ fn read_usage(pid: i32) -> Option<Usage> {
+ let stat = std::fs::read_to_string(format!("/proc/{pid}/stat")).ok()?;
+ Some(Usage {
+ ticks: live_ticks(&stat)?,
+ switches: tasks(pid)
+ .filter_map(|tid| {
+ std::fs::read_to_string(format!("/proc/{pid}/task/{tid}/status")).ok()
+ })
+ .map(|status| status_switches(&status))
+ .sum(),
+ })
+ }
+
+ /// utime + stime (stat fields 14 and 15, counted after the comm, which
+ /// may contain spaces and parentheses); `None` for an unreaped zombie,
+ /// whose stat file outlives it.
+ fn live_ticks(stat: &str) -> Option<u64> {
+ let (_, fields) = stat.rsplit_once(')')?;
+ let mut fields = fields.split_whitespace();
+ if fields.next()? == "Z" {
+ return None;
+ }
+ let mut fields = fields.skip(10);
+ let utime: u64 = fields.next()?.parse().ok()?;
+ let stime: u64 = fields.next()?.parse().ok()?;
+ Some(utime + stime)
+ }
+
+ fn status_switches(status: &str) -> u64 {
+ status
+ .lines()
+ .filter_map(|line| {
+ let (key, value) = line.split_once(':')?;
+ matches!(
+ key,
+ "voluntary_ctxt_switches" | "nonvoluntary_ctxt_switches"
+ )
+ .then(|| value.trim().parse::<u64>().ok())?
+ })
+ .sum()
+ }
+
+ fn tasks(pid: i32) -> impl Iterator<Item = String> {
+ std::fs::read_dir(format!("/proc/{pid}/task"))
+ .into_iter()
+ .flatten()
+ .filter_map(|entry| Some(entry.ok()?.file_name().to_string_lossy().into_owned()))
+ }
+
+ /// The direct children of every thread of `pid`.
+ fn children(pid: i32) -> Vec<i32> {
+ tasks(pid)
+ .filter_map(|tid| {
+ std::fs::read_to_string(format!("/proc/{pid}/task/{tid}/children")).ok()
+ })
+ .flat_map(|list| {
+ list.split_whitespace()
+ .filter_map(|child| child.parse().ok())
+ .collect::<Vec<i32>>()
+ })
+ .collect()
+ }
+
+ #[cfg(test)]
+ mod tests {
+ #[test]
+ fn counters_parse_from_proc_text() {
+ let stat =
+ "4242 (pa worker (x)) S 1 4242 4242 0 -1 4194560 900 0 0 0 31 7 0 0 20 0 9 0";
+ assert_eq!(super::live_ticks(stat), Some(38));
+ assert_eq!(super::live_ticks(&stat.replacen(") S", ") Z", 1)), None);
+ let status = "Name:\tprime-agent\nvoluntary_ctxt_switches:\t120\nnonvoluntary_ctxt_switches:\t5\n";
+ assert_eq!(super::status_switches(status), 125);
+ }
+
+ #[test]
+ fn the_snapshot_reads_real_proc_counters_down_to_descendants() {
+ let mut child = std::process::Command::new("/bin/cat")
+ .stdin(std::process::Stdio::piped())
+ .spawn()
+ .expect("spawn the mock descendant");
+ let own = std::process::id() as i32;
+ let seen = super::snapshot(&[own]);
+ let _ = child.kill();
+ let _ = child.wait();
+ assert!(seen.contains_key(&(child.id() as i32)), "{seen:?}");
+ assert!(seen[&own].switches > 0);
+ }
+ }
+}
+
+#[cfg(not(target_os = "linux"))]
+mod idle {
+ use super::{Measured, PtyChild};
+
+ pub(super) fn measure(_child: &mut PtyChild, _daemon: i32) -> anyhow::Result<Measured> {
+ anyhow::bail!("idle-cost reads /proc and runs on Linux only")
+ }
+}
+
#[cfg(test)]
mod tests {
use super::*;
@@ -863,7 +1254,11 @@ mod tests {
Scenario::ColdStart,
] {
let measured = corpus_for(scenario, &cold, &warm);
- let result = build_result(measured, scenario, vec![1.0, 2.0]);
+ let run = Measured {
+ settled: vec![1.0],
+ ..Measured::default()
+ };
+ let result = build_result(measured, scenario, vec![run]);
let expected = if scenario == Scenario::ColdStart {
&cold
} else {
@@ -982,10 +1377,17 @@ mod tests {
}
std::fs::write(&sidecar, "written by the mock scenario")?;
launches.push(index);
- Ok(index as f64)
+ Ok(Measured {
+ settled: vec![index as f64],
+ ..Measured::default()
+ })
})
.expect("sample the mock scenario");
- assert_eq!((launches, samples), (vec![0, 1, 2], vec![1.0, 2.0]));
+ let samples: Vec<Vec<f64>> = samples.into_iter().map(|run| run.settled).collect();
+ assert_eq!(
+ (launches, samples),
+ (vec![0, 1, 2], vec![vec![1.0], vec![2.0]])
+ );
}
#[test]
@@ -1022,11 +1424,8 @@ mod tests {
ctx.liveness = Duration::from_secs(10);
let (sample, launched) = launch(&ctx, Scenario::WarmStart, 0)
.expect("the daemon must drain its measured UI connection before exit");
- assert!(sample >= 0.0);
- assert!(
- launched.ready_ms.is_some(),
- "the actual mock paint/attach ran"
- );
+ assert!(sample.settled[0] >= 0.0);
+ assert!(launched.ready.is_some(), "the actual mock paint/attach ran");
assert_eq!(
std::fs::read_to_string(run_socket(&ctx, 0).with_extension("drained"))
.expect("the mock daemon reached normal drain completion"),
diff --git a/crates/pa-perf/src/corpus/artifacts.rs b/crates/pa-perf/src/corpus/artifacts.rs
index 75a4674a71..c1c4417fa6 100644
--- a/crates/pa-perf/src/corpus/artifacts.rs
+++ b/crates/pa-perf/src/corpus/artifacts.rs
@@ -194,6 +194,7 @@ pub(crate) fn spawn_child(
marker: None,
parent_session: Some(parent_file.display().to_string()),
rlm_depth: None,
+ ordinals: false,
},
rng,
)?;
diff --git a/crates/pa-perf/src/corpus/mod.rs b/crates/pa-perf/src/corpus/mod.rs
index 7c5d9d6ac5..ef5780be2e 100644
--- a/crates/pa-perf/src/corpus/mod.rs
+++ b/crates/pa-perf/src/corpus/mod.rs
@@ -5,6 +5,8 @@
pub(crate) mod artifacts;
pub(crate) mod rows;
+pub use rows::ORDINAL_PREFIX;
+
use std::path::{Path, PathBuf};
use anyhow::Context;
@@ -293,6 +295,8 @@ pub fn generate(options: &CorpusOptions) -> anyhow::Result<Manifest> {
marker: marker.clone(),
parent_session: None,
rlm_depth: Some(0),
+ // The huge session is the transcript-paging target.
+ ordinals: index == 0,
},
&mut rng,
)?;
@@ -378,6 +382,7 @@ pub fn generate(options: &CorpusOptions) -> anyhow::Result<Manifest> {
marker: None,
parent_session: None,
rlm_depth: Some(0),
+ ordinals: false,
},
&mut rng,
)?;
diff --git a/crates/pa-perf/src/corpus/rows.rs b/crates/pa-perf/src/corpus/rows.rs
index 7f157a44bb..9ea9580c71 100644
--- a/crates/pa-perf/src/corpus/rows.rs
+++ b/crates/pa-perf/src/corpus/rows.rs
@@ -118,8 +118,16 @@ pub(crate) struct SessionSpec {
pub marker: Option<String>,
pub parent_session: Option<String>,
pub rlm_depth: Option<u64>,
+ /// Prefix every generated text block with its ordinal (`ORDINAL_PREFIX` +
+ /// n, counting up in transcript order) so a scrolled screen names how far
+ /// up it is; the marker turn stays untagged.
+ pub ordinals: bool,
}
+/// The block-ordinal token: `ord` + the ordinal's digits. No generated
+/// word contains `ord`, so the token is unambiguous on any screen.
+pub const ORDINAL_PREFIX: &str = "ord";
+
/// What one generated session produced.
pub(crate) struct Written {
pub rows: usize,
@@ -321,6 +329,11 @@ pub(crate) fn write_session_file(
// The conversation: fill to `rows`, keeping a compaction boundary near
// 60% for long sessions and the marker on the final user message.
let target = spec.rows as u64;
+ let mut ordinals = spec.ordinals.then_some(0u64..);
+ let mut tag = |text: String| match ordinals.as_mut().and_then(Iterator::next) {
+ Some(ordinal) => format!("{ORDINAL_PREFIX}{ordinal} {text}"),
+ None => text,
+ };
let mut compaction_at = (target >= 1500).then(|| target * 6 / 10);
let mut last_user_text = String::new();
while rows.rows_written() < target {
@@ -348,7 +361,7 @@ pub(crate) fn write_session_file(
let marker_turn = spec.marker.is_some() && target - rows.rows_written() <= 4;
let text = match &spec.marker {
Some(marker) if marker_turn => marker.clone(),
- _ => user_text(rng),
+ _ => tag(user_text(rng)),
};
let content = if rng.chance(30) {
UserContent::Blocks(vec![UserContentBlock::Text(TextContent {
@@ -387,7 +400,7 @@ pub(crate) fn write_session_file(
let mut blocks = Vec::new();
if !marker_turn && rng.chance(60) {
blocks.push(AssistantContentBlock::Thinking(ThinkingContent {
- thinking: assistant_text(rng),
+ thinking: tag(assistant_text(rng)),
thinking_signature: None,
redacted: None,
rest: serde_json::Map::new(),
@@ -397,7 +410,7 @@ pub(crate) fn write_session_file(
text: if marker_turn {
"Done.".to_string()
} else {
- assistant_text(rng)
+ tag(assistant_text(rng))
},
text_signature: None,
rest: serde_json::Map::new(),
@@ -456,7 +469,7 @@ pub(crate) fn write_session_file(
tool_call_id: call_id,
tool_name: name.to_string(),
content: vec![UserContentBlock::Text(TextContent {
- text: tool_output(rng),
+ text: tag(tool_output(rng)),
text_signature: None,
rest: serde_json::Map::new(),
})],
@@ -505,6 +518,7 @@ mod tests {
marker: Some("final user marker".to_string()),
parent_session: None,
rlm_depth: Some(0),
+ ordinals: false,
}
}
diff --git a/crates/pa-perf/src/report.rs b/crates/pa-perf/src/report.rs
index 4dfd2993dc..4a2e668a90 100644
--- a/crates/pa-perf/src/report.rs
+++ b/crates/pa-perf/src/report.rs
@@ -45,14 +45,34 @@ pub struct Report {
#[must_use]
pub fn report(results: &[BenchResult], baseline: &Baseline) -> Report {
let mut rows = Vec::new();
+ let mut notes = Vec::new();
let mut regressions = Vec::new();
for result in results {
let key = format!("{}/{}", key_of(result.scenario), result.profile);
let median = result.median_ms;
+ let first_paint = result
+ .first_paint
+ .as_ref()
+ .map_or("—".to_string(), |series| {
+ format!("{:.1}", series.median_ms)
+ });
+ let worst_gap = result.worst_gap.as_ref().map_or("—".to_string(), |series| {
+ format!("{:.1} / {:.1}", series.p95_ms, series.max_ms)
+ });
+ if let Some(wakeups) = &result.wakeups {
+ let mut sorted = wakeups.clone();
+ sorted.sort_unstable();
+ notes.push(format!(
+ "- {key}: {median:.0} ms CPU, {} context switches (median of {} run(s); {})",
+ sorted[sorted.len() / 2],
+ sorted.len(),
+ result.needle
+ ));
+ }
+ let p95 = result.p95_ms;
let Some(entry) = baseline.entries.get(&key) else {
rows.push(format!(
- "| {} | {median:.1} | {:.1} | — | — | (no baseline) |",
- key, result.p95_ms
+ "| {key} | {median:.1} | {p95:.1} | {first_paint} | {worst_gap} | — | — | (no baseline) |"
));
continue;
};
@@ -66,17 +86,19 @@ pub fn report(results: &[BenchResult], baseline: &Baseline) -> Report {
format!("{:+.0}%", delta * 100.0)
};
rows.push(format!(
- "| {} | {median:.1} | {:.1} | {:.1} | {:+.0}% | {verdict} |",
- key,
- result.p95_ms,
+ "| {key} | {median:.1} | {p95:.1} | {first_paint} | {worst_gap} | {:.1} | {:+.0}% | {verdict} |",
entry.median_ms,
delta * 100.0
));
}
- let markdown = format!(
- "| scenario/profile | median ms | p95 ms | baseline ms | delta | verdict |\n|---|---|---|---|---|---|\n{}",
+ let mut markdown = format!(
+ "| scenario/profile | median ms | p95 ms | first-paint median ms | worst gap ms (p95 / max) | baseline ms | delta | verdict |\n|---|---|---|---|---|---|---|---|\n{}",
rows.join("\n")
);
+ if !notes.is_empty() {
+ markdown.push_str("\n\n");
+ markdown.push_str(&notes.join("\n"));
+ }
Report {
markdown,
regressions,
diff --git a/crates/pa-perf/tests/report_regression.rs b/crates/pa-perf/tests/report_regression.rs
index e1007c471b..a7066b1781 100644
--- a/crates/pa-perf/tests/report_regression.rs
+++ b/crates/pa-perf/tests/report_regression.rs
@@ -24,6 +24,9 @@ fn result(scenario: Scenario, median_ms: f64) -> BenchResult {
child_sessions: 12,
archive: 3,
},
+ first_paint: None,
+ worst_gap: None,
+ wakeups: None,
}
}
@@ -72,3 +75,24 @@ fn a_missing_baseline_prints_no_baseline() {
assert!(report.markdown.contains("open-session/small"));
assert!(report.markdown.contains("(no baseline)"));
}
+
+#[test]
+fn results_written_before_the_secondary_series_still_report() {
+ let old = serde_json::json!({
+ "schema": 1,
+ "scenario": "agents-view",
+ "profile": "small",
+ "runs": 1,
+ "samples_ms": [30.0],
+ "median_ms": 30.0,
+ "p95_ms": 30.0,
+ "needle": "",
+ "binary": "",
+ "corpus": { "root": "", "sessions": 10, "total_rows": 3500, "child_sessions": 12, "archive": 3 },
+ });
+ let parsed: BenchResult = serde_json::from_value(old).expect("an old result file parses");
+ let report = report(&[parsed], &baseline(1_000.0, false));
+ assert!(report
+ .markdown
+ .contains("| agents-view/small | 30.0 | 30.0 | — | — |"));
+}