A self-improving RLM agent for coding workflows and long-running autonomous tasks.
Rust
pa-perf: two-mark agents view, transcript paging/top, paint-gap gauge, idle cost (#3723)
Sebastian Müllerc971d48e3c0c
diff --git a/crates/pa-perf/Cargo.toml b/crates/pa-perf/Cargo.tomlindex 7f1ed12b61..cec2ccdd2b 100644--- a/crates/pa-perf/Cargo.toml+++ b/crates/pa-perf/Cargo.toml@@ -22,7 +22,8 @@ vt100 = "0.15"# pty + poll for the bench harness: the same nix version the workspace# already resolves for pa-core/pa-cli.[target.'cfg(unix)'.dependencies]-nix = { version = "0.29", features = ["term", "signal", "process", "fs", "poll"] }+# `feature` provides sysconf(CLK_TCK) for idle-cost's /proc tick counts.+nix = { version = "0.29", features = ["term", "signal", "process", "fs", "poll", "feature"] }[dependencies.pa-types]path = "../pa-types"diff --git a/crates/pa-perf/README.md b/crates/pa-perf/README.mdindex f6442e40d6..ab4b1e252d 100644--- a/crates/pa-perf/README.md+++ b/crates/pa-perf/README.md@@ -119,6 +119,40 @@ the liveness deadline only fails broken runs):| `open-session` | cold | `--resume <huge session>` -> the huge session's tail marker (the final user row; its reply is one line) || `session-switch` | cold | enter on the uniquely-filtered switch-b row -> B's transcript marker (the final user row; its reply is one line) || `at-completion` | cold | the `@` keystroke -> the first completion menu row |+| `transcript-paging` | cold | `--resume <huge session>`, then 20 PageUps: each keystroke -> the first frame whose topmost block ordinal is above the previous frame's (one sample per page) |+| `transcript-top` | cold | `--resume <huge session>`, then the `tui.viewport.top` key -> the first frame showing the session's first block (ordinal 0); the head is never in the attach tail, so this is a daemon page fetch |+| `idle-cost` | cold | owned daemon + TUI, ready, then a 30 s idle window: CPU ms (utime+stime) of the daemon, its descendants, and the TUI, plus their context switches (medium only, Linux only) |++First paint: `agents-view` and `agents-view-warm` also record the first+usable paint of the same run (`first_paint`: the first frame whose roster+lists a loaded row); the median stays the settled mark the baseline keys+mean. `cold-start`/`cold-ready` (and `warm-*`) are already a+first-paint/ready pair from one launch. `open-session` and `session-switch`+have no separate first paint: their marker is the first usable frame.++Block ordinals: the corpus prefixes every generated text block of the huge+session with `ord<n>`, `n` counting up from 0 in transcript order. The 20+PageUps stay inside the attach tail (the daemon serves the last 512+messages), so `transcript-paging` measures the TUI's own scroll and render;+`transcript-top` measures the daemon's head-page fetch (`get_messages`).++Stall gauge (`worst_gap`): each run's longest no-output stretch inside its+measured windows (trigger -> mark). The report prints its p95 and max; it+never gates.++idle-cost walks `/proc` from the owned pids down+(`/proc/<pid>/task/*/children`, never by name) and sums CPU ticks (`stat`)+and context switches (`task/*/status`) over the processes alive at the+window's end; a process that exits inside the window is not counted, and+the run fails if the daemon or the TUI exits. It runs once per suite on the+medium leg: `pa-perf run` skips it elsewhere and on non-Linux, and+`bench --scenario idle-cost` errors on non-Linux. CPU resolution is one+clock tick (10 ms at 100 Hz).++Screen needles see only completed frames: the chat surface brackets its+paints in synchronized output (mode 2026), and the terminal the bench+presents as (ghostty) shows such a frame only at its end. The agents view+paints unbracketed, so its needles see bytes as they arrive.Mechanics (the repo's pty-e2e discipline): the TUI child runs in its ownsession with NO controlling tty (`setsid` pre-exec; vendored crossterm@@ -139,9 +173,12 @@ pid/name-based kill.pa-perf report --baseline crates/pa-perf/baseline.json --results results/*.json [--markdown OUT]```-Prints the aggregate table (scenario/profile, median, p95, baseline, delta)-and exits nonzero ONLY when a non-report-only median exceeds twice the-baseline. Noisy scenarios (>20% spread across 3 CI runs) are marked+Prints the aggregate table (scenario/profile, median, p95, first-paint+median, worst gap p95 / max, baseline, delta), then one line per idle-cost+result (CPU ms and context switches), and exits nonzero ONLY when a+non-report-only median exceeds twice the baseline. The first-paint and+worst-gap columns never gate; scenarios without a baseline entry print as+`(no baseline)`. Noisy scenarios (>20% spread across 3 CI runs) are marked`report_only` in the baseline: their deltas always print, the gate neverfails. `baseline.json` is committed from CI numbers, not laptop/sandboxones.diff --git a/crates/pa-perf/src/bench/mod.rs b/crates/pa-perf/src/bench/mod.rsindex 95d74d364f..7a7ebf6c9f 100644--- a/crates/pa-perf/src/bench/mod.rs+++ b/crates/pa-perf/src/bench/mod.rs@@ -34,6 +34,15 @@ pub enum Scenario {SessionSwitch,/// The `@` keystroke -> the first completion menu row.AtCompletion,+ /// `--resume <huge session>`, then each PageUp -> the first frame scrolled+ /// up a page (one sample per page).+ TranscriptPaging,+ /// `--resume <huge session>`, then the top key -> the session's first+ /// block (a daemon page fetch).+ TranscriptTop,+ /// CPU ms of the owned daemon, its descendants, and the TUI over an idle+ /// window after ready. Linux-only (/proc).+ IdleCost,}/// Every bench knob for one scenario over one corpus.@@ -86,6 +95,26 @@ pub struct BenchResult {pub needle: String,pub binary: String,pub corpus: BenchCorpus,+ /// The first usable paint of the same runs (the median above stays the+ /// settled mark, the baseline's meaning).+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub first_paint: Option<Series>,+ /// Each run's longest no-output gap inside its measured windows (a+ /// flag, never gated).+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub worst_gap: Option<Series>,+ /// idle-cost: each run's context switches over the idle window.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub wakeups: Option<Vec<u64>>,+}++/// A secondary per-run series (ms) with its summary statistics.+#[derive(Debug, Serialize, Deserialize)]+pub struct Series {+ pub samples_ms: Vec<f64>,+ pub median_ms: f64,+ pub p95_ms: f64,+ pub max_ms: f64,}/// The corpus summary the result records (baseline keys use scenario/profile).diff --git a/crates/pa-perf/src/bench/pty.rs b/crates/pa-perf/src/bench/pty.rsindex d5dfbab810..8d76f88d19 100644--- a/crates/pa-perf/src/bench/pty.rs+++ b/crates/pa-perf/src/bench/pty.rs@@ -21,6 +21,12 @@ pub(crate) const KITTY_ANSWER: &[u8] = b"\x1b[?7u\x1b[?62;c";/// The idle poll slice between empty reads (poll(2) drives the wakeups).const IDLE_POLL: Duration = Duration::from_millis(25);+/// The synchronized-output bracket (mode 2026): the terminal the bench+/// presents as (ghostty) shows only completed frames, so screen predicates+/// must too.+const SYNC_BEGIN: &[u8] = b"\x1b[?2026h";+const SYNC_END: &[u8] = b"\x1b[?2026l";+/// One product child on a pty, plus the collected master stream and the/// decoded screen (a vt100 model): byte needles cover literal paints; the/// screen model covers content that only exists as cell diffs (the agents@@ -36,6 +42,10 @@ pub(crate) struct PtyChild {kitty_answered: bool,/// Set only after the exited child's final PTY bytes have drained.exited: Option<std::process::ExitStatus>,+ /// When each read chunk arrived (the stall gauge's no-output gaps).+ arrivals: Vec<Instant>,+ /// Inside an unfinished synchronized-output frame.+ in_frame: bool,}impl PtyChild {@@ -84,6 +94,8 @@ impl PtyChild {screen: vt100::Parser::new(rows, cols, 0),kitty_answered: false,exited: None,+ arrivals: Vec::new(),+ in_frame: false,})}@@ -194,9 +206,17 @@ impl PtyChild {/// Include the previous read's overlap when a terminal query is split.fn receive(&mut self, chunk: &[u8]) -> anyhow::Result<()> {+ self.arrivals.push(Instant::now());let from = self.output.len().saturating_sub(KITTY_QUERY.len() - 1);+ let frame_from = self.output.len().saturating_sub(SYNC_BEGIN.len() - 1);self.output.extend_from_slice(chunk);self.screen.process(chunk);+ let tail = &self.output[frame_from..];+ match (rfind(tail, SYNC_BEGIN), rfind(tail, SYNC_END)) {+ (Some(begin), end) if end.is_none_or(|end| begin > end) => self.in_frame = true,+ (_, Some(_)) => self.in_frame = false,+ _ => {}+ }if !self.kitty_answered && find(&self.output[from..], KITTY_QUERY).is_some() {self.write(KITTY_ANSWER)?;self.kitty_answered = true;@@ -234,7 +254,7 @@ impl PtyChild {settled: impl Fn(&str) -> bool,) -> anyhow::Result<Instant> {loop {- if settled(&self.screen_text()) {+ if !self.in_frame && settled(&self.screen_text()) {return Ok(Instant::now());}if let Some(status) = self.exited {@@ -254,10 +274,34 @@ impl PtyChild {}/// The decoded screen's text (rows joined; trailing blanks trimmed).- fn screen_text(&mut self) -> String {+ pub(crate) fn screen_text(&self) -> String {self.screen.screen().contents().trim_end().to_string()}+ /// Keep reading (and answering) the child's output until `deadline`: a+ /// measured window must not stall the child on a full pty.+ ///+ /// # Errors+ ///+ /// Errors when the child exits inside the window.+ #[cfg(target_os = "linux")]+ pub(crate) fn drain_until(&mut self, deadline: Instant) -> anyhow::Result<()> {+ loop {+ self.pump(Some(deadline))?;+ if let Some(status) = self.exited {+ anyhow::bail!("child exited with {status} inside the measured window");+ }+ if Instant::now() >= deadline {+ return Ok(());+ }+ }+ }++ /// The longest no-output stretch from `from` until now.+ pub(crate) fn longest_gap_since(&self, from: Instant) -> Duration {+ longest_gap(&self.arrivals, from, Instant::now())+ }+/// The child's pid (teardown signals this pid only).pub(crate) fn pid(&self) -> i32 {self.child.id() as i32@@ -306,6 +350,27 @@ fn find(haystack: &[u8], needle: &[u8]) -> Option<usize> {haystack.windows(needle.len()).position(|w| w == needle)}+fn rfind(haystack: &[u8], needle: &[u8]) -> Option<usize> {+ haystack.windows(needle.len()).rposition(|w| w == needle)+}++/// The longest stretch of `from..=to` in which no chunk arrived; the window+/// edges bound the first and last stretch (`arrivals` is ascending).+fn longest_gap(arrivals: &[Instant], from: Instant, to: Instant) -> Duration {+ let first = arrivals.partition_point(|at| *at < from);+ let mut previous = from;+ let mut longest = Duration::ZERO;+ for &at in arrivals[first..]+ .iter()+ .take_while(|at| **at <= to)+ .chain(std::iter::once(&to))+ {+ longest = longest.max(at.saturating_duration_since(previous));+ previous = at;+ }+ longest+}+#[cfg(test)]mod tests {use super::*;@@ -335,6 +400,49 @@ mod tests {}}+ #[test]+ fn screen_predicates_see_only_completed_frames_at_every_read_boundary() {+ let dir = tempfile::tempdir().expect("temp dir");+ let frame = [SYNC_BEGIN, b"painted".as_slice(), SYNC_END].concat();+ for split in 1..frame.len() {+ let mut child = PtyChild::spawn(+ std::path::Path::new("/bin/cat"),+ &[],+ &[],+ dir.path(),+ 120,+ 36,+ )+ .expect("spawn mock terminal peer");+ child.receive(&frame[..split]).expect("first chunk");+ assert_eq!(+ child.in_frame,+ split >= SYNC_BEGIN.len(),+ "split at {split}"+ );+ child.receive(&frame[split..]).expect("second chunk");+ assert!(!child.in_frame, "the frame end was missed at {split}");+ }+ }++ #[test]+ fn the_longest_gap_is_bounded_by_the_window_edges() {+ let start = Instant::now();+ let at = |ms| start + Duration::from_millis(ms);+ let arrivals = [at(0), at(5), at(40), at(45), at(200)];+ // Arrivals before the window never shorten its leading stretch.+ assert_eq!(+ longest_gap(&arrivals, at(10), at(42)),+ Duration::from_millis(30)+ );+ // The trailing stretch runs to the window end, not the next arrival.+ assert_eq!(+ longest_gap(&arrivals, at(10), at(100)),+ Duration::from_millis(55)+ );+ assert_eq!(longest_gap(&[], at(0), at(7)), Duration::from_millis(7));+ }+#[test]fn exited_children_fail_missing_needles_without_waiting_for_liveness_timeout() {let directory = tempfile::tempdir().expect("mock directory");@@ -367,6 +475,26 @@ mod tests {.contains("exit status: 7"));}+ #[cfg(target_os = "linux")]+ #[test]+ fn a_child_exit_seen_by_the_last_drain_pump_fails_the_window() {+ let directory = tempfile::tempdir().expect("mock directory");+ let mut child = PtyChild::spawn(+ std::path::Path::new("/bin/sh"),+ &["-c".into(), "exit 7".into()],+ &[],+ directory.path(),+ 120,+ 36,+ )+ .expect("spawn exiting mock");+ child.child.wait().expect("the mock exits");+ let error = child+ .drain_until(Instant::now())+ .expect_err("a dead child fails the window");+ assert!(error.to_string().contains("exit status: 7"), "{error}");+ }+#[test]fn final_output_is_drained_before_reporting_child_exit() {let directory = tempfile::tempdir().expect("mock directory");diff --git a/crates/pa-perf/src/bench/runner.rs b/crates/pa-perf/src/bench/runner.rsindex 419154aa69..311415962c 100644--- a/crates/pa-perf/src/bench/runner.rs+++ b/crates/pa-perf/src/bench/runner.rs@@ -8,8 +8,8 @@ use std::time::{Duration, Instant};use anyhow::Context;-use crate::bench::{BenchCorpus, BenchOptions, BenchResult, RunOptions, Scenario};-use crate::corpus::{self, CacheMode, CorpusOptions, Manifest};+use crate::bench::{BenchCorpus, BenchOptions, BenchResult, RunOptions, Scenario, Series};+use crate::corpus::{self, CacheMode, CorpusOptions, Manifest, Profile};use super::pty::PtyChild;@@ -24,6 +24,53 @@ const FIRST_PAINT_NEEDLE: &[u8] = "▗▄▄█▀".as_bytes();/// "0 (0%)".const READY_NEEDLE: &str = "0 (0%)";+const PAGE_UP: &[u8] = b"\x1b[5~";+/// Few enough pages to stay inside the attach tail (TUI-local scrolling).+const PAGES: usize = 20;+/// The product's default `tui.viewport.top` key (shift+alt+up).+const VIEWPORT_TOP: &[u8] = b"\x1b[1;4A";++const IDLE_WINDOW: Duration = Duration::from_secs(30);++/// One run's measurements (transcript-paging settles once per page).+#[derive(Debug, Default)]+struct Measured {+ settled: Vec<f64>,+ first_paint: Option<f64>,+ worst_gap: Option<f64>,+ wakeups: Option<u64>,+}++impl Measured {+ fn single(mark: Mark) -> Self {+ Self {+ settled: vec![mark.ms],+ worst_gap: Some(mark.gap_ms),+ ..Self::default()+ }+ }+}++/// A trigger -> mark interval and its longest no-output gap.+#[derive(Debug, Clone, Copy)]+struct Mark {+ ms: f64,+ gap_ms: f64,+}++impl Mark {+ fn since(child: &PtyChild, started: Instant) -> Self {+ Self {+ ms: millis(started.elapsed()),+ gap_ms: millis(child.longest_gap_since(started)),+ }+ }+}++fn millis(duration: Duration) -> f64 {+ duration.as_secs_f64() * 1000.0+}+/// One bench run's anchored paths.struct RunCtx {binary: PathBuf,@@ -98,10 +145,10 @@ pub(super) fn run(options: &BenchOptions) -> anyhow::Result<BenchResult> {&options.corpus,Duration::from_secs(options.liveness_secs),)?;- let samples = sample_runs(&ctx, options.runs, |index| {- scenario_once(&ctx, options.scenario, index)+ let runs = sample_runs(&ctx, options.runs, |index| {+ Ok(launch(&ctx, options.scenario, index)?.0)})?;- let result = build_result(&ctx, options.scenario, samples);+ let result = build_result(&ctx, options.scenario, runs);write_result(&result, &options.json)?;Ok(result)}@@ -112,9 +159,9 @@ pub(super) fn run(options: &BenchOptions) -> anyhow::Result<BenchResult> {fn sample_runs(ctx: &RunCtx,runs: usize,- mut scenario: impl FnMut(usize) -> anyhow::Result<f64>,-) -> anyhow::Result<Vec<f64>> {- let mut samples = Vec::with_capacity(runs + 1);+ mut scenario: impl FnMut(usize) -> anyhow::Result<Measured>,+) -> anyhow::Result<Vec<Measured>> {+ let mut measured = Vec::with_capacity(runs);let total = runs + 1;for index in 0..total {restore(ctx)?;@@ -124,44 +171,41 @@ fn sample_runs(outcome?;continue;}- let sample = outcome.context("a sampled run failed")?;- samples.push(sample);+ let run = outcome.context("a sampled run failed")?;println!(- "run {index}/{runs}: {sample:.1}ms (wall {:.1}s)",+ "run {index}/{runs}: {:.1?}ms (wall {:.1}s)",+ run.settled,started.elapsed().as_secs_f32());+ measured.push(run);}- Ok(samples)-}--/// One scenario pass; returns the sample in ms.-fn scenario_once(ctx: &RunCtx, scenario: Scenario, run: usize) -> anyhow::Result<f64> {- let (sample, _) = launch(ctx, scenario, run)?;- Ok(sample)+ Ok(measured)}/// The ONE launch path: the pre-started daemon for warm-* scenarios, the/// scenario's own resume target + first needle, the post-launch mechanics,/// and the daemon teardown on every path. Both `run()` (single scenario)/// and `run_suite()` call this.-fn launch(ctx: &RunCtx, scenario: Scenario, run: usize) -> anyhow::Result<(f64, Launch)> {+fn launch(ctx: &RunCtx, scenario: Scenario, run: usize) -> anyhow::Result<(Measured, Launch)> {let socket = run_socket(ctx, run);std::fs::create_dir_all(socket.parent().expect("a socket dir"))?;let mut daemon = if matches!(scenario,- Scenario::WarmStart | Scenario::WarmReady | Scenario::AgentsViewWarm+ Scenario::WarmStart | Scenario::WarmReady | Scenario::AgentsViewWarm | Scenario::IdleCost) {// AgentsViewWarm pre-starts so both TUIs share the daemon the first// populate warms; the single-scenario path cannot express that (its// second launch would spawn a second daemon), so it runs the first- // populate only and the sample is the suite's warm number.+ // populate only and the sample is the suite's warm number. IdleCost+ // pre-starts so the daemon is an owned pid whose /proc it can read.Some(start_daemon(ctx, &socket)?)} else {None};- let outcome: anyhow::Result<(f64, Launch)> = (|| {+ let daemon_pid = daemon.as_ref().map(|daemon| daemon.child.id() as i32);+ let outcome: anyhow::Result<(Measured, Launch)> = (|| {let mut launched = spawn_for_scenario(ctx, &socket, scenario, run)?;- let sample = launch_scenario(ctx, &mut launched, scenario)?;+ let sample = launch_scenario(ctx, &mut launched, scenario, daemon_pid)?;// The measured client must disconnect before daemon shutdown.// Keeping it alive while waiting for daemon exit can hold server// connection draining open or start the client's reconnect path.@@ -229,7 +273,9 @@ pub(super) fn run_suite(options: &RunOptions) -> anyhow::Result<Vec<BenchResult>let cold_ctx = RunCtx::load(&options.binary, &options.out.join("corpus-cold"), liveness)?;let warm_ctx = RunCtx::load(&options.binary, &options.out.join("corpus-warm"), liveness)?;- let mut samples: BTreeMap<Scenario, Vec<f64>> = BTreeMap::new();+ let mut samples: BTreeMap<Scenario, Vec<Measured>> = BTreeMap::new();+ // Each scenario's share of the suite's wall time (the CI budget).+ let mut wall: BTreeMap<Scenario, Duration> = BTreeMap::new();let total = options.runs + 1;for index in 0..total {let started = Instant::now();@@ -238,51 +284,84 @@ pub(super) fn run_suite(options: &RunOptions) -> anyhow::Result<Vec<BenchResult>// per launch; the cold-start/cold-ready (and warm-*) pairs share// one launch each.restore(&cold_ctx)?;- let (start_sample, launched) = launch(&cold_ctx, Scenario::ColdStart, index)?;- let ready_sample = launched- .ready_ms- .context("the cold launch never attached")?;- push(&mut samples, index, Scenario::ColdStart, start_sample);- push(&mut samples, index, Scenario::ColdReady, ready_sample);+ let (start, launched) = launch(&cold_ctx, Scenario::ColdStart, index)?;+ let ready = launched.ready.context("the cold launch never attached")?;+ push(&mut samples, index, Scenario::ColdStart, start);+ push(+ &mut samples,+ index,+ Scenario::ColdReady,+ Measured::single(ready),+ );for scenario in [Scenario::AgentsView,Scenario::AgentsViewWarm,Scenario::OpenSession,Scenario::SessionSwitch,Scenario::AtCompletion,+ Scenario::TranscriptPaging,+ Scenario::TranscriptTop,] {+ let launched = Instant::now();restore(&cold_ctx)?;- let (sample, _) = launch(&cold_ctx, scenario, index)?;- push(&mut samples, index, scenario, sample);+ let (measured, _) = launch(&cold_ctx, scenario, index)?;+ push(&mut samples, index, scenario, measured);+ *wall.entry(scenario).or_default() += launched.elapsed();}restore(&warm_ctx)?;- let (start_sample, launched) = launch(&warm_ctx, Scenario::WarmStart, index)?;- let ready_sample = launched- .ready_ms- .context("the warm launch never attached")?;- push(&mut samples, index, Scenario::WarmStart, start_sample);- push(&mut samples, index, Scenario::WarmReady, ready_sample);+ let (start, launched) = launch(&warm_ctx, Scenario::WarmStart, index)?;+ let ready = launched.ready.context("the warm launch never attached")?;+ push(&mut samples, index, Scenario::WarmStart, start);+ push(+ &mut samples,+ index,+ Scenario::WarmReady,+ Measured::single(ready),+ );println!("run {index}/{} complete ({:.1}s)",options.runs,started.elapsed().as_secs_f32());}+ // One idle window per suite on one profile: an idle ready session's+ // cost is per process, not per corpus size, so one leg pays the window.+ if options.profile == Profile::Medium {+ if cfg!(target_os = "linux") {+ let launched = Instant::now();+ restore(&cold_ctx)?;+ let (measured, _) = launch(&cold_ctx, Scenario::IdleCost, total)?;+ samples.insert(Scenario::IdleCost, vec![measured]);+ wall.insert(Scenario::IdleCost, launched.elapsed());+ } else {+ println!("idle-cost skipped: it reads /proc (Linux-only)");+ }+ }+ for (scenario, spent) in &wall {+ println!(+ "wall {}: {:.1}s",+ crate::report::key_of(*scenario),+ spent.as_secs_f32()+ );+ }let results_dir = options.out.join("results");std::fs::create_dir_all(&results_dir)?;let mut results = Vec::new();- for (scenario, mut samples) in samples {- samples.sort_by(f64::total_cmp);- if samples.len() != options.runs {+ for (scenario, runs) in samples {+ let expected = if scenario == Scenario::IdleCost {+ 1+ } else {+ options.runs+ };+ if runs.len() != expected {anyhow::bail!(- "{scenario:?} collected {} samples, expected {}",- samples.len(),- options.runs+ "{scenario:?} collected {} runs, expected {expected}",+ runs.len(),);}let ctx = corpus_for(scenario, &cold_ctx, &warm_ctx);- let result = build_result(ctx, scenario, samples);+ let result = build_result(ctx, scenario, runs);let name = crate::report::key_of(scenario);// Per-profile unique names: the CI legs merge their artifacts into// one results dir, and same-named files would overwrite each other.@@ -309,14 +388,22 @@ fn corpus_for<'a>(scenario: Scenario, cold: &'a RunCtx, warm: &'a RunCtx) -> &'a| Scenario::AgentsViewWarm| Scenario::OpenSession| Scenario::SessionSwitch- | Scenario::AtCompletion => cold,+ | Scenario::AtCompletion+ | Scenario::TranscriptPaging+ | Scenario::TranscriptTop+ | Scenario::IdleCost => cold,}}-/// Push a sample unless this is the discarded warm-up run.-fn push(samples: &mut BTreeMap<Scenario, Vec<f64>>, index: usize, scenario: Scenario, sample: f64) {+/// Push a run unless this is the discarded warm-up run.+fn push(+ samples: &mut BTreeMap<Scenario, Vec<Measured>>,+ index: usize,+ scenario: Scenario,+ measured: Measured,+) {if index > 0 {- samples.entry(scenario).or_default().push(sample);+ samples.entry(scenario).or_default().push(measured);}}@@ -328,7 +415,7 @@ fn spawn_for_scenario(run: usize,) -> anyhow::Result<Launch> {let (resume, first_needle) = match scenario {- Scenario::OpenSession => (+ Scenario::OpenSession | Scenario::TranscriptPaging | Scenario::TranscriptTop => (Some(ctx.manifest.markers.huge.file.clone()),ctx.manifest.markers.huge.needle.as_bytes().to_vec(),),@@ -345,7 +432,11 @@ fn spawn_for_scenario(let mut launch = Launch::spawn(ctx, socket, resume.as_deref(), &first_needle, dump)?;if matches!(scenario,- Scenario::ColdStart | Scenario::WarmStart | Scenario::ColdReady | Scenario::WarmReady+ Scenario::ColdStart+ | Scenario::WarmStart+ | Scenario::ColdReady+ | Scenario::WarmReady+ | Scenario::IdleCost) {launch.wait_ready()?;}@@ -354,16 +445,28 @@ fn spawn_for_scenario(/// The scenario mechanics after the launch needle; a completed launch/// never dumps its stream.-fn launch_scenario(ctx: &RunCtx, launch: &mut Launch, scenario: Scenario) -> anyhow::Result<f64> {- let sample = post_launch(ctx, launch, scenario)?;+fn launch_scenario(+ ctx: &RunCtx,+ launch: &mut Launch,+ scenario: Scenario,+ daemon_pid: Option<i32>,+) -> anyhow::Result<Measured> {+ let measured = post_launch(ctx, launch, scenario, daemon_pid)?;launch.completed = true;- Ok(sample)+ Ok(measured)}/// The scenario mechanics after the launch needle.-fn post_launch(ctx: &RunCtx, launch: &mut Launch, scenario: Scenario) -> anyhow::Result<f64> {+fn post_launch(+ ctx: &RunCtx,+ launch: &mut Launch,+ scenario: Scenario,+ daemon_pid: Option<i32>,+) -> anyhow::Result<Measured> {match scenario {- Scenario::ColdStart | Scenario::WarmStart | Scenario::OpenSession => Ok(launch.first_ms),+ Scenario::ColdStart | Scenario::WarmStart | Scenario::OpenSession => {+ Ok(Measured::single(launch.first))+ }Scenario::AgentsView => agents_view_populate(ctx, &mut launch.child),Scenario::AgentsViewWarm => {// The first populate warms the daemon's in-process scan cache;@@ -375,13 +478,21 @@ fn post_launch(ctx: &RunCtx, launch: &mut Launch, scenario: Scenario) -> anyhow:.join("failed-streams").join("agents-view-warm-second.bin");let mut second = Launch::spawn(ctx, &socket, None, FIRST_PAINT_NEEDLE, dump)?;- let sample = agents_view_populate(ctx, &mut second.child)?;+ let measured = agents_view_populate(ctx, &mut second.child)?;second.completed = true;- Ok(sample)+ Ok(measured)}Scenario::SessionSwitch => switch_to_target_b(ctx, &mut launch.child),Scenario::AtCompletion => at_completion(ctx, &mut launch.child),- Scenario::ColdReady | Scenario::WarmReady => Ok(launch.ready_ms.unwrap_or(launch.first_ms)),+ Scenario::TranscriptPaging => transcript_paging(ctx, &mut launch.child),+ Scenario::TranscriptTop => transcript_top(ctx, &mut launch.child),+ Scenario::IdleCost => idle::measure(+ &mut launch.child,+ daemon_pid.expect("launch pre-starts the idle-cost daemon"),+ ),+ Scenario::ColdReady | Scenario::WarmReady => {+ Ok(Measured::single(launch.ready.unwrap_or(launch.first)))+ }}}@@ -390,8 +501,8 @@ fn post_launch(ctx: &RunCtx, launch: &mut Launch, scenario: Scenario) -> anyhow:struct Launch {child: PtyChild,socket: PathBuf,- first_ms: f64,- ready_ms: Option<f64>,+ first: Mark,+ ready: Option<Mark>,spawned: Instant,liveness: Duration,/// Where the raw pty stream lands when the launch does NOT complete (the@@ -434,10 +545,13 @@ impl Launch {"the first paint"};let mut launch = Self {- first_ms: 0.0,+ first: Mark {+ ms: 0.0,+ gap_ms: 0.0,+ },child,socket: socket.to_path_buf(),- ready_ms: None,+ ready: None,spawned: started,liveness: ctx.liveness,stream_dump: Some(stream_dump),@@ -449,7 +563,7 @@ impl Launch {what,started + ctx.liveness,)?;- launch.first_ms = started.elapsed().as_secs_f64() * 1000.0;+ launch.first = Mark::since(&launch.child, started);Ok(launch)}@@ -461,7 +575,7 @@ impl Launch {"the attach pill",self.spawned + self.liveness,)?;- self.ready_ms = Some(self.spawned.elapsed().as_secs_f64() * 1000.0);+ self.ready = Some(Mark::since(&self.child, self.spawned));Ok(())}}@@ -485,22 +599,28 @@ impl Drop for Launch {/// cell diffs, so the byte stream cannot see it — the screen model does)./// The count includes the TUI's own fresh session (rendered as one more/// inactive row).-fn agents_view_populate(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<f64> {+fn agents_view_populate(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<Measured> {let started = Instant::now();child.write(b"\x1b[D")?;+ child.wait_screen("the first roster row", started + ctx.liveness, |screen| {+ roster_counts(screen).is_some_and(|(loaded, _)| loaded >= 1)+ })?;+ let first_paint = millis(started.elapsed());wait_roster(ctx, child, ctx.manifest.markers.roster_inactive + 1)?;- Ok(started.elapsed().as_secs_f64() * 1000.0)+ Ok(Measured {+ first_paint: Some(first_paint),+ ..Measured::single(Mark::since(child, started))+ })}-/// The agents view opens on its first catalog page: the `Inactive (N)`-/// heading counts the loaded rows and the `M older sessions not loaded`-/// footer the rest, so the page is settled once both account for every-/// expected session. An unpaged roster has no footer.-fn roster_settled(screen: &str, expected: usize) -> bool {+/// The agents view's roster accounting: the `Inactive (N)` heading counts+/// the loaded rows and the `M older sessions not loaded` footer the rest+/// (an unpaged roster has no footer). `None` while no roster is on screen.+fn roster_counts(screen: &str) -> Option<(usize, usize)> {let loaded = screen.split_once("Inactive (").and_then(|(_, rest)| rest.split_once(')'))- .and_then(|(number, _)| number.parse::<usize>().ok());+ .and_then(|(number, _)| number.parse::<usize>().ok())?;let unloaded = screen.lines().find_map(|line| {@@ -508,7 +628,13 @@ fn roster_settled(screen: &str, expected: usize) -> bool {number.parse::<usize>().ok()}).unwrap_or(0);- loaded.is_some_and(|loaded| loaded + unloaded == expected)+ Some((loaded, unloaded))+}++/// The first catalog page is settled once the loaded and not-yet-loaded+/// counts account for every expected session.+fn roster_settled(screen: &str, expected: usize) -> bool {+ roster_counts(screen).is_some_and(|(loaded, unloaded)| loaded + unloaded == expected)}fn wait_roster(ctx: &RunCtx, child: &mut PtyChild, expected: usize) -> anyhow::Result<Instant> {@@ -522,8 +648,10 @@ fn wait_roster(ctx: &RunCtx, child: &mut PtyChild, expected: usize) -> anyhow::R/// From the opened session A: open the agents view, uniquely filter to/// switch-b, and time enter -> B's transcript marker. The `zq` token in/// switch-b's name matches no other row (no generated text carries the-/// letter z), so Enter always drills THE row.-fn switch_to_target_b(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<f64> {+/// letter z), so Enter always drills THE row. (No first-paint mark: the+/// agents view stays up until B's first chat frame, which already carries+/// the marker.)+fn switch_to_target_b(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<Measured> {child.write(b"\x1b[D")?;// Wait the settled roster before typing: keys racing the scan flood// were the CI-runner failure (the query echoed nothing). The resumed@@ -548,14 +676,14 @@ fn switch_to_target_b(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<f64>"the switch-b transcript",Instant::now() + ctx.liveness,)?;- Ok(started.elapsed().as_secs_f64() * 1000.0)+ Ok(Measured::single(Mark::since(child, started)))}/// The `@` completion: the trigger is the `@` keystroke alone; the needle/// is the first painted completion menu row (the menu lists the project/// tree's entries — `src/` is a top-level dir of the generated project/// that the pre-menu screen never contains).-fn at_completion(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<f64> {+fn at_completion(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<Measured> {let started = Instant::now();child.write(b"@")?;// The menu lists the project tree's entries — `src/` is a top-level@@ -566,7 +694,72 @@ fn at_completion(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<f64> {"the completion menu row",Instant::now() + ctx.liveness,)?;- Ok(started.elapsed().as_secs_f64() * 1000.0)+ Ok(Measured::single(Mark::since(child, started)))+}++/// From the resumed tail: PAGES PageUps, each timed to the first frame whose+/// topmost block ordinal is above the previous frame's.+fn transcript_paging(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<Measured> {+ wait_ordinal_frame(ctx, child)?;+ let first = topmost_ordinal(&child.screen_text());+ let mut pages = Vec::with_capacity(PAGES);+ for page in 1..=PAGES {+ let before = topmost_ordinal(&child.screen_text())+ .with_context(|| format!("no block ordinal on screen before page {page}"))?;+ let started = Instant::now();+ child.write(PAGE_UP)?;+ child.wait_screen(+ &format!("page {page} above block ordinal {before}"),+ started + ctx.liveness,+ |screen| topmost_ordinal(screen).is_some_and(|top| top < before),+ )?;+ pages.push(Mark::since(child, started));+ }+ println!(+ "transcript-paging: topmost block ordinal {first:?} -> {:?}",+ topmost_ordinal(&child.screen_text())+ );+ Ok(Measured {+ settled: pages.iter().map(|page| page.ms).collect(),+ worst_gap: pages.iter().map(|page| page.gap_ms).reduce(f64::max),+ ..Measured::default()+ })+}++/// The top key -> the first frame showing block ordinal 0. The head is never+/// in the attach tail, so the TUI fetches it from the daemon.+fn transcript_top(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<Measured> {+ wait_ordinal_frame(ctx, child)?;+ let started = Instant::now();+ child.write(VIEWPORT_TOP)?;+ child.wait_screen(+ "the session's first block",+ started + ctx.liveness,+ |screen| topmost_ordinal(screen) == Some(0),+ )?;+ Ok(Measured::single(Mark::since(child, started)))+}++fn wait_ordinal_frame(ctx: &RunCtx, child: &mut PtyChild) -> anyhow::Result<Instant> {+ child.wait_screen(+ "an ordinal-tagged transcript frame",+ Instant::now() + ctx.liveness,+ |screen| topmost_ordinal(screen).is_some(),+ )+}++/// The smallest block ordinal on screen is the topmost block.+fn topmost_ordinal(screen: &str) -> Option<u64> {+ screen+ .match_indices(corpus::ORDINAL_PREFIX)+ .filter_map(|(at, prefix)| {+ let digits = &screen[at + prefix.len()..];+ let end = digits+ .find(|c: char| !c.is_ascii_digit())+ .unwrap_or(digits.len());+ digits[..end].parse().ok()+ })+ .min()}/// The per-run isolated socket path.@@ -735,14 +928,16 @@ fn restore(ctx: &RunCtx) -> anyhow::Result<()> {Ok(())}-/// The result object for one scenario's samples.-fn build_result(ctx: &RunCtx, scenario: Scenario, samples: Vec<f64>) -> BenchResult {- let (median_ms, p95_ms) = stats(&samples);+/// The result object for one scenario's runs (settled marks pooled).+fn build_result(ctx: &RunCtx, scenario: Scenario, runs: Vec<Measured>) -> BenchResult {+ let mut samples: Vec<f64> = runs.iter().flat_map(|run| run.settled.clone()).collect();+ samples.sort_by(f64::total_cmp);+ let (median_ms, p95_ms, _) = stats(&samples);BenchResult {schema: 1,scenario,profile: crate::report::profile_key(ctx.manifest.profile),- runs: samples.len(),+ runs: runs.len(),median_ms,p95_ms,samples_ms: samples,@@ -755,11 +950,27 @@ fn build_result(ctx: &RunCtx, scenario: Scenario, samples: Vec<f64>) -> BenchReschild_sessions: ctx.manifest.counts.child_sessions,archive: ctx.manifest.counts.archive,},+ first_paint: series(runs.iter().map(|run| run.first_paint).collect()),+ worst_gap: series(runs.iter().map(|run| run.worst_gap).collect()),+ wakeups: runs.iter().map(|run| run.wakeups).collect(),}}-/// Median + p95 over the samples.-fn stats(samples: &[f64]) -> (f64, f64) {+/// The series when every run measured it.+fn series(samples: Option<Vec<f64>>) -> Option<Series> {+ let mut samples = samples?;+ samples.sort_by(f64::total_cmp);+ let (median_ms, p95_ms, max_ms) = stats(&samples);+ Some(Series {+ samples_ms: samples,+ median_ms,+ p95_ms,+ max_ms,+ })+}++/// Median, p95, and max over the samples.+fn stats(samples: &[f64]) -> (f64, f64, f64) {let mut sorted = samples.to_vec();sorted.sort_by(f64::total_cmp);let median = if sorted.len() % 2 == 1 {@@ -768,7 +979,11 @@ fn stats(samples: &[f64]) -> (f64, f64) {(sorted[sorted.len() / 2 - 1] + sorted[sorted.len() / 2]) / 2.0};let p95_index = ((sorted.len() as f64 - 1.0) * 0.95).ceil() as usize;- (median, sorted[p95_index.min(sorted.len() - 1)])+ (+ median,+ sorted[p95_index.min(sorted.len() - 1)],+ sorted[sorted.len() - 1],+ )}fn needle_of(scenario: Scenario, ctx: &RunCtx) -> String {@@ -794,6 +1009,17 @@ fn needle_of(scenario: Scenario, ctx: &RunCtx) -> String {ctx.manifest.markers.switch_b.needle),Scenario::AtCompletion => "the first @ completion menu row".to_string(),+ Scenario::TranscriptPaging => format!(+ "per PageUp ({PAGES} per run) on the huge session: the first frame whose topmost block ordinal is above the previous frame's"+ ),+ Scenario::TranscriptTop => {+ "the top key on the huge session -> the first frame showing its first block (ordinal 0)"+ .to_string()+ }+ Scenario::IdleCost => format!(+ "CPU ms of the owned daemon, its descendants, and the TUI over a {}s idle window after ready",+ IDLE_WINDOW.as_secs()+ ),}}@@ -802,6 +1028,171 @@ fn write_result(result: &BenchResult, path: &Path) -> anyhow::Result<()> {std::fs::write(path, json).with_context(|| format!("write {}", path.display()))}+/// idle-cost: processes are found from the owned pids down (/proc+/// children), never by name.+#[cfg(target_os = "linux")]+mod idle {+ use std::collections::BTreeMap;+ use std::time::Instant;++ use anyhow::Context;++ use super::{Measured, PtyChild, IDLE_WINDOW};++ /// utime+stime (clock ticks) and context switches summed over threads.+ #[derive(Debug, Clone, Copy, Default)]+ struct Usage {+ ticks: u64,+ switches: u64,+ }++ pub(super) fn measure(child: &mut PtyChild, daemon: i32) -> anyhow::Result<Measured> {+ let roots = [daemon, child.pid()];+ let before = snapshot(&roots);+ child.drain_until(Instant::now() + IDLE_WINDOW)?;+ let after = snapshot(&roots);+ anyhow::ensure!(+ after.contains_key(&daemon),+ "the daemon exited inside the idle window"+ );+ // A process that exited inside the window is gone from `after`, so+ // its share is not counted.+ let delta = after.iter().fold(Usage::default(), |sum, (pid, end)| {+ let start = before.get(pid).copied().unwrap_or_default();+ Usage {+ ticks: sum.ticks + end.ticks.saturating_sub(start.ticks),+ switches: sum.switches + end.switches.saturating_sub(start.switches),+ }+ });+ let ticks_per_second = nix::unistd::sysconf(nix::unistd::SysconfVar::CLK_TCK)?+ .context("CLK_TCK is defined on Linux")?;+ let cpu_ms = delta.ticks as f64 * 1000.0 / ticks_per_second as f64;+ println!(+ "idle-cost: {cpu_ms:.0}ms CPU, {} context switches, {} processes",+ delta.switches,+ after.len()+ );+ Ok(Measured {+ settled: vec![cpu_ms],+ wakeups: Some(delta.switches),+ ..Measured::default()+ })+ }++ /// Every live process under `roots` (inclusive) with its counters.+ fn snapshot(roots: &[i32]) -> BTreeMap<i32, Usage> {+ let mut usage = BTreeMap::new();+ let mut stack = roots.to_vec();+ while let Some(pid) = stack.pop() {+ let Some(counters) = read_usage(pid) else {+ continue;+ };+ usage.insert(pid, counters);+ stack.extend(children(pid));+ }+ usage+ }++ fn read_usage(pid: i32) -> Option<Usage> {+ let stat = std::fs::read_to_string(format!("/proc/{pid}/stat")).ok()?;+ Some(Usage {+ ticks: live_ticks(&stat)?,+ switches: tasks(pid)+ .filter_map(|tid| {+ std::fs::read_to_string(format!("/proc/{pid}/task/{tid}/status")).ok()+ })+ .map(|status| status_switches(&status))+ .sum(),+ })+ }++ /// utime + stime (stat fields 14 and 15, counted after the comm, which+ /// may contain spaces and parentheses); `None` for an unreaped zombie,+ /// whose stat file outlives it.+ fn live_ticks(stat: &str) -> Option<u64> {+ let (_, fields) = stat.rsplit_once(')')?;+ let mut fields = fields.split_whitespace();+ if fields.next()? == "Z" {+ return None;+ }+ let mut fields = fields.skip(10);+ let utime: u64 = fields.next()?.parse().ok()?;+ let stime: u64 = fields.next()?.parse().ok()?;+ Some(utime + stime)+ }++ fn status_switches(status: &str) -> u64 {+ status+ .lines()+ .filter_map(|line| {+ let (key, value) = line.split_once(':')?;+ matches!(+ key,+ "voluntary_ctxt_switches" | "nonvoluntary_ctxt_switches"+ )+ .then(|| value.trim().parse::<u64>().ok())?+ })+ .sum()+ }++ fn tasks(pid: i32) -> impl Iterator<Item = String> {+ std::fs::read_dir(format!("/proc/{pid}/task"))+ .into_iter()+ .flatten()+ .filter_map(|entry| Some(entry.ok()?.file_name().to_string_lossy().into_owned()))+ }++ /// The direct children of every thread of `pid`.+ fn children(pid: i32) -> Vec<i32> {+ tasks(pid)+ .filter_map(|tid| {+ std::fs::read_to_string(format!("/proc/{pid}/task/{tid}/children")).ok()+ })+ .flat_map(|list| {+ list.split_whitespace()+ .filter_map(|child| child.parse().ok())+ .collect::<Vec<i32>>()+ })+ .collect()+ }++ #[cfg(test)]+ mod tests {+ #[test]+ fn counters_parse_from_proc_text() {+ let stat =+ "4242 (pa worker (x)) S 1 4242 4242 0 -1 4194560 900 0 0 0 31 7 0 0 20 0 9 0";+ assert_eq!(super::live_ticks(stat), Some(38));+ assert_eq!(super::live_ticks(&stat.replacen(") S", ") Z", 1)), None);+ let status = "Name:\tprime-agent\nvoluntary_ctxt_switches:\t120\nnonvoluntary_ctxt_switches:\t5\n";+ assert_eq!(super::status_switches(status), 125);+ }++ #[test]+ fn the_snapshot_reads_real_proc_counters_down_to_descendants() {+ let mut child = std::process::Command::new("/bin/cat")+ .stdin(std::process::Stdio::piped())+ .spawn()+ .expect("spawn the mock descendant");+ let own = std::process::id() as i32;+ let seen = super::snapshot(&[own]);+ let _ = child.kill();+ let _ = child.wait();+ assert!(seen.contains_key(&(child.id() as i32)), "{seen:?}");+ assert!(seen[&own].switches > 0);+ }+ }+}++#[cfg(not(target_os = "linux"))]+mod idle {+ use super::{Measured, PtyChild};++ pub(super) fn measure(_child: &mut PtyChild, _daemon: i32) -> anyhow::Result<Measured> {+ anyhow::bail!("idle-cost reads /proc and runs on Linux only")+ }+}+#[cfg(test)]mod tests {use super::*;@@ -863,7 +1254,11 @@ mod tests {Scenario::ColdStart,] {let measured = corpus_for(scenario, &cold, &warm);- let result = build_result(measured, scenario, vec![1.0, 2.0]);+ let run = Measured {+ settled: vec![1.0],+ ..Measured::default()+ };+ let result = build_result(measured, scenario, vec![run]);let expected = if scenario == Scenario::ColdStart {&cold} else {@@ -982,10 +1377,17 @@ mod tests {}std::fs::write(&sidecar, "written by the mock scenario")?;launches.push(index);- Ok(index as f64)+ Ok(Measured {+ settled: vec![index as f64],+ ..Measured::default()+ })}).expect("sample the mock scenario");- assert_eq!((launches, samples), (vec![0, 1, 2], vec![1.0, 2.0]));+ let samples: Vec<Vec<f64>> = samples.into_iter().map(|run| run.settled).collect();+ assert_eq!(+ (launches, samples),+ (vec![0, 1, 2], vec![vec![1.0], vec![2.0]])+ );}#[test]@@ -1022,11 +1424,8 @@ mod tests {ctx.liveness = Duration::from_secs(10);let (sample, launched) = launch(&ctx, Scenario::WarmStart, 0).expect("the daemon must drain its measured UI connection before exit");- assert!(sample >= 0.0);- assert!(- launched.ready_ms.is_some(),- "the actual mock paint/attach ran"- );+ assert!(sample.settled[0] >= 0.0);+ assert!(launched.ready.is_some(), "the actual mock paint/attach ran");assert_eq!(std::fs::read_to_string(run_socket(&ctx, 0).with_extension("drained")).expect("the mock daemon reached normal drain completion"),diff --git a/crates/pa-perf/src/corpus/artifacts.rs b/crates/pa-perf/src/corpus/artifacts.rsindex 75a4674a71..c1c4417fa6 100644--- a/crates/pa-perf/src/corpus/artifacts.rs+++ b/crates/pa-perf/src/corpus/artifacts.rs@@ -194,6 +194,7 @@ pub(crate) fn spawn_child(marker: None,parent_session: Some(parent_file.display().to_string()),rlm_depth: None,+ ordinals: false,},rng,)?;diff --git a/crates/pa-perf/src/corpus/mod.rs b/crates/pa-perf/src/corpus/mod.rsindex 7c5d9d6ac5..ef5780be2e 100644--- a/crates/pa-perf/src/corpus/mod.rs+++ b/crates/pa-perf/src/corpus/mod.rs@@ -5,6 +5,8 @@pub(crate) mod artifacts;pub(crate) mod rows;+pub use rows::ORDINAL_PREFIX;+use std::path::{Path, PathBuf};use anyhow::Context;@@ -293,6 +295,8 @@ pub fn generate(options: &CorpusOptions) -> anyhow::Result<Manifest> {marker: marker.clone(),parent_session: None,rlm_depth: Some(0),+ // The huge session is the transcript-paging target.+ ordinals: index == 0,},&mut rng,)?;@@ -378,6 +382,7 @@ pub fn generate(options: &CorpusOptions) -> anyhow::Result<Manifest> {marker: None,parent_session: None,rlm_depth: Some(0),+ ordinals: false,},&mut rng,)?;diff --git a/crates/pa-perf/src/corpus/rows.rs b/crates/pa-perf/src/corpus/rows.rsindex 7f157a44bb..9ea9580c71 100644--- a/crates/pa-perf/src/corpus/rows.rs+++ b/crates/pa-perf/src/corpus/rows.rs@@ -118,8 +118,16 @@ pub(crate) struct SessionSpec {pub marker: Option<String>,pub parent_session: Option<String>,pub rlm_depth: Option<u64>,+ /// Prefix every generated text block with its ordinal (`ORDINAL_PREFIX` ++ /// n, counting up in transcript order) so a scrolled screen names how far+ /// up it is; the marker turn stays untagged.+ pub ordinals: bool,}+/// The block-ordinal token: `ord` + the ordinal's digits. No generated+/// word contains `ord`, so the token is unambiguous on any screen.+pub const ORDINAL_PREFIX: &str = "ord";+/// What one generated session produced.pub(crate) struct Written {pub rows: usize,@@ -321,6 +329,11 @@ pub(crate) fn write_session_file(// The conversation: fill to `rows`, keeping a compaction boundary near// 60% for long sessions and the marker on the final user message.let target = spec.rows as u64;+ let mut ordinals = spec.ordinals.then_some(0u64..);+ let mut tag = |text: String| match ordinals.as_mut().and_then(Iterator::next) {+ Some(ordinal) => format!("{ORDINAL_PREFIX}{ordinal} {text}"),+ None => text,+ };let mut compaction_at = (target >= 1500).then(|| target * 6 / 10);let mut last_user_text = String::new();while rows.rows_written() < target {@@ -348,7 +361,7 @@ pub(crate) fn write_session_file(let marker_turn = spec.marker.is_some() && target - rows.rows_written() <= 4;let text = match &spec.marker {Some(marker) if marker_turn => marker.clone(),- _ => user_text(rng),+ _ => tag(user_text(rng)),};let content = if rng.chance(30) {UserContent::Blocks(vec![UserContentBlock::Text(TextContent {@@ -387,7 +400,7 @@ pub(crate) fn write_session_file(let mut blocks = Vec::new();if !marker_turn && rng.chance(60) {blocks.push(AssistantContentBlock::Thinking(ThinkingContent {- thinking: assistant_text(rng),+ thinking: tag(assistant_text(rng)),thinking_signature: None,redacted: None,rest: serde_json::Map::new(),@@ -397,7 +410,7 @@ pub(crate) fn write_session_file(text: if marker_turn {"Done.".to_string()} else {- assistant_text(rng)+ tag(assistant_text(rng))},text_signature: None,rest: serde_json::Map::new(),@@ -456,7 +469,7 @@ pub(crate) fn write_session_file(tool_call_id: call_id,tool_name: name.to_string(),content: vec![UserContentBlock::Text(TextContent {- text: tool_output(rng),+ text: tag(tool_output(rng)),text_signature: None,rest: serde_json::Map::new(),})],@@ -505,6 +518,7 @@ mod tests {marker: Some("final user marker".to_string()),parent_session: None,rlm_depth: Some(0),+ ordinals: false,}}diff --git a/crates/pa-perf/src/report.rs b/crates/pa-perf/src/report.rsindex 4dfd2993dc..4a2e668a90 100644--- a/crates/pa-perf/src/report.rs+++ b/crates/pa-perf/src/report.rs@@ -45,14 +45,34 @@ pub struct Report {#[must_use]pub fn report(results: &[BenchResult], baseline: &Baseline) -> Report {let mut rows = Vec::new();+ let mut notes = Vec::new();let mut regressions = Vec::new();for result in results {let key = format!("{}/{}", key_of(result.scenario), result.profile);let median = result.median_ms;+ let first_paint = result+ .first_paint+ .as_ref()+ .map_or("—".to_string(), |series| {+ format!("{:.1}", series.median_ms)+ });+ let worst_gap = result.worst_gap.as_ref().map_or("—".to_string(), |series| {+ format!("{:.1} / {:.1}", series.p95_ms, series.max_ms)+ });+ if let Some(wakeups) = &result.wakeups {+ let mut sorted = wakeups.clone();+ sorted.sort_unstable();+ notes.push(format!(+ "- {key}: {median:.0} ms CPU, {} context switches (median of {} run(s); {})",+ sorted[sorted.len() / 2],+ sorted.len(),+ result.needle+ ));+ }+ let p95 = result.p95_ms;let Some(entry) = baseline.entries.get(&key) else {rows.push(format!(- "| {} | {median:.1} | {:.1} | — | — | (no baseline) |",- key, result.p95_ms+ "| {key} | {median:.1} | {p95:.1} | {first_paint} | {worst_gap} | — | — | (no baseline) |"));continue;};@@ -66,17 +86,19 @@ pub fn report(results: &[BenchResult], baseline: &Baseline) -> Report {format!("{:+.0}%", delta * 100.0)};rows.push(format!(- "| {} | {median:.1} | {:.1} | {:.1} | {:+.0}% | {verdict} |",- key,- result.p95_ms,+ "| {key} | {median:.1} | {p95:.1} | {first_paint} | {worst_gap} | {:.1} | {:+.0}% | {verdict} |",entry.median_ms,delta * 100.0));}- let markdown = format!(- "| scenario/profile | median ms | p95 ms | baseline ms | delta | verdict |\n|---|---|---|---|---|---|\n{}",+ let mut markdown = format!(+ "| scenario/profile | median ms | p95 ms | first-paint median ms | worst gap ms (p95 / max) | baseline ms | delta | verdict |\n|---|---|---|---|---|---|---|---|\n{}",rows.join("\n"));+ if !notes.is_empty() {+ markdown.push_str("\n\n");+ markdown.push_str(¬es.join("\n"));+ }Report {markdown,regressions,diff --git a/crates/pa-perf/tests/report_regression.rs b/crates/pa-perf/tests/report_regression.rsindex e1007c471b..a7066b1781 100644--- a/crates/pa-perf/tests/report_regression.rs+++ b/crates/pa-perf/tests/report_regression.rs@@ -24,6 +24,9 @@ fn result(scenario: Scenario, median_ms: f64) -> BenchResult {child_sessions: 12,archive: 3,},+ first_paint: None,+ worst_gap: None,+ wakeups: None,}}@@ -72,3 +75,24 @@ fn a_missing_baseline_prints_no_baseline() {assert!(report.markdown.contains("open-session/small"));assert!(report.markdown.contains("(no baseline)"));}++#[test]+fn results_written_before_the_secondary_series_still_report() {+ let old = serde_json::json!({+ "schema": 1,+ "scenario": "agents-view",+ "profile": "small",+ "runs": 1,+ "samples_ms": [30.0],+ "median_ms": 30.0,+ "p95_ms": 30.0,+ "needle": "",+ "binary": "",+ "corpus": { "root": "", "sessions": 10, "total_rows": 3500, "child_sessions": 12, "archive": 3 },+ });+ let parsed: BenchResult = serde_json::from_value(old).expect("an old result file parses");+ let report = report(&[parsed], &baseline(1_000.0, false));+ assert!(report+ .markdown+ .contains("| agents-view/small | 30.0 | 30.0 | — | — |"));+}