Code at the speed of thought – Zed is a high-performance, multiplayer code editor from the creators of Atom and Tree-sitter.
gpui: Add bench_metrics for multi-metric Criterion benchmarks (#64753)
Anthony Eide683fd7b465e
diff --git a/.agents/skills/gpui-bench/SKILL.md b/.agents/skills/gpui-bench/SKILL.mdindex 8aa142b8364..8a1a2777835 100644--- a/.agents/skills/gpui-bench/SKILL.md+++ b/.agents/skills/gpui-bench/SKILL.md@@ -132,6 +132,58 @@ The attribute supports options such as:Inspect the current macro before depending on an option because the API is evolving.+Register benchmark functions with `gpui::bench_group!` and `gpui::bench_main!` (drop-in for Criterion's `criterion_group!`/`criterion_main!`, including the `name = ...; config = ...; targets = ...` form). The generated functions take `&mut criterion::Criterion<gpui::BenchMeasurement>`; `bench_group!` supplies that measurement from the environment.++### Measurements: wall time, hardware counters, and rusage++`gpui::BenchMeasurement` (from the gpui-independent `bench_metrics` crate) records several metrics per Criterion sample. Criterion analyzes one of them (the *primary*, shown on its `time:` line with confidence intervals and baseline comparison); the rest are *secondaries*, printed per iteration in the GPUI bench report as median/min/max. `BENCH_MEASUREMENT` selects the arrangement:++| `BENCH_MEASUREMENT` | Criterion analyzes | Reported as secondaries |+|---|---|---|+| unset (default) | wall time | every counter the machine supports (below) |+| `instructions` | process-wide retired instructions; fails fast if counters are unavailable | the rest, plus wall time |+| `foreground-instructions` | the benchmark thread's retired instructions; fails fast if unavailable | the rest, plus wall time |+| `wall-time` | wall time | nothing; no counters are opened |++For a CI gate on frame cost prefer `foreground-instructions`: it excludes background parsing and GPU-driver submission threads, whose work varies run to run, and on Apple Silicon the process-wide count includes driver jitter that widens the interval (about ±0.6% versus ±0.01% for the foreground thread).++Default secondaries, each skipped with a one-time note when unavailable:++- **instructions** (process-wide) and **foreground instructions** (the benchmark thread only, where GPUI polls tasks, lays out, paints, and submits). Near-deterministic, so a ~0.1% spread is typical and small regressions are detectable. Blind to waits, memory stalls, and GPU shader work. Their ratio, **foreground share of instructions**, shows work moving between the foreground and background threads even when the total is flat; with a headless renderer it also exposes driver-thread submission work (about 7% of a small Metal frame).+- **cycles** and **IPC** (`instructions / cycles`). Falling IPC with flat instructions means worse cache or branch behavior, not more work.+- **branch misses** and **cache misses** (last-level), for diagnosing an IPC drop. Linux only.+- **foreground context switches**: voluntary switches on the benchmark thread per iteration, i.e. how many times a frame blocked on a lock, channel, or fence. This is the metric that catches the waits instructions cannot see. Linux only.+- **page faults**: minor faults per iteration, a privilege-free proxy for fresh memory touched.++Run one benchmark process per machine at a time. Hardware counters are per task, so instruction counts stay correct next to a concurrent `cargo bench`, but wall time, IPC, and cache misses share the hardware and degrade, and two processes requesting counters can exceed a core's counter slots, at which point the kernel multiplexes and the counts become scaled estimates (a warning is printed once). Within one process, benchmarks already run sequentially; process-wide counters can still attribute a previous benchmark's trailing background work to the next benchmark's first sample, which the foreground-thread metrics are immune to.++Platform support: on Linux, hardware counters come from `perf_event_open` and need `CAP_PERFMON` or `/proc/sys/kernel/perf_event_paranoid` at 2 or below; Docker's default seccomp profile and many cloud VMs block it, and the default mode then reports wall time plus the `getrusage` metrics. On Apple Silicon, instructions and cycles (both scopes) come from `proc_pid_rusage` and `thread_selfcounts` with no privileges; Intel Macs have no source. Criterion baselines are keyed by benchmark id, not measurement, so use distinct `--save-baseline` names when switching the primary on one machine.++Never present instructions alone for a latency claim; pair them with wall time and the frame metrics from the same run. Other Criterion measurements plug in with `criterion_group!` and `config = Criterion::default().with_measurement(gpui::BenchMeasurement::new(primary).with_secondary("name", other))`.++#### Benchmarks outside GPUI++`bench_metrics` has no GPUI dependency, so any Criterion benchmark in the workspace (`rope`, `sum_tree`, `text`, ...) gets the same metrics and the same `BENCH_MEASUREMENT` switch. Add `bench_metrics.workspace = true` to `[dev-dependencies]`, take `&mut Criterion<bench_metrics::BenchMeasurement>` in the benchmark functions, and register them with:++```rust+criterion_group! {+ name = benches;+ config = Criterion::default()+ .with_measurement(bench_metrics::BenchMeasurement::from_env_or_exit());+ targets = rope_benchmarks+}+```++That alone switches Criterion's analyzed metric with `BENCH_MEASUREMENT=instructions`. To also print the per-iteration secondaries, run each routine through a `MetricReport`, which counts iterations for you:++```rust+let report = bench_metrics::MetricReport::new();+group.bench_function("append", |b| report.iter(b, || rope.append(small.clone())));+report.print(" ");+```++Criterion's `Throughput` lines keep working alongside. Page faults per iteration are a useful allocation-pressure signal for data-structure crates even without a counting allocator.+### `bench_iter`: synchronous functions and compute workUse `bench_iter` for synchronous application or compute code:@@ -197,8 +249,9 @@ On macOS, the headless renderer uses Metal without showing a window. `bench_rendOn platforms without a headless renderer, GPUI still performs CPU-side window work, but presenting discards the scene and does not measure real GPU submission. State this limitation in results.-Depending on the pinned revision, `BenchReport` can include:+`BenchReport` is printed once per benchmark, and once per input for `inputs = ...` benchmarks, labeled with the Criterion benchmark id so the two can be correlated. Depending on the pinned revision, it can include:+- secondary measurements per iteration (median, min, max): instructions, foreground instructions, cycles, IPC, branch and cache misses, foreground context switches, page faults;- dirty-to-draw duration;- draw and present intervals;- invalidations per frame;@@ -359,6 +412,7 @@ Before accepting a GPUI benchmark, verify:- [ ] Cache state is controlled and documented.- [ ] A responsiveness signal competes with the heavy work when foreground starvation matters.- [ ] Frame metrics and completion throughput are both reported.+- [ ] Instructions and IPC (Linux and Apple Silicon) and foreground context switches (Linux) are reported next to wall time, and none is presented alone as the whole result.- [ ] Final correctness, ordering, and work counts are asserted.- [ ] The workload completes under bounded smoke and measured runs.- [ ] The same benchmark code runs on baseline and candidate.diff --git a/.rules b/.rulesindex c0d62fcdbe6..4f07c39e70c 100644--- a/.rules+++ b/.rules@@ -148,6 +148,7 @@ When an agent opens or updates a pull request, it must:- Use one bullet under `Release Notes:`:- `- Added ...`, `- Fixed ...`, or `- Improved ...` for user-facing changes, or- `- N/A` for docs-only and other non-user-facing changes.+- Changes to `gpui` (including its benchmark and test harnesses) must have a bullet prefixed with `[GPUI]` describing the improvement even when nothing is user-facing in Zed itself, since GPUI is consumed as a library (CI enforces this), for example `- [GPUI] Improved benchmarks to report retired CPU instructions alongside wall time on Linux`.- Format release notes exactly with a blank line after the heading, for example:```diff --git a/Cargo.lock b/Cargo.lockindex 1ed2119bc9e..633872e6929 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -657,6 +657,15 @@ dependencies = ["equator",]+[[package]]+name = "alloca"+version = "0.4.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "e5a7d05ea6aea7e9e64d25b9156ba2fee3fdd659e34e41063cd2fc7cd020d7f4"+dependencies = [+ "cc",+]+[[package]]name = "allocator-api2"version = "0.2.21"@@ -2112,6 +2121,16 @@ version = "0.5.2"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "3a8241f3ebb85c056b509d4327ad0358fbbba6ffb340bf388f26350aeda225b1"+[[package]]+name = "bench_metrics"+version = "0.1.0"+dependencies = [+ "anyhow",+ "criterion",+ "libc",+ "perf-event2",+]+[[package]]name = "benchmarks"version = "0.1.0"@@ -2187,7 +2206,7 @@ dependencies = ["bitflags 2.13.1","cexpr","clang-sys",- "itertools 0.11.0",+ "itertools 0.13.0","log","prettyplease","proc-macro2",@@ -2592,6 +2611,12 @@ dependencies = ["pkg-config",]+[[package]]+name = "c-enum"+version = "0.2.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "cd17eb909a8c6a894926bfcc3400a4bb0e732f5a57d37b1f14e8b29e329bace8"+[[package]]name = "cached"version = "0.56.0"@@ -4341,25 +4366,24 @@ dependencies = [[[package]]name = "criterion"-version = "0.5.1"+version = "0.8.2"source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "f2b12d017a929603d80db1831cd3a24082f8137ce19c69e6447f54f5fc8d692f"+checksum = "950046b2aa2492f9a536f5f4f9a3de7b9e2476e575e05bd6c333371add4d98f3"dependencies = [+ "alloca","anes","cast","ciborium","clap","criterion-plot",- "is-terminal",- "itertools 0.10.5",+ "itertools 0.13.0","num-traits",- "once_cell","oorandom",+ "page_size","plotters","rayon","regex","serde",- "serde_derive","serde_json","tinytemplate","walkdir",@@ -4367,12 +4391,12 @@ dependencies = [[[package]]name = "criterion-plot"-version = "0.5.0"+version = "0.8.2"source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "6b50826342786a51a89e2da3a28f1c32b06e387201bc2d19791f622c673706b1"+checksum = "d8d80a2f4f5b554395e47b5d8305bc3d27813bacb73493eb1001e8f76dae29ea"dependencies = ["cast",- "itertools 0.10.5",+ "itertools 0.13.0",][[package]]@@ -7507,6 +7531,7 @@ dependencies = ["async-channel 2.5.0","async-task","backtrace",+ "bench_metrics","bindgen","bitflags 2.13.1","chrono",@@ -9131,9 +9156,9 @@ dependencies = [[[package]]name = "itertools"-version = "0.11.0"+version = "0.13.0"source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "b1c173a5686ce8bfa551b3563d0c2170bf24ca44da99c7ca4bfdab5418c3fe57"+checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186"dependencies = ["either",]@@ -12965,6 +12990,41 @@ dependencies = ["serde_json",]+[[package]]+name = "perf-event-data"+version = "0.1.8"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "575828d9d7d205188048eb1508560607a03d21eafdbba47b8cade1736c1c28e1"+dependencies = [+ "bitflags 2.13.1",+ "c-enum",+ "perf-event-open-sys2",+]++[[package]]+name = "perf-event-open-sys2"+version = "5.0.6"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "d9c25955321465255e437600b54296983fab1feac2cd0c38958adeb26dbae49e"+dependencies = [+ "libc",+ "memoffset",+]++[[package]]+name = "perf-event2"+version = "0.7.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "0939b8fad77dfaeb29ebbd35faaeaadbf833167f30975f1b8993bbba09ea0a0f"+dependencies = [+ "bitflags 2.13.1",+ "c-enum",+ "libc",+ "memmap2",+ "perf-event-data",+ "perf-event-open-sys2",+]+[[package]]name = "pest"version = "2.8.3"diff --git a/Cargo.toml b/Cargo.tomlindex 0e5908bbe45..41db7bfee5a 100644--- a/Cargo.toml+++ b/Cargo.toml@@ -20,6 +20,7 @@ members = ["crates/auto_update_ui","crates/aws_http_client","crates/bedrock",+ "crates/bench_metrics","crates/benchmarks","crates/breadcrumbs","crates/buffer_diff",@@ -291,6 +292,7 @@ anthropic = { path = "crates/anthropic" }askpass = { path = "crates/askpass" }assets = { path = "crates/assets" }audio = { path = "crates/audio" }+bench_metrics = { path = "crates/bench_metrics" }auto_update = { path = "crates/auto_update" }auto_update_ui = { path = "crates/auto_update_ui" }aws_http_client = { path = "crates/aws_http_client" }@@ -597,7 +599,8 @@ core-text = "21"core-video = { version = "0.5.2", features = ["metal"] }cpal = "0.17"crash-handler = "0.8"-criterion = { version = "0.5", features = ["html_reports"] }+criterion = { version = "0.8.2", features = ["html_reports"] }+perf-event = { package = "perf-event2", version = "0.7.4" }crossbeam = "0.8.4"ctor = "1.0.12"dap-types = { git = "https://github.com/zed-industries/dap-types", rev = "1b461b310481d01e02b2603c16d7144b926339f8" }diff --git a/crates/bench_metrics/Cargo.toml b/crates/bench_metrics/Cargo.tomlnew file mode 100644index 00000000000..8715ba0da5a--- /dev/null+++ b/crates/bench_metrics/Cargo.toml@@ -0,0 +1,24 @@+[package]+name = "bench_metrics"+version = "0.1.0"+edition.workspace = true+publish = false+license = "Apache-2.0"++[lints]+workspace = true++[lib]+path = "src/bench_metrics.rs"+doctest = false++[features]+test-support = []++[dependencies]+anyhow.workspace = true+criterion.workspace = true+libc.workspace = true++[target.'cfg(target_os = "linux")'.dependencies]+perf-event.workspace = truediff --git a/crates/bench_metrics/LICENSE-APACHE b/crates/bench_metrics/LICENSE-APACHEnew file mode 120000index 00000000000..1cd601d0a3a--- /dev/null+++ b/crates/bench_metrics/LICENSE-APACHE@@ -0,0 +1 @@+../../LICENSE-APACHE\ No newline at end of filediff --git a/crates/bench_metrics/src/bench_metrics.rs b/crates/bench_metrics/src/bench_metrics.rsnew file mode 100644index 00000000000..3d157b629d3--- /dev/null+++ b/crates/bench_metrics/src/bench_metrics.rs@@ -0,0 +1,1801 @@+//! Criterion measurements beyond wall time, and a way to report several of+//! them from one benchmark run.+//!+//! Criterion analyzes exactly one scalar per benchmark. [`BenchMeasurement`]+//! wraps that *primary* measurement together with any number of *secondary*+//! measurements taken over the same iterations, whose per-iteration values a+//! benchmark harness records into a [`MetricReport`]. The measurements here+//! are process- or thread-scoped counters: Linux hardware performance+//! counters ([`HardwareCounter`]) and `getrusage` statistics+//! ([`ResourceCounter`]). Nothing in this crate depends on GPUI, so any+//! Criterion benchmark can use it:+//!+//! ```ignore+//! fn benches(c: &mut Criterion<bench_metrics::BenchMeasurement>) {+//! let report = bench_metrics::MetricReport::new();+//! c.bench_function("append", |b| report.iter(b, || work()));+//! report.print(" ");+//! }+//!+//! criterion_group! {+//! name = group;+//! config = Criterion::default()+//! .with_measurement(bench_metrics::BenchMeasurement::from_env_or_exit());+//! targets = benches+//! }+//! criterion_main!(group);+//! ```++use std::{+ cell::{Cell, RefCell},+ rc::Rc,+};++use anyhow::{Result, anyhow};+use criterion::measurement::{Measurement, ValueFormatter};++/// The environment variable [`BenchMeasurement::from_env`] reads.+pub const BENCH_MEASUREMENT_ENV_VAR: &str = "BENCH_MEASUREMENT";++/// Which threads a counter covers.+#[derive(Clone, Copy, Debug, Eq, PartialEq)]+pub enum MetricScope {+ /// Every thread in the benchmark process, including threads created+ /// after the counter was opened.+ Process,+ /// Only the thread that constructed the counter. In a Criterion benchmark+ /// that is the thread running the benchmark routine; in GPUI benchmarks it+ /// is also the foreground thread where task polls, layout, and paint run.+ CallingThread,+}++impl MetricScope {+ fn label(self) -> &'static str {+ match self {+ Self::Process => "the benchmark process",+ Self::CallingThread => "the calling thread",+ }+ }+}++/// Formats event counts with K/M/G prefixes and a unit noun.+#[derive(Clone, Copy)]+pub struct CountFormatter {+ unit: &'static str,+}++impl CountFormatter {+ /// Creates a formatter whose values are counts of `unit`, e.g. `"cycles"`.+ pub const fn new(unit: &'static str) -> Self {+ Self { unit }+ }+}++impl ValueFormatter for CountFormatter {+ fn scale_values(&self, typical_value: f64, values: &mut [f64]) -> &'static str {+ let (scale, prefix) = if typical_value < 1_000.0 {+ (1.0, "")+ } else if typical_value < 1_000_000.0 {+ (1_000.0, "K ")+ } else if typical_value < 1_000_000_000.0 {+ (1_000_000.0, "M ")+ } else {+ (1_000_000_000.0, "G ")+ };+ for value in values {+ *value /= scale;+ }+ // `&'static str` is required, so the prefixed unit is interned once.+ intern(prefix, self.unit)+ }++ fn scale_throughputs(+ &self,+ _typical_value: f64,+ throughput: &criterion::Throughput,+ values: &mut [f64],+ ) -> &'static str {+ let (units, per) = match throughput {+ criterion::Throughput::Bits(units) => (*units, "/bit"),+ criterion::Throughput::Bytes(units) | criterion::Throughput::BytesDecimal(units) => {+ (*units, "/byte")+ }+ criterion::Throughput::Elements(units)+ | criterion::Throughput::ElementsAndBytes {+ elements: units, ..+ } => (*units, "/element"),+ };+ for value in values {+ *value /= units as f64;+ }+ intern(self.unit, per)+ }++ fn scale_for_machines(&self, _values: &mut [f64]) -> &'static str {+ self.unit+ }+}++/// Returns a `'static` concatenation of two strings, leaking each distinct+/// combination once. Criterion's `ValueFormatter` requires `&'static str`+/// units, and the set of combinations is a handful of fixed labels.+fn intern(first: &str, second: &str) -> &'static str {+ use std::sync::{Mutex, OnceLock};++ static INTERNED: OnceLock<Mutex<Vec<&'static str>>> = OnceLock::new();+ let combined = format!("{first}{second}");+ let mut interned = INTERNED+ .get_or_init(Default::default)+ .lock()+ .unwrap_or_else(|poisoned| poisoned.into_inner());+ if let Some(existing) = interned.iter().find(|existing| **existing == combined) {+ return existing;+ }+ let leaked: &'static str = Box::leak(combined.into_boxed_str());+ interned.push(leaked);+ leaked+}++// ---------------------------------------------------------------------------+// Hardware performance counters+// ---------------------------------------------------------------------------++/// A CPU event counted by [`HardwareCounter`].+#[derive(Clone, Copy, Debug, Eq, PartialEq)]+pub enum HardwareEvent {+ /// Retired userspace instructions: near-deterministic, so small+ /// regressions are detectable, but blind to waits and memory stalls.+ Instructions,+ /// Userspace CPU cycles. `instructions / cycles` is IPC; a falling IPC with+ /// flat instructions means worse cache or branch behavior.+ Cycles,+ /// Mispredicted branches. Linux only.+ BranchMisses,+ /// Last-level cache misses, i.e. accesses served from memory. Linux only.+ CacheMisses,+ /// Last-level cache accesses. Linux only.+ CacheReferences,+}++impl HardwareEvent {+ /// The unit noun used in reports.+ pub fn unit(self) -> &'static str {+ match self {+ Self::Instructions => "instructions",+ Self::Cycles => "cycles",+ Self::BranchMisses => "branch misses",+ Self::CacheMisses => "cache misses",+ Self::CacheReferences => "cache references",+ }+ }++ /// The event's `perf` name, also used in error messages on every platform.+ fn name(self) -> &'static str {+ match self {+ Self::Instructions => "instructions",+ Self::Cycles => "cpu-cycles",+ Self::BranchMisses => "branch-misses",+ Self::CacheMisses => "cache-misses",+ Self::CacheReferences => "cache-references",+ }+ }+}++/// Criterion measurement for a CPU hardware event.+///+/// The counter is opened once, at construction; each `Measurement::start`/`end`+/// reads it and reports the difference, so a sample costs one read per counter+/// (tens to hundreds of nanoseconds). Which events and scopes are available+/// depends on the platform backend:+///+/// * **Linux** (`perf_event_open`): every event, both scopes. Needs+/// `CAP_PERFMON` or `perf_event_paranoid <= 2`. See [`perf`] for how+/// process scope, hybrid CPUs, and multiplexing are handled.+/// * **macOS** (`proc_pid_rusage` and `thread_selfcounts`): instructions and+/// cycles, both scopes, on Apple Silicon without privileges. Intel Macs and+/// the cache/branch events are unsupported.+///+/// Unsupported combinations fail at construction with an actionable error, so+/// callers can skip them (as [`BenchMeasurement::from_env`] does) rather than+/// measure zeros.+pub struct HardwareCounter {+ event: HardwareEvent,+ scope: MetricScope,+ backend: backend::Counter,+ formatter: CountFormatter,+}++impl HardwareCounter {+ /// Opens a counter for `event` over `scope`.+ pub fn new(event: HardwareEvent, scope: MetricScope) -> Result<Self> {+ let backend = backend::Counter::open(event, scope)?;+ let this = Self {+ event,+ scope,+ backend,+ formatter: CountFormatter::new(event.unit()),+ };+ this.verify_counting()?;+ Ok(this)+ }++ /// The counted event.+ pub fn event(&self) -> HardwareEvent {+ self.event+ }++ /// The covered threads.+ pub fn scope(&self) -> MetricScope {+ self.scope+ }++ /// Confirms the counter advances for work on this thread. A virtual machine+ /// can accept the open yet never schedule the event, and Intel Macs report+ /// zero; failing here is clearer than measuring zeros.+ fn verify_counting(&self) -> Result<()> {+ let before = self.backend.snapshot();+ std::hint::black_box((0..10_000_u64).fold(0_u64, |total, value| total.wrapping_add(value)));+ let after = self.backend.snapshot();+ if self.backend.advanced(&before, &after) {+ Ok(())+ } else {+ Err(anyhow!(+ "the {} counter for {} opened but does not advance; hardware performance \+ counters are unavailable to this machine or CI runner",+ self.event.name(),+ self.scope.label()+ ))+ }+ }+}++impl Measurement for HardwareCounter {+ type Intermediate = backend::Snapshot;+ type Value = f64;++ fn start(&self) -> Self::Intermediate {+ self.backend.snapshot()+ }++ fn end(&self, start: Self::Intermediate) -> Self::Value {+ let end = self.backend.snapshot();+ self.backend.delta(&start, &end)+ }++ fn add(&self, first: &Self::Value, second: &Self::Value) -> Self::Value {+ first + second+ }++ fn zero(&self) -> Self::Value {+ 0.0+ }++ fn to_f64(&self, value: &Self::Value) -> f64 {+ *value+ }++ fn formatter(&self) -> &dyn ValueFormatter {+ &self.formatter+ }+}++#[cfg(target_os = "macos")]+use darwin as backend;+/// The platform backend behind [`HardwareCounter`]. Each provides a `Counter`+/// with `open`, `snapshot`, `advanced`, and `delta`, and a `Snapshot` type.+#[cfg(target_os = "linux")]+use perf as backend;+#[cfg(not(any(target_os = "linux", target_os = "macos")))]+use unsupported as backend;++/// Linux backend: `perf_event_open` through the `perf-event2` crate, whose+/// items are referenced through the `perf_event` path so this module's own+/// helpers are distinguishable from the crate's.+///+/// With [`MetricScope::Process`], one inherited counter is opened for every+/// thread that exists at construction. Inheritance extends each counter to+/// the threads that thread later creates, and live descendants are summed on+/// read, so the whole process is covered for the counter's lifetime. With+/// [`MetricScope::CallingThread`], a single non-inherited counter observes the+/// constructing thread only.+///+/// Hybrid CPUs (separate performance and efficiency core PMUs) need one event+/// per PMU, of which only the one matching the CPU a thread runs on can be+/// scheduled; each PMU's encoding is read from sysfs. If the kernel+/// multiplexes a conventional counter with other profiling, its count is+/// scaled by `time_enabled / time_running` and a warning is printed once.+#[cfg(target_os = "linux")]+pub mod perf {+ use super::*;++ pub(super) struct Counter {+ event: HardwareEvent,+ counters: RefCell<Vec<PerfCounter>>,+ multiplexing_reported: Cell<bool>,+ }++ struct PerfCounter {+ counter: perf_event::Counter,+ /// Generic events on conventional CPUs are scaled when multiplexed.+ /// Hybrid per-PMU events are not, since a thread on the other core type+ /// shows as "not running" for that PMU by design.+ scale_for_multiplexing: bool,+ }++ /// Every counter's state at one instant.+ #[doc(hidden)]+ pub struct Snapshot(Vec<CounterState>);++ #[derive(Clone, Copy)]+ struct CounterState {+ count: u64,+ time_enabled: std::time::Duration,+ time_running: std::time::Duration,+ }++ impl Counter {+ pub(super) fn open(event: HardwareEvent, scope: MetricScope) -> Result<Self> {+ let hybrid_events = perf::hybrid_pmu_events(event)?;+ let mut counters = Vec::new();+ match scope {+ MetricScope::CallingThread => {+ Self::open_target(+ event,+ CounterTarget::CallingThread,+ &hybrid_events,+ &mut counters,+ )?;+ }+ MetricScope::Process => {+ for thread_id in perf::process_thread_ids()? {+ Self::open_target(+ event,+ CounterTarget::ThreadAndDescendants(thread_id),+ &hybrid_events,+ &mut counters,+ )?;+ }+ }+ }+ if counters.is_empty() {+ return Err(anyhow!(+ "no Linux {} counter could be opened for {}: the CPU does not expose this event",+ event.name(),+ scope.label()+ ));+ }+ for counter in &mut counters {+ counter+ .counter+ .enable()+ .map_err(|error| Self::operation_error(event, "enable", error))?;+ }+ Ok(Self {+ event,+ counters: RefCell::new(counters),+ multiplexing_reported: Cell::new(false),+ })+ }++ fn open_target(+ event: HardwareEvent,+ target: CounterTarget,+ hybrid_events: &[(u32, u64)],+ counters: &mut Vec<PerfCounter>,+ ) -> Result<()> {+ let opened: Vec<(std::io::Result<perf_event::Counter>, bool)> =+ if hybrid_events.is_empty() {+ let mut builder = perf_event::Builder::new(generic_event(event));+ target.configure(&mut builder);+ vec![(builder.build(), true)]+ } else {+ hybrid_events+ .iter()+ .map(|&(pmu_type, config)| {+ let mut builder =+ perf_event::Builder::new(perf_event::events::Raw::new(config));+ builder.attrs_mut().type_ = pmu_type;+ target.configure(&mut builder);+ (builder.build(), false)+ })+ .collect()+ };+ for (result, scale_for_multiplexing) in opened {+ match result {+ Ok(counter) => counters.push(PerfCounter {+ counter,+ scale_for_multiplexing,+ }),+ // The thread exited between enumeration and open.+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => {}+ Err(error) => return Err(Self::open_error(event, target, error)),+ }+ }+ Ok(())+ }++ fn open_error(+ event: HardwareEvent,+ target: CounterTarget,+ error: std::io::Error,+ ) -> anyhow::Error {+ anyhow!(+ "failed to open Linux {} counter for {target}: {error}. Grant this benchmark \+ CAP_PERFMON or adjust /proc/sys/kernel/perf_event_paranoid according to your CI \+ security policy",+ event.name()+ )+ }++ fn operation_error(+ event: HardwareEvent,+ operation: &str,+ error: std::io::Error,+ ) -> anyhow::Error {+ anyhow!("Linux {} counter {operation} failed: {error}", event.name())+ }++ pub(super) fn snapshot(&self) -> Snapshot {+ Snapshot(+ self.counters+ .borrow_mut()+ .iter_mut()+ .map(|counter| {+ let data = counter.counter.read_full().unwrap_or_else(|error| {+ panic!("{}", Self::operation_error(self.event, "read", error))+ });+ CounterState {+ count: data.count(),+ time_enabled: data+ .time_enabled()+ .expect("time-enabled counter data was requested"),+ time_running: data+ .time_running()+ .expect("time-running counter data was requested"),+ }+ })+ .collect(),+ )+ }++ pub(super) fn advanced(&self, before: &Snapshot, after: &Snapshot) -> bool {+ before+ .0+ .iter()+ .zip(&after.0)+ .any(|(before, after)| after.time_running > before.time_running)+ }++ pub(super) fn delta(&self, start: &Snapshot, end: &Snapshot) -> f64 {+ let counters = self.counters.borrow();+ let mut total = 0.0;+ let mut any_counter_ran = false;+ for ((counter, start), end) in counters.iter().zip(&start.0).zip(&end.0) {+ let count = end.count.saturating_sub(start.count) as f64;+ let time_enabled = end.time_enabled.saturating_sub(start.time_enabled);+ let time_running = end.time_running.saturating_sub(start.time_running);+ if time_running.is_zero() {+ // Process-wide coverage includes threads that slept for the+ // whole interval, and hybrid PMUs whose core type this thread+ // never ran on.+ continue;+ }+ any_counter_ran = true;+ if counter.scale_for_multiplexing && time_running < time_enabled {+ if !self.multiplexing_reported.replace(true) {+ eprintln!(+ "{} counters were multiplexed by the kernel; counts are scaled using \+ time_enabled/time_running",+ self.event.name()+ );+ }+ total += count * time_enabled.as_secs_f64() / time_running.as_secs_f64();+ } else {+ total += count;+ }+ }+ if !any_counter_ran {+ panic!(+ "Linux {} counters did not run during the measured interval; ensure hardware \+ performance counters are available to this machine or CI runner",+ self.event.name()+ );+ }+ total+ }+ }++ fn generic_event(event: HardwareEvent) -> perf_event::events::Hardware {+ use perf_event::events::Hardware;+ match event {+ HardwareEvent::Instructions => Hardware::INSTRUCTIONS,+ HardwareEvent::Cycles => Hardware::CPU_CYCLES,+ HardwareEvent::BranchMisses => Hardware::BRANCH_MISSES,+ HardwareEvent::CacheMisses => Hardware::CACHE_MISSES,+ HardwareEvent::CacheReferences => Hardware::CACHE_REFERENCES,+ }+ }++ #[derive(Clone, Copy)]+ enum CounterTarget {+ /// `pid = 0` to `perf_event_open`: the calling thread, with no inheritance.+ CallingThread,+ /// A specific thread plus every thread it creates afterwards.+ ThreadAndDescendants(i32),+ }++ impl CounterTarget {+ fn configure(self, builder: &mut perf_event::Builder) {+ use perf_event::ReadFormat;++ match self {+ Self::CallingThread => builder.observe_self().inherit(false),+ Self::ThreadAndDescendants(thread_id) => {+ builder.observe_pid(thread_id).inherit(true)+ }+ };+ builder.read_format(ReadFormat::TOTAL_TIME_ENABLED | ReadFormat::TOTAL_TIME_RUNNING);+ }+ }++ impl std::fmt::Display for CounterTarget {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ match self {+ Self::CallingThread => write!(f, "the calling thread"),+ Self::ThreadAndDescendants(thread_id) => write!(f, "thread {thread_id}"),+ }+ }+ }++ fn process_thread_ids() -> Result<Vec<i32>> {+ let mut thread_ids = Vec::new();+ for entry in std::fs::read_dir("/proc/self/task")+ .map_err(|error| anyhow!("failed to enumerate benchmark process threads: {error}"))?+ {+ let entry = entry.map_err(|error| {+ anyhow!("failed to enumerate a benchmark process thread: {error}")+ })?;+ if let Some(thread_id) = entry+ .file_name()+ .to_str()+ .and_then(|thread_id| thread_id.parse().ok())+ {+ thread_ids.push(thread_id);+ }+ }+ Ok(thread_ids)+ }++ /// Returns `(pmu_type, config)` for `event` on each hybrid CPU PMU, or an+ /// empty list on conventional CPUs where the generic hardware event suffices.+ ///+ /// A PMU (performance monitoring unit) is the per-core block of counter+ /// registers; Linux exposes each as a device under+ /// `/sys/bus/event_source/devices/`. Conventional CPUs have one, `cpu`, and+ /// the generic events target it. Hybrid CPUs have `cpu_core` (performance+ /// cores) and `cpu_atom` (efficiency cores) with distinct event encodings,+ /// and the kernel refuses the generic event for a task that may run on+ /// either, so one raw event per PMU is needed.+ ///+ /// Each PMU publishes its events as `term=value,...` strings and the bit+ /// layout of each term under `format/`, e.g. `cache-misses` is+ /// `event=0x2e,umask=0x41` with `event` at `config:0-7` and `umask` at+ /// `config:8-15`.+ fn hybrid_pmu_events(event: HardwareEvent) -> Result<Vec<(u32, u64)>> {+ use anyhow::Context as _;++ let mut events = Vec::new();+ for pmu in ["cpu_core", "cpu_atom"] {+ let pmu_path = std::path::Path::new("/sys/bus/event_source/devices").join(pmu);+ if !pmu_path.exists() {+ continue;+ }+ let pmu_type = std::fs::read_to_string(pmu_path.join("type"))+ .with_context(|| format!("failed to read Linux {pmu} performance-counter type"))?+ .trim()+ .parse::<u32>()+ .with_context(|| format!("invalid Linux {pmu} performance-counter type"))?;+ let encoding = std::fs::read_to_string(pmu_path.join("events").join(event.name()))+ .with_context(|| format!("failed to read Linux {pmu} {} event", event.name()))?;+ let mut config = 0_u64;+ for term in encoding.trim().split(',') {+ let (name, value) = term.split_once('=').unwrap_or((term, "1"));+ let value = value+ .strip_prefix("0x")+ .map_or_else(|| value.parse::<u64>(), |hex| u64::from_str_radix(hex, 16))+ .with_context(|| {+ format!("invalid Linux {pmu} {} event term {term:?}", event.name())+ })?;+ let layout = std::fs::read_to_string(pmu_path.join("format").join(name))+ .with_context(|| format!("failed to read Linux {pmu} format for {name:?}"))?;+ let bits = layout.trim().strip_prefix("config:").ok_or_else(|| {+ anyhow!(+ "unsupported Linux {pmu} format {:?} for {name:?}",+ layout.trim()+ )+ })?;+ let (low, high) = match bits.split_once('-') {+ Some((low, high)) => (low.parse::<u32>()?, high.parse::<u32>()?),+ None => {+ let bit = bits.parse::<u32>()?;+ (bit, bit)+ }+ };+ let width = high - low + 1;+ let mask = if width >= 64 {+ u64::MAX+ } else {+ (1_u64 << width) - 1+ };+ config |= (value & mask) << low;+ }+ events.push((pmu_type, config));+ }+ Ok(events)+ }++ #[cfg(test)]+ mod tests {+ use super::*;++ #[test]+ fn perf_permission_error_is_actionable() {+ let error = Counter::open_error(+ HardwareEvent::Instructions,+ CounterTarget::ThreadAndDescendants(42),+ std::io::Error::from(std::io::ErrorKind::PermissionDenied),+ );+ let message = error.to_string();+ assert!(message.contains("CAP_PERFMON"));+ assert!(message.contains("perf_event_paranoid"));+ assert!(message.contains("thread 42"));+ }+ }+}++/// macOS backend. Apple Silicon exposes retired instructions and cycles to+/// unprivileged processes through two XNU interfaces: `proc_pid_rusage` with+/// `RUSAGE_INFO_V4` for the whole process (every thread, including exited+/// ones), and `thread_selfcounts` for the calling thread. Both are plain+/// reads with no counter to open or close. Intel Macs report zero for both,+/// which [`HardwareCounter::new`] detects, and macOS virtual machines (CI+/// runners included) reject `thread_selfcounts` with `ENOTSUP` while still+/// serving `proc_pid_rusage`. Branch and cache events require the private+/// `kperf` framework and root, so they are unsupported here.+#[cfg(target_os = "macos")]+pub mod darwin {+ use super::*;++ pub(super) struct Counter {+ event: HardwareEvent,+ scope: MetricScope,+ }++ /// `[instructions, cycles]` at one instant.+ #[doc(hidden)]+ pub struct Snapshot([u64; 2]);++ /// The leading fields of `struct rusage_info_v4` from `<sys/resource.h>`,+ /// up to and including the two counters this backend reads. Only a+ /// prefix is declared: the kernel copies out `sizeof` the flavor's full+ /// struct, so the buffer below is padded to that size.+ #[repr(C)]+ struct RusageInfoV4Prefix {+ ri_uuid: [u8; 16],+ ri_user_time: u64,+ ri_system_time: u64,+ ri_pkg_idle_wkups: u64,+ ri_interrupt_wkups: u64,+ ri_pageins: u64,+ ri_wired_size: u64,+ ri_resident_size: u64,+ ri_phys_footprint: u64,+ ri_proc_start_abstime: u64,+ ri_proc_exit_abstime: u64,+ ri_child_user_time: u64,+ ri_child_system_time: u64,+ ri_child_pkg_idle_wkups: u64,+ ri_child_interrupt_wkups: u64,+ ri_child_pageins: u64,+ ri_child_elapsed_abstime: u64,+ ri_diskio_bytesread: u64,+ ri_diskio_byteswritten: u64,+ ri_cpu_time_qos_default: u64,+ ri_cpu_time_qos_maintenance: u64,+ ri_cpu_time_qos_background: u64,+ ri_cpu_time_qos_utility: u64,+ ri_cpu_time_qos_legacy: u64,+ ri_cpu_time_qos_user_initiated: u64,+ ri_cpu_time_qos_user_interactive: u64,+ ri_billed_system_time: u64,+ ri_serviced_system_time: u64,+ ri_logical_writes: u64,+ ri_lifetime_max_phys_footprint: u64,+ ri_instructions: u64,+ ri_cycles: u64,+ }++ /// `sizeof(struct rusage_info_v4)`: the prefix above plus the four trailing+ /// `u64` fields (`ri_billed_energy`, `ri_serviced_energy`,+ /// `ri_interval_max_phys_footprint`, `ri_runnable_time`).+ const RUSAGE_INFO_V4_SIZE: usize = std::mem::size_of::<RusageInfoV4Prefix>() + 4 * 8;+ const RUSAGE_INFO_V4: libc::c_int = 4;+ /// `THSC_CPI` in `<sys/thread_selfcounts.h>` (not in `libc`).+ const THREAD_SELFCOUNTS_CPI: libc::c_int = 1;++ #[repr(C, align(8))]+ struct RusageInfoV4Buffer([u8; RUSAGE_INFO_V4_SIZE]);++ unsafe extern "C" {+ fn proc_pid_rusage(+ pid: libc::c_int,+ flavor: libc::c_int,+ buffer: *mut RusageInfoV4Buffer,+ ) -> libc::c_int;+ fn thread_selfcounts(+ kind: libc::c_int,+ buffer: *mut u64,+ size: libc::size_t,+ ) -> libc::c_int;+ }++ impl Counter {+ pub(super) fn open(event: HardwareEvent, scope: MetricScope) -> Result<Self> {+ if !matches!(event, HardwareEvent::Instructions | HardwareEvent::Cycles) {+ return Err(anyhow!(+ "macOS exposes only instructions and cycles without root; {} needs kperf",+ event.name()+ ));+ }+ let this = Self { event, scope };+ this.read().map_err(|error| {+ anyhow!(+ "{error}; hardware counters are unavailable for {} on this Mac (Intel Macs and \+ macOS virtual machines such as CI runners do not expose them)",+ scope.label()+ )+ })?;+ Ok(this)+ }++ fn read(&self) -> Result<[u64; 2]> {+ match self.scope {+ MetricScope::Process => {+ let mut buffer = RusageInfoV4Buffer([0; RUSAGE_INFO_V4_SIZE]);+ // SAFETY: the buffer is the flavor's full size and aligned+ // for its `u64` fields; the kernel fills it on success.+ let status = unsafe {+ proc_pid_rusage(+ std::process::id() as libc::c_int,+ RUSAGE_INFO_V4,+ &mut buffer,+ )+ };+ if status != 0 {+ return Err(anyhow!(+ "proc_pid_rusage failed: {}",+ std::io::Error::last_os_error()+ ));+ }+ // SAFETY: the buffer is at least as large and as aligned as+ // the prefix struct, and every field is a plain integer.+ let info = unsafe { &*(buffer.0.as_ptr() as *const RusageInfoV4Prefix) };+ Ok([info.ri_instructions, info.ri_cycles])+ }+ MetricScope::CallingThread => {+ let mut counts = [0_u64; 2];+ // SAFETY: `counts` holds the two `u64`s the CPI flavor writes.+ let status = unsafe {+ thread_selfcounts(+ THREAD_SELFCOUNTS_CPI,+ counts.as_mut_ptr(),+ std::mem::size_of_val(&counts),+ )+ };+ if status != 0 {+ return Err(anyhow!(+ "thread_selfcounts failed: {}",+ std::io::Error::last_os_error()+ ));+ }+ Ok(counts)+ }+ }+ }++ fn value(&self, snapshot: &Snapshot) -> u64 {+ match self.event {+ HardwareEvent::Instructions => snapshot.0[0],+ _ => snapshot.0[1],+ }+ }++ pub(super) fn snapshot(&self) -> Snapshot {+ Snapshot(self.read().unwrap_or_else(|error| panic!("{error}")))+ }++ pub(super) fn advanced(&self, before: &Snapshot, after: &Snapshot) -> bool {+ self.value(after) > self.value(before)+ }++ pub(super) fn delta(&self, start: &Snapshot, end: &Snapshot) -> f64 {+ self.value(end).saturating_sub(self.value(start)) as f64+ }+ }+}++/// Fallback backend for platforms without a hardware-counter source.+#[cfg(not(any(target_os = "linux", target_os = "macos")))]+pub mod unsupported {+ use super::*;++ pub(super) struct Counter;++ #[doc(hidden)]+ pub struct Snapshot;++ impl Counter {+ pub(super) fn open(event: HardwareEvent, scope: MetricScope) -> Result<Self> {+ Err(anyhow!(+ "hardware counters ({} for {}) are unavailable on this platform",+ event.name(),+ scope.label()+ ))+ }++ pub(super) fn snapshot(&self) -> Snapshot {+ Snapshot+ }++ pub(super) fn advanced(&self, _: &Snapshot, _: &Snapshot) -> bool {+ false+ }++ pub(super) fn delta(&self, _: &Snapshot, _: &Snapshot) -> f64 {+ 0.0+ }+ }+}++// ---------------------------------------------------------------------------+// getrusage statistics+// ---------------------------------------------------------------------------++/// A statistic reported by `getrusage`, counted by [`ResourceCounter`].+#[derive(Clone, Copy, Debug, Eq, PartialEq)]+pub enum ResourceMetric {+ /// Times the thread blocked and gave up the CPU: lock waits, channel+ /// receives, sleeps, and I/O. Preemptions are not included, since they+ /// reflect the machine rather than the code.+ VoluntaryContextSwitches,+ /// Page faults served without disk I/O: first touches of freshly mapped+ /// memory, a privilege-free proxy for allocation pressure.+ MinorPageFaults,+}++impl ResourceMetric {+ /// The unit noun used in reports.+ pub fn unit(self) -> &'static str {+ match self {+ Self::VoluntaryContextSwitches => "context switches",+ Self::MinorPageFaults => "page faults",+ }+ }++ #[cfg(unix)]+ fn read(self, usage: &libc::rusage) -> u64 {+ let value = match self {+ Self::VoluntaryContextSwitches => usage.ru_nvcsw,+ Self::MinorPageFaults => usage.ru_minflt,+ };+ value.max(0) as u64+ }+}++/// Criterion measurement for a `getrusage` statistic.+///+/// Unlike hardware counters this needs no privileges and works on every Unix.+/// [`MetricScope::CallingThread`] uses `RUSAGE_THREAD`, which only Linux+/// provides; [`MetricScope::Process`] uses `RUSAGE_SELF`, which sums every+/// thread in the process.+pub struct ResourceCounter {+ metric: ResourceMetric,+ scope: MetricScope,+ formatter: CountFormatter,+}++impl ResourceCounter {+ /// Creates a counter for `metric` over `scope`.+ pub fn new(metric: ResourceMetric, scope: MetricScope) -> Result<Self> {+ let this = Self {+ metric,+ scope,+ formatter: CountFormatter::new(metric.unit()),+ };+ this.read()?;+ Ok(this)+ }++ /// The counted statistic.+ pub fn metric(&self) -> ResourceMetric {+ self.metric+ }++ /// The covered threads.+ pub fn scope(&self) -> MetricScope {+ self.scope+ }++ #[cfg(unix)]+ fn read(&self) -> Result<u64> {+ let who = match self.scope {+ MetricScope::Process => libc::RUSAGE_SELF,+ #[cfg(target_os = "linux")]+ MetricScope::CallingThread => libc::RUSAGE_THREAD,+ #[cfg(not(target_os = "linux"))]+ MetricScope::CallingThread => {+ return Err(anyhow!(+ "getrusage statistics for {} need RUSAGE_THREAD, which only Linux provides",+ self.scope.label()+ ));+ }+ };+ // SAFETY: `rusage` is plain old data that `getrusage` fully initializes+ // on success, and `who` is one of the constants the call accepts.+ let mut usage: libc::rusage = unsafe { std::mem::zeroed() };+ let status = unsafe { libc::getrusage(who, &mut usage) };+ if status != 0 {+ return Err(anyhow!(+ "getrusage failed: {}",+ std::io::Error::last_os_error()+ ));+ }+ Ok(self.metric.read(&usage))+ }++ #[cfg(not(unix))]+ fn read(&self) -> Result<u64> {+ Err(anyhow!(+ "getrusage statistics for {} are unavailable on this platform",+ self.scope.label()+ ))+ }+}++impl Measurement for ResourceCounter {+ type Intermediate = u64;+ type Value = f64;++ fn start(&self) -> Self::Intermediate {+ self.read().unwrap_or_else(|error| panic!("{error}"))+ }++ fn end(&self, start: Self::Intermediate) -> Self::Value {+ let end = self.read().unwrap_or_else(|error| panic!("{error}"));+ end.saturating_sub(start) as f64+ }++ fn add(&self, first: &Self::Value, second: &Self::Value) -> Self::Value {+ first + second+ }++ fn zero(&self) -> Self::Value {+ 0.0+ }++ fn to_f64(&self, value: &Self::Value) -> f64 {+ *value+ }++ fn formatter(&self) -> &dyn ValueFormatter {+ &self.formatter+ }+}++// ---------------------------------------------------------------------------+// Composite measurement+// ---------------------------------------------------------------------------++/// A Criterion measurement that records several metrics per sample.+///+/// Criterion analyzes one scalar per benchmark, fixed in the `Criterion<M>`+/// type. This type erases the concrete [`Measurement`] Criterion analyzes (the+/// *primary*) so benchmark code needs no type parameter, and takes any number+/// of *secondary* measurements over the same iterations. A harness records+/// their per-iteration values into a [`MetricReport`]; see+/// [`MetricReport::iter`] for plain Criterion benchmarks.+///+/// Every measurement already reduces to `f64` for Criterion's statistics, so+/// the erased value is that `f64` and each measurement's own formatter still+/// labels it.+pub struct BenchMeasurement {+ primary: Box<dyn ErasedMeasurement>,+ secondaries: Rc<SecondaryMeasurements>,+}++/// The name under which the primary's per-sample total is available to+/// [`BenchMeasurement::with_ratio`] until [`BenchMeasurement::named`] renames it.+pub const PRIMARY_METRIC_NAME: &str = "primary";++impl BenchMeasurement {+ /// Wraps the Criterion measurement whose values Criterion analyzes.+ pub fn new<M>(primary: M) -> Self+ where+ M: Measurement + 'static,+ M::Intermediate: 'static,+ {+ Self {+ primary: Box::new(ErasedMeasurementCell::new(primary)),+ secondaries: Rc::new(SecondaryMeasurements {+ primary_name: PRIMARY_METRIC_NAME,+ ..Default::default()+ }),+ }+ }++ /// Names the primary so ratios can refer to it, e.g. `named("cycles")`+ /// then `with_ratio("IPC", "instructions", "cycles")`. The primary is not+ /// itself reported as a secondary; Criterion already reports it.+ pub fn named(mut self, name: &'static str) -> Self {+ self.secondaries_mut().primary_name = name;+ self+ }++ /// Adds a measurement taken alongside the primary on every sample.+ ///+ /// `name` labels the metric in reports, e.g. `"instructions"`.+ /// Secondaries start before and end after the primary, so their own+ /// readout stays outside the primary's measured interval.+ pub fn with_secondary<M>(mut self, name: &'static str, measurement: M) -> Self+ where+ M: Measurement + 'static,+ M::Intermediate: 'static,+ {+ self.secondaries_mut()+ .metrics+ .push(Rc::new(SecondaryMetric {+ name,+ measurement: Box::new(ErasedMeasurementCell::new(measurement)),+ total: Cell::new(0.0),+ }));+ self+ }++ /// Adds `name` to reports as the ratio of two secondaries' totals over+ /// each sample, e.g. `with_ratio("IPC", "instructions", "cycles")`.+ /// Unknown names are reported as unavailable rather than failing, so a+ /// ratio can be declared even when one input was skipped on this machine.+ pub fn with_ratio(+ mut self,+ name: &'static str,+ numerator: &'static str,+ denominator: &'static str,+ ) -> Self {+ self.secondaries_mut().ratios.push(RatioMetric {+ name,+ numerator,+ denominator,+ });+ self+ }++ fn secondaries_mut(&mut self) -> &mut SecondaryMeasurements {+ Rc::get_mut(&mut self.secondaries)+ .expect("secondaries are only shared once measurement starts")+ }++ /// Returns the measurement configured by [`BENCH_MEASUREMENT_ENV_VAR`].+ ///+ /// * unset: Criterion analyzes wall time; every counter this machine+ /// supports is a secondary: process and calling-thread instructions,+ /// cycles, and their ratios where [`HardwareCounter`] has a backend+ /// (Linux, Apple Silicon); branch and cache misses on Linux; voluntary+ /// context switches and minor page faults from `getrusage`. Counters+ /// that fail to open are skipped with a note printed once.+ /// * `wall-time`: wall time only, no counters.+ /// * `instructions`: Criterion analyzes process-wide retired instructions.+ /// * `foreground-instructions`: Criterion analyzes the benchmark thread's+ /// retired instructions, which excludes background and driver threads+ /// and is the tighter regression gate for frame cost.+ ///+ /// In the instruction modes wall time becomes a secondary, and construction+ /// fails when counters are unavailable so CI does not silently measure+ /// something else.+ pub fn from_env() -> Result<Self> {+ match std::env::var(BENCH_MEASUREMENT_ENV_VAR).as_deref() {+ Err(std::env::VarError::NotPresent) => {+ let measurement = Self::new(criterion::measurement::WallTime);+ Ok(measurement.with_default_secondaries(None))+ }+ Ok("wall-time") => Ok(Self::new(criterion::measurement::WallTime)),+ Ok("instructions") => Self::with_instructions_primary(MetricScope::Process),+ Ok("foreground-instructions") => {+ Self::with_instructions_primary(MetricScope::CallingThread)+ }+ Ok(value) => Err(anyhow!(+ "unsupported {BENCH_MEASUREMENT_ENV_VAR} value {value:?}; expected \"wall-time\", \+ \"instructions\", or \"foreground-instructions\""+ )),+ Err(error) => Err(anyhow!(+ "{BENCH_MEASUREMENT_ENV_VAR} is not valid Unicode: {error}"+ )),+ }+ }++ /// [`Self::from_env`], exiting the process with an actionable message when+ /// the requested measurement can't be used on this machine. For use in+ /// `criterion_group!` `config` expressions, where there is nothing to+ /// propagate an error to.+ pub fn from_env_or_exit() -> Self {+ Self::from_env().unwrap_or_else(|error| {+ eprintln!("failed to select benchmark measurement: {error:#}");+ std::process::exit(2);+ })+ }++ fn with_instructions_primary(scope: MetricScope) -> Result<Self> {+ let instructions = HardwareCounter::new(HardwareEvent::Instructions, scope)?;+ let name = match scope {+ MetricScope::Process => "instructions",+ MetricScope::CallingThread => "foreground instructions",+ };+ Ok(Self::new(instructions)+ .named(name)+ .with_default_secondaries(Some(scope))+ .with_secondary("wall time", criterion::measurement::WallTime))+ }++ /// Adds every counter this machine supports, leaving out the instruction+ /// count for `primary_instructions` when that scope is already the+ /// primary.+ fn with_default_secondaries(mut self, primary_instructions: Option<MetricScope>) -> Self {+ // Every hardware counter needs the same access, so a denial of the+ // first one is reported once for all of them rather than once per+ // event. Events a platform lacks are still noted individually.+ let probe = if primary_instructions.is_some() {+ Ok(None)+ } else {+ HardwareCounter::new(HardwareEvent::Instructions, MetricScope::Process).map(Some)+ };+ match probe {+ Err(error) => note_unavailable("hardware counters", &error),+ Ok(instructions) => {+ if let Some(instructions) = instructions {+ self = self.with_secondary("instructions", instructions);+ }+ let hardware =+ |this: Self, name, event, scope| match HardwareCounter::new(event, scope) {+ Ok(counter) => this.with_secondary(name, counter),+ Err(error) => {+ note_unavailable(name, &error);+ this+ }+ };+ // Whichever instruction scope is not the primary is reported+ // as a secondary, so both are always present for the ratio.+ let (other_scope, other_name) = match primary_instructions {+ Some(MetricScope::CallingThread) => (MetricScope::Process, "instructions"),+ _ => (MetricScope::CallingThread, "foreground instructions"),+ };+ self = hardware(self, other_name, HardwareEvent::Instructions, other_scope);+ self = hardware(self, "cycles", HardwareEvent::Cycles, MetricScope::Process);+ self = hardware(+ self,+ "branch misses",+ HardwareEvent::BranchMisses,+ MetricScope::Process,+ );+ self = hardware(+ self,+ "cache misses",+ HardwareEvent::CacheMisses,+ MetricScope::Process,+ );+ self = self.with_ratio("IPC", "instructions", "cycles").with_ratio(+ "foreground share of instructions",+ "foreground instructions",+ "instructions",+ );+ }+ }+ let resource = |this: Self, name, metric, scope| match ResourceCounter::new(metric, scope) {+ Ok(counter) => this.with_secondary(name, counter),+ Err(error) => {+ note_unavailable(name, &error);+ this+ }+ };+ self = resource(+ self,+ "foreground context switches",+ ResourceMetric::VoluntaryContextSwitches,+ MetricScope::CallingThread,+ );+ self = resource(+ self,+ "page faults",+ ResourceMetric::MinorPageFaults,+ MetricScope::Process,+ );+ self+ }+}++/// Prints why a default secondary is unavailable, once per metric per process,+/// so a machine without perf access reports it without spamming every group.+fn note_unavailable(name: &str, error: &anyhow::Error) {+ use std::sync::{Mutex, OnceLock};++ static NOTED: OnceLock<Mutex<Vec<String>>> = OnceLock::new();+ let mut noted = NOTED+ .get_or_init(Default::default)+ .lock()+ .unwrap_or_else(|poisoned| poisoned.into_inner());+ if noted.iter().any(|noted| noted == name) {+ return;+ }+ noted.push(name.to_owned());+ eprintln!("benchmark metric {name:?} unavailable on this machine: {error:#}");+}++/// Object-safe view of a Criterion measurement.+///+/// Criterion always pairs `start` with `end` on the same thread and never+/// nests them, so the intermediate value can be parked in the measurement+/// between the two calls instead of flowing through the generic+/// `Intermediate` type.+trait ErasedMeasurement {+ fn start(&self);+ fn end(&self) -> f64;+ fn formatter(&self) -> &dyn ValueFormatter;+}++struct ErasedMeasurementCell<M: Measurement> {+ measurement: M,+ intermediate: RefCell<Option<M::Intermediate>>,+}++impl<M: Measurement> ErasedMeasurementCell<M> {+ fn new(measurement: M) -> Self {+ Self {+ measurement,+ intermediate: RefCell::new(None),+ }+ }+}++impl<M: Measurement> ErasedMeasurement for ErasedMeasurementCell<M> {+ fn start(&self) {+ let previous = self+ .intermediate+ .borrow_mut()+ .replace(self.measurement.start());+ assert!(+ previous.is_none(),+ "Measurement::start called again before Measurement::end"+ );+ }++ fn end(&self) -> f64 {+ let intermediate = self+ .intermediate+ .borrow_mut()+ .take()+ .expect("Measurement::end called without a matching Measurement::start");+ self.measurement.to_f64(&self.measurement.end(intermediate))+ }++ fn formatter(&self) -> &dyn ValueFormatter {+ self.measurement.formatter()+ }+}++/// A secondary metric and its total since the report last drained it.+struct SecondaryMetric {+ name: &'static str,+ measurement: Box<dyn ErasedMeasurement>,+ total: Cell<f64>,+}++struct RatioMetric {+ name: &'static str,+ numerator: &'static str,+ denominator: &'static str,+}++#[derive(Default)]+struct SecondaryMeasurements {+ metrics: Vec<Rc<SecondaryMetric>>,+ ratios: Vec<RatioMetric>,+ primary_name: &'static str,+ /// The primary's total since the report last drained it, so ratios can+ /// combine it with secondaries.+ primary_total: Cell<f64>,+}++impl SecondaryMeasurements {+ /// Returns each metric's total since the previous call and resets it.+ fn take_totals(&self) -> Vec<(Rc<SecondaryMetric>, f64)> {+ self.metrics+ .iter()+ .map(|metric| (metric.clone(), metric.total.replace(0.0)))+ .collect()+ }++ fn take_primary_total(&self) -> (&'static str, f64) {+ (self.primary_name, self.primary_total.replace(0.0))+ }+}++thread_local! {+ /// The secondaries of the `BenchMeasurement` most recently started on this+ /// thread. Criterion owns the measurement and hands benchmark code only a+ /// `Bencher`, so this is how a harness reaches the secondary totals after+ /// Criterion's iteration loop returns.+ static ACTIVE_SECONDARIES: RefCell<Option<Rc<SecondaryMeasurements>>> =+ const { RefCell::new(None) };+}++fn active_secondaries() -> Option<Rc<SecondaryMeasurements>> {+ ACTIVE_SECONDARIES.with(|active| active.borrow().clone())+}++impl Measurement for BenchMeasurement {+ type Intermediate = ();+ type Value = f64;++ fn start(&self) -> Self::Intermediate {+ ACTIVE_SECONDARIES.with(|active| *active.borrow_mut() = Some(self.secondaries.clone()));+ for metric in &self.secondaries.metrics {+ metric.measurement.start();+ }+ self.primary.start();+ }++ fn end(&self, (): Self::Intermediate) -> Self::Value {+ let value = self.primary.end();+ self.secondaries+ .primary_total+ .set(self.secondaries.primary_total.get() + value);+ for metric in self.secondaries.metrics.iter().rev() {+ metric+ .total+ .set(metric.total.get() + metric.measurement.end());+ }+ value+ }++ fn add(&self, first: &Self::Value, second: &Self::Value) -> Self::Value {+ first + second+ }++ fn zero(&self) -> Self::Value {+ 0.0+ }++ fn to_f64(&self, value: &Self::Value) -> f64 {+ *value+ }++ fn formatter(&self) -> &dyn ValueFormatter {+ self.primary.formatter()+ }+}++// ---------------------------------------------------------------------------+// Report+// ---------------------------------------------------------------------------++/// Per-iteration values of every secondary metric and ratio, one entry per+/// Criterion sample.+#[derive(Clone, Default)]+pub struct MetricReport {+ samples: Rc<RefCell<Vec<MetricSamples>>>,+}++struct MetricSamples {+ name: &'static str,+ /// `None` for ratios, which are unitless.+ formatter: Option<Rc<SecondaryMetric>>,+ values: Vec<f64>,+}++impl MetricReport {+ /// Creates an empty report.+ pub fn new() -> Self {+ Self::default()+ }++ /// Returns whether any sample has been recorded.+ pub fn is_empty(&self) -> bool {+ self.samples.borrow().is_empty()+ }++ /// Returns every recorded per-iteration value (or ratio) for `name`, in+ /// sample order, for programmatic checks on top of the printed summary.+ pub fn values(&self, name: &str) -> Option<Vec<f64>> {+ self.samples+ .borrow()+ .iter()+ .find(|samples| samples.name == name)+ .map(|samples| samples.values.clone())+ }++ /// Records the per-iteration value of every active secondary metric for+ /// the Criterion sample that just finished, using the totals accumulated+ /// since the previous call, and each ratio of those totals.+ ///+ /// `iterations` is counted by the caller because Criterion's `Bencher`+ /// does not expose it, and `iter_batched` with `BatchSize::PerIteration`+ /// ends the measurement once per iteration rather than once per sample.+ pub fn record_sample(&self, iterations: u64) {+ let Some(secondaries) = active_secondaries() else {+ return;+ };+ let totals = secondaries.take_totals();+ let (primary_name, primary_total) = secondaries.take_primary_total();+ if iterations == 0 {+ return;+ }+ let mut samples = self.samples.borrow_mut();+ for (metric, total) in &totals {+ Self::push(+ &mut samples,+ metric.name,+ Some(metric),+ total / iterations as f64,+ );+ }+ for ratio in &secondaries.ratios {+ let total_named = |name| {+ if name == primary_name {+ return Some(primary_total);+ }+ totals+ .iter()+ .find(|(metric, _)| metric.name == name)+ .map(|(_, total)| *total)+ };+ if let (Some(numerator), Some(denominator)) =+ (total_named(ratio.numerator), total_named(ratio.denominator))+ && denominator > 0.0+ {+ Self::push(&mut samples, ratio.name, None, numerator / denominator);+ }+ }+ }++ fn push(+ samples: &mut Vec<MetricSamples>,+ name: &'static str,+ metric: Option<&Rc<SecondaryMetric>>,+ value: f64,+ ) {+ match samples.iter_mut().find(|samples| samples.name == name) {+ Some(samples) => samples.values.push(value),+ None => samples.push(MetricSamples {+ name,+ formatter: metric.cloned(),+ values: vec![value],+ }),+ }+ }++ /// Discards secondary totals accumulated outside a measured interval,+ /// e.g. by Criterion iterating a different benchmark on this thread.+ pub fn discard_pending(&self) {+ if let Some(secondaries) = active_secondaries() {+ secondaries.take_totals();+ secondaries.take_primary_total();+ }+ }++ /// `Bencher::iter` for plain Criterion benchmarks: runs `routine` under+ /// Criterion's loop and records this sample's secondary metrics.+ pub fn iter<O>(+ &self,+ bencher: &mut criterion::Bencher<'_, BenchMeasurement>,+ mut routine: impl FnMut() -> O,+ ) {+ self.discard_pending();+ let mut iterations = 0;+ bencher.iter(|| {+ iterations += 1;+ routine()+ });+ self.record_sample(iterations);+ }++ /// Prints one line per metric, each prefixed with `indent`, to stderr.+ pub fn print(&self, indent: &str) {+ for samples in self.samples.borrow().iter() {+ let mut sorted = samples.values.clone();+ sorted.sort_by(f64::total_cmp);+ let Some((&min, &max)) = sorted.first().zip(sorted.last()) else {+ continue;+ };+ let median = sorted[sorted.len() / 2];+ match &samples.formatter {+ Some(metric) => {+ let mut scaled = [median, min, max];+ let unit = metric+ .measurement+ .formatter()+ .scale_values(median, &mut scaled);+ let [median, min, max] = scaled;+ eprintln!(+ "{indent}{} per iteration: median {median:.3} {unit} (min {min:.3}, max \+ {max:.3}, samples {})",+ samples.name,+ sorted.len()+ );+ }+ None => eprintln!(+ "{indent}{}: median {median:.3} (min {min:.3}, max {max:.3}, samples {})",+ samples.name,+ sorted.len()+ ),+ }+ }+ }+}++/// A deterministic measurement for tests: counts increments of a shared+/// counter between `start` and `end`.+#[cfg(any(test, feature = "test-support"))]+pub struct FakeCounter {+ counter: std::sync::Arc<std::sync::atomic::AtomicU64>,+ formatter: CountFormatter,+}++#[cfg(any(test, feature = "test-support"))]+impl FakeCounter {+ /// Creates a measurement reading `counter`.+ pub fn new(counter: std::sync::Arc<std::sync::atomic::AtomicU64>) -> Self {+ Self {+ counter,+ formatter: CountFormatter::new("ticks"),+ }+ }+}++#[cfg(any(test, feature = "test-support"))]+impl Measurement for FakeCounter {+ type Intermediate = u64;+ type Value = u64;++ fn start(&self) -> u64 {+ self.counter.load(std::sync::atomic::Ordering::SeqCst)+ }++ fn end(&self, start: u64) -> u64 {+ self.counter.load(std::sync::atomic::Ordering::SeqCst) - start+ }++ fn add(&self, first: &u64, second: &u64) -> u64 {+ first + second+ }++ fn zero(&self) -> u64 {+ 0+ }++ fn to_f64(&self, value: &u64) -> f64 {+ *value as f64+ }++ fn formatter(&self) -> &dyn ValueFormatter {+ &self.formatter+ }+}++#[cfg(test)]+mod tests {+ use super::*;+ use std::sync::{+ Arc,+ atomic::{AtomicU64, Ordering},+ };++ fn fast_criterion(measurement: BenchMeasurement) -> criterion::Criterion<BenchMeasurement> {+ criterion::Criterion::default()+ .with_measurement(measurement)+ .without_plots()+ .sample_size(10)+ .warm_up_time(std::time::Duration::from_millis(1))+ .measurement_time(std::time::Duration::from_millis(1))+ }++ #[test]+ fn erased_measurement_preserves_wrapped_values() {+ let counter = Arc::new(AtomicU64::new(0));+ let measurement = BenchMeasurement::new(FakeCounter::new(counter.clone()));++ let intermediate = measurement.start();+ counter.fetch_add(5, Ordering::SeqCst);+ assert_eq!(measurement.end(intermediate), 5.0);++ let intermediate = measurement.start();+ counter.fetch_add(3, Ordering::SeqCst);+ assert_eq!(measurement.end(intermediate), 3.0);+ }++ #[test]+ fn report_normalizes_secondaries_per_iteration_and_computes_ratios() {+ let primary = Arc::new(AtomicU64::new(0));+ let secondary = Arc::new(AtomicU64::new(0));+ let report = MetricReport::new();+ let mut criterion = fast_criterion(+ BenchMeasurement::new(FakeCounter::new(primary.clone()))+ .named("primary ticks")+ .with_secondary("ticks", FakeCounter::new(secondary.clone()))+ .with_secondary("double ticks", FakeCounter::new(secondary.clone()))+ .with_ratio("tick ratio", "double ticks", "ticks")+ .with_ratio("ticks per primary", "ticks", "primary ticks")+ .with_ratio("unavailable ratio", "ticks", "missing"),+ );++ criterion.bench_function("report", |bencher| {+ // Fixture work outside the measured loop is not attributed to any+ // iteration.+ secondary.fetch_add(1_000_000, Ordering::SeqCst);+ report.iter(bencher, || {+ primary.fetch_add(1, Ordering::SeqCst);+ secondary.fetch_add(11, Ordering::SeqCst);+ });+ });++ let samples = report.samples.borrow();+ let values = |name: &str| {+ samples+ .iter()+ .find(|samples| samples.name == name)+ .map(|samples| samples.values.clone())+ };+ let ticks = values("ticks").expect("ticks were recorded");+ assert!(!ticks.is_empty());+ assert!(+ ticks.iter().all(|value| *value == 11.0),+ "each sample's total should be divided by its iteration count: {ticks:?}"+ );+ let ratio = values("tick ratio").expect("ratio was recorded");+ assert!(+ ratio.iter().all(|value| *value == 1.0),+ "ratio of identical counters should be 1: {ratio:?}"+ );+ let per_primary =+ values("ticks per primary").expect("ratio against the primary was recorded");+ assert!(+ per_primary.iter().all(|value| *value == 11.0),+ "secondary 11 per iteration over primary 1 per iteration: {per_primary:?}"+ );+ assert!(values("unavailable ratio").is_none());+ }++ #[test]+ fn count_formatter_scales_with_prefixes() {+ let formatter = CountFormatter::new("cycles");+ let mut values = [2_500_000.0];+ assert_eq!(formatter.scale_values(2_500_000.0, &mut values), "M cycles");+ assert_eq!(values, [2.5]);+ let mut values = [12.0];+ assert_eq!(formatter.scale_values(12.0, &mut values), "cycles");+ assert_eq!(formatter.scale_for_machines(&mut values), "cycles");+ }++ #[cfg(unix)]+ #[test]+ fn resource_counter_counts_page_faults_and_context_switches() {+ let faults = ResourceCounter::new(ResourceMetric::MinorPageFaults, MetricScope::Process)+ .expect("RUSAGE_SELF is always available");+ let start = faults.start();+ // Touch fresh pages: a 4 MiB allocation faults in ~1000 pages.+ let buffer = vec![1_u8; 4 << 20];+ std::hint::black_box(&buffer);+ assert!(faults.end(start) > 100.0);++ #[cfg(target_os = "linux")]+ {+ let switches = ResourceCounter::new(+ ResourceMetric::VoluntaryContextSwitches,+ MetricScope::CallingThread,+ )+ .expect("RUSAGE_THREAD is available on Linux");+ let start = switches.start();+ for _ in 0..5 {+ std::thread::sleep(std::time::Duration::from_micros(100));+ }+ assert!(switches.end(start) >= 5.0);+ }+ }++ /// Returns a closure measuring a background thread spinning `iterations`+ /// times, or `None` when this machine has no usable hardware counters.+ fn background_spin_instructions(scope: MetricScope) -> Option<impl Fn(u64) -> f64> {+ let counter = match HardwareCounter::new(HardwareEvent::Instructions, scope) {+ Ok(counter) => counter,+ Err(error) => {+ let message = error.to_string();+ assert!(+ message.contains("CAP_PERFMON")+ || message.contains("does not advance")+ || message.contains("unavailable on this platform")+ || message.contains("unavailable for"),+ "unavailable counters should have an actionable error: {message}"+ );+ return None;+ }+ };+ Some(move |iterations: u64| {+ let start = counter.start();+ let worker = std::thread::spawn(move || {+ let mut total = 0_u64;+ for value in 0..iterations {+ total = std::hint::black_box(total.wrapping_add(value));+ }+ total+ });+ std::hint::black_box(worker.join().expect("spin thread should finish"));+ counter.end(start)+ })+ }++ #[test]+ fn process_instructions_include_threads_spawned_after_open() {+ let Some(measure) = background_spin_instructions(MetricScope::Process) else {+ return;+ };+ let idle = measure(0);+ let busy = measure(1_000_000);+ assert!(+ busy > idle + 500_000.0,+ "work on a thread spawned after the counter opened should be counted: \+ idle={idle}, busy={busy}"+ );+ }++ #[test]+ fn calling_thread_instructions_exclude_other_threads() {+ let Some(measure) = background_spin_instructions(MetricScope::CallingThread) else {+ return;+ };+ let idle = measure(0);+ let busy = measure(1_000_000);+ assert!(+ busy < idle + 100_000.0,+ "another thread's work should not count toward the calling thread: \+ idle={idle}, busy={busy}"+ );+ }++ #[test]+ fn hardware_events_open_or_fail_actionably() {+ for event in [+ HardwareEvent::Cycles,+ HardwareEvent::BranchMisses,+ HardwareEvent::CacheMisses,+ HardwareEvent::CacheReferences,+ ] {+ match HardwareCounter::new(event, MetricScope::Process) {+ Ok(counter) => {+ let start = counter.start();+ std::hint::black_box((0..100_000_u64).fold(0_u64, |t, v| t.wrapping_add(v)));+ let count = counter.end(start);+ assert!(count >= 0.0);+ }+ Err(error) => {+ let message = error.to_string();+ assert!(+ message.contains("CAP_PERFMON")+ || message.contains("does not expose")+ || message.contains("does not advance")+ || message.contains("needs kperf")+ || message.contains("unavailable on this platform")+ || message.contains("unavailable for"),+ "{event:?}: {message}"+ );+ }+ }+ }+ }+}diff --git a/crates/benchmarks/benches/display_map.rs b/crates/benchmarks/benches/display_map.rsindex bad8d232772..06117a665f0 100644--- a/crates/benchmarks/benches/display_map.rs+++ b/crates/benchmarks/benches/display_map.rs@@ -1,4 +1,4 @@-use criterion::{BenchmarkId, Criterion, black_box, criterion_group, criterion_main};+use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};use editor::{EditorStyle, MultiBuffer, display_map::*};use gpui::{AppContext as _, HighlightStyle, Hsla, TestDispatcher, font, px};use itertools::Itertools;@@ -6,6 +6,7 @@ use multi_buffer::MultiBufferOffset;use project::project_settings::DiagnosticSeverity;use rand::{Rng, SeedableRng, rngs::StdRng};use settings::SettingsStore;+use std::hint::black_box;use std::{num::NonZeroU32, time::Duration};use text::Bias;use util::RandomCharIter;diff --git a/crates/benchmarks/benches/edit_file_tool.rs b/crates/benchmarks/benches/edit_file_tool.rsindex 204628cfe80..f364e260729 100644--- a/crates/benchmarks/benches/edit_file_tool.rs+++ b/crates/benchmarks/benches/edit_file_tool.rs@@ -16,9 +16,7 @@ use benchmarks::bench_utils::{RUST_FUNCTION_BODY_LINES, RUST_FUNCTION_LINES, RUST_MODULE_HEADER_LINES, random_rust_file,rust_file_line_count, rust_identifier as identifier,};-use criterion::{- BatchSize, BenchmarkId, Criterion, Throughput, black_box, criterion_group, criterion_main,-};+use criterion::{BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main};use editor::{Editor, EditorStyle};use futures::{StreamExt as _, pin_mut, task::noop_waker};use gpui::{@@ -32,6 +30,7 @@ use prompt_store::ProjectContext;use rand::{Rng as _, SeedableRng as _, rngs::StdRng};use serde_json::{Value, json};use settings::{Settings as _, SettingsStore};+use std::hint::black_box;use ui::IntoElement as _;const SEED: u64 = 0x5EED_5EED;diff --git a/crates/edit_prediction/benches/kept_rate.rs b/crates/edit_prediction/benches/kept_rate.rsindex 44defe5d19d..8db3c274d22 100644--- a/crates/edit_prediction/benches/kept_rate.rs+++ b/crates/edit_prediction/benches/kept_rate.rs@@ -1,5 +1,6 @@-use criterion::{BenchmarkId, Criterion, black_box, criterion_group, criterion_main};+use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};use edit_prediction::metrics::compute_kept_rate;+use std::hint::black_box;fn repeated_function_lines(line_count: usize) -> String {let mut text = String::with_capacity(line_count * 32);diff --git a/crates/edit_prediction/benches/ts_error_count.rs b/crates/edit_prediction/benches/ts_error_count.rsindex 518f9ab5bc2..c6e77f609a9 100644--- a/crates/edit_prediction/benches/ts_error_count.rs+++ b/crates/edit_prediction/benches/ts_error_count.rs@@ -1,6 +1,6 @@use std::sync::Arc;-use criterion::{BenchmarkId, Criterion, black_box, criterion_group, criterion_main};+use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};use edit_prediction::metrics::count_tree_sitter_errors;use fs::FakeFs;use gpui::{AppContext as _, TestAppContext};@@ -8,6 +8,7 @@ use language::{Buffer, BufferSnapshot, LanguageRegistry};use languages::init as init_languages;use node_runtime::NodeRuntime;use settings::SettingsStore;+use std::hint::black_box;struct ParsedCase {label: String,diff --git a/crates/gpui/Cargo.toml b/crates/gpui/Cargo.tomlindex e767de582ff..1cebc98f702 100644--- a/crates/gpui/Cargo.toml+++ b/crates/gpui/Cargo.toml@@ -26,7 +26,7 @@ test-support = ["x11","proptest",]-bench-support = ["profiler", "dep:criterion"]+bench-support = ["profiler", "dep:criterion", "dep:bench_metrics"]# Preserve the published feature name while consumers migrate to `bench-support`.bench = ["bench-support"]inspector = ["gpui_macros/inspector"]@@ -53,6 +53,7 @@ backtrace = { workspace = true, optional = true }bitflags.workspace = truecollections.workspace = true+bench_metrics = { workspace = true, optional = true }criterion = { workspace = true, optional = true }ctor.workspace = truederive_more.workspace = true@@ -131,13 +132,13 @@ objc2-metal = { version = "0.3", optional = true }scap = { workspace = true, optional = true }-[target.'cfg(target_os = "windows")'.dependencies]windows = { workspace = true, features = ["Win32_Foundation", "Win32_System_Power"] }[dev-dependencies]backtrace.workspace = true+bench_metrics = { workspace = true, features = ["test-support"] }collections = { workspace = true, features = ["test-support"] }env_logger.workspace = truegpui_platform = { workspace = true, features = ["font-kit", "wayland", "x11"] }diff --git a/crates/gpui/src/app.rs b/crates/gpui/src/app.rsindex b62cb7e6764..612d3df5882 100644--- a/crates/gpui/src/app.rs+++ b/crates/gpui/src/app.rs@@ -21,7 +21,10 @@ use slotmap::SlotMap;pub use async_context::*;#[cfg(feature = "bench-support")]-pub use bench_context::{BenchAppContext, BenchReport, BenchWindowContext, bench_platform};+pub use bench_context::{+ BenchAppContext, BenchMeasurement, BenchReport, BenchWindowContext, MetricReport,+ bench_platform,+};use collections::{FxHashMap, FxHashSet, HashMap, TypeIdHashMap, TypeIdHashSet, VecDeque};pub use context::*;pub use entity_map::*;diff --git a/crates/gpui/src/app/bench_context.rs b/crates/gpui/src/app/bench_context.rsindex c9e5484d298..15eef07eb57 100644--- a/crates/gpui/src/app/bench_context.rs+++ b/crates/gpui/src/app/bench_context.rs@@ -10,6 +10,7 @@ use std::{};use anyhow::{Result, anyhow};+pub use bench_metrics::{BenchMeasurement, MetricReport};use hdrhistogram::Histogram;use crate::{@@ -101,6 +102,7 @@ pub struct ForegroundWorkSummary {#[derive(Clone)]pub struct BenchReport {frame_snapshot: Rc<RefCell<WindowFrameSnapshot>>,+ metrics: MetricReport,frame_budget_nanos: u128,}@@ -127,10 +129,18 @@ impl BenchReport {);Self {frame_snapshot: Rc::new(RefCell::new(WindowFrameSnapshot::new())),+ metrics: MetricReport::new(),frame_budget_nanos,}}+ /// Returns the per-iteration secondary metrics (retired instructions,+ /// cycles, context switches, ...) recorded alongside Criterion's primary+ /// measurement.+ pub fn metrics(&self) -> &MetricReport {+ &self.metrics+ }+fn record_frame_timings<'i>(&self, events: impl IntoIterator<Item = &'i FrameEvent>) {let mut snapshot = self.frame_snapshot.borrow_mut();// `.ok()` on `record`: this operation is infallible (the histograms auto-resize).@@ -239,15 +249,15 @@ impl BenchReport {}/// Prints this report to stderr.- pub fn print(&self, benchmark_name: Option<&'static str>) {+ pub fn print(&self, benchmark_name: &str) {let frame_snapshot = self.frame_snapshot.borrow();- if frame_snapshot.is_empty() {+ if frame_snapshot.is_empty() && self.metrics.is_empty() {return;}- let benchmark_name = benchmark_name.unwrap_or("unknown benchmark");eprintln!("GPUI bench report (all observed iterations): {benchmark_name}");eprintln!(" note: includes Criterion warmup/calibration");+ self.metrics.print(" ");self.print_histogram("window dirty-to-draw", &frame_snapshot.dirty_to_draw);self.print_histogram("window draw", &frame_snapshot.draw);self.print_histogram("window present interval", &frame_snapshot.present_interval);@@ -488,7 +498,7 @@ pub struct BenchAppContext<'a, 'measurement> {background_executor: BackgroundExecutor,foreground_executor: ForegroundExecutor,benchmark_name: Option<&'static str>,- bencher: Rc<RefCell<Option<&'a mut criterion::Bencher<'measurement>>>>,+ bencher: Rc<RefCell<Option<&'a mut criterion::Bencher<'measurement, BenchMeasurement>>>>,report: BenchReport,}@@ -501,7 +511,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {pub fn new(platform: Rc<dyn Platform>,benchmark_name: Option<&'static str>,- bencher: &'a mut criterion::Bencher<'measurement>,+ bencher: &'a mut criterion::Bencher<'measurement, BenchMeasurement>,) -> Self {Self::build(platform, benchmark_name, bencher, BenchReport::default())}@@ -515,7 +525,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {pub fn new_with_platform_and_report(platform: Rc<dyn Platform>,benchmark_name: Option<&'static str>,- bencher: &'a mut criterion::Bencher<'measurement>,+ bencher: &'a mut criterion::Bencher<'measurement, BenchMeasurement>,report: BenchReport,) -> Self {Self::build(platform, benchmark_name, bencher, report)@@ -524,7 +534,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {fn build(platform: Rc<dyn Platform>,benchmark_name: Option<&'static str>,- bencher: &'a mut criterion::Bencher<'measurement>,+ bencher: &'a mut criterion::Bencher<'measurement, BenchMeasurement>,report: BenchReport,) -> Self {let background_executor = platform.background_executor();@@ -662,12 +672,16 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {let bencher = self.take_bencher("bench_iter");self.dispatch_pending_frames(|| true);let collector = TraceScope::start(self.foreground_journal_collector());+ self.report.metrics.discard_pending();+ let mut iterations = 0;let mut benchmark = || {+ iterations += 1;benchmark(self);self.dispatch_pending_frames(|| true);};bencher.iter(&mut benchmark);let events = collector.finish();+ self.report.metrics.record_sample(iterations);self.report.record_frame_timings(events.frame_events.iter());self.report.record_foreground_events(events.foreground_events());@@ -722,6 +736,8 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {let mut benchmark_context = self.clone();let foreground_executor = self.foreground_executor.clone();let report = self.report.clone();+ report.metrics.discard_pending();+ let iterations = std::cell::Cell::new(0);bencher.iter_batched_ref(|| {@@ -740,6 +756,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {}},|measured_input| {+ iterations.set(iterations.get() + 1);let task = benchmark(&mut measured_input.input, &mut benchmark_context);benchmark_context.dispatch_pending_frames(|| true);let output = Rc::new(RefCell::new(None));@@ -759,6 +776,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {},criterion::BatchSize::PerIteration,);+ report.metrics.record_sample(iterations.get());self.replace_bencher(bencher);}@@ -789,7 +807,10 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {let dispatcher = self.background_executor.dispatcher().clone();self.dispatch_pending_frames(|| true);let collector = TraceScope::start(self.foreground_journal_collector());+ self.report.metrics.discard_pending();+ let mut iterations = 0;let mut benchmark = || {+ iterations += 1;dispatcher.as_threaded().expect("validated in BenchAppContext::build")@@ -808,6 +829,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {bencher.iter(&mut benchmark);let events = collector.finish();+ self.report.metrics.record_sample(iterations);self.report.record_frame_timings(events.frame_events.iter());self.report.record_foreground_events(events.foreground_events());@@ -853,6 +875,8 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {.as_threaded().expect("validated in BenchAppContext::build");let report = self.report.clone();+ report.metrics.discard_pending();+ let iterations = std::cell::Cell::new(0);bencher.iter_batched_ref(|| {@@ -867,6 +891,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {}},|measured_input| {+ iterations.set(iterations.get() + 1);let (state, window, stopped) = &mut measured_input.input;let started = Instant::now();let check_deadline = || {@@ -932,6 +957,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {},criterion::BatchSize::PerIteration,);+ report.metrics.record_sample(iterations.get());self.replace_bencher(bencher);}@@ -1009,13 +1035,16 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {}}- fn take_bencher(&self, benchmark_kind: &str) -> &'a mut criterion::Bencher<'measurement> {+ fn take_bencher(+ &self,+ benchmark_kind: &str,+ ) -> &'a mut criterion::Bencher<'measurement, BenchMeasurement> {self.bencher.borrow_mut().take().unwrap_or_else(|| {panic!("cannot start {benchmark_kind}: benchmark measurement is already running")})}- fn replace_bencher(&self, bencher: &'a mut criterion::Bencher<'measurement>) {+ fn replace_bencher(&self, bencher: &'a mut criterion::Bencher<'measurement, BenchMeasurement>) {let previous = self.bencher.borrow_mut().replace(bencher);assert!(previous.is_none(),@@ -1323,11 +1352,96 @@ impl VisualContext for BenchWindowContext<'_, '_> {#[cfg(test)]mod tests {- use std::{rc::Rc, sync::Arc};+ use std::{+ rc::Rc,+ sync::{+ Arc,+ atomic::{AtomicU64, Ordering},+ },+ };use super::*;use crate::profiler::journal::install_test_foreground_journal;+ /// Runs `benchmark` under a wall-time primary with a fake counter as the+ /// secondary metric and returns the per-iteration values the report+ /// recorded for it.+ fn secondary_values_per_iteration(+ counter: Arc<AtomicU64>,+ benchmark: impl Fn(&mut BenchAppContext, &Arc<AtomicU64>),+ ) -> Vec<u64> {+ let platform = bench_platform(None, Arc::new(crate::NoopTextSystem::new()));+ let report = BenchReport::default();+ let mut criterion = criterion::Criterion::default()+ .with_measurement(+ BenchMeasurement::new(criterion::measurement::WallTime).with_secondary(+ "fake counter",+ bench_metrics::FakeCounter::new(counter.clone()),+ ),+ )+ .without_plots()+ .sample_size(10)+ .warm_up_time(Duration::from_millis(1))+ .measurement_time(Duration::from_millis(1));++ criterion.bench_function("secondary_values_per_iteration", |bencher| {+ let mut cx = BenchAppContext::new_with_platform_and_report(+ platform.clone(),+ Some("secondary_values_per_iteration"),+ bencher,+ report.clone(),+ );+ // Fixture work outside the measured loop must not be attributed+ // to any iteration.+ counter.fetch_add(1_000_000, Ordering::SeqCst);+ benchmark(&mut cx, &counter);+ cx.teardown();+ });++ report+ .metrics()+ .values("fake counter")+ .expect("the fake counter was recorded")+ .into_iter()+ .map(|value| value as u64)+ .collect()+ }++ #[test]+ fn secondary_metric_is_normalized_per_iteration_for_bench_iter() {+ let values = secondary_values_per_iteration(Arc::new(AtomicU64::new(0)), |cx, counter| {+ cx.bench_iter(|_| {+ counter.fetch_add(11, Ordering::SeqCst);+ });+ });+ assert!(!values.is_empty());+ assert!(+ values.iter().all(|value| *value == 11),+ "each sample's total should be divided by its iteration count: {values:?}"+ );+ }++ #[test]+ fn secondary_metric_is_accumulated_across_per_iteration_batches() {+ let values = secondary_values_per_iteration(Arc::new(AtomicU64::new(0)), |cx, counter| {+ cx.bench_batched_task(+ |_| {+ // Setup runs outside every measured interval.+ counter.fetch_add(1_000, Ordering::SeqCst);+ },+ |(), cx| {+ counter.fetch_add(11, Ordering::SeqCst);+ cx.background_spawn(async {})+ },+ );+ });+ assert!(!values.is_empty());+ assert!(+ values.iter().all(|value| *value == 11),+ "PerIteration batches should sum to one per-iteration value: {values:?}"+ );+ }+#[test]fn foreground_work_reports_long_task_without_window_draw() {let (journal, _journal_guard) = install_test_foreground_journal(1024, 64);@@ -1421,6 +1535,7 @@ mod tests {let name = "bench_task_reports_long_task_without_window";let mut criterion = criterion::Criterion::default()+ .with_measurement(BenchMeasurement::new(criterion::measurement::WallTime)).without_plots().sample_size(10).warm_up_time(Duration::from_millis(1))diff --git a/crates/gpui/src/gpui.rs b/crates/gpui/src/gpui.rsindex 51585b5e5ef..387729e8a3e 100644--- a/crates/gpui/src/gpui.rs+++ b/crates/gpui/src/gpui.rs@@ -112,14 +112,35 @@ pub use spring::*;/// Defines a Criterion benchmark group for benchmarks annotated with [`gpui::bench`].///-/// This mirrors `criterion::criterion_group!` so GPUI benchmark files can keep the-/// same shape as ordinary Criterion benchmarks.+/// This mirrors `criterion::criterion_group!`, but the group measures with the+/// `gpui::BenchMeasurement` configured by `BENCH_MEASUREMENT` (see+/// `BenchMeasurement::from_env`). By default Criterion analyzes wall time+/// while retired instructions, cycles, IPC, context switches, and the other+/// counters this machine supports are printed per iteration in the GPUI bench+/// report. `BENCH_MEASUREMENT=instructions` makes Criterion analyze+/// process-wide instructions instead, `foreground-instructions` the benchmark+/// thread's alone, and `wall-time` disables all counters. A+/// `config = ...` expression may set any other Criterion option; its+/// measurement is replaced. To measure with something else, call+/// `criterion::criterion_group!` directly with+/// `config = criterion::Criterion::default().with_measurement(gpui::BenchMeasurement::new(...))`.////// [`gpui::bench`]: crate::bench#[macro_export]macro_rules! bench_group {- ($($tokens:tt)*) => {- criterion::criterion_group!($($tokens)*);+ (name = $name:ident; config = $config:expr; targets = $($target:path),+ $(,)?) => {+ criterion::criterion_group! {+ name = $name;+ config = ($config).with_measurement($crate::BenchMeasurement::from_env_or_exit());+ targets = $($target),++ }+ };+ ($name:ident, $($target:path),+ $(,)?) => {+ $crate::bench_group! {+ name = $name;+ config = criterion::Criterion::default();+ targets = $($target),++ }};}diff --git a/crates/gpui_macros/src/bench.rs b/crates/gpui_macros/src/bench.rsindex 8fea39ef0ec..eb1350bc781 100644--- a/crates/gpui_macros/src/bench.rs+++ b/crates/gpui_macros/src/bench.rs@@ -83,10 +83,13 @@ pub fn bench(args: TokenStream, function: TokenStream) -> TokenStream {let sample_size =sample_size.map(|sample_size| quote! { group.sample_size(#sample_size); });quote! {- let report = #report_expr;let mut group = criterion.benchmark_group(#group_name);#sample_sizefor input in #inputs {+ // One report per input: per-iteration metrics differ across+ // inputs, so blending them would make the summary meaningless.+ let report = #report_expr;+ let report_name = format!("{}/{}/{}", #group_name, #input_name, input);group.bench_with_input(criterion::BenchmarkId::new(#input_name, &input), &input, {let report = report.clone();move |bencher, input| {@@ -105,9 +108,9 @@ pub fn bench(args: TokenStream, function: TokenStream) -> TokenStream {cx.teardown();}});+ report.print(&report_name);}group.finish();- report.print(Some(stringify!(#outer_fn_name)));}} else {if let Some(input_name) = input_name {@@ -148,14 +151,14 @@ pub fn bench(args: TokenStream, function: TokenStream) -> TokenStream {cx.teardown();}});- report.print(Some(stringify!(#outer_fn_name)));+ report.print(stringify!(#outer_fn_name));}};TokenStream::from(quote! {#inner_fn- fn #outer_fn_name(criterion: &mut criterion::Criterion) {+ fn #outer_fn_name(criterion: &mut criterion::Criterion<gpui::BenchMeasurement>) {#benchmark}diff --git a/crates/language/benches/highlight_map.rs b/crates/language/benches/highlight_map.rsindex 678bd08a8db..ab449d51e61 100644--- a/crates/language/benches/highlight_map.rs+++ b/crates/language/benches/highlight_map.rs@@ -1,6 +1,7 @@-use criterion::{BenchmarkId, Criterion, black_box, criterion_group, criterion_main};+use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};use gpui::rgba;use language::build_highlight_map;+use std::hint::black_box;use theme::SyntaxTheme;fn syntax_theme(highlight_names: &[&str]) -> SyntaxTheme {diff --git a/crates/rope/benches/rope_benchmark.rs b/crates/rope/benches/rope_benchmark.rsindex 8599328aacf..707acef1d5d 100644--- a/crates/rope/benches/rope_benchmark.rs+++ b/crates/rope/benches/rope_benchmark.rs@@ -1,11 +1,10 @@use std::ops::Range;-use criterion::{- BatchSize, BenchmarkId, Criterion, Throughput, black_box, criterion_group, criterion_main,-};+use criterion::{BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main};use rand::prelude::*;use rand::rngs::StdRng;use rope::{Point, Rope};+use std::hint::black_box;use sum_tree::Bias;use util::RandomCharIter;diff --git a/crates/streaming_diff/benches/streaming_diff.rs b/crates/streaming_diff/benches/streaming_diff.rsindex cbf83b7557e..3067ba4ee56 100644--- a/crates/streaming_diff/benches/streaming_diff.rs+++ b/crates/streaming_diff/benches/streaming_diff.rs@@ -1,7 +1,6 @@-use criterion::{- BatchSize, BenchmarkId, Criterion, Throughput, black_box, criterion_group, criterion_main,-};+use criterion::{BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main};use rand::{Rng as _, SeedableRng as _, rngs::StdRng};+use std::hint::black_box;use streaming_diff::StreamingDiff;const SEED: u64 = 0x5EED_5EED;