The design language that makes your AI harness better at design.
JavaScript
Measure what each rule costs the edit hook (#1052)
Paul Bakaus6af45ac9a5eb
diff --git a/CLAUDE.md b/CLAUDE.mdindex 1e8e89b3e..25baed71e 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -369,6 +369,14 @@ The rule logic lives in `crates/core`: every check, the browser rule adapters ovOrder for a new rule: fixture here first, registry row in `crates/foundation/src/registry.rs`, the check in `crates/core` against that fixture, oracle case + golden, `cargo xtask bundle` to refresh the two tracked live assets, then `bun run build && bun run test` with a binary present. Rule counts quoted in `README.md` / `README.npm.md` are validated by `generateCounts` against `crates/live/assets/antipatterns.json`.+**Hook cost is part of a rule.** The edit hook is a new process on every Edit or Write in millions of agent sessions, and it runs the whole text engine (or, for `.html`, the whole static HTML engine) cold: `IMMEDIATE_TIER_RULES` only filters which findings are shown per edit, it does not skip rules. So every text or static-HTML rule is paid on every edit, regex compilation and lazy tables included, and a millisecond matters. Before landing a new or changed check in those engines, measure it on main and on the branch and put both numbers in the PR:++```bash+cargo run --release -p impeccable-html --example rule_cost -- --runs 9 tests/fixtures/antipatterns <real project source>+```++It writes `rule-cost.json`: per edit, cold, per engine, the shared parse and cascade work, and every profiled unit with the rules it emitted (mean, p50, p95 ms). A rule inside a shared pass (`color-rules`, `html-patterns`, ...) moves that pass's number. Prefer the detection that touches less: one regex over many, a tag- or property-gated check over a per-element walk, work that rides an existing pass over a new one. A check that needs layout or pixels belongs in the browser engine, not the hook. The Detector Lab's Rules page charts these numbers per rule.+### Rule packs (downstream crates adding rules)A crate that depends on this workspace can add rules without forking it: implement `impeccable_core::rule_pack::RulePack` (text plus the two browser DOM hooks) and, for the static engine, `impeccable_html::StaticRulePack`, call `impeccable_core::rule_pack::install(&PACK)` at startup, and hand the pack to the engine through `TextOptions` / `ScanOptions`, `DetectHtmlOptions`, `StaticHtmlEngine`, or `BrowserConfig`. Every hook runs after the built-ins and before inline ignores, so built-in output with no pack installed is byte-identical, which the oracle enforces. The registry keeps `ANTIPATTERNS` as the built-in list and `registry::extend` appends a pack's rows, panicking on an id collision. `crates/wasm --features detect` exposes the two file engines as JSON exports (`detect_text_json`, `detect_html_source_json`) for hosts that cannot exec the binary; Pristine consumes that path. Full contract in `docs/ENGINE.md` ("Rule packs"). The shipped `impeccable` binary installs no pack, and nothing in this repo should start doing so.diff --git a/crates/detect/src/regex_matchers.rs b/crates/detect/src/regex_matchers.rsindex 1f9b5b5e1..85af544dd 100644--- a/crates/detect/src/regex_matchers.rs+++ b/crates/detect/src/regex_matchers.rs@@ -2245,6 +2245,8 @@ pub fn analyzer_rule_id(i: usize) -> String {"marketing-buzzword","aphoristic-cadence","dark-glow",+ "radial-halo",+ "marquee",];IDS.get(i).map(|s| s.to_string())diff --git a/crates/html/examples/rule_cost.rs b/crates/html/examples/rule_cost.rsnew file mode 100644index 000000000..0f7506d50--- /dev/null+++ b/crates/html/examples/rule_cost.rs@@ -0,0 +1,376 @@+//! What each detector rule costs the edit hook, per edit, in milliseconds.+//!+//! cargo run --release -p impeccable-html --example rule_cost -- \+//! [--runs 9] [--out rule-cost.json] <file-or-dir>...+//!+//! The hook (`crates/hook/src/hook.rs`, `run_hook`) is a new process on every+//! Edit or Write, and it runs a whole engine over the edited file: the static+//! HTML engine for `.html` / `.htm`, the text engine for every other+//! `ALLOWED_EXTS` file. The per-edit tier only filters the findings shown+//! afterwards; it does not skip rules. So a rule costs every edit of a file+//! its engine reads, and what it costs is a cold run: its regexes compile and+//! its lazy tables build in that process.+//!+//! This measures exactly that. For every sample it starts itself again as a+//! fresh process (`--one <file>`), runs the file's engine the way the hook+//! does (inline ignores on, no design system, no rule pack) with the+//! engines' own profiler switched on (`impeccable_detect::profiler`), and+//! reports the timed units. Runs are interleaved across files so drift on the+//! machine spreads over every file instead of landing on one.+//!+//! A unit is what the engine times as one step. In the text engine that is a+//! line matcher or a page analyzer, named by its rule id. In the HTML engine+//! several rules share one pass over the elements (`color-rules`,+//! `quality-rules`, ...); their cost cannot be split, and the output says+//! which rules a unit emitted on the inputs.+//!+//! `--one <file>` prints one sample as a JSON line and is what the driver+//! spawns; it is also handy on its own.++use std::collections::{BTreeMap, BTreeSet};+use std::path::{Path, PathBuf};+use std::process::Command;+use std::rc::Rc;+use std::time::Instant;++use impeccable_detect::detect_text::{detect_text, TextOptions};+use impeccable_detect::engines::{HtmlEngine, ScanOptions};+use impeccable_detect::profiler::DetectorProfile;+use impeccable_html::StaticHtmlEngine;+use serde_json::{json, Value};++/// `crates/hook/src/hook_lib.rs` `ALLOWED_EXTS`: the files the hook scans.+const HOOK_EXTS: &[&str] = &[+ "tsx", "jsx", "html", "htm", "vue", "svelte", "astro", "css", "scss", "sass", "less", "ts",+ "js",+];++fn is_html(path: &Path) -> bool {+ matches!(ext(path).as_str(), "html" | "htm")+}++fn ext(path: &Path) -> String {+ path.extension()+ .and_then(|e| e.to_str())+ .unwrap_or("")+ .to_ascii_lowercase()+}++/// One cold sample: the engine the hook would pick, over one file, profiled.+/// A file that cannot be read or scanned is an error, never an empty sample.+fn one(path: &Path) -> Result<Value, String> {+ let file = path.to_string_lossy().to_string();+ // As the hook reads it (`hook.rs`): bytes, decoded lossily.+ let bytes = std::fs::read(path).map_err(|e| format!("cannot read {file}: {e}"))?;+ let content = String::from_utf8_lossy(&bytes).into_owned();+ let profile = Rc::new(DetectorProfile::new());+ let html = is_html(path);+ let started = Instant::now();+ let findings = if html {+ let options = ScanOptions {+ inline_ignores: true,+ profile: Some(profile.clone()),+ ..Default::default()+ };+ StaticHtmlEngine::default()+ .detect_html(&file, &options, &mut std::io::sink())+ .map_err(|e| format!("the HTML engine failed on {file}: {}", e.message))?+ } else {+ let options = TextOptions {+ profile: Some(&profile),+ inline_ignores: true,+ ..Default::default()+ };+ detect_text(&content, &file, &options)+ };+ let total_ms = started.elapsed().as_secs_f64() * 1000.0;+ let events: Vec<Value> = profile+ .events+ .borrow()+ .iter()+ .map(|e| json!({ "engine": e.engine, "phase": e.phase, "unit": e.rule_id, "ms": e.ms, "rules": e.finding_ids.clone().unwrap_or_default() }))+ .collect();+ Ok(json!({+ "file": file,+ "engine": if html { "static-html" } else { "text" },+ "bytes": bytes.len(),+ "totalMs": total_ms,+ "findings": findings.len(),+ "events": events,+ }))+}++/// The files the hook would scan under `path`. A symlink met while walking is+/// skipped (a link to a sibling would weigh its files twice, a link to a+/// parent would loop); a path given on the command line is followed.+fn collect(path: &Path, out: &mut Vec<PathBuf>, top: bool) {+ let link = std::fs::symlink_metadata(path)+ .map(|m| m.file_type().is_symlink())+ .unwrap_or(false);+ if link && !top {+ return;+ }+ if path.is_dir() {+ let name = path.file_name().and_then(|n| n.to_str()).unwrap_or("");+ if matches!(+ name,+ "node_modules" | ".git" | "dist" | "target" | ".next" | "build"+ ) {+ return;+ }+ let mut entries: Vec<PathBuf> = std::fs::read_dir(path)+ .map(|d| d.flatten().map(|e| e.path()).collect())+ .unwrap_or_default();+ entries.sort();+ for entry in entries {+ collect(&entry, out, false);+ }+ } else if HOOK_EXTS.contains(&ext(path).as_str()) {+ out.push(path.to_path_buf());+ }+}++/// Engine work no rule owns: reading, parsing, the cascade, selector+/// matching, extracting style blocks. Every edit pays it.+const SHARED_PHASES: &[&str] = &[+ "setup",+ // Reading a page's local linked stylesheets (`cascade/build.rs`).+ "preprocess",+ "parse-html",+ "parse-css",+ "cascade",+ "selector-match",+ "extract",+];++/// An event that only wraps others: the HTML engine times its call into the+/// text-content analyzers (`engine.rs`, `Meta::new("page", "text-content")`),+/// and each analyzer inside records its own `text-content` event.+fn is_container(e: &Value) -> bool {+ e["phase"] == "page" && e["unit"] == "text-content"+}++fn stats(mut samples: Vec<f64>) -> Value {+ if samples.is_empty() {+ return json!(null);+ }+ samples.sort_by(|a, b| a.partial_cmp(b).unwrap());+ let pick = |p: f64| {+ samples[(((p / 100.0) * samples.len() as f64).ceil() as usize).clamp(1, samples.len()) - 1]+ };+ let mean = samples.iter().sum::<f64>() / samples.len() as f64;+ let r = |v: f64| (v * 1000.0).round() / 1000.0;+ json!({ "mean": r(mean), "p50": r(pick(50.0)), "p95": r(pick(95.0)), "max": r(*samples.last().unwrap()) })+}++fn registry(root: &Path) -> BTreeSet<String> {+ let text = std::fs::read_to_string(root.join("crates/live/assets/antipatterns.json"))+ .unwrap_or_default();+ let rows: Vec<Value> = serde_json::from_str(&text).unwrap_or_default();+ rows.iter()+ .filter_map(|r| r["id"].as_str().map(str::to_string))+ .collect()+}++fn git(root: &Path, args: &[&str]) -> String {+ Command::new("git")+ .args(args)+ .current_dir(root)+ .output()+ .ok()+ .map(|o| String::from_utf8_lossy(&o.stdout).trim().to_string())+ .unwrap_or_default()+}++fn main() {+ let args: Vec<String> = std::env::args().skip(1).collect();+ if let Some(i) = args.iter().position(|a| a == "--one") {+ let Some(path) = args.get(i + 1) else {+ eprintln!("usage: rule_cost --one <file>");+ std::process::exit(1);+ };+ match one(Path::new(path)) {+ Ok(sample) => println!("{sample}"),+ Err(e) => {+ eprintln!("{e}");+ std::process::exit(2);+ }+ }+ return;+ }++ let flag = |name: &str| {+ args.iter()+ .position(|a| a == name)+ .and_then(|i| args.get(i + 1))+ .cloned()+ };+ let runs: usize = flag("--runs").and_then(|v| v.parse().ok()).unwrap_or(9);+ let out = flag("--out").unwrap_or_else(|| "rule-cost.json".into());+ let mut inputs = Vec::new();+ let mut skip = false;+ for a in &args {+ if skip {+ skip = false;+ continue;+ }+ if a == "--runs" || a == "--out" {+ skip = true;+ continue;+ }+ if !Path::new(a).exists() {+ eprintln!("rule_cost: no such file or directory: {a}");+ std::process::exit(1);+ }+ collect(Path::new(a), &mut inputs, true);+ }+ if inputs.is_empty() {+ eprintln!("usage: rule_cost [--runs N] [--out rule-cost.json] <file-or-dir>...");+ std::process::exit(1);+ }+ let root = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../..");+ let rules = registry(&root);+ let me = std::env::current_exe().expect("own path");++ // engine -> samples; a sample is one cold run of one file.+ let mut samples: BTreeMap<String, Vec<Value>> = BTreeMap::new();+ let mut process_ms: BTreeMap<String, Vec<f64>> = BTreeMap::new();+ for run in 0..runs {+ eprint!("\rrun {}/{runs}", run + 1);+ for path in &inputs {+ let started = Instant::now();+ let output = Command::new(&me)+ .arg("--one")+ .arg(path)+ .output()+ .expect("spawn");+ let wall = started.elapsed().as_secs_f64() * 1000.0;+ // A failed sample stops the run: a report with holes in it is not a measurement.+ let sample = match serde_json::from_slice::<Value>(&output.stdout) {+ Ok(sample) if output.status.success() => sample,+ _ => {+ eprintln!(+ "\nrule_cost: the sample of {} failed ({}): {}",+ path.display(),+ output.status,+ String::from_utf8_lossy(&output.stderr).trim()+ );+ std::process::exit(1);+ }+ };+ let engine = sample["engine"].as_str().unwrap_or("?").to_string();+ process_ms.entry(engine.clone()).or_default().push(wall);+ samples.entry(engine).or_default().push(sample);+ }+ }+ eprintln!();++ let mut engines = serde_json::Map::new();+ let mut units_out = Vec::new();+ let mut by_rule: BTreeMap<String, Vec<Value>> = BTreeMap::new();+ for (engine, list) in &samples {+ // unit -> per-sample ms (0 where it did not run), phases, rules seen.+ let mut unit_ms: BTreeMap<String, Vec<f64>> = BTreeMap::new();+ let mut unit_phases: BTreeMap<String, BTreeSet<String>> = BTreeMap::new();+ let mut unit_rules: BTreeMap<String, BTreeSet<String>> = BTreeMap::new();+ let (mut totals, mut shared, mut unprofiled) = (vec![], vec![], vec![]);+ for (i, sample) in list.iter().enumerate() {+ let total = sample["totalMs"].as_f64().unwrap_or(0.0);+ let (mut shared_ms, mut attributed) = (0.0, 0.0);+ for e in sample["events"]+ .as_array()+ .into_iter()+ .flatten()+ .filter(|e| !is_container(e))+ {+ let ms = e["ms"].as_f64().unwrap_or(0.0);+ let phase = e["phase"].as_str().unwrap_or("");+ if SHARED_PHASES.contains(&phase) {+ shared_ms += ms;+ continue;+ }+ attributed += ms;+ let unit = e["unit"].as_str().unwrap_or("?").to_string();+ let slot = unit_ms.entry(unit.clone()).or_default();+ slot.resize(list.len(), 0.0);+ slot[i] += ms;+ unit_phases+ .entry(unit.clone())+ .or_default()+ .insert(phase.to_string());+ let seen = unit_rules.entry(unit.clone()).or_default();+ for r in e["rules"]+ .as_array()+ .into_iter()+ .flatten()+ .filter_map(|r| r.as_str())+ {+ seen.insert(r.to_string());+ }+ if rules.contains(&unit) {+ seen.insert(unit);+ }+ }+ totals.push(total);+ shared.push(shared_ms);+ unprofiled.push((total - shared_ms - attributed).max(0.0));+ }+ let n = list.len();+ engines.insert(+ engine.clone(),+ json!({+ "samples": n,+ "files": n / runs.max(1),+ "perEdit": stats(totals),+ "shared": stats(shared),+ "unprofiled": stats(unprofiled),+ "process": stats(process_ms.remove(engine).unwrap_or_default()),+ }),+ );+ for (unit, mut ms) in unit_ms {+ ms.resize(n, 0.0);+ let emitted: Vec<String> = unit_rules+ .remove(&unit)+ .unwrap_or_default()+ .into_iter()+ .filter(|r| rules.contains(r))+ .collect();+ let row = json!({+ "engine": engine,+ "unit": unit,+ "phases": unit_phases.remove(&unit).unwrap_or_default(),+ "rules": emitted,+ "perEdit": stats(ms),+ });+ for r in &emitted {+ by_rule.entry(r.clone()).or_default().push(json!({ "engine": engine, "unit": unit, "sharedWith": emitted.len() - 1, "perEdit": row["perEdit"] }));+ }+ units_out.push(row);+ }+ }+ units_out.sort_by(|a, b| {+ b["perEdit"]["mean"]+ .as_f64()+ .partial_cmp(&a["perEdit"]["mean"].as_f64())+ .unwrap()+ });++ let report = json!({+ "commit": git(&root, &["rev-parse", "HEAD"]),+ "dirty": !git(&root, &["status", "--porcelain", "--untracked-files=no"]).is_empty(),+ "commitDate": git(&root, &["log", "-1", "--format=%cI"]),+ "generatedAtUnix": std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_secs()).unwrap_or(0),+ "machine": format!("{} {}", std::env::consts::OS, std::env::consts::ARCH),+ "runs": runs,+ "inputs": inputs.len(),+ "method": "Each sample is a fresh process running one file through the engine the hook picks for it, cold, with the engines' profiler on. perEdit stats are over samples (file x run), in milliseconds. A unit that did not run on a file counts 0 there. Units sharing a pass are not split: sharedWith counts the other rules in the same unit.",+ "engines": engines,+ "units": units_out,+ "rules": by_rule,+ // Registry rules no hook unit emitted on these inputs: browser-only+ // rules (layout, pixels), or rules the inputs never triggered.+ "notSeen": rules.iter().filter(|r| !by_rule.contains_key(*r)).collect::<Vec<_>>(),+ });+ std::fs::write(&out, serde_json::to_string_pretty(&report).unwrap()).expect("write report");+ eprintln!("wrote {out}: {} inputs x {runs} runs", inputs.len());+}