The design language that makes your AI harness better at design.
JavaScript
Add test:rust-changed: test the crates a change reaches, not the whole workspace
Paul Bakausfa95cbf6fb43
diff --git a/AGENTS.md b/AGENTS.mdindex 38afae21f..85f916d48 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -8,6 +8,7 @@- `cargo build --release -p impeccable` - build this checkout's runtime into `target/release/impeccable`.- `cargo test --workspace` - run the Rust workspace tests.+- `bun run test:rust-changed` - run the Rust tests of the crates your change touches (against the merge base with `origin/main`, working tree included) plus every crate that depends on them; `--print` shows the selection, `--base <ref>` changes the base. Full log in `target/test-rust-changed.log`, one summary line on stdout.- `bun run build` - source-first build: regenerate `dist/`, derived site assets, and validation output without syncing tracked harness folders.- `bun run build:release` - release/distribution build: run the full build and sync tracked root harness folders plus `plugin/`.- `bun run rebuild` - clean and rebuild everything from scratch without syncing tracked harness folders.@@ -46,7 +47,7 @@ For Rust, follow the surrounding crate's conventions and workspace formatting coTests use Bun's test runner plus Node's built-in `--test`. Name tests `*.test.js` or `*.test.mjs` and place new fixtures near the behavior they cover, usually under `tests/fixtures/`. Prefer targeted test runs while iterating, then finish with `bun run test`. If you change generated outputs or provider transforms, verify both source parsing and at least one affected provider path in `dist/`.-For runtime changes under `crates/`, add a failing regression in the affected crate, run its focused tests, then `cargo test --workspace`. Rebuild with `cargo build --release -p impeccable` and run `IMPECCABLE_BIN="$PWD/target/release/impeccable" bun run test` so the oracle exercises the changed source, not an older downloaded release. Review intended oracle changes by hand; never overwrite goldens just to make a regression pass. `tests/oracle/vectors/calls/` contains frozen function-level vectors and must not be regenerated.+For runtime changes under `crates/`, add a failing regression in the affected crate, run its focused tests (`cargo test -p <crate> <filter>`), then `bun run test:rust-changed`. The whole `cargo test --workspace` takes about 10 minutes on a 10-core laptop, almost all of it in the Chrome-driven `impeccable-browser` tests (about 5.5 min) and the `impeccable` CLI tests (about 2.3 min), against under 2 minutes to compile the workspace from cold; the script skips the crates a change cannot reach, and selects the whole workspace by itself when `Cargo.toml`, `Cargo.lock` or the toolchain moves. CI's `rust` jobs run the whole workspace on every PR that touches Rust. Run any cargo command that takes more than a minute as a background command and wait for its exit notification; do not poll with `sleep` loops or `pgrep -f` (it matches the polling shell's own command line and never exits). Rebuild with `cargo build --release -p impeccable` and run `IMPECCABLE_BIN="$PWD/target/release/impeccable" bun run test` so the oracle exercises the changed source, not an older downloaded release. Review intended oracle changes by hand; never overwrite goldens just to make a regression pass. `tests/oracle/vectors/calls/` contains frozen function-level vectors and must not be regenerated.For changes to the live-mode page JS (`skill/scripts/live-browser*.js`) or an `ENGINE_VERSION` bump, also run `bun run test:live-e2e` (kept out of the default suite because it does real `npm install` per fixture and boots framework dev servers). Scope to one fixture with `IMPECCABLE_E2E_ONLY=<fixture-name>` while iterating; pass `IMPECCABLE_E2E_DEBUG=1` for page-DOM and dev-server-log dumps on failure. Schema and authoring guide for new fixtures live in `tests/framework-fixtures/README.md`.diff --git a/CLAUDE.md b/CLAUDE.mdindex 828f63255..dad81a83f 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -163,7 +163,7 @@ bun run test:plugin-e2e # Just the plugin loader E2E (also part of the defbun run test:cleanup # Kill live servers a previous run of THIS checkout left behind```-Unit tests (build orchestration, transformers, validators) run via `bun test`. Everything that spawns the engine binary (`tests/oracle.test.mjs`, `tests/framework-fixtures.test.mjs`) runs via `node --test`; both skip cleanly when no binary is found (`bun run fetch:engine` or `IMPECCABLE_BIN`). The `test` script handles this split automatically. Runtime unit and integration tests live under `crates/` and run with `cargo test --workspace`; the oracle goldens pin observable verb behavior across the same workspace.+Unit tests (build orchestration, transformers, validators) run via `bun test`. Everything that spawns the engine binary (`tests/oracle.test.mjs`, `tests/framework-fixtures.test.mjs`) runs via `node --test`; both skip cleanly when no binary is found (`bun run fetch:engine` or `IMPECCABLE_BIN`). The `test` script handles this split automatically. Runtime unit and integration tests live under `crates/` and run with `cargo test --workspace` (about 10 minutes, mostly the Chrome-driven `impeccable-browser` tests; `bun run test:rust-changed` runs only the crates a change reaches, see AGENTS.md); the oracle goldens pin observable verb behavior across the same workspace.### Live servers must not outlive their test process@@ -193,7 +193,7 @@ The default suite does not cover everything. When a change touches one of these| `ENGINE_VERSION` bump | `bun run test:new-work-e2e` | Playwright, offline, no API cost || `plugin/`, `skill/agents/`, `scripts/build.js`, plugin manifest validator | `bun run test:plugin-e2e` | ~1 s; already in the default suite, needs the `claude` CLI |-For verb-level behavior changes in `crates/`, run focused crate tests and `cargo test --workspace`, then `cargo build --release -p impeccable`. Run `IMPECCABLE_BIN="$PWD/target/release/impeccable" bun run test` to exercise the changed source rather than an older downloaded release. Add a new oracle case when the contract grows and review golden changes by hand. See `docs/ENGINE.md` for browser-bundle checks and generated assets.+For verb-level behavior changes in `crates/`, run focused crate tests and `bun run test:rust-changed` (CI runs the whole workspace), then `cargo build --release -p impeccable`. Start long cargo runs as background commands and wait for the exit notification instead of polling with `sleep` or `pgrep`. Run `IMPECCABLE_BIN="$PWD/target/release/impeccable" bun run test` to exercise the changed source rather than an older downloaded release. Add a new oracle case when the contract grows and review golden changes by hand. See `docs/ENGINE.md` for browser-bundle checks and generated assets.**Plugin loader E2E** (`tests/plugin-e2e.test.mjs`, in the default suite): installs the committed `./plugin` subtree into a real Claude Code, sandboxed via `CLAUDE_CONFIG_DIR` in a temp dir, and asserts the component inventory from `claude plugin details`: the skill parses, every `plugin/agents/*.md` is visible, hooks are discovered. This is the only check that catches loader-contract surprises the unit guards can't know about (PR #494 shipped an `agents` manifest key that silently loaded zero agents; `claude plugin validate` never flags plugin-manifest problems). Runs in about a second; skips cleanly when the `claude` CLI is not on PATH. The known contract itself (allowed manifest keys, no `agents` key, trailing-slash `skills` path, source agents shipped) is pinned deterministically by `scripts/lib/validate-plugin-manifest.js`, unit-tested in `tests/validate-plugin-manifest.test.js` and enforced as a `bun run build` gate. Never add a key to the generated plugin manifest without verifying it against a real install and extending `KNOWN_LOADER_KEYS`.diff --git a/package.json b/package.jsonindex e43cb54eb..e9cf17fc0 100644--- a/package.json+++ b/package.json@@ -68,6 +68,7 @@"test:skill-behavior": "node scripts/run-tests.mjs skill-behavior","test:skill-workflow": "node scripts/run-tests.mjs skill-workflow","test:live-svelte-adapter-deepseek": "node scripts/run-tests.mjs live-svelte-adapter-deepseek",+ "test:rust-changed": "node scripts/test-rust-changed.mjs","smoke:hooks": "node scripts/smoke-provider-hooks.mjs","audit": "bun audit --audit-level=moderate","prepack": "cp README.md README.repo.md && cp README.npm.md README.md",diff --git a/scripts/test-rust-changed.mjs b/scripts/test-rust-changed.mjsnew file mode 100644index 000000000..9dcba68c8--- /dev/null+++ b/scripts/test-rust-changed.mjs@@ -0,0 +1,163 @@+#!/usr/bin/env node+// Run the Rust tests for the crates a change touches and every workspace+// crate that depends on them, instead of the whole workspace.+//+// node scripts/test-rust-changed.mjs [--base <ref>] [--print] [-- <cargo test args>]+//+// The change is the working tree (committed, staged, unstaged and untracked)+// against the merge base with `--base` (default origin/main). A file under+// crates/<dir>/ selects that crate. A file outside crates/ selects every crate+// whose sources name its path or one of its directories (include_str!, a+// fixture dir), so fixtures, browser-bundle/, ui/ and skill/scripts/ edits+// reach the crates that read them. Cargo.toml, Cargo.lock,+// rust-toolchain.toml and .cargo/ select the whole workspace. Then every+// workspace crate that depends on a selected one, dev-dependencies included,+// joins the run.+//+// The full cargo output goes to target/test-rust-changed.log; stdout gets the+// selection, then one summary line (and the failing test names). The exit+// code is cargo's. `--print` only prints the selection.+import fs from 'node:fs';+import path from 'node:path';+import { execFileSync, spawn } from 'node:child_process';+import { fileURLToPath } from 'node:url';++const ROOT = path.resolve(fileURLToPath(new URL('..', import.meta.url)));+const WORKSPACE_FILES = [/^Cargo\.(toml|lock)$/, /^rust-toolchain(\.toml)?$/, /^\.cargo\//];++const argv = process.argv.slice(2);+const dashDash = argv.indexOf('--');+const own = dashDash === -1 ? argv : argv.slice(0, dashDash);+const passthrough = dashDash === -1 ? [] : argv.slice(dashDash + 1);+if (own.includes('--help') || own.includes('-h')) {+ console.log('usage: node scripts/test-rust-changed.mjs [--base <ref>] [--print] [-- <cargo test args>]');+ process.exit(0);+}+const baseAt = own.indexOf('--base');+const base = baseAt === -1 ? 'origin/main' : own[baseAt + 1];+if (!base) fail('--base needs a ref');+const printOnly = own.includes('--print');++const git = (args) => execFileSync('git', args, { cwd: ROOT, encoding: 'utf-8', maxBuffer: 64 * 1024 * 1024 });+const lines = (text) => text.split(/\r?\n/).filter(Boolean);++let mergeBase;+try {+ mergeBase = git(['merge-base', base, 'HEAD']).trim();+} catch {+ fail(`no merge base with ${base}; fetch it first or pass --base <ref>`);+}+const changed = [...new Set([+ ...lines(git(['diff', '--name-only', mergeBase])),+ ...lines(git(['ls-files', '--others', '--exclude-standard'])),+])];++const metadata = JSON.parse(execFileSync('cargo', ['metadata', '--format-version', '1', '--no-deps'], {+ cwd: ROOT, encoding: 'utf-8', maxBuffer: 64 * 1024 * 1024,+}));+const packages = metadata.packages.map((p) => ({+ name: p.name,+ dir: path.relative(ROOT, path.dirname(p.manifest_path)).split(path.sep).join('/'),+ deps: p.dependencies.filter((d) => d.path).map((d) => d.name),+}));+const byDir = new Map(packages.map((p) => [p.dir, p.name]));++const whole = changed.filter((f) => WORKSPACE_FILES.some((re) => re.test(f)));+const selected = new Set();+const reasons = new Map();+const pick = (name, why) => {+ if (!selected.has(name)) reasons.set(name, why);+ selected.add(name);+};++const external = [];+for (const file of changed) {+ const m = /^(crates\/[^/]+)\//.exec(file);+ if (m) {+ const name = byDir.get(m[1]);+ if (name) pick(name, file);+ } else if (!whole.includes(file)) {+ external.push(file);+ }+}+for (const file of external) {+ // The file itself, then each directory above it down to two segments:+ // `tests/fixtures/antipatterns/x.html` is also read as+ // `tests/fixtures/antipatterns/` and `tests/fixtures/`.+ const parts = file.split('/');+ const terms = [file];+ for (let i = parts.length - 1; i >= 2; i--) terms.push(`${parts.slice(0, i).join('/')}/`);+ let hits = '';+ try {+ hits = git(['grep', '-l', '-F', ...terms.flatMap((t) => ['-e', t]), '--', 'crates']);+ } catch {+ continue; // git grep exits 1 when nothing matches+ }+ for (const hit of lines(hits)) {+ const name = byDir.get(hit.split('/').slice(0, 2).join('/'));+ if (name) pick(name, file);+ }+}++// Reverse dependencies: anything in the workspace that depends on a selected+// crate is selected too.+for (let grew = true; grew;) {+ grew = false;+ for (const p of packages) {+ if (selected.has(p.name)) continue;+ const dep = p.deps.find((d) => selected.has(d));+ if (dep) {+ pick(p.name, `depends on ${dep}`);+ grew = true;+ }+ }+}++if (whole.length) {+ console.log(`Whole workspace: ${whole.join(', ')} changed against ${base}.`);+} else if (!selected.size) {+ console.log(`No Rust crate is affected by the ${changed.length} changed file(s) against ${base}.`);+ process.exit(0);+} else {+ console.log(`${selected.size} of ${packages.length} crates against ${base}:`);+ for (const name of [...selected].sort()) console.log(` ${name} (${reasons.get(name)})`);+}++const cargoArgs = ['test', ...(whole.length ? ['--workspace'] : [...selected].sort().flatMap((n) => ['-p', n])), '--no-fail-fast', ...passthrough];+console.log(`cargo ${cargoArgs.join(' ')}`);+if (printOnly) process.exit(0);++const logPath = path.join(metadata.target_directory, 'test-rust-changed.log');+fs.mkdirSync(path.dirname(logPath), { recursive: true });+const log = fs.createWriteStream(logPath);+console.log(`Log: ${logPath}`);+const started = Date.now();+const child = spawn('cargo', cargoArgs, { cwd: ROOT, stdio: ['ignore', 'pipe', 'pipe'] });+let output = '';+for (const stream of [child.stdout, child.stderr]) {+ stream.on('data', (chunk) => {+ log.write(chunk);+ output += chunk;+ });+}+child.on('close', (code, signal) => {+ log.end();+ let passed = 0;+ let failed = 0;+ for (const m of output.matchAll(/^test result: \w+\. (\d+) passed; (\d+) failed/gm)) {+ passed += Number(m[1]);+ failed += Number(m[2]);+ }+ const failures = [...output.matchAll(/^test (\S+) \.\.\. FAILED$/gm)].map((m) => m[1]);+ const secs = Math.round((Date.now() - started) / 1000);+ const status = code === 0 ? 'ok' : `FAILED (exit ${code ?? signal})`;+ console.log(`${status}: ${passed} passed, ${failed} failed in ${secs}s`);+ for (const name of failures) console.log(` FAILED ${name}`);+ if (code !== 0 && !failures.length) console.log(` see the log for the error: ${logPath}`);+ process.exit(code ?? 1);+});++function fail(message) {+ console.error(`test-rust-changed: ${message}`);+ process.exit(2);+}