Skip to content

The design language that makes your AI harness better at design.

JavaScript

Add test:rust-changed: test the crates a change reaches, not the whole workspace

Paul Bakausfa95cbf6fb43
diff --git a/AGENTS.md b/AGENTS.md
index 38afae21f..85f916d48 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -8,6 +8,7 @@
- `cargo build --release -p impeccable` - build this checkout's runtime into `target/release/impeccable`.
- `cargo test --workspace` - run the Rust workspace tests.
+- `bun run test:rust-changed` - run the Rust tests of the crates your change touches (against the merge base with `origin/main`, working tree included) plus every crate that depends on them; `--print` shows the selection, `--base <ref>` changes the base. Full log in `target/test-rust-changed.log`, one summary line on stdout.
- `bun run build` - source-first build: regenerate `dist/`, derived site assets, and validation output without syncing tracked harness folders.
- `bun run build:release` - release/distribution build: run the full build and sync tracked root harness folders plus `plugin/`.
- `bun run rebuild` - clean and rebuild everything from scratch without syncing tracked harness folders.
@@ -46,7 +47,7 @@ For Rust, follow the surrounding crate's conventions and workspace formatting co
Tests use Bun's test runner plus Node's built-in `--test`. Name tests `*.test.js` or `*.test.mjs` and place new fixtures near the behavior they cover, usually under `tests/fixtures/`. Prefer targeted test runs while iterating, then finish with `bun run test`. If you change generated outputs or provider transforms, verify both source parsing and at least one affected provider path in `dist/`.
-For runtime changes under `crates/`, add a failing regression in the affected crate, run its focused tests, then `cargo test --workspace`. Rebuild with `cargo build --release -p impeccable` and run `IMPECCABLE_BIN="$PWD/target/release/impeccable" bun run test` so the oracle exercises the changed source, not an older downloaded release. Review intended oracle changes by hand; never overwrite goldens just to make a regression pass. `tests/oracle/vectors/calls/` contains frozen function-level vectors and must not be regenerated.
+For runtime changes under `crates/`, add a failing regression in the affected crate, run its focused tests (`cargo test -p <crate> <filter>`), then `bun run test:rust-changed`. The whole `cargo test --workspace` takes about 10 minutes on a 10-core laptop, almost all of it in the Chrome-driven `impeccable-browser` tests (about 5.5 min) and the `impeccable` CLI tests (about 2.3 min), against under 2 minutes to compile the workspace from cold; the script skips the crates a change cannot reach, and selects the whole workspace by itself when `Cargo.toml`, `Cargo.lock` or the toolchain moves. CI's `rust` jobs run the whole workspace on every PR that touches Rust. Run any cargo command that takes more than a minute as a background command and wait for its exit notification; do not poll with `sleep` loops or `pgrep -f` (it matches the polling shell's own command line and never exits). Rebuild with `cargo build --release -p impeccable` and run `IMPECCABLE_BIN="$PWD/target/release/impeccable" bun run test` so the oracle exercises the changed source, not an older downloaded release. Review intended oracle changes by hand; never overwrite goldens just to make a regression pass. `tests/oracle/vectors/calls/` contains frozen function-level vectors and must not be regenerated.
For changes to the live-mode page JS (`skill/scripts/live-browser*.js`) or an `ENGINE_VERSION` bump, also run `bun run test:live-e2e` (kept out of the default suite because it does real `npm install` per fixture and boots framework dev servers). Scope to one fixture with `IMPECCABLE_E2E_ONLY=<fixture-name>` while iterating; pass `IMPECCABLE_E2E_DEBUG=1` for page-DOM and dev-server-log dumps on failure. Schema and authoring guide for new fixtures live in `tests/framework-fixtures/README.md`.
diff --git a/CLAUDE.md b/CLAUDE.md
index 828f63255..dad81a83f 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -163,7 +163,7 @@ bun run test:plugin-e2e # Just the plugin loader E2E (also part of the def
bun run test:cleanup # Kill live servers a previous run of THIS checkout left behind
```
-Unit tests (build orchestration, transformers, validators) run via `bun test`. Everything that spawns the engine binary (`tests/oracle.test.mjs`, `tests/framework-fixtures.test.mjs`) runs via `node --test`; both skip cleanly when no binary is found (`bun run fetch:engine` or `IMPECCABLE_BIN`). The `test` script handles this split automatically. Runtime unit and integration tests live under `crates/` and run with `cargo test --workspace`; the oracle goldens pin observable verb behavior across the same workspace.
+Unit tests (build orchestration, transformers, validators) run via `bun test`. Everything that spawns the engine binary (`tests/oracle.test.mjs`, `tests/framework-fixtures.test.mjs`) runs via `node --test`; both skip cleanly when no binary is found (`bun run fetch:engine` or `IMPECCABLE_BIN`). The `test` script handles this split automatically. Runtime unit and integration tests live under `crates/` and run with `cargo test --workspace` (about 10 minutes, mostly the Chrome-driven `impeccable-browser` tests; `bun run test:rust-changed` runs only the crates a change reaches, see AGENTS.md); the oracle goldens pin observable verb behavior across the same workspace.
### Live servers must not outlive their test process
@@ -193,7 +193,7 @@ The default suite does not cover everything. When a change touches one of these
| `ENGINE_VERSION` bump | `bun run test:new-work-e2e` | Playwright, offline, no API cost |
| `plugin/`, `skill/agents/`, `scripts/build.js`, plugin manifest validator | `bun run test:plugin-e2e` | ~1 s; already in the default suite, needs the `claude` CLI |
-For verb-level behavior changes in `crates/`, run focused crate tests and `cargo test --workspace`, then `cargo build --release -p impeccable`. Run `IMPECCABLE_BIN="$PWD/target/release/impeccable" bun run test` to exercise the changed source rather than an older downloaded release. Add a new oracle case when the contract grows and review golden changes by hand. See `docs/ENGINE.md` for browser-bundle checks and generated assets.
+For verb-level behavior changes in `crates/`, run focused crate tests and `bun run test:rust-changed` (CI runs the whole workspace), then `cargo build --release -p impeccable`. Start long cargo runs as background commands and wait for the exit notification instead of polling with `sleep` or `pgrep`. Run `IMPECCABLE_BIN="$PWD/target/release/impeccable" bun run test` to exercise the changed source rather than an older downloaded release. Add a new oracle case when the contract grows and review golden changes by hand. See `docs/ENGINE.md` for browser-bundle checks and generated assets.
**Plugin loader E2E** (`tests/plugin-e2e.test.mjs`, in the default suite): installs the committed `./plugin` subtree into a real Claude Code, sandboxed via `CLAUDE_CONFIG_DIR` in a temp dir, and asserts the component inventory from `claude plugin details`: the skill parses, every `plugin/agents/*.md` is visible, hooks are discovered. This is the only check that catches loader-contract surprises the unit guards can't know about (PR #494 shipped an `agents` manifest key that silently loaded zero agents; `claude plugin validate` never flags plugin-manifest problems). Runs in about a second; skips cleanly when the `claude` CLI is not on PATH. The known contract itself (allowed manifest keys, no `agents` key, trailing-slash `skills` path, source agents shipped) is pinned deterministically by `scripts/lib/validate-plugin-manifest.js`, unit-tested in `tests/validate-plugin-manifest.test.js` and enforced as a `bun run build` gate. Never add a key to the generated plugin manifest without verifying it against a real install and extending `KNOWN_LOADER_KEYS`.
diff --git a/package.json b/package.json
index e43cb54eb..e9cf17fc0 100644
--- a/package.json
+++ b/package.json
@@ -68,6 +68,7 @@
"test:skill-behavior": "node scripts/run-tests.mjs skill-behavior",
"test:skill-workflow": "node scripts/run-tests.mjs skill-workflow",
"test:live-svelte-adapter-deepseek": "node scripts/run-tests.mjs live-svelte-adapter-deepseek",
+ "test:rust-changed": "node scripts/test-rust-changed.mjs",
"smoke:hooks": "node scripts/smoke-provider-hooks.mjs",
"audit": "bun audit --audit-level=moderate",
"prepack": "cp README.md README.repo.md && cp README.npm.md README.md",
diff --git a/scripts/test-rust-changed.mjs b/scripts/test-rust-changed.mjs
new file mode 100644
index 000000000..9dcba68c8
--- /dev/null
+++ b/scripts/test-rust-changed.mjs
@@ -0,0 +1,163 @@
+#!/usr/bin/env node
+// Run the Rust tests for the crates a change touches and every workspace
+// crate that depends on them, instead of the whole workspace.
+//
+// node scripts/test-rust-changed.mjs [--base <ref>] [--print] [-- <cargo test args>]
+//
+// The change is the working tree (committed, staged, unstaged and untracked)
+// against the merge base with `--base` (default origin/main). A file under
+// crates/<dir>/ selects that crate. A file outside crates/ selects every crate
+// whose sources name its path or one of its directories (include_str!, a
+// fixture dir), so fixtures, browser-bundle/, ui/ and skill/scripts/ edits
+// reach the crates that read them. Cargo.toml, Cargo.lock,
+// rust-toolchain.toml and .cargo/ select the whole workspace. Then every
+// workspace crate that depends on a selected one, dev-dependencies included,
+// joins the run.
+//
+// The full cargo output goes to target/test-rust-changed.log; stdout gets the
+// selection, then one summary line (and the failing test names). The exit
+// code is cargo's. `--print` only prints the selection.
+import fs from 'node:fs';
+import path from 'node:path';
+import { execFileSync, spawn } from 'node:child_process';
+import { fileURLToPath } from 'node:url';
+
+const ROOT = path.resolve(fileURLToPath(new URL('..', import.meta.url)));
+const WORKSPACE_FILES = [/^Cargo\.(toml|lock)$/, /^rust-toolchain(\.toml)?$/, /^\.cargo\//];
+
+const argv = process.argv.slice(2);
+const dashDash = argv.indexOf('--');
+const own = dashDash === -1 ? argv : argv.slice(0, dashDash);
+const passthrough = dashDash === -1 ? [] : argv.slice(dashDash + 1);
+if (own.includes('--help') || own.includes('-h')) {
+ console.log('usage: node scripts/test-rust-changed.mjs [--base <ref>] [--print] [-- <cargo test args>]');
+ process.exit(0);
+}
+const baseAt = own.indexOf('--base');
+const base = baseAt === -1 ? 'origin/main' : own[baseAt + 1];
+if (!base) fail('--base needs a ref');
+const printOnly = own.includes('--print');
+
+const git = (args) => execFileSync('git', args, { cwd: ROOT, encoding: 'utf-8', maxBuffer: 64 * 1024 * 1024 });
+const lines = (text) => text.split(/\r?\n/).filter(Boolean);
+
+let mergeBase;
+try {
+ mergeBase = git(['merge-base', base, 'HEAD']).trim();
+} catch {
+ fail(`no merge base with ${base}; fetch it first or pass --base <ref>`);
+}
+const changed = [...new Set([
+ ...lines(git(['diff', '--name-only', mergeBase])),
+ ...lines(git(['ls-files', '--others', '--exclude-standard'])),
+])];
+
+const metadata = JSON.parse(execFileSync('cargo', ['metadata', '--format-version', '1', '--no-deps'], {
+ cwd: ROOT, encoding: 'utf-8', maxBuffer: 64 * 1024 * 1024,
+}));
+const packages = metadata.packages.map((p) => ({
+ name: p.name,
+ dir: path.relative(ROOT, path.dirname(p.manifest_path)).split(path.sep).join('/'),
+ deps: p.dependencies.filter((d) => d.path).map((d) => d.name),
+}));
+const byDir = new Map(packages.map((p) => [p.dir, p.name]));
+
+const whole = changed.filter((f) => WORKSPACE_FILES.some((re) => re.test(f)));
+const selected = new Set();
+const reasons = new Map();
+const pick = (name, why) => {
+ if (!selected.has(name)) reasons.set(name, why);
+ selected.add(name);
+};
+
+const external = [];
+for (const file of changed) {
+ const m = /^(crates\/[^/]+)\//.exec(file);
+ if (m) {
+ const name = byDir.get(m[1]);
+ if (name) pick(name, file);
+ } else if (!whole.includes(file)) {
+ external.push(file);
+ }
+}
+for (const file of external) {
+ // The file itself, then each directory above it down to two segments:
+ // `tests/fixtures/antipatterns/x.html` is also read as
+ // `tests/fixtures/antipatterns/` and `tests/fixtures/`.
+ const parts = file.split('/');
+ const terms = [file];
+ for (let i = parts.length - 1; i >= 2; i--) terms.push(`${parts.slice(0, i).join('/')}/`);
+ let hits = '';
+ try {
+ hits = git(['grep', '-l', '-F', ...terms.flatMap((t) => ['-e', t]), '--', 'crates']);
+ } catch {
+ continue; // git grep exits 1 when nothing matches
+ }
+ for (const hit of lines(hits)) {
+ const name = byDir.get(hit.split('/').slice(0, 2).join('/'));
+ if (name) pick(name, file);
+ }
+}
+
+// Reverse dependencies: anything in the workspace that depends on a selected
+// crate is selected too.
+for (let grew = true; grew;) {
+ grew = false;
+ for (const p of packages) {
+ if (selected.has(p.name)) continue;
+ const dep = p.deps.find((d) => selected.has(d));
+ if (dep) {
+ pick(p.name, `depends on ${dep}`);
+ grew = true;
+ }
+ }
+}
+
+if (whole.length) {
+ console.log(`Whole workspace: ${whole.join(', ')} changed against ${base}.`);
+} else if (!selected.size) {
+ console.log(`No Rust crate is affected by the ${changed.length} changed file(s) against ${base}.`);
+ process.exit(0);
+} else {
+ console.log(`${selected.size} of ${packages.length} crates against ${base}:`);
+ for (const name of [...selected].sort()) console.log(` ${name} (${reasons.get(name)})`);
+}
+
+const cargoArgs = ['test', ...(whole.length ? ['--workspace'] : [...selected].sort().flatMap((n) => ['-p', n])), '--no-fail-fast', ...passthrough];
+console.log(`cargo ${cargoArgs.join(' ')}`);
+if (printOnly) process.exit(0);
+
+const logPath = path.join(metadata.target_directory, 'test-rust-changed.log');
+fs.mkdirSync(path.dirname(logPath), { recursive: true });
+const log = fs.createWriteStream(logPath);
+console.log(`Log: ${logPath}`);
+const started = Date.now();
+const child = spawn('cargo', cargoArgs, { cwd: ROOT, stdio: ['ignore', 'pipe', 'pipe'] });
+let output = '';
+for (const stream of [child.stdout, child.stderr]) {
+ stream.on('data', (chunk) => {
+ log.write(chunk);
+ output += chunk;
+ });
+}
+child.on('close', (code, signal) => {
+ log.end();
+ let passed = 0;
+ let failed = 0;
+ for (const m of output.matchAll(/^test result: \w+\. (\d+) passed; (\d+) failed/gm)) {
+ passed += Number(m[1]);
+ failed += Number(m[2]);
+ }
+ const failures = [...output.matchAll(/^test (\S+) \.\.\. FAILED$/gm)].map((m) => m[1]);
+ const secs = Math.round((Date.now() - started) / 1000);
+ const status = code === 0 ? 'ok' : `FAILED (exit ${code ?? signal})`;
+ console.log(`${status}: ${passed} passed, ${failed} failed in ${secs}s`);
+ for (const name of failures) console.log(` FAILED ${name}`);
+ if (code !== 0 && !failures.length) console.log(` see the log for the error: ${logPath}`);
+ process.exit(code ?? 1);
+});
+
+function fail(message) {
+ console.error(`test-rust-changed: ${message}`);
+ process.exit(2);
+}