Skip to content

An open source design system that's fully customizable and agent ready

TypeScript

feat(cli): self-check in the agent-init prompt to stop CSS/Tailwind drift (#4044)

Joeyc899736a94de
diff --git a/internal/vibe-tests/prompt-purity-test/.gitignore b/internal/vibe-tests/prompt-purity-test/.gitignore
new file mode 100644
index 0000000000..cdba529df3
--- /dev/null
+++ b/internal/vibe-tests/prompt-purity-test/.gitignore
@@ -0,0 +1,8 @@
+# Runtime artifacts default to os.tmpdir()/astryx-purity (out of the repo).
+# These patterns only matter if you pass --out pointing inside the repo.
+sandbox/
+results/
+runs/
+purity-config-*.json
+purity-summary-*.json
+*.log
diff --git a/internal/vibe-tests/prompt-purity-test/ITERATE.md b/internal/vibe-tests/prompt-purity-test/ITERATE.md
new file mode 100644
index 0000000000..dec3920caf
--- /dev/null
+++ b/internal/vibe-tests/prompt-purity-test/ITERATE.md
@@ -0,0 +1,141 @@
+# ITERATE — autonomous prompt-purity search (driven by `/loop`)
+
+This is the per-tick playbook for **automatically evolving the astryx-init prompt** until
+agents reliably stop veering into raw CSS/Tailwind (or catch themselves when they do).
+It is meant to be run by the `/loop` command, self-paced (no fixed interval), gated on
+each harness run completing.
+
+**Invocation** (from the repo root, in a live Cursor agent session):
+
+```
+/loop follow internal/vibe-tests/prompt-purity-test/ITERATE.md
+```
+
+No interval => dynamic mode: run one iteration now, then wake when the current harness
+run finishes and run the next. This is FULLY AUTONOMOUS with NO iteration cap — it stops
+only on a confirmed winner or a genuine plateau (see Stop conditions). Token spend is
+accepted: a better shipped prompt saves far more downstream.
+
+Prereq (once): the `cursor-agent` CLI must be logged in (`cursor-agent status`). If it
+is not, stop and tell the user; do not loop.
+
+---
+
+## State (survives across ticks)
+
+Keep a running log so each tick can compare against every prior attempt and never
+re-test a wording:
+
+- Path: `"$TMPDIR/astryx-purity/_loop/iterations-log.json"` (outside the repo, like all
+ runtime artifacts). Create it on the first tick if absent: `{ "tick": 0, "champion":
+ "B-selfcheck", "history": [] }`.
+- Each `history[]` entry: `{ tick, expId, ranConditions, variantHashes, rates: { <cond>:
+ { veeredUncaught, caughtGivenVeered, finalPurityMean, n } }, note }`.
+- "champion" = the best variant found so far (lowest `veeredUncaught`, tie-broken by
+ higher `finalPurityMean`). Seed champion = `B-selfcheck`.
+
+`A-control` is the immutable baseline — never edit `variants/A-control.md`.
+
+---
+
+## One tick
+
+### 1. Read where we are
+- Read `iterations-log.json`. Identify the current `champion`.
+- If `history` is empty, this is the seed tick: run `A-control`, `B-selfcheck`,
+ `C-strong` together (exploration settings below) to establish the baseline and pick
+ the first champion. Skip step 2 on the seed tick.
+
+### 2. Author ONE new challenger (skip on the seed tick)
+- Based on the last run's per-run detail in `purity-summary.json` (which markers survived
+ in `veeredUncaught` runs, whether `ranDocRetrieval` was low, whether catches happened
+ late), write ONE new variant wording that targets the observed failure mode. Ideas to
+ vary (change one axis at a time so you learn what matters):
+ - imperative strength / ordering (self-check first vs last)
+ - explicit marker checklist vs terse rule
+ - mandating `astryx component`/`search` verification vs not
+ - "re-read the file" framing vs "before you output" framing
+ - length (a terse one-liner sometimes beats a wall of text)
+- Write it to a NEW file `variants/g<NN>-<slug>.md` (never overwrite A/B/C or a prior
+ generation — history must stay reproducible). Add a matching entry to
+ `conditions.json` `conditions[]`: `{ "id": "g<NN>-<slug>", "role": "candidate",
+ "variant": "g<NN>-<slug>" }`.
+- Do NOT test a wording whose text hash already appears in `history`.
+
+### 3. Run (exploration = cheap, fast)
+- Compare exactly THREE conditions so runs stay focused: control, champion, challenger.
+- Explore settings: `--prompts dd-1,tc-6 --reps 3`.
+- Start it in the BACKGROUND and arm the completion wake (dynamic `/loop`). Generate an
+ id first so the wake payload carries it:
+
+```bash
+EXP=$(node -e "console.log(require('crypto').randomBytes(4).toString('hex'))")
+( node internal/vibe-tests/prompt-purity-test/run-purity.mjs \
+ --exp "$EXP" --conditions A-control,<champion>,<challenger> --prompts dd-1,tc-6 --reps 3 \
+ --concurrency 6 ; \
+ echo "AGENT_LOOP_WAKE_purity {\"expId\":\"$EXP\",\"phase\":\"explore\"}" ) 2>&1
+```
+
+ Launch via the Shell tool with `block_until_ms: 0` and `notify_on_output` on
+ `^AGENT_LOOP_WAKE_purity`. Also arm ONE long fallback heartbeat (e.g. `sleep 1800`)
+ per the loop skill. Then END THE TURN — do not block.
+
+### 4. On wake — evaluate
+- Read `"$TMPDIR/astryx-purity/$EXP/purity-summary.json"`.
+- Append a `history` entry (rates for each condition + variant hashes).
+- Compare challenger vs champion: challenger becomes the new champion if its
+ `veeredUncaught` mean is lower (tie-break higher `finalPurityMean`). Update `champion`.
+- Report a one-line tick summary to the user: control vs champion vs challenger on
+ `veeredUncaught`, `caughtGivenVeered`, purity — and whether the champion changed.
+
+### 5. Decide: confirm, continue, or stop
+- **Promote to confirmation** when the champion beats `A-control` in exploration
+ (`veeredUncaught` lower AND purity higher). Run a CONFIRMATION:
+ `--conditions A-control,<champion> --prompts dd-1,dd-5,wd-1,wd-4,tc-6,ps-1 --reps 7`
+ (K>=5, full curated set), same background+wake pattern with `"phase":"confirm"`.
+- On a confirmation wake: if the champion is **decisive** — its `veeredUncaught` Wilson
+ upper bound is below control's lower bound AND mean purity is higher — go to Ship.
+ Otherwise record it, keep searching (back to step 2).
+- **Continue** (no interval): re-arm and start the next tick (step 2).
+
+---
+
+## Stop conditions
+
+Stop the loop (kill any watcher/heartbeat PID, do not re-arm) when ANY holds:
+
+- **Winner confirmed** — a champion passed the confirmation run decisively. Go to Ship.
+- **Plateau** — the best `veeredUncaught` has not improved across the last 3 distinct
+ challengers. Report the best-so-far; ship only if it also beats control decisively,
+ else report that no variant reliably beats control and hand back to the user.
+- **Floor** — `A-control` already shows ~0 `veeredUncaught` on the confirmation set
+ (nothing to fix); report and stop.
+- The user says stop.
+
+---
+
+## Ship (only after a confirmed winner)
+
+This completes Phase 4 / the `ship` to-do:
+
+1. Bake the winning wording into `generateCompressedIndex()` in
+ `packages/cli/src/commands/agent-docs.mjs` — add the self-check lines to the emitted
+ block (respect the `stylex` / `tailwind` / `css` styling-system branches).
+2. Update the test `packages/cli/src/commands/agent-docs.test.mjs` to assert the new
+ self-check text is present in the generated index.
+3. Run the CLI tests for that file.
+4. Report: the winning variant text, its confirmation rates vs control, and the diff.
+5. Stop the loop.
+
+---
+
+## Reporting & safety
+
+- Every tick: emit a short status (which conditions ran, the funnel deltas, champion
+ change, next action). On stop: say why and give the final winner + rates.
+- All runs stay under `"$TMPDIR/astryx-purity"` (never the repo), so parallel Cursor
+ work, file-watchers, and `git status` are untouched. Only `conditions.json` +
+ `variants/g*.md` (and, on Ship, the two CLI files) change in the repo.
+- Never edit `variants/A-control.md`. Never place `expectedComponents` in any prompt.
+- One loop only — before arming, check for an already-running `AGENT_LOOP_WAKE_purity`
+ watcher and reuse it.
diff --git a/internal/vibe-tests/prompt-purity-test/PLAN.md b/internal/vibe-tests/prompt-purity-test/PLAN.md
new file mode 100644
index 0000000000..c236de8eb5
--- /dev/null
+++ b/internal/vibe-tests/prompt-purity-test/PLAN.md
@@ -0,0 +1,138 @@
+# Prompt Purity Vibe Test
+
+**Status:** implemented (harness); iterating on variants
+**Related:** `../cli-discovery-test/PLAN.md` (blueprint), `../src/universal-eval.ts` (reused detectors), `packages/cli/src/commands/agent-docs.mjs` (the prompt under test)
+
+---
+
+## 1. The question
+
+Agents are already decent at using Astryx — but on hard tasks they **veer** into raw
+CSS / Tailwind / inline styles / hardcoded values instead of Astryx components +
+`xstyle`/tokens. We want to change the **injected init prompt** so the agent
+**self-verifies** its output and **catches itself** before finishing.
+
+> Does adding a post-generation self-check to the `astryx init` prompt make agents
+> reliably (a) *not* leave raw CSS/Tailwind in the final file, and (b) when they do
+> start to veer, **catch and correct it** — measured from the actual run trajectory?
+
+## 2. What varies (independent variable)
+
+The text that `astryx init --features agents` splices between `<!-- ASTRYX:START -->`
+markers, generated by `generateCompressedIndex()` in
+`packages/cli/src/commands/agent-docs.mjs`. We do **not** edit that source during
+experimentation. Instead each condition = a real `astryx init` AGENTS.md + one
+**variant** self-check block spliced inside the markers (`variants/*.md`,
+`conditions.json`). Only the injected prompt varies; environment, prompts, and model
+are held constant (Checker Protocol §2). The winning variant is baked into
+`generateCompressedIndex()` only at the end.
+
+## 3. What we measure (dependent variables)
+
+Per run, from the **purity timeline** — the ordered snapshots of the output `.tsx`
+captured live during the run (`run-purity.ts` via `fs.watch` + a poll fallback),
+classified in `purity-eval.ts`:
+
+- `neverVeered` — no non-Astryx styling marker ever appeared. (best)
+- `caught` — a marker appeared at some step but the **final** file is clean. (the
+ self-check working: veered, then caught)
+- `veeredUncaught` — a marker is still present at the end. (the bad outcome to minimize)
+- `finalPurity` — 0–100 graded score of the final file.
+
+**Veer markers** (see `purity-eval.ts`, tunable in one place): `className=`,
+`style={{…}}`, `.css` import / `@apply`, Tailwind utility classes, hardcoded
+`#hex`/`rgb()` colors (hard markers → drive the binary "veered"); hardcoded
+`px`/`rem`, styled raw elements (soft markers → affect the score only).
+
+Secondary signals: `ranAstryxSelfCheck` (did it run `astryx search/component/docs`
+to verify — from the logging shim), and a transcript scan for "caught-in-reasoning"
+(proposed CSS then reconsidered before writing).
+
+## 4. Harness
+
+- `setup-purity.mjs` — builds one isolated project per `condition × prompt × rep`
+ **under an out-of-workspace sandbox** (`os.tmpdir()/astryx-purity/<expId>` by
+ default; `--out` to override). Bare base = `@astryxdesign/core` + theme symlinked,
+ a **StyleX compiler plugin listed in `package.json`** so the CLI detects the
+ `stylex` styling system (the Astryx path), a logging `astryx` shim, then real
+ `astryx init` + the variant splice. Writes `tasks/<id>.json` + `purity-config-<expId>.json`.
+- `run-purity.ts` — runner. Shells out to the `cursor-agent` CLI headless (`--print
+ --output-format stream-json --model claude-opus-4-8-max-fast --force --trust
+ --workspace <sandbox project>`), snapshots the `.tsx` on every change, persists the
+ streamed events to `transcript.jsonl`, records runtime facts to `run.json`. Bounded
+ concurrency (`--concurrency`, default 3) + per-run timeout so it never starves
+ interactive Cursor usage.
+- `purity-eval.ts` — veer-marker detectors + timeline classifier (also reuses
+ `evaluate()` from `../src/universal-eval` for a supplementary graded score).
+- `purity-aggregate.ts` — per-condition Wilson 95% CI rates + comparison vs control;
+ writes `purity-summary-<expId>.json`.
+- `run-purity.mjs` — one-command orchestrator (setup → run → aggregate); the unit
+ `/loop` would drive.
+- `smoke-test.ts` — **no-LLM** verification of the classifier + setup wiring.
+
+## 5. Isolation (never interferes with parallel Cursor work)
+
+- All runtime artifacts live under an **out-of-workspace sandbox** (temp dir /
+ worktree), never in the open repo, so file-watchers, other Cursor agents, and
+ `git status` are untouched.
+- Each agent runs with `cwd` = its sandbox project and `settingSources: []`; repo
+ packages are only **read** via symlink. `fs.watch` watches only the sandbox.
+- Uses the `cursor-agent` CLI scoped to the sandbox via `--workspace`; bounded
+ concurrency + per-run timeout keep interactive Cursor usage responsive. No separate
+ API key needed — it uses your existing `cursor-agent` login.
+
+## 6. Procedure
+
+One prerequisite for LIVE runs (the no-LLM smoke test and `--dry-run` need nothing):
+the `cursor-agent` CLI must be logged in.
+
+```bash
+cursor-agent status # should say "Logged in as ..."; if not: cursor-agent login
+```
+
+```bash
+# no-LLM harness check (zero model calls) — always safe to run
+npx tsx prompt-purity-test/smoke-test.ts
+
+# one-shot: setup -> run -> aggregate (pilot default: dd-1, tc-6; A vs B; K=3)
+node prompt-purity-test/run-purity.mjs --conditions A-control,B-selfcheck --prompts dd-1,tc-6 --reps 3
+
+# or step by step
+node prompt-purity-test/setup-purity.mjs --exp <id> --conditions A-control,B-selfcheck --prompts dd-1,tc-6 --reps 3
+npx tsx prompt-purity-test/run-purity.ts --experiment <id> --concurrency 3
+npx tsx prompt-purity-test/purity-aggregate.ts --experiment <id>
+```
+
+## 7. Decision rule
+
+A variant **wins** when its `veeredUncaught` rate is significantly below control
+(its upper Wilson bound clears control's lower bound) and/or `caught | veered` and
+mean `finalPurity` are significantly higher, holding across the curated hard prompts
+at K ≥ 5. Then bake it into `generateCompressedIndex()`.
+
+## 8. Checker Protocol compliance
+
+- §2 only the SUT varies — one bare base, single-variant splice, identical prompts/env/model.
+- §3 never leak the answer — `expectedComponents` is eval-only, never in the task prompt.
+- §4 representative environment — realistic "installed @astryxdesign/core" project.
+- §5 context-free agents — fresh SDK agent per run, `settingSources: []`, K independent reps.
+
+## 9. Autonomous iteration (`/loop`)
+
+Once the prereqs are in place, the whole search runs itself. In a live Cursor agent
+session:
+
+```
+/loop follow internal/vibe-tests/prompt-purity-test/ITERATE.md
+```
+
+Dynamic (self-paced) mode: it runs one iteration now, then wakes when each harness run
+finishes (event-gated on the orchestrator's completion line) and runs the next. Per tick
+it authors ONE new challenger variant aimed at the last run's failure mode, compares
+control vs champion vs challenger cheaply (K=3 on `dd-1,tc-6`), promotes the best, then
+confirms a leader at K>=5 on the full curated set. It is fully autonomous with NO
+iteration cap — it stops only on a confirmed winner (then bakes the wording into
+`generateCompressedIndex()` and updates the CLI test — Phase 4) or a genuine plateau.
+Cross-tick memory lives in `"$TMPDIR/astryx-purity/_loop/iterations-log.json"`; all runs
+stay in the tmpdir sandbox, so your parallel Cursor work is never touched. See
+`ITERATE.md` for the full per-tick procedure and stop conditions.
diff --git a/internal/vibe-tests/prompt-purity-test/conditions.json b/internal/vibe-tests/prompt-purity-test/conditions.json
new file mode 100644
index 0000000000..2554368160
--- /dev/null
+++ b/internal/vibe-tests/prompt-purity-test/conditions.json
@@ -0,0 +1,28 @@
+{
+ "name": "prompt-purity",
+ "description": "A/B the astryx-init injected prompt (generateCompressedIndex in packages/cli/src/commands/agent-docs.mjs). Each condition = a fresh `astryx init` AGENTS.md + one variant self-check block spliced inside the <!-- ASTRYX:START/END --> markers. Only the injected prompt varies; environment, prompts, and model are held constant (Checker Protocol §2).",
+ "control": "A-control",
+ "conditions": [
+ {
+ "id": "A-control",
+ "label": "Injected prompt as-is (no self-check)",
+ "role": "control",
+ "variant": "A-control",
+ "hypothesis": "On hard tasks the agent veers into raw CSS/Tailwind and leaves it in the final file."
+ },
+ {
+ "id": "B-selfcheck",
+ "label": "+ post-generation self-check block",
+ "role": "candidate",
+ "variant": "B-selfcheck",
+ "hypothesis": "A re-read-and-fix step makes the agent catch veers it introduced, raising final purity."
+ },
+ {
+ "id": "C-strong",
+ "label": "+ stronger imperative self-check with CLI verify",
+ "role": "candidate",
+ "variant": "C-strong",
+ "hypothesis": "A more forceful, checklist-style self-check with a mandated `astryx component` verify catches more veers, at some token cost."
+ }
+ ]
+}
diff --git a/internal/vibe-tests/prompt-purity-test/prompts.json b/internal/vibe-tests/prompt-purity-test/prompts.json
new file mode 100644
index 0000000000..aab0d25fb2
--- /dev/null
+++ b/internal/vibe-tests/prompt-purity-test/prompts.json
@@ -0,0 +1,20 @@
+{
+ "name": "prompt-purity-prompts",
+ "description": "Curated veer-prone subset referenced BY ID from ../test-sets/default.json. Hard, layout/data/theming-heavy tasks are where agents abandon the design system for raw CSS/Tailwind. expectedComponents is eval-only and is NEVER shown to the agent (Checker Protocol §3). Referenced by id to avoid answer-leak and drift.",
+ "source": "../test-sets/default.json",
+ "selectionStrategy": "hardest, most veer-prone tasks across distinct categories; K>=5 reps per prompt per condition since veering is stochastic.",
+ "promptIds": [
+ {"id": "dd-1", "category": "data-display", "complexity": "complex"},
+ {"id": "dd-5", "category": "data-display", "complexity": "complex"},
+ {"id": "wd-1", "category": "workflow-description", "complexity": "complex"},
+ {"id": "wd-4", "category": "workflow-description", "complexity": "complex"},
+ {"id": "tc-6", "category": "theme-customization", "complexity": "complex"},
+ {"id": "ps-1", "category": "page-setup", "complexity": "moderate"}
+ ],
+ "pilotDefault": ["dd-1", "tc-6"],
+ "notes": [
+ "tc-6 explicitly asks for the XDS theme system 'not inline styles' — a direct veer probe.",
+ "dd-1/dd-5 (tables) and wd-4 (a full internal tool) strongly tempt hand-rolled markup + CSS.",
+ "IDs verified against ../test-sets/default.json. Do NOT copy prompt text or expectedComponents here."
+ ]
+}
diff --git a/internal/vibe-tests/prompt-purity-test/purity-aggregate.ts b/internal/vibe-tests/prompt-purity-test/purity-aggregate.ts
new file mode 100644
index 0000000000..8494461e5e
--- /dev/null
+++ b/internal/vibe-tests/prompt-purity-test/purity-aggregate.ts
@@ -0,0 +1,423 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+
+/**
+ * @file purity-aggregate.ts
+ * @input --experiment <id> [--out <base>] (reads <sandbox>/<id>/purity-config.json)
+ * @output Per-condition purity funnel with Wilson 95% CIs + comparison vs control; writes purity-summary.json
+ * @position internal/vibe-tests/prompt-purity-test — scores the prompt-purity experiment
+ *
+ * Reconstructs each run's purity timeline from its captured snapshots (the objective
+ * record of what the agent actually wrote, moment to moment), classifies veered/caught
+ * via purity-eval.ts, and reports per-condition rates with Wilson 95% CIs so "reliably
+ * catches its veers" is legible. Ground-truth CLI usage comes from the shim log.
+ *
+ * Usage: npx tsx purity-aggregate.ts --experiment <id> [--out <base>]
+ */
+
+import * as fs from 'node:fs';
+import * as os from 'node:os';
+import * as path from 'node:path';
+import {
+ classifyTimeline,
+ parseResultEvent,
+ estimateCostUSD,
+ emptyUsage,
+ addUsage,
+ type Classification,
+ type TokenUsage,
+} from './purity-eval.js';
+
+const DOC_RETRIEVAL_CMDS = new Set(['component', 'build', 'search', 'template', 'docs', 'swizzle']);
+
+function sandboxBase(out?: string): string {
+ return out || process.env.ASTRYX_PURITY_OUT || path.join(os.tmpdir(), 'astryx-purity');
+}
+
+/** Wilson score interval for a binomial proportion (95% by default). */
+function wilson(successes: number, n: number, z = 1.96): {p: number; lo: number; hi: number} {
+ if (n === 0) {return {p: 0, lo: 0, hi: 0};}
+ const phat = successes / n;
+ const z2 = z * z;
+ const denom = 1 + z2 / n;
+ const center = phat + z2 / (2 * n);
+ const margin = z * Math.sqrt((phat * (1 - phat) + z2 / (4 * n)) / n);
+ return {p: phat, lo: Math.max(0, (center - margin) / denom), hi: Math.min(1, (center + margin) / denom)};
+}
+
+interface Rate {
+ count: number;
+ n: number;
+ pct: number;
+ lo: number;
+ hi: number;
+}
+function rate<T>(items: T[], pick: (x: T) => boolean): Rate {
+ const s = items.filter(pick).length;
+ const {p, lo, hi} = wilson(s, items.length);
+ return {count: s, n: items.length, pct: p, lo, hi};
+}
+const fmtRate = (r: Rate) => `${(r.pct * 100).toFixed(0)}% [${(r.lo * 100).toFixed(0)}-${(r.hi * 100).toFixed(0)}] (${r.count}/${r.n})`;
+
+interface RunResult {
+ taskId: string;
+ basePromptId: string;
+ category: string;
+ condition: string;
+ status: string;
+ model: string;
+ completed: boolean;
+ ranAstryxCli: boolean;
+ ranDocRetrieval: boolean;
+ cliCommands: string[];
+ usage: TokenUsage | null;
+ estCostUSD: number | null;
+ apiDurationMs: number | null;
+ cls: Classification;
+}
+
+function readSnapshots(runDir: string): string[] {
+ const snapDir = path.join(runDir, 'snapshots');
+ if (!fs.existsSync(snapDir)) {return [];}
+ return fs
+ .readdirSync(snapDir)
+ .filter(f => f.endsWith('.tsx'))
+ .sort() // NNNN-<ts> prefix sorts chronologically
+ .map(f => fs.readFileSync(path.join(snapDir, f), 'utf-8'));
+}
+
+function readTranscriptText(runDir: string, cap = 200_000): string {
+ const p = path.join(runDir, 'transcript.jsonl');
+ if (!fs.existsSync(p)) {return '';}
+ let text = '';
+ for (const line of fs.readFileSync(p, 'utf-8').split('\n')) {
+ if (!line.trim()) {continue;}
+ try {
+ const ev = JSON.parse(line) as {message?: {content?: unknown}; text?: unknown};
+ const content = ev?.message?.content;
+ if (Array.isArray(content)) {
+ for (const block of content as Array<{type?: string; text?: string}>) {
+ if (block?.type === 'text' && typeof block.text === 'string') {text += block.text + '\n';}
+ }
+ } else if (typeof ev?.text === 'string') {
+ text += ev.text + '\n';
+ }
+ } catch {
+ /* skip malformed line */
+ }
+ if (text.length > cap) {break;}
+ }
+ return text;
+}
+
+function subcommand(argv: string[]): string | null {
+ return argv.find(a => !a.startsWith('-')) ?? null;
+}
+
+function readInvocationLog(logPath: string): {commands: string[]; subs: string[]} {
+ if (!fs.existsSync(logPath)) {return {commands: [], subs: []};}
+ const entries = fs
+ .readFileSync(logPath, 'utf-8')
+ .trim()
+ .split('\n')
+ .filter(Boolean)
+ .map(l => {
+ try {
+ return JSON.parse(l) as {argv: string[]};
+ } catch {
+ return null;
+ }
+ })
+ .filter((e): e is {argv: string[]} => e !== null);
+ return {
+ commands: entries.map(e => e.argv.join(' ')),
+ subs: entries.map(e => subcommand(e.argv)).filter((s): s is string => s !== null),
+ };
+}
+
+interface TaskRecord {
+ taskId: string;
+ basePromptId: string;
+ category: string;
+ condition: string;
+ runDir: string;
+ outputFile?: string;
+ invocationLog: string;
+}
+
+function evaluateRun(task: TaskRecord): RunResult {
+ const versions = readSnapshots(task.runDir);
+ // Fallback: if no snapshots were captured but a final file exists, classify that.
+ if (versions.length === 0 && task.outputFile && fs.existsSync(task.outputFile)) {
+ const finalContent = fs.readFileSync(task.outputFile, 'utf-8');
+ if (finalContent.trim()) {versions.push(finalContent);}
+ }
+ const transcriptText = readTranscriptText(task.runDir);
+ const cls = classifyTimeline(versions, {transcriptText});
+
+ const {commands, subs} = readInvocationLog(task.invocationLog);
+
+ let status = 'unknown';
+ let model = 'unknown';
+ let usage: TokenUsage | null = null;
+ let estCostUSD: number | null = null;
+ let apiDurationMs: number | null = null;
+ const runJsonPath = path.join(task.runDir, 'run.json');
+ if (fs.existsSync(runJsonPath)) {
+ try {
+ const rj = JSON.parse(fs.readFileSync(runJsonPath, 'utf-8'));
+ status = rj.status ?? 'unknown';
+ model = rj.model ?? 'unknown';
+ usage = rj.usage ?? null;
+ estCostUSD = rj.estCostUSD ?? null;
+ apiDurationMs = rj.apiDurationMs ?? null;
+ } catch {
+ /* ignore */
+ }
+ }
+ // Backfill from the transcript's result line for runs written before cost was
+ // first-class (e.g. an in-flight/seed run), so token + $ data is available everywhere.
+ if (!usage) {
+ const res = parseResultEvent(path.join(task.runDir, 'transcript.jsonl'));
+ if (res?.usage) {
+ usage = res.usage;
+ apiDurationMs = apiDurationMs ?? res.apiDurationMs;
+ }
+ }
+ if (usage && estCostUSD == null) {
+ estCostUSD = estimateCostUSD(usage, model === 'unknown' ? 'default' : model);
+ }
+
+ return {
+ taskId: task.taskId,
+ basePromptId: task.basePromptId,
+ category: task.category,
+ condition: task.condition,
+ status,
+ model,
+ completed: !cls.noOutput,
+ ranAstryxCli: commands.length > 0,
+ ranDocRetrieval: subs.some(s => DOC_RETRIEVAL_CMDS.has(s)),
+ cliCommands: commands,
+ usage,
+ estCostUSD,
+ apiDurationMs,
+ cls,
+ };
+}
+
+function conditionRuns(condDir: string): RunResult[] {
+ const tasksDir = path.join(condDir, 'tasks');
+ if (!fs.existsSync(tasksDir)) {return [];}
+ return fs
+ .readdirSync(tasksDir)
+ .filter(f => f.endsWith('.json'))
+ .map(f => JSON.parse(fs.readFileSync(path.join(tasksDir, f), 'utf-8')))
+ .map(evaluateRun);
+}
+
+function mean(nums: number[]): number {
+ return nums.length ? nums.reduce((a, b) => a + b, 0) / nums.length : 0;
+}
+
+const fmtNum = (n: number) => Math.round(n).toLocaleString('en-US');
+
+interface CostBlock {
+ n: number;
+ inputTokens: number;
+ outputTokens: number;
+ cacheReadTokens: number;
+ cacheWriteTokens: number;
+ avgInputTokens: number;
+ avgOutputTokens: number;
+ estCostUSD: number;
+}
+
+function costOf(runs: RunResult[]): CostBlock {
+ let usage = emptyUsage();
+ let estCostUSD = 0;
+ let n = 0;
+ for (const r of runs) {
+ if (!r.usage) {continue;}
+ usage = addUsage(usage, r.usage);
+ estCostUSD += r.estCostUSD ?? 0;
+ n++;
+ }
+ return {
+ n,
+ inputTokens: usage.inputTokens,
+ outputTokens: usage.outputTokens,
+ cacheReadTokens: usage.cacheReadTokens,
+ cacheWriteTokens: usage.cacheWriteTokens,
+ avgInputTokens: n ? Math.round(usage.inputTokens / n) : 0,
+ avgOutputTokens: n ? Math.round(usage.outputTokens / n) : 0,
+ estCostUSD,
+ };
+}
+
+function costByPrompt(runs: RunResult[]): Record<string, CostBlock> {
+ const groups: Record<string, RunResult[]> = {};
+ for (const r of runs) {(groups[r.basePromptId] ??= []).push(r);}
+ const out: Record<string, CostBlock> = {};
+ for (const [pid, rs] of Object.entries(groups)) {out[pid] = costOf(rs);}
+ return out;
+}
+
+function main() {
+ const argv = process.argv.slice(2);
+ const get = (flag: string) => {
+ const i = argv.indexOf(flag);
+ return i !== -1 ? argv[i + 1] : undefined;
+ };
+ const expId = get('--experiment');
+ if (!expId) {
+ console.error('Usage: purity-aggregate.ts --experiment <id> [--out <base>]');
+ process.exit(1);
+ }
+ const expDir = path.join(sandboxBase(get('--out')), expId);
+ const configPath = path.join(expDir, 'purity-config.json');
+ if (!fs.existsSync(configPath)) {
+ console.error(`No config at ${configPath}. Run setup-purity.mjs (and run-purity.ts) first.`);
+ process.exit(1);
+ }
+ const config = JSON.parse(fs.readFileSync(configPath, 'utf-8'));
+ const control: string = config.control ?? 'A-control';
+
+ type CondReport = {
+ n: number;
+ completed: Rate;
+ neverVeered: Rate;
+ caughtGivenVeered: Rate;
+ veeredUncaught: Rate;
+ ranDocRetrieval: Rate;
+ finalPurityMean: number;
+ cost: CostBlock & {byPrompt: Record<string, CostBlock>};
+ runs: unknown[];
+ };
+ const perCondition: Record<string, CondReport> = {};
+ const summary: Record<string, unknown> = {
+ experimentId: expId,
+ generatedAt: new Date().toISOString(),
+ control,
+ conditions: perCondition,
+ };
+
+ console.log(`\nPrompt Purity — experiment ${expId}`);
+ console.log(`Funnel over runs that produced output. Rates carry a Wilson 95% CI.\n`);
+ console.log(
+ `${'condition'.padEnd(14)} ${'completed'.padEnd(16)} ${'neverVeered'.padEnd(20)} ${'caught|veered'.padEnd(20)} ${'veeredUncaught'.padEnd(20)} ${'ranCLI'.padEnd(16)} purity`,
+ );
+ console.log('-'.repeat(126));
+
+ for (const cond of config.conditions as string[]) {
+ const condDir = config.conditionDirs[cond];
+ const runs = conditionRuns(condDir);
+ const done = runs.filter(r => r.completed);
+ const veered = done.filter(r => r.cls.everVeered);
+
+ const completed = rate(runs, r => r.completed);
+ const neverVeered = rate(done, r => r.cls.neverVeered);
+ const caughtGivenVeered = rate(veered, r => r.cls.caught);
+ const veeredUncaught = rate(done, r => r.cls.veeredUncaught);
+ const ranCli = rate(done, r => r.ranDocRetrieval);
+ const purityMean = mean(done.map(r => r.cls.finalPurity));
+
+ console.log(
+ `${cond.padEnd(14)} ${fmtRate(completed).padEnd(16)} ${fmtRate(neverVeered).padEnd(20)} ${fmtRate(caughtGivenVeered).padEnd(20)} ${fmtRate(veeredUncaught).padEnd(20)} ${fmtRate(ranCli).padEnd(16)} ${purityMean.toFixed(0)}`,
+ );
+
+ perCondition[cond] = {
+ n: runs.length,
+ completed,
+ neverVeered,
+ caughtGivenVeered,
+ veeredUncaught,
+ ranDocRetrieval: ranCli,
+ finalPurityMean: purityMean,
+ cost: {...costOf(runs), byPrompt: costByPrompt(runs)},
+ runs: runs.map(r => ({
+ taskId: r.taskId,
+ basePromptId: r.basePromptId,
+ category: r.category,
+ status: r.status,
+ completed: r.completed,
+ everVeered: r.cls.everVeered,
+ caught: r.cls.caught,
+ veeredUncaught: r.cls.veeredUncaught,
+ neverVeered: r.cls.neverVeered,
+ finalPurity: r.cls.finalPurity,
+ peakHardCount: r.cls.peakHardCount,
+ finalHardCount: r.cls.finalHardCount,
+ removedHardCount: r.cls.removedHardCount,
+ finalMarkers: r.cls.finalMarkers,
+ ranDocRetrieval: r.ranDocRetrieval,
+ cliCommands: r.cliCommands,
+ caughtInReasoning: r.cls.caughtInReasoning,
+ usage: r.usage,
+ estCostUSD: r.estCostUSD,
+ apiDurationMs: r.apiDurationMs,
+ })),
+ };
+ }
+
+ // ── Comparison vs control ──────────────────────────────────────────
+ const base = perCondition[control];
+ if (base) {
+ console.log(`\nvs control (${control}) — lower veeredUncaught + higher purity is better:\n`);
+ for (const cond of config.conditions as string[]) {
+ if (cond === control) {continue;}
+ const c = perCondition[cond];
+ const vuDelta = (c.veeredUncaught.pct - base.veeredUncaught.pct) * 100;
+ const purDelta = c.finalPurityMean - base.finalPurityMean;
+ const decisive = c.veeredUncaught.hi < base.veeredUncaught.lo;
+ console.log(
+ ` ${cond.padEnd(14)} veeredUncaught ${vuDelta >= 0 ? '+' : ''}${vuDelta.toFixed(0)}pp purity ${purDelta >= 0 ? '+' : ''}${purDelta.toFixed(0)} ${decisive ? 'DECISIVE (CI clears control)' : '(CIs overlap — need more reps)'}`,
+ );
+ }
+ }
+
+ // ── Cost (exact tokens; $ is an estimate) ──────────────────────────
+ const totals = emptyUsage();
+ let totalCost = 0;
+ let totalRuns = 0;
+ console.log(`\nCost — exact token counts; $ is an ESTIMATE (per-model table, cache-read priced low):\n`);
+ console.log(
+ `${'condition'.padEnd(14)} ${'runs'.padEnd(5)} ${'input'.padEnd(12)} ${'output'.padEnd(12)} ${'cacheRead'.padEnd(14)} ${'cacheWrite'.padEnd(13)} ${'avgOut'.padEnd(8)} est$`,
+ );
+ console.log('-'.repeat(100));
+ for (const cond of config.conditions as string[]) {
+ const c = perCondition[cond].cost as CostBlock;
+ console.log(
+ `${cond.padEnd(14)} ${String(c.n).padEnd(5)} ${fmtNum(c.inputTokens).padEnd(12)} ${fmtNum(c.outputTokens).padEnd(12)} ${fmtNum(c.cacheReadTokens).padEnd(14)} ${fmtNum(c.cacheWriteTokens).padEnd(13)} ${fmtNum(c.avgOutputTokens).padEnd(8)} $${c.estCostUSD.toFixed(2)}`,
+ );
+ totals.inputTokens += c.inputTokens;
+ totals.outputTokens += c.outputTokens;
+ totals.cacheReadTokens += c.cacheReadTokens;
+ totals.cacheWriteTokens += c.cacheWriteTokens;
+ totalCost += c.estCostUSD;
+ totalRuns += c.n;
+ }
+ console.log('-'.repeat(100));
+ console.log(
+ `${'TOTAL'.padEnd(14)} ${String(totalRuns).padEnd(5)} ${fmtNum(totals.inputTokens).padEnd(12)} ${fmtNum(totals.outputTokens).padEnd(12)} ${fmtNum(totals.cacheReadTokens).padEnd(14)} ${fmtNum(totals.cacheWriteTokens).padEnd(13)} ${''.padEnd(8)} $${totalCost.toFixed(2)}`,
+ );
+ console.log(`\n($ is an estimate — on a Cursor plan real billing may be request-based; token counts above are exact.)`);
+
+ summary.cost = {
+ total: {
+ runs: totalRuns,
+ inputTokens: totals.inputTokens,
+ outputTokens: totals.outputTokens,
+ cacheReadTokens: totals.cacheReadTokens,
+ cacheWriteTokens: totals.cacheWriteTokens,
+ estCostUSD: totalCost,
+ },
+ note: 'Token counts are exact (cursor-agent stream-json result events). estCostUSD is an estimate from the per-model PRICING table in purity-eval.ts; re-scale for graphs.',
+ };
+
+ const outPath = path.join(expDir, 'purity-summary.json');
+ fs.writeFileSync(outPath, JSON.stringify(summary, null, 2));
+ console.log(`\nSummary: ${outPath}`);
+ console.log(`Read: a variant wins when its veeredUncaught CI clears control's and purity is higher, across K>=5 reps.\n`);
+}
+
+main();
diff --git a/internal/vibe-tests/prompt-purity-test/purity-eval.ts b/internal/vibe-tests/prompt-purity-test/purity-eval.ts
new file mode 100644
index 0000000000..75270c642f
--- /dev/null
+++ b/internal/vibe-tests/prompt-purity-test/purity-eval.ts
@@ -0,0 +1,343 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+
+/**
+ * @file purity-eval.ts
+ * @input generated .tsx code (single version) or an ordered array of versions (a run's snapshot timeline)
+ * @output veer-marker hits, a 0-100 purity score, and a per-run veered/caught classification
+ * @position internal/vibe-tests/prompt-purity-test — the "did it veer, did we catch it" detector
+ *
+ * This is the ONE place that defines what counts as "veering off Astryx". Tune here.
+ *
+ * HARD markers (drive the binary "veered" flag): unambiguous non-Astryx styling —
+ * className=, inline style={{...}}, imported .css / @apply / @import, Tailwind
+ * utility classes. In the Astryx (stylex) path these should never appear; styling
+ * is Astryx components + the `xstyle` prop + tokens.
+ * SOFT markers (affect the score, NOT the binary flag): hardcoded #hex/rgb() colors
+ * and px/rem sizes. Kept soft because a hex inside a theme/token definition
+ * (createTheme/defineVars) is legitimate Astryx, and we don't want to punish that.
+ */
+
+import * as fs from 'node:fs';
+
+export type VeerType =
+ | 'className'
+ | 'inlineStyle'
+ | 'cssImport'
+ | 'tailwind'
+ | 'hardcodedColor'
+ | 'hardcodedSize';
+
+export type Severity = 'hard' | 'soft';
+
+export interface MarkerHit {
+ type: VeerType;
+ severity: Severity;
+ count: number;
+ example: string;
+}
+
+export interface Classification {
+ versions: number;
+ noOutput: boolean;
+ everVeered: boolean;
+ finalClean: boolean;
+ /** everVeered && finalClean — tried CSS/Tailwind, then fixed it (the self-check working). */
+ caught: boolean;
+ /** everVeered && !finalClean — non-Astryx styling still present at the end (bad). */
+ veeredUncaught: boolean;
+ /** produced clean output and never introduced a hard marker (best). */
+ neverVeered: boolean;
+ finalPurity: number;
+ peakHardCount: number;
+ finalHardCount: number;
+ /** how many hard veers were present at peak but gone by the end (evidence of catching). */
+ removedHardCount: number;
+ finalMarkers: MarkerHit[];
+ /** secondary, transcript-based: agent mentioned CSS/Tailwind in reasoning. */
+ mentionedVeerInReasoning: boolean;
+ /** mentioned a veer but never wrote a hard marker — caught itself pre-write. */
+ caughtInReasoning: boolean;
+}
+
+// ── className string extraction (for Tailwind detection) ──────────────
+
+/** Pull the contents of className="..." / '...' / {`...`} literals. */
+function extractClassNameLiterals(code: string): string[] {
+ const out: string[] = [];
+ const re = /className\s*=\s*(?:"([^"]*)"|'([^']*)'|\{\s*`([^`]*)`\s*\}|\{\s*"([^"]*)"\s*\}|\{\s*'([^']*)'\s*\})/g;
+ let m: RegExpExecArray | null;
+ while ((m = re.exec(code)) !== null) {
+ const val = m[1] ?? m[2] ?? m[3] ?? m[4] ?? m[5];
+ if (val) {out.push(val);}
+ }
+ return out;
+}
+
+const TW_TOKEN =
+ /^(?:(?:sm|md|lg|xl|2xl|hover|focus|focus-visible|active|disabled|group-hover|dark|first|last|odd|even)?:)*(?:flex|grid|block|inline-block|inline-flex|inline|hidden|container|isolate|(?:p|m|px|py|pt|pb|pl|pr|mx|my|mt|mb|ml|mr|gap|gap-x|gap-y|space-x|space-y|w|h|min-w|max-w|min-h|max-h|top|bottom|left|right|inset|size)-[\w./%[\]#-]+|text-(?:xs|sm|base|lg|xl|\dxl|left|center|right|justify|[a-z]+-\d{2,3})|leading-\w+|tracking-\w+|font-(?:thin|extralight|light|normal|medium|semibold|bold|extrabold|black|mono|sans|serif)|bg-[\w./%[\]#-]+|border(?:-[\w./%[\]#-]+)?|rounded(?:-[\w]+)?|shadow(?:-[\w]+)?|ring(?:-[\w]+)?|items-\w+|justify-\w+|content-\w+|self-\w+|place-\w+|flex-\w+|grow(?:-\d)?|shrink(?:-\d)?|basis-[\w./%-]+|order-\w+|col-\w+|row-\w+|grid-cols-\w+|grid-rows-\w+|z-\d+|opacity-\d+|cursor-\w+|select-\w+|pointer-events-\w+|overflow-\w+|whitespace-\w+|truncate|uppercase|lowercase|capitalize|italic|underline|absolute|relative|fixed|sticky|static|transition(?:-\w+)?|duration-\d+|ease-\w+|animate-\w+|divide-[\w./%-]+)$/;
+
+function countTailwindTokens(code: string): {count: number; example: string} {
+ let count = 0;
+ let example = '';
+ for (const literal of extractClassNameLiterals(code)) {
+ for (const tok of literal.split(/\s+/).filter(Boolean)) {
+ if (TW_TOKEN.test(tok)) {
+ count++;
+ if (!example) {example = tok;}
+ }
+ }
+ }
+ return {count, example};
+}
+
+// ── Marker detectors ─────────────────────────────────────────────────
+
+interface Detector {
+ type: VeerType;
+ severity: Severity;
+ detect: (code: string) => {count: number; example: string};
+}
+
+function regexDetector(type: VeerType, severity: Severity, re: RegExp): Detector {
+ return {
+ type,
+ severity,
+ detect: (code: string) => {
+ const matches = code.match(re) || [];
+ return {count: matches.length, example: matches[0]?.trim().slice(0, 60) ?? ''};
+ },
+ };
+}
+
+const DETECTORS: Detector[] = [
+ regexDetector('className', 'hard', /\bclassName\s*=/g),
+ regexDetector('inlineStyle', 'hard', /\bstyle\s*=\s*\{\{/g),
+ regexDetector('cssImport', 'hard', /(?:import[^;\n]*["'][^"']+\.css["']|@apply\b|@import\b)/g),
+ {type: 'tailwind', severity: 'hard', detect: countTailwindTokens},
+ regexDetector('hardcodedColor', 'soft', /#[0-9a-fA-F]{3,8}\b|\b(?:rgb|rgba|hsl|hsla)\s*\(/g),
+ regexDetector('hardcodedSize', 'soft', /\b\d{2,}(?:px|rem)\b/g),
+];
+
+/** All marker hits (count > 0) for a single code string. */
+export function detectMarkers(code: string): MarkerHit[] {
+ const hits: MarkerHit[] = [];
+ for (const d of DETECTORS) {
+ const {count, example} = d.detect(code);
+ if (count > 0) {hits.push({type: d.type, severity: d.severity, count, example});}
+ }
+ return hits;
+}
+
+export function hardMarkers(code: string): MarkerHit[] {
+ return detectMarkers(code).filter(h => h.severity === 'hard');
+}
+
+export function hardMarkerCount(code: string): number {
+ return hardMarkers(code).reduce((n, h) => n + h.count, 0);
+}
+
+// Per-type penalty weight and cap for the graded purity score.
+const PENALTY: Record<VeerType, {weight: number; cap: number}> = {
+ className: {weight: 10, cap: 40},
+ inlineStyle: {weight: 10, cap: 40},
+ cssImport: {weight: 20, cap: 40},
+ tailwind: {weight: 3, cap: 30},
+ hardcodedColor: {weight: 4, cap: 20},
+ hardcodedSize: {weight: 2, cap: 16},
+};
+
+/** 0-100 purity of a single code string (100 = pure Astryx, 0 = drowning in raw CSS/TW). */
+export function purityScore(code: string): number {
+ if (!code || !code.trim()) {return 0;}
+ let penalty = 0;
+ for (const h of detectMarkers(code)) {
+ const {weight, cap} = PENALTY[h.type];
+ penalty += Math.min(cap, h.count * weight);
+ }
+ return Math.max(0, Math.min(100, 100 - penalty));
+}
+
+const REASONING_VEER =
+ /\b(tailwind|plain css|raw css|vanilla css|inline styles?|hand-?roll|hardcod\w*|className)\b/i;
+
+/**
+ * Classify a run from its ordered snapshot timeline (each entry = the full file
+ * contents at that moment). `transcriptText` is optional corroborating reasoning text.
+ */
+export function classifyTimeline(
+ versions: string[],
+ opts: {transcriptText?: string} = {},
+): Classification {
+ const nonEmpty = versions.filter(v => v && v.trim().length > 0);
+ const noOutput = nonEmpty.length === 0;
+
+ const hardCounts = versions.map(v => hardMarkerCount(v));
+ const everVeered = hardCounts.some(n => n > 0);
+ const peakHardCount = hardCounts.length ? Math.max(...hardCounts) : 0;
+
+ const final = versions.length ? versions[versions.length - 1] : '';
+ const finalMarkers = detectMarkers(final);
+ const finalHardCount = finalMarkers.filter(h => h.severity === 'hard').reduce((n, h) => n + h.count, 0);
+ const finalClean = !noOutput && finalHardCount === 0;
+ const finalPurity = purityScore(final);
+
+ const caught = everVeered && finalClean;
+ const veeredUncaught = everVeered && !finalClean;
+ const neverVeered = !everVeered && !noOutput;
+
+ const mentionedVeerInReasoning = opts.transcriptText
+ ? REASONING_VEER.test(opts.transcriptText)
+ : false;
+ const caughtInReasoning = mentionedVeerInReasoning && neverVeered;
+
+ return {
+ versions: versions.length,
+ noOutput,
+ everVeered,
+ finalClean,
+ caught,
+ veeredUncaught,
+ neverVeered,
+ finalPurity,
+ peakHardCount,
+ finalHardCount,
+ removedHardCount: Math.max(0, peakHardCount - finalHardCount),
+ finalMarkers,
+ mentionedVeerInReasoning,
+ caughtInReasoning,
+ };
+}
+
+/**
+ * Supplementary graded quality via the shared deterministic scorer. Lazy + guarded so
+ * the core classifier stays dependency-free (and the no-LLM smoke test never needs it).
+ */
+export async function gradedQuality(code: string): Promise<number | null> {
+ try {
+ const mod = (await import('../src/universal-eval.js')) as {
+ evaluate: (code: string, target: string) => unknown;
+ getAverageScore?: (score: unknown) => number;
+ };
+ const score = mod.evaluate(code, 'astryx');
+ return typeof mod.getAverageScore === 'function' ? mod.getAverageScore(score) : null;
+ } catch {
+ return null;
+ }
+}
+
+// ── Cost: parse the cursor-agent terminal `result` event + estimate $ ─────────
+//
+// Token counts are EXACT (from the CLI's stream-json result line). The $ is an
+// ESTIMATE from the per-model table below — on a Cursor plan actual billing may be
+// request-based, so treat $ as directional and re-scale PRICING for your graphs.
+// The raw token counts are the hard numbers to plot.
+
+export interface TokenUsage {
+ inputTokens: number;
+ outputTokens: number;
+ cacheReadTokens: number;
+ cacheWriteTokens: number;
+}
+
+export interface AgentResult {
+ subtype: string | null;
+ isError: boolean;
+ durationMs: number | null;
+ apiDurationMs: number | null;
+ usage: TokenUsage | null;
+}
+
+export interface ModelPricing {
+ inputPerM: number;
+ outputPerM: number;
+ cacheReadPerM: number;
+ cacheWritePerM: number;
+}
+
+/** USD per 1M tokens. Estimates — edit freely; cache-read is intentionally cheap. */
+export const PRICING: Record<string, ModelPricing> = {
+ // Opus 4.x class
+ 'claude-opus-4-8-max-fast': {inputPerM: 15, outputPerM: 75, cacheReadPerM: 1.5, cacheWritePerM: 18.75},
+ 'claude-opus-4-8-thinking-max-fast': {inputPerM: 15, outputPerM: 75, cacheReadPerM: 1.5, cacheWritePerM: 18.75},
+ default: {inputPerM: 15, outputPerM: 75, cacheReadPerM: 1.5, cacheWritePerM: 18.75},
+};
+
+export function pricingFor(model: string): ModelPricing {
+ return PRICING[model] ?? PRICING.default;
+}
+
+export function estimateCostUSD(usage: TokenUsage, model: string): number {
+ const p = pricingFor(model);
+ return (
+ (usage.inputTokens * p.inputPerM +
+ usage.outputTokens * p.outputPerM +
+ usage.cacheReadTokens * p.cacheReadPerM +
+ usage.cacheWriteTokens * p.cacheWritePerM) /
+ 1_000_000
+ );
+}
+
+const numOrNull = (v: unknown): number | null => (typeof v === 'number' && Number.isFinite(v) ? v : null);
+
+/**
+ * Parse the last `type:"result"` line out of a cursor-agent stream-json transcript.
+ * Tolerant to camelCase (cursor-agent) and snake_case (Claude-style) field names.
+ */
+export function parseResultFromText(text: string): AgentResult | null {
+ if (!text) {return null;}
+ const lines = text.split('\n');
+ for (let i = lines.length - 1; i >= 0; i--) {
+ const line = lines[i];
+ if (!line.includes('"type":"result"') && !line.includes('"type": "result"')) {continue;}
+ try {
+ const ev = JSON.parse(line) as {
+ type?: string;
+ subtype?: string;
+ is_error?: boolean;
+ duration_ms?: number;
+ duration_api_ms?: number;
+ usage?: Record<string, unknown>;
+ };
+ if (ev?.type !== 'result') {continue;}
+ const u: Record<string, unknown> = ev.usage ?? {};
+ const usage: TokenUsage = {
+ inputTokens: numOrNull(u.inputTokens ?? u.input_tokens) ?? 0,
+ outputTokens: numOrNull(u.outputTokens ?? u.output_tokens) ?? 0,
+ cacheReadTokens: numOrNull(u.cacheReadTokens ?? u.cache_read_input_tokens ?? u.cacheReadInputTokens) ?? 0,
+ cacheWriteTokens:
+ numOrNull(u.cacheWriteTokens ?? u.cache_creation_input_tokens ?? u.cacheWriteInputTokens) ?? 0,
+ };
+ return {
+ subtype: ev.subtype ?? null,
+ isError: !!ev.is_error,
+ durationMs: numOrNull(ev.duration_ms),
+ apiDurationMs: numOrNull(ev.duration_api_ms) ?? numOrNull(ev.duration_ms),
+ usage,
+ };
+ } catch {
+ /* malformed line — keep scanning upward */
+ }
+ }
+ return null;
+}
+
+export function parseResultEvent(transcriptPath: string): AgentResult | null {
+ try {
+ return parseResultFromText(fs.readFileSync(transcriptPath, 'utf-8'));
+ } catch {
+ return null;
+ }
+}
+
+export function emptyUsage(): TokenUsage {
+ return {inputTokens: 0, outputTokens: 0, cacheReadTokens: 0, cacheWriteTokens: 0};
+}
+
+export function addUsage(a: TokenUsage, b: TokenUsage): TokenUsage {
+ return {
+ inputTokens: a.inputTokens + b.inputTokens,
+ outputTokens: a.outputTokens + b.outputTokens,
+ cacheReadTokens: a.cacheReadTokens + b.cacheReadTokens,
+ cacheWriteTokens: a.cacheWriteTokens + b.cacheWriteTokens,
+ };
+}
diff --git a/internal/vibe-tests/prompt-purity-test/run-purity.mjs b/internal/vibe-tests/prompt-purity-test/run-purity.mjs
new file mode 100644
index 0000000000..2495c73aea
--- /dev/null
+++ b/internal/vibe-tests/prompt-purity-test/run-purity.mjs
@@ -0,0 +1,78 @@
+#!/usr/bin/env node
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+
+/**
+ * @file run-purity.mjs
+ * @input CLI flags (forwarded): --conditions, --prompts, --sample, --reps, --out, --concurrency, --model, --dry-run
+ * @output Runs setup -> SDK run -> aggregate for one fresh experiment id
+ * @position internal/vibe-tests/prompt-purity-test — one-command orchestrator
+ *
+ * This is the unit the /loop command would drive: tweak variants/*.md, then re-run
+ * this to get a fresh A/B comparison. Requires the cursor-agent CLI to be logged in
+ * (`cursor-agent status`) unless --dry-run.
+ *
+ * Usage:
+ * node run-purity.mjs # A vs B, pilot prompts, 3 reps
+ * node run-purity.mjs --conditions A-control,B-selfcheck,C-strong --sample 6 --reps 5 --concurrency 4
+ * node run-purity.mjs --dry-run # wire-check without model calls
+ *
+ * For AUTONOMOUS iteration, don't drive this file directly with /loop — follow the
+ * playbook, which self-paces (event-gated on each run completing), evolves challenger
+ * variants, confirms at K>=5, and ships the winner:
+ * /loop follow internal/vibe-tests/prompt-purity-test/ITERATE.md
+ */
+
+import * as crypto from 'node:crypto';
+import * as path from 'node:path';
+import {execFileSync} from 'node:child_process';
+import {fileURLToPath} from 'node:url';
+
+const EXP_DIR = path.dirname(fileURLToPath(import.meta.url));
+
+function passthrough(argv, flag) {
+ const i = argv.indexOf(flag);
+ return i !== -1 ? argv[i + 1] : undefined;
+}
+
+function forward(argv, flags) {
+ const out = [];
+ for (const f of flags) {
+ const v = passthrough(argv, f);
+ if (v !== undefined) out.push(f, v);
+ }
+ return out;
+}
+
+function run(cmd, args) {
+ console.log(`\n$ ${cmd} ${args.join(' ')}\n`);
+ execFileSync(cmd, args, {stdio: 'inherit', cwd: EXP_DIR});
+}
+
+function main() {
+ const argv = process.argv.slice(2);
+ const dryRun = argv.includes('--dry-run');
+ const expId = passthrough(argv, '--exp') ?? crypto.randomBytes(4).toString('hex');
+
+ // 1. setup — build isolated sandbox projects per condition x prompt x rep
+ run('node', [
+ path.join(EXP_DIR, 'setup-purity.mjs'),
+ '--exp', expId,
+ ...forward(argv, ['--out', '--conditions', '--prompts', '--sample', '--reps']),
+ ]);
+
+ // 2. run — Cursor SDK, captures snapshots + transcript per run
+ run('npx', [
+ 'tsx',
+ path.join(EXP_DIR, 'run-purity.ts'),
+ '--experiment', expId,
+ ...forward(argv, ['--out', '--concurrency', '--model', '--conditions', '--timeout']),
+ ...(dryRun ? ['--dry-run'] : []),
+ ]);
+
+ // 3. aggregate — per-condition Wilson-CI funnel + comparison vs control
+ run('npx', ['tsx', path.join(EXP_DIR, 'purity-aggregate.ts'), '--experiment', expId, ...forward(argv, ['--out'])]);
+
+ console.log(`\nExperiment ${expId} complete.`);
+}
+
+main();
diff --git a/internal/vibe-tests/prompt-purity-test/run-purity.ts b/internal/vibe-tests/prompt-purity-test/run-purity.ts
new file mode 100644
index 0000000000..7d3332a450
--- /dev/null
+++ b/internal/vibe-tests/prompt-purity-test/run-purity.ts
@@ -0,0 +1,367 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+
+/**
+ * @file run-purity.ts
+ * @input --experiment <id> [--out <base>] [--concurrency N] [--model <id>] [--timeout ms] [--conditions ...] [--dry-run]
+ * @output Per task: <runDir>/transcript.jsonl, <runDir>/snapshots/NNNN-*.tsx, <runDir>/run.json
+ * @position internal/vibe-tests/prompt-purity-test — the runner
+ *
+ * Runs each task's context-free agent via the `cursor-agent` CLI (headless) on Opus 4.8
+ * 1M Max Fast, capturing the FULL trajectory so purity-aggregate.ts can tell whether the
+ * agent veered off Astryx and whether it caught itself.
+ *
+ * Why cursor-agent (not the SDK): it ships with Cursor, uses your existing login (no npm
+ * dependency, no registry/API-key setup), and `--print --output-format stream-json`
+ * streams every message + tool call. Confirm auth with `cursor-agent status`.
+ *
+ * ISOLATION (never interferes with parallel Cursor work):
+ * - Each agent runs with cwd + --workspace = the sandbox project (under $TMPDIR); it
+ * can't touch your main checkout. Repo packages are only symlinked (read-only).
+ * - fs.watch watches only the sandbox project dir.
+ * - Bounded concurrency (--concurrency, default 3), per-run timeout, killed on stop.
+ *
+ * Usage:
+ * npx tsx run-purity.ts --experiment <id>
+ * npx tsx run-purity.ts --experiment <id> --concurrency 4 --model claude-opus-4-8-max-fast
+ * npx tsx run-purity.ts --experiment <id> --dry-run # exercise the pipeline, no model calls
+ */
+
+import * as fs from 'node:fs';
+import * as os from 'node:os';
+import * as path from 'node:path';
+import {spawn, spawnSync} from 'node:child_process';
+import {parseResultFromText, estimateCostUSD, type TokenUsage, type AgentResult} from './purity-eval.js';
+
+const DEFAULT_MODEL = 'claude-opus-4-8-max-fast'; // Opus 4.8 1M Max Fast
+const DEFAULT_AGENT = 'cursor-agent';
+const DEFAULT_TIMEOUT_MS = 15 * 60 * 1000;
+
+const ensureDir = (d: string) => fs.mkdirSync(d, {recursive: true});
+const writeJson = (p: string, o: unknown) => fs.writeFileSync(p, JSON.stringify(o, null, 2));
+
+function sandboxBase(out?: string): string {
+ return out || process.env.ASTRYX_PURITY_OUT || path.join(os.tmpdir(), 'astryx-purity');
+}
+
+interface TaskFile {
+ taskId: string;
+ basePromptId: string;
+ category: string;
+ condition: string;
+ projectDir: string;
+ outputFile: string;
+ runDir: string;
+ taskPrompt: string;
+}
+
+interface RunJson {
+ taskId: string;
+ condition: string;
+ basePromptId: string;
+ category: string;
+ model: string;
+ status: string;
+ error: string | null;
+ subtype: string | null;
+ isError: boolean;
+ startedAt: string;
+ finishedAt: string;
+ durationMs: number;
+ apiDurationMs: number | null;
+ usage: TokenUsage | null;
+ estCostUSD: number | null;
+ snapshotCount: number;
+ outputExists: boolean;
+}
+
+// ── Live snapshotting of the output file (the purity timeline) ────────
+
+function startSnapshotter(targetFile: string, snapDir: string) {
+ ensureDir(snapDir);
+ let seq = 0;
+ let last: string | null = null;
+ let debounce: NodeJS.Timeout | null = null;
+
+ const capture = () => {
+ try {
+ if (!fs.existsSync(targetFile)) {return;}
+ const content = fs.readFileSync(targetFile, 'utf-8');
+ if (content === last) {return;}
+ last = content;
+ const name = `${String(seq++).padStart(4, '0')}-${Date.now()}.tsx`;
+ fs.writeFileSync(path.join(snapDir, name), content);
+ } catch {
+ /* transient read during write — the poll will catch the settled version */
+ }
+ };
+
+ const dir = path.dirname(targetFile);
+ ensureDir(dir);
+ const base = path.basename(targetFile);
+ let watcher: fs.FSWatcher | null = null;
+ try {
+ watcher = fs.watch(dir, (_event, filename) => {
+ if (filename && filename.toString() !== base) {return;}
+ if (debounce) {clearTimeout(debounce);}
+ debounce = setTimeout(capture, 80);
+ });
+ } catch {
+ /* fs.watch unsupported on some FS — the poll below is the fallback */
+ }
+ const poll = setInterval(capture, 500);
+
+ return {
+ stop: () => {
+ if (debounce) {clearTimeout(debounce);}
+ clearInterval(poll);
+ try {
+ watcher?.close();
+ } catch {
+ /* ignore */
+ }
+ capture(); // guaranteed final snapshot
+ },
+ };
+}
+
+// ── Bounded concurrency pool ─────────────────────────────────────────
+
+async function pool<T, R>(items: T[], limit: number, worker: (item: T, i: number) => Promise<R>): Promise<R[]> {
+ const results = new Array<R>(items.length);
+ let cursor = 0;
+ const runners = Array.from({length: Math.min(Math.max(1, limit), items.length || 1)}, async () => {
+ while (true) {
+ const i = cursor++;
+ if (i >= items.length) {break;}
+ results[i] = await worker(items[i], i);
+ }
+ });
+ await Promise.all(runners);
+ return results;
+}
+
+// ── One run (spawn cursor-agent headless) ────────────────────────────
+
+function runOne(cfg: {model: string; agent: string; timeoutMs: number}, task: TaskFile): Promise<RunJson> {
+ return new Promise(resolve => {
+ ensureDir(task.runDir);
+ const transcriptPath = path.join(task.runDir, 'transcript.jsonl');
+ const snapDir = path.join(task.runDir, 'snapshots');
+ const snap = startSnapshotter(task.outputFile, snapDir);
+ const startedAt = new Date().toISOString();
+ const t0 = Date.now();
+
+ let done = false;
+ let transcriptBuf = '';
+ const finalize = (status: string, error: string | null) => {
+ if (done) {return;}
+ done = true;
+ snap.stop();
+ const snapshotCount = fs.existsSync(snapDir) ? fs.readdirSync(snapDir).filter(f => f.endsWith('.tsx')).length : 0;
+ const outputExists = fs.existsSync(task.outputFile) && fs.readFileSync(task.outputFile, 'utf-8').trim().length > 0;
+ // Parse the CLI's terminal `result` line for exact token usage + api duration,
+ // so cost data is first-class in run.json (durable even if transcripts are pruned).
+ const result: AgentResult | null = parseResultFromText(transcriptBuf);
+ const usage: TokenUsage | null = result?.usage ?? null;
+ const estCostUSD = usage ? estimateCostUSD(usage, cfg.model) : null;
+ const runJson: RunJson = {
+ taskId: task.taskId,
+ condition: task.condition,
+ basePromptId: task.basePromptId,
+ category: task.category,
+ model: cfg.model,
+ status,
+ error,
+ subtype: result?.subtype ?? null,
+ isError: result?.isError ?? status !== 'finished',
+ startedAt,
+ finishedAt: new Date().toISOString(),
+ durationMs: Date.now() - t0,
+ apiDurationMs: result?.apiDurationMs ?? null,
+ usage,
+ estCostUSD,
+ snapshotCount,
+ outputExists,
+ };
+ writeJson(path.join(task.runDir, 'run.json'), runJson);
+ const tok = usage ? ` tok(out=${usage.outputTokens} cacheR=${usage.cacheReadTokens})` : '';
+ console.log(` [${status.padEnd(13)}] ${task.condition}/${task.taskId} snapshots=${snapshotCount} output=${outputExists}${tok}${error ? ` (${error.slice(0, 80)})` : ''}`);
+ resolve(runJson);
+ };
+
+ const args = [
+ '--print',
+ '--output-format', 'stream-json',
+ '--model', cfg.model,
+ '--force', // allow write/shell non-interactively
+ '--trust', // trust the sandbox workspace (headless)
+ '--workspace', task.projectDir,
+ task.taskPrompt,
+ ];
+
+ let child;
+ try {
+ child = spawn(cfg.agent, args, {cwd: task.projectDir, env: process.env});
+ } catch (e) {
+ finalize('startup_error', e instanceof Error ? e.message : String(e));
+ return;
+ }
+
+ const out = fs.createWriteStream(transcriptPath);
+ child.stdout.on('data', d => {
+ const s = d.toString();
+ if (transcriptBuf.length < 8_000_000) {transcriptBuf += s;} // keep in memory to parse the result line
+ out.write(s); // persist full transcript to disk
+ });
+ let stderr = '';
+ child.stderr.on('data', d => {
+ if (stderr.length < 4000) {stderr += d.toString();}
+ });
+
+ let killed = false;
+ const timer = setTimeout(() => {
+ killed = true;
+ try {
+ child.kill('SIGTERM');
+ } catch {
+ /* ignore */
+ }
+ setTimeout(() => {
+ try {
+ child.kill('SIGKILL');
+ } catch {
+ /* ignore */
+ }
+ }, 5000);
+ }, cfg.timeoutMs);
+
+ child.on('error', (e: NodeJS.ErrnoException) => {
+ clearTimeout(timer);
+ finalize(e.code === 'ENOENT' ? 'startup_error' : 'exception', e.message || String(e));
+ });
+ child.on('close', code => {
+ clearTimeout(timer);
+ out.end();
+ const status = killed ? 'timeout' : code === 0 ? 'finished' : 'error';
+ finalize(status, status === 'finished' ? null : stderr.trim().slice(-300) || `exit ${code}`);
+ });
+ });
+}
+
+// ── Main ─────────────────────────────────────────────────────────────
+
+function parseArgs(argv: string[]) {
+ const get = (flag: string) => {
+ const i = argv.indexOf(flag);
+ return i !== -1 ? argv[i + 1] : undefined;
+ };
+ const getInt = (flag: string, def: number) => {
+ const v = get(flag);
+ return v ? parseInt(v, 10) : def;
+ };
+ const conditions = get('--conditions');
+ return {
+ experiment: get('--experiment'),
+ out: get('--out'),
+ concurrency: getInt('--concurrency', 3),
+ model: get('--model') ?? DEFAULT_MODEL,
+ agent: get('--agent') ?? DEFAULT_AGENT,
+ timeoutMs: getInt('--timeout', DEFAULT_TIMEOUT_MS),
+ conditions: conditions ? conditions.split(',').map(s => s.trim()) : undefined,
+ dryRun: argv.includes('--dry-run'),
+ };
+}
+
+function loadTasks(configDirs: Record<string, string>, conditions: string[]): TaskFile[] {
+ const perCond: TaskFile[][] = [];
+ for (const cond of conditions) {
+ const condDir = configDirs[cond];
+ if (!condDir) {throw new Error(`Condition "${cond}" not found in config.conditionDirs`);}
+ const tasksDir = path.join(condDir, 'tasks');
+ const arr: TaskFile[] = [];
+ if (fs.existsSync(tasksDir)) {
+ for (const f of fs.readdirSync(tasksDir).filter(f => f.endsWith('.json'))) {
+ arr.push(JSON.parse(fs.readFileSync(path.join(tasksDir, f), 'utf-8')) as TaskFile);
+ }
+ }
+ perCond.push(arr);
+ }
+ // Interleave round-robin across conditions so partial progress stays balanced and no
+ // single condition monopolizes an early rate-limit window.
+ const tasks: TaskFile[] = [];
+ const max = Math.max(0, ...perCond.map(a => a.length));
+ for (let i = 0; i < max; i++) {for (const arr of perCond) {if (i < arr.length) {tasks.push(arr[i]);}}}
+ return tasks;
+}
+
+/** Confirm cursor-agent is present and logged in (non-fatal warning otherwise). */
+function checkAgent(agent: string): boolean {
+ const r = spawnSync(agent, ['status'], {encoding: 'utf-8'});
+ if (r.error) {
+ console.error(`Could not run "${agent}" (${r.error.message}). Install the Cursor CLI or pass --agent <path>.`);
+ return false;
+ }
+ const text = `${r.stdout ?? ''}${r.stderr ?? ''}`;
+ if (!/logged in|Logged in/i.test(text) && !process.env.CURSOR_API_KEY) {
+ console.warn(`[warn] "${agent} status" does not report a login. Run "${agent} login" or set CURSOR_API_KEY.`);
+ } else {
+ console.log(` Auth: ${text.trim().split('\n')[0] ?? 'ok'}`);
+ }
+ return true;
+}
+
+async function main() {
+ const args = parseArgs(process.argv.slice(2));
+ if (!args.experiment) {
+ console.error('Usage: run-purity.ts --experiment <id> [--out <base>] [--concurrency N] [--model <id>] [--dry-run]');
+ process.exit(1);
+ }
+ const expDir = path.join(sandboxBase(args.out), args.experiment);
+ const configPath = path.join(expDir, 'purity-config.json');
+ if (!fs.existsSync(configPath)) {
+ console.error(`No config at ${configPath}. Run setup-purity.mjs first.`);
+ process.exit(1);
+ }
+ const config = JSON.parse(fs.readFileSync(configPath, 'utf-8'));
+ const conditions = args.conditions ?? config.conditions;
+ const tasks = loadTasks(config.conditionDirs, conditions);
+
+ console.log(`\nPrompt Purity — run (experiment ${args.experiment})`);
+ console.log(` Conditions: ${conditions.join(', ')}`);
+ console.log(` Tasks: ${tasks.length}`);
+ console.log(` Model: ${args.model}`);
+ console.log(` Concurrency: ${args.concurrency}${args.dryRun ? ' (DRY RUN — no model calls)' : ''}\n`);
+
+ if (args.dryRun) {
+ for (const task of tasks) {
+ ensureDir(task.runDir);
+ writeJson(path.join(task.runDir, 'run.json'), {
+ taskId: task.taskId,
+ condition: task.condition,
+ basePromptId: task.basePromptId,
+ category: task.category,
+ status: 'dry-run',
+ snapshotCount: 0,
+ outputExists: false,
+ });
+ console.log(` [dry-run ] ${task.condition}/${task.taskId}`);
+ }
+ console.log(`\nDry run complete. ${tasks.length} run dirs prepared.\n`);
+ return;
+ }
+
+ if (!checkAgent(args.agent)) {process.exit(1);}
+
+ const t0 = Date.now();
+ const results = await pool(tasks, args.concurrency, task => runOne({model: args.model, agent: args.agent, timeoutMs: args.timeoutMs}, task));
+
+ const byStatus: Record<string, number> = {};
+ for (const r of results) {byStatus[r.status] = (byStatus[r.status] ?? 0) + 1;}
+ console.log(`\nDone in ${((Date.now() - t0) / 1000).toFixed(0)}s. Status: ${Object.entries(byStatus).map(([k, v]) => `${k}=${v}`).join(', ')}`);
+ console.log(`\nNext: npx tsx ${path.relative(process.cwd(), path.join(import.meta.dirname, 'purity-aggregate.ts'))} --experiment ${args.experiment}${args.out ? ` --out ${args.out}` : ''}\n`);
+}
+
+main().catch(e => {
+ console.error(e);
+ process.exit(1);
+});
diff --git a/internal/vibe-tests/prompt-purity-test/setup-purity.mjs b/internal/vibe-tests/prompt-purity-test/setup-purity.mjs
new file mode 100644
index 0000000000..9b836f2c7d
--- /dev/null
+++ b/internal/vibe-tests/prompt-purity-test/setup-purity.mjs
@@ -0,0 +1,378 @@
+#!/usr/bin/env node
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+
+/**
+ * @file setup-purity.mjs
+ * @input CLI flags: --exp, --out, --conditions, --prompts, --sample, --reps
+ * @output One <sandbox>/<expId>/<conditionId>/ tree per condition + <sandbox>/<expId>/purity-config.json
+ * @position internal/vibe-tests/prompt-purity-test — prompt-purity A/B experiment
+ *
+ * Builds isolated, realistic "installed @astryxdesign/core" projects whose ONLY
+ * guidance is a VARIANT of the astryx-init prompt (the independent variable). Each
+ * CONDITION = a fresh `astryx init` AGENTS.md + one variant self-check block spliced
+ * inside the <!-- ASTRYX:START/END --> markers (see conditions.json + variants/*.md).
+ * Only the injected prompt varies (Checker Protocol §2).
+ *
+ * ISOLATION: everything is created under an OUT-OF-WORKSPACE sandbox root
+ * (os.tmpdir()/astryx-purity by default, --out to override) so it never touches your
+ * open repo, editor file-watchers, other Cursor agents, or `git status`. Repo
+ * packages are only symlinked (read-only).
+ *
+ * GROUND TRUTH: every project gets an `astryx` bin SHIM at node_modules/.bin that
+ * logs argv to <project>/.astryx-invocations.log before exec-ing the real CLI, so we
+ * know objectively whether the agent ran the CLI to self-verify.
+ *
+ * Usage:
+ * node setup-purity.mjs # A vs B, curated pilot prompts, 3 reps
+ * node setup-purity.mjs --conditions A-control,B-selfcheck,C-strong --sample 6 --reps 5
+ * node setup-purity.mjs --exp abcd1234 --prompts dd-1,tc-6 --reps 3
+ */
+
+import * as fs from 'node:fs';
+import * as os from 'node:os';
+import * as path from 'node:path';
+import * as crypto from 'node:crypto';
+import {execFileSync} from 'node:child_process';
+import {fileURLToPath} from 'node:url';
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+const EXP_DIR = __dirname; // internal/vibe-tests/prompt-purity-test
+const VIBE_DIR = path.resolve(EXP_DIR, '..'); // internal/vibe-tests
+const REPO_ROOT = path.resolve(VIBE_DIR, '../..');
+const REAL_CLI_BIN = path.join(REPO_ROOT, 'packages', 'cli', 'bin', 'astryx.mjs');
+
+// ── Helpers ──────────────────────────────────────────────────────────
+
+const genId = () => crypto.randomBytes(4).toString('hex');
+const timestamp = () => new Date().toISOString();
+const ensureDir = dir => fs.mkdirSync(dir, {recursive: true});
+const readJson = p => JSON.parse(fs.readFileSync(p, 'utf-8'));
+
+/** Default sandbox base — OUTSIDE the repo. Overridable with --out or $ASTRYX_PURITY_OUT. */
+export function sandboxBase(out) {
+ return out || process.env.ASTRYX_PURITY_OUT || path.join(os.tmpdir(), 'astryx-purity');
+}
+
+function symlink(target, linkPath, type) {
+ ensureDir(path.dirname(linkPath));
+ if (fs.existsSync(linkPath)) return;
+ fs.symlinkSync(target, linkPath, type);
+}
+
+/** Curated prompt IDs from prompts.json, explicit --prompts, or a stratified sample. */
+function selectPrompts({sample, promptIds}) {
+ const testSet = readJson(path.join(VIBE_DIR, 'test-sets', 'default.json'));
+ const all = Array.isArray(testSet) ? testSet : testSet.prompts;
+ const byId = new Map(all.map(p => [p.id, p]));
+ const lookup = id => {
+ const p = byId.get(id);
+ if (!p) throw new Error(`Prompt id not found in default.json: ${id}`);
+ return p;
+ };
+
+ if (promptIds && promptIds.length > 0) return promptIds.map(lookup);
+ if (sample) return stratified(all, sample);
+
+ const curated = readJson(path.join(EXP_DIR, 'prompts.json'));
+ return curated.promptIds.map(({id}) => lookup(id));
+}
+
+function stratified(prompts, n) {
+ const byCat = new Map();
+ for (const p of prompts) {
+ const arr = byCat.get(p.category) ?? [];
+ arr.push(p);
+ byCat.set(p.category, arr);
+ }
+ const cats = [...byCat.keys()];
+ const out = [];
+ for (const c of cats) {
+ out.push(byCat.get(c)[Math.floor(Math.random() * byCat.get(c).length)]);
+ if (out.length >= n) return out.slice(0, n);
+ }
+ let i = 0;
+ while (out.length < n && i < cats.length * prompts.length) {
+ const avail = byCat.get(cats[i % cats.length]).filter(p => !out.includes(p));
+ if (avail.length) out.push(avail[Math.floor(Math.random() * avail.length)]);
+ i++;
+ }
+ return out.slice(0, n);
+}
+
+// ── Base project + ground-truth shim ─────────────────────────────────
+
+/**
+ * Realistic "installed @astryxdesign/core" base. Crucially lists a StyleX *compiler*
+ * plugin in devDependencies so the CLI's detectStylingSystem() returns 'stylex' and
+ * the injected prompt recommends the Astryx xstyle/StyleX path (what we're testing) —
+ * NOT the plain-CSS fallback. The plugin is only referenced in package.json (read by
+ * detection); it is never installed/compiled here.
+ */
+function createBaseProject(projectDir) {
+ ensureDir(projectDir);
+ const pkg = {
+ name: 'my-app',
+ private: true,
+ type: 'module',
+ dependencies: {
+ '@astryxdesign/core': '*',
+ '@astryxdesign/theme-neutral': '*',
+ '@stylexjs/stylex': '^0.10',
+ react: '^19',
+ 'react-dom': '^19',
+ },
+ devDependencies: {
+ '@stylexjs/babel-plugin': '^0.10',
+ },
+ };
+ fs.writeFileSync(path.join(projectDir, 'package.json'), JSON.stringify(pkg, null, 2));
+
+ const nm = path.join(projectDir, 'node_modules');
+ symlink(path.join(REPO_ROOT, 'packages', 'core'), path.join(nm, '@astryxdesign', 'core'), 'dir');
+ symlink(path.join(REPO_ROOT, 'packages', 'themes', 'neutral'), path.join(nm, '@astryxdesign', 'theme-neutral'), 'dir');
+ // CLI is symlinked (so `npx astryx` resolves) but NOT listed in package.json.
+ symlink(path.join(REPO_ROOT, 'packages', 'cli'), path.join(nm, '@astryxdesign', 'cli'), 'dir');
+
+ installLoggingShim(projectDir);
+}
+
+/**
+ * Executable `astryx` shim in node_modules/.bin that appends each invocation to
+ * <project>/.astryx-invocations.log, then execs the real CLI. Absolute paths baked in.
+ */
+function installLoggingShim(projectDir) {
+ const binDir = path.join(projectDir, 'node_modules', '.bin');
+ ensureDir(binDir);
+ const logPath = path.join(projectDir, '.astryx-invocations.log');
+ const shim = `#!/usr/bin/env node
+// GENERATED shim — logs invocations for the prompt-purity experiment, then execs the real CLI.
+import {appendFileSync} from 'node:fs';
+import {spawnSync} from 'node:child_process';
+const LOG = ${JSON.stringify(logPath)};
+const REAL = ${JSON.stringify(REAL_CLI_BIN)};
+const argv = process.argv.slice(2);
+try { appendFileSync(LOG, JSON.stringify({ts: new Date().toISOString(), argv}) + '\\n'); } catch {}
+const r = spawnSync(process.execPath, [REAL, ...argv], {stdio: 'inherit'});
+process.exit(r.status ?? 0);
+`;
+ const shimPath = path.join(binDir, 'astryx');
+ fs.writeFileSync(shimPath, shim);
+ fs.chmodSync(shimPath, 0o755);
+}
+
+// ── Variant splice (the independent variable) ────────────────────────
+
+const MARKER_END = '<!-- ASTRYX:END -->';
+const LEGACY_MARKER_END = '<!-- XDS:END -->';
+
+/**
+ * Splice a variant's self-check text inside the injected ASTRYX block of AGENTS.md.
+ * HTML comments are stripped; if nothing remains (the control), inject nothing.
+ */
+function applyVariant(projectDir, variantId) {
+ const variantPath = path.join(EXP_DIR, 'variants', `${variantId}.md`);
+ if (!fs.existsSync(variantPath)) {
+ throw new Error(`Variant file not found: ${variantPath}`);
+ }
+ const raw = fs.readFileSync(variantPath, 'utf-8');
+ const text = raw.replace(/<!--[\s\S]*?-->/g, '').trim();
+ if (!text) return {injected: false}; // control / no-op
+
+ const agentsPath = path.join(projectDir, 'AGENTS.md');
+ if (!fs.existsSync(agentsPath)) {
+ throw new Error(`AGENTS.md missing at ${agentsPath} — did \`astryx init\` run?`);
+ }
+ let content = fs.readFileSync(agentsPath, 'utf-8');
+ const block = `\n\nPURITY SELF-CHECK\n${text}\n`;
+ if (content.includes(MARKER_END)) {
+ content = content.replace(MARKER_END, `${block}${MARKER_END}`);
+ } else if (content.includes(LEGACY_MARKER_END)) {
+ content = content.replace(LEGACY_MARKER_END, `${block}${LEGACY_MARKER_END}`);
+ } else {
+ content = content.trimEnd() + block;
+ }
+ fs.writeFileSync(agentsPath, content);
+ return {injected: true};
+}
+
+/** Run the real `astryx init` (direct bin, NOT the shim) to write AGENTS.md. */
+function runInit(projectDir) {
+ execFileSync(
+ process.execPath,
+ [REAL_CLI_BIN, 'init', '--features', 'agents', '--agent-docs-path', 'AGENTS.md'],
+ {cwd: projectDir, stdio: 'pipe'},
+ );
+}
+
+// ── Task prompt (IDENTICAL across conditions except the AGENTS.md variant) ──
+
+function taskPrompt(prompt, projectDir, outputFile) {
+ return `First read AGENTS.md in this directory for component-library guidance, then follow it.
+
+You are building a React + TypeScript (.tsx) UI using the Astryx design system. Your
+project is at ${projectDir}.
+
+## Task
+
+${prompt.prompt}
+
+## Output
+
+Write a single, complete, self-contained .tsx file to:
+ ${outputFile}
+
+Then write a short metadata file next to it at:
+ ${path.join(projectDir, 'result-meta.json')}
+
+with this shape:
+{
+ "completedAt": "<ISO timestamp>",
+ "docsRead": [<component/doc names you looked up>],
+ "selfCheck": {
+ "reReadOutput": <true|false>,
+ "foundNonAstryx": <true|false>,
+ "fixed": <true|false>
+ }
+}
+
+Write ONLY valid TSX to the .tsx file (no markdown fences, no prose).`;
+}
+
+// ── Build one condition ──────────────────────────────────────────────
+
+function buildCondition({expDir, conditionId, variantId, prompts, reps}) {
+ const condDir = path.join(expDir, conditionId);
+ ensureDir(path.join(condDir, 'tasks'));
+ ensureDir(path.join(condDir, 'projects'));
+ ensureDir(path.join(condDir, 'runs'));
+
+ const taskIds = [];
+ for (const prompt of prompts) {
+ for (let k = 1; k <= reps; k++) {
+ const taskId = reps > 1 ? `${prompt.id}__r${k}` : prompt.id;
+ taskIds.push(taskId);
+ const projectDir = path.join(condDir, 'projects', taskId);
+ const outputFile = path.join(projectDir, 'output.tsx');
+
+ createBaseProject(projectDir);
+ runInit(projectDir);
+ const {injected} = applyVariant(projectDir, variantId);
+
+ const task = {
+ taskId,
+ basePromptId: prompt.id,
+ category: prompt.category,
+ complexity: prompt.complexity,
+ condition: conditionId,
+ variant: variantId,
+ variantInjected: injected,
+ prompt: prompt.prompt,
+ expectedComponents: prompt.expectedComponents, // EVAL-ONLY; never placed in taskPrompt
+ projectDir,
+ outputFile,
+ agentsPath: path.join(projectDir, 'AGENTS.md'),
+ invocationLog: path.join(projectDir, '.astryx-invocations.log'),
+ runDir: path.join(condDir, 'runs', taskId),
+ taskPrompt: taskPrompt(prompt, projectDir, outputFile),
+ createdAt: timestamp(),
+ };
+ fs.writeFileSync(path.join(condDir, 'tasks', `${taskId}.json`), JSON.stringify(task, null, 2));
+ }
+ }
+
+ const manifest = {
+ experimentCondition: conditionId,
+ variant: variantId,
+ createdAt: timestamp(),
+ reps,
+ prompts: prompts.map(p => ({id: p.id, category: p.category})),
+ taskIds,
+ };
+ fs.writeFileSync(path.join(condDir, 'manifest.json'), JSON.stringify(manifest, null, 2));
+ return {conditionId, condDir, taskIds};
+}
+
+// ── Main ─────────────────────────────────────────────────────────────
+
+function parseArgs(argv) {
+ const get = flag => {
+ const i = argv.indexOf(flag);
+ return i !== -1 ? argv[i + 1] : undefined;
+ };
+ const conditionsArg = get('--conditions');
+ return {
+ exp: get('--exp'),
+ out: get('--out'),
+ conditions: conditionsArg ? conditionsArg.split(',').map(s => s.trim()) : undefined,
+ sample: get('--sample') ? parseInt(get('--sample'), 10) : undefined,
+ reps: get('--reps') ? parseInt(get('--reps'), 10) : undefined,
+ promptIds: get('--prompts') ? get('--prompts').split(',').map(s => s.trim()) : undefined,
+ };
+}
+
+function main() {
+ const args = parseArgs(process.argv.slice(2));
+ const conditionsDef = readJson(path.join(EXP_DIR, 'conditions.json'));
+ const byId = new Map(conditionsDef.conditions.map(c => [c.id, c]));
+
+ const conditionIds = args.conditions ?? ['A-control', 'B-selfcheck'];
+ for (const id of conditionIds) {
+ if (!byId.has(id)) {
+ throw new Error(`Unknown condition "${id}". Known: ${[...byId.keys()].join(', ')}`);
+ }
+ }
+
+ // Pilot default prompts come from prompts.json#pilotDefault when nothing specified.
+ let promptIds = args.promptIds;
+ if (!promptIds && !args.sample) {
+ const curated = readJson(path.join(EXP_DIR, 'prompts.json'));
+ promptIds = curated.pilotDefault ?? curated.promptIds.map(p => p.id);
+ }
+ const reps = args.reps ?? 3;
+ const prompts = selectPrompts({sample: args.sample, promptIds});
+
+ const expId = args.exp ?? genId();
+ const base = sandboxBase(args.out);
+ const expDir = path.join(base, expId);
+ ensureDir(expDir);
+
+ console.log(`\nPrompt Purity Vibe Test — experiment ${expId}`);
+ console.log(` Sandbox: ${expDir}`);
+ console.log(` Conditions: ${conditionIds.join(', ')}`);
+ console.log(` Prompts: ${prompts.map(p => p.id).join(', ')} x ${reps} rep(s)`);
+ console.log(` Runs/cond: ${prompts.length * reps}\n`);
+
+ const built = [];
+ for (const id of conditionIds) {
+ const c = byId.get(id);
+ process.stdout.write(` building ${id} (variant ${c.variant})...`);
+ built.push(buildCondition({expDir, conditionId: id, variantId: c.variant, prompts, reps}));
+ process.stdout.write(' done\n');
+ }
+
+ const config = {
+ experimentId: expId,
+ experiment: 'prompt-purity',
+ createdAt: timestamp(),
+ sandboxRoot: expDir,
+ control: conditionsDef.control ?? 'A-control',
+ reps,
+ promptIds: prompts.map(p => p.id),
+ conditions: conditionIds,
+ conditionDirs: Object.fromEntries(built.map(b => [b.conditionId, b.condDir])),
+ };
+ const configPath = path.join(expDir, 'purity-config.json');
+ fs.writeFileSync(configPath, JSON.stringify(config, null, 2));
+
+ console.log(`\nConditions ready:`);
+ for (const b of built) console.log(` ${b.conditionId.padEnd(14)} ${b.taskIds.length} tasks`);
+ console.log(`\nConfig: ${configPath}`);
+ console.log(`\nNext:`);
+ console.log(` npx tsx ${path.relative(process.cwd(), path.join(EXP_DIR, 'run-purity.ts'))} --experiment ${expId}${args.out ? ` --out ${args.out}` : ''}`);
+ console.log(` npx tsx ${path.relative(process.cwd(), path.join(EXP_DIR, 'purity-aggregate.ts'))} --experiment ${expId}${args.out ? ` --out ${args.out}` : ''}\n`);
+
+ // Machine-readable line for the orchestrator to capture the id.
+ console.log(`PURITY_EXPERIMENT_ID=${expId}`);
+}
+
+main();
diff --git a/internal/vibe-tests/prompt-purity-test/smoke-test.ts b/internal/vibe-tests/prompt-purity-test/smoke-test.ts
new file mode 100644
index 0000000000..4568400f84
--- /dev/null
+++ b/internal/vibe-tests/prompt-purity-test/smoke-test.ts
@@ -0,0 +1,220 @@
+// Copyright (c) Meta Platforms, Inc. and affiliates.
+
+/**
+ * @file smoke-test.ts
+ * @input none (self-contained; runs setup into a temp dir)
+ * @output PASS/FAIL per check; exit 1 on any failure
+ * @position internal/vibe-tests/prompt-purity-test — no-LLM verification of the harness
+ *
+ * Verifies the harness mechanism WITHOUT any model calls:
+ * A. the veer/catch classifier labels fabricated snapshot timelines correctly
+ * B. setup-purity.mjs wires a real condition correctly (variant spliced into AGENTS.md,
+ * stylex styling detected, logging shim installed, NO expectedComponents leak)
+ * C. run-purity.ts --dry-run drives the pipeline headlessly (best-effort)
+ *
+ * Usage: npx tsx smoke-test.ts
+ */
+
+import * as assert from 'node:assert';
+import * as fs from 'node:fs';
+import * as os from 'node:os';
+import * as path from 'node:path';
+import * as crypto from 'node:crypto';
+import {execFileSync} from 'node:child_process';
+import {classifyTimeline, detectMarkers, purityScore} from './purity-eval.js';
+
+const EXP_DIR = import.meta.dirname;
+let failures = 0;
+let passed = 0;
+
+function check(name: string, fn: () => void) {
+ try {
+ fn();
+ passed++;
+ console.log(` PASS ${name}`);
+ } catch (e) {
+ failures++;
+ console.error(` FAIL ${name}\n ${e instanceof Error ? e.message : String(e)}`);
+ }
+}
+
+/** Best-effort check: reports a WARN instead of failing the suite. */
+function softCheck(name: string, fn: () => void) {
+ try {
+ fn();
+ passed++;
+ console.log(` PASS ${name}`);
+ } catch (e) {
+ console.warn(` WARN ${name}\n ${e instanceof Error ? e.message : String(e)}`);
+ }
+}
+
+// ── Fixtures ─────────────────────────────────────────────────────────
+
+const PURE = `import {Card, VStack, Heading, Text, Button} from '@astryxdesign/core';
+export default function App() {
+ return (
+ <Card>
+ <VStack gap="md">
+ <Heading level={2}>Users</Heading>
+ <Text>All good.</Text>
+ <Button variant="primary">Add</Button>
+ </VStack>
+ </Card>
+ );
+}`;
+
+const DIRTY = `import './styles.css';
+export default function App() {
+ return (
+ <div className="flex flex-col p-4 gap-2 bg-gray-100 rounded-lg" style={{color: '#ffffff', padding: '16px'}}>
+ <span className="text-sm font-semibold">Users</span>
+ </div>
+ );
+}`;
+
+// ── A. classifier ────────────────────────────────────────────────────
+
+console.log('\nA. Veer/catch classifier (fabricated timelines)');
+
+check('detectMarkers finds className/inlineStyle/cssImport/tailwind/hardcodedColor in dirty code', () => {
+ const types = new Set(detectMarkers(DIRTY).map(h => h.type));
+ for (const t of ['className', 'inlineStyle', 'cssImport', 'tailwind', 'hardcodedColor'] as const) {
+ assert.ok(types.has(t), `expected marker "${t}"`);
+ }
+});
+
+check('detectMarkers finds nothing hard in pure Astryx code', () => {
+ const hard = detectMarkers(PURE).filter(h => h.severity === 'hard');
+ assert.equal(hard.length, 0, `unexpected hard markers: ${hard.map(h => h.type).join(', ')}`);
+});
+
+check('purityScore: pure high (>90), dirty low (<50)', () => {
+ assert.ok(purityScore(PURE) > 90, `pure=${purityScore(PURE)}`);
+ assert.ok(purityScore(DIRTY) < 50, `dirty=${purityScore(DIRTY)}`);
+});
+
+check('neverVeered: single clean write', () => {
+ const c = classifyTimeline([PURE]);
+ assert.ok(c.neverVeered && !c.everVeered && !c.caught && !c.veeredUncaught && !c.noOutput, JSON.stringify(c));
+});
+
+check('caught: dirty then clean (veered, then fixed)', () => {
+ const c = classifyTimeline([DIRTY, PURE]);
+ assert.ok(c.everVeered, 'should have veered');
+ assert.ok(c.caught, 'should be caught');
+ assert.ok(!c.veeredUncaught && !c.neverVeered, JSON.stringify(c));
+ assert.ok(c.removedHardCount > 0, 'should record removed hard markers');
+});
+
+check('veeredUncaught: clean then dirty (veered, not fixed)', () => {
+ const c = classifyTimeline([PURE, DIRTY]);
+ assert.ok(c.everVeered && c.veeredUncaught, JSON.stringify(c));
+ assert.ok(!c.caught && !c.neverVeered, JSON.stringify(c));
+});
+
+check('noOutput: empty timeline', () => {
+ const c = classifyTimeline([]);
+ assert.ok(c.noOutput && !c.everVeered && !c.neverVeered, JSON.stringify(c));
+});
+
+check('caughtInReasoning: mentioned Tailwind but never wrote a hard marker', () => {
+ const c = classifyTimeline([PURE], {transcriptText: 'I considered using Tailwind here but Astryx has a component for it.'});
+ assert.ok(c.mentionedVeerInReasoning && c.neverVeered && c.caughtInReasoning, JSON.stringify(c));
+});
+
+// ── B. setup wiring ──────────────────────────────────────────────────
+
+console.log('\nB. setup-purity.mjs wiring (real astryx init into a temp sandbox)');
+
+const smokeOut = fs.mkdtempSync(path.join(os.tmpdir(), 'purity-smoke-'));
+const expId = 'smoke' + crypto.randomBytes(2).toString('hex');
+let setupOk = false;
+
+check('setup-purity.mjs builds A-control + B-selfcheck for dd-1 (1 rep)', () => {
+ execFileSync(
+ process.execPath,
+ [
+ path.join(EXP_DIR, 'setup-purity.mjs'),
+ '--exp', expId,
+ '--out', smokeOut,
+ '--conditions', 'A-control,B-selfcheck',
+ '--prompts', 'dd-1',
+ '--reps', '1',
+ ],
+ {stdio: 'pipe'},
+ );
+ assert.ok(fs.existsSync(path.join(smokeOut, expId, 'purity-config.json')), 'purity-config.json missing');
+ setupOk = true;
+});
+
+const proj = (cond: string) => path.join(smokeOut, expId, cond, 'projects', 'dd-1');
+
+check('control AGENTS.md has NO self-check; candidate AGENTS.md HAS it', () => {
+ assert.ok(setupOk, 'setup did not complete');
+ const a = fs.readFileSync(path.join(proj('A-control'), 'AGENTS.md'), 'utf-8');
+ const b = fs.readFileSync(path.join(proj('B-selfcheck'), 'AGENTS.md'), 'utf-8');
+ assert.ok(!a.includes('PURITY SELF-CHECK'), 'control should not contain the self-check block');
+ assert.ok(b.includes('PURITY SELF-CHECK'), 'candidate should contain the self-check block');
+ assert.ok(b.includes('re-read the file'), 'candidate should contain the B-selfcheck text');
+});
+
+check('injected prompt recommends the Astryx xstyle/StyleX path (stylex detected)', () => {
+ assert.ok(setupOk, 'setup did not complete');
+ const a = fs.readFileSync(path.join(proj('A-control'), 'AGENTS.md'), 'utf-8');
+ assert.ok(/xstyle|StyleX/i.test(a), 'AGENTS.md should reference xstyle/StyleX (styling system should detect as stylex)');
+});
+
+check('base package.json lists a StyleX compiler plugin (drives stylex detection)', () => {
+ assert.ok(setupOk, 'setup did not complete');
+ const pkg = JSON.parse(fs.readFileSync(path.join(proj('A-control'), 'package.json'), 'utf-8'));
+ assert.ok(pkg.devDependencies?.['@stylexjs/babel-plugin'], 'missing @stylexjs/babel-plugin');
+});
+
+check('logging astryx shim is installed', () => {
+ assert.ok(setupOk, 'setup did not complete');
+ const shim = path.join(proj('A-control'), 'node_modules', '.bin', 'astryx');
+ assert.ok(fs.existsSync(shim), 'shim missing');
+ assert.ok(fs.readFileSync(shim, 'utf-8').includes('.astryx-invocations.log'), 'shim does not log invocations');
+});
+
+check('no answer leak: expectedComponents never appears in the task prompt', () => {
+ assert.ok(setupOk, 'setup did not complete');
+ const task = JSON.parse(fs.readFileSync(path.join(smokeOut, expId, 'B-selfcheck', 'tasks', 'dd-1.json'), 'utf-8'));
+ assert.ok(!task.taskPrompt.includes('expectedComponents'), 'taskPrompt leaks the expectedComponents key');
+ for (const comp of task.expectedComponents ?? []) {
+ assert.ok(!task.taskPrompt.includes(comp), `taskPrompt leaks expected component "${comp}"`);
+ }
+});
+
+// ── C. runner dry-run (best-effort) ──────────────────────────────────
+
+console.log('\nC. run-purity.ts --dry-run (headless pipeline; best-effort)');
+
+softCheck('dry-run creates run.json with status "dry-run"', () => {
+ assert.ok(setupOk, 'setup did not complete');
+ try {
+ execFileSync('npx', ['tsx', path.join(EXP_DIR, 'run-purity.ts'), '--experiment', expId, '--out', smokeOut, '--dry-run'], {
+ stdio: 'pipe',
+ cwd: EXP_DIR,
+ });
+ } catch (e) {
+ // Non-fatal: environment may not resolve `npx tsx` here. The classifier + wiring
+ // checks are the load-bearing ones; surface this as a soft note instead.
+ throw new Error(`could not run dry-run via "npx tsx" (${e instanceof Error ? e.message : String(e)}) — run it manually to verify`, {cause: e});
+ }
+ const runJson = path.join(smokeOut, expId, 'A-control', 'runs', 'dd-1', 'run.json');
+ assert.ok(fs.existsSync(runJson), 'run.json not created by dry-run');
+ assert.equal(JSON.parse(fs.readFileSync(runJson, 'utf-8')).status, 'dry-run');
+});
+
+// ── Cleanup + report ─────────────────────────────────────────────────
+
+try {
+ fs.rmSync(smokeOut, {recursive: true, force: true});
+} catch {
+ /* ignore */
+}
+
+console.log(`\n${failures === 0 ? 'ALL CHECKS PASSED' : `${failures} CHECK(S) FAILED`} (${passed} passed, ${failures} failed)\n`);
+process.exit(failures === 0 ? 0 : 1);
diff --git a/internal/vibe-tests/prompt-purity-test/variants/A-control.md b/internal/vibe-tests/prompt-purity-test/variants/A-control.md
new file mode 100644
index 0000000000..89a8e4da1f
--- /dev/null
+++ b/internal/vibe-tests/prompt-purity-test/variants/A-control.md
@@ -0,0 +1,7 @@
+<!--
+Control condition: NO self-check block is appended.
+This file is intentionally comment-only. setup-purity.mjs strips HTML comments and,
+finding no remaining text, injects nothing — so the agent sees the astryx-init prompt
+exactly as `generateCompressedIndex()` emits it today. This is the baseline the
+candidate variants (B, C) are measured against.
+-->
diff --git a/internal/vibe-tests/prompt-purity-test/variants/B-selfcheck.md b/internal/vibe-tests/prompt-purity-test/variants/B-selfcheck.md
new file mode 100644
index 0000000000..5d40221435
--- /dev/null
+++ b/internal/vibe-tests/prompt-purity-test/variants/B-selfcheck.md
@@ -0,0 +1,7 @@
+SELF-CHECK (before you finish, every time): re-read the file you wrote. If you see
+className=, style={{...}}, an imported .css / @apply, Tailwind utility classes, raw
+<div>/<span> used for layout, or hardcoded #hex/rgb()/px/rem — you veered off Astryx.
+For each, replace it with the Astryx component or the xstyle prop + token. If unsure a
+component exists, run `astryx search "<thing>"` / `astryx component <Name>` — do not
+hand-roll it in CSS. Finish only when the file is Astryx components + xstyle/tokens
+with zero raw CSS/Tailwind.
diff --git a/internal/vibe-tests/prompt-purity-test/variants/C-strong.md b/internal/vibe-tests/prompt-purity-test/variants/C-strong.md
new file mode 100644
index 0000000000..426ffa7c48
--- /dev/null
+++ b/internal/vibe-tests/prompt-purity-test/variants/C-strong.md
@@ -0,0 +1,15 @@
+STOP — MANDATORY SELF-AUDIT before you output the final file. Astryx is the ONLY
+styling system here; raw CSS/Tailwind is a failure, not a fallback. Run this checklist
+against your file and fix every hit before finishing:
+
+1. Grep your own output for: `className=`, `style={{`, `import "*.css"`, `@apply`,
+ Tailwind utilities (e.g. `flex`, `p-4`, `text-sm`, `bg-*`, `rounded-*`), raw
+ `<div>/<span>` doing layout, and hardcoded values (`#hex`, `rgb(`, `NNpx`, `NNrem`).
+2. For EACH hit: delete it and use the Astryx component (layout, spacing, surfaces,
+ typography all have one) or the `xstyle` prop with a token. Never a raw element +
+ CSS when a component exists.
+3. If you are not 100% sure a component/prop exists, you MUST verify with
+ `astryx component <Name>` or `astryx search "<thing>"` — do NOT guess and do NOT
+ fall back to CSS.
+4. Only finish when a re-read shows ZERO items from step 1. If any remain, you have
+ not finished — fix them and re-audit.
diff --git a/packages/cli/src/commands/agent-docs.mjs b/packages/cli/src/commands/agent-docs.mjs
index 3deef40a64..b1fb93479c 100644
--- a/packages/cli/src/commands/agent-docs.mjs
+++ b/packages/cli/src/commands/agent-docs.mjs
@@ -197,6 +197,20 @@ export function generateCompressedIndex(version, {coreDir, runPrefix = getRunPre
lines.push("- Custom styling: component props first; else style/className with tokens — var(--color-*|--spacing-*|--radius-*). No raw hex/px. (No StyleX/Tailwind compiler here — don't use xstyle/utility classes.)");
}
lines.push('- Tokens for every value (`astryx docs tokens`). Brand/accent via `astryx theme` — never override --color-* in :root.');
+ // Self-check — post-generation pass. Validated via vibe tests (internal/vibe-tests/
+ // prompt-purity-test): on complex multi-step UIs the rules above alone still leave raw
+ // CSS in ~11-13% of runs; a re-read-and-fix pass cuts that ~4x at negligible token cost.
+ // The fix names the sanctioned escape hatch for the configured system.
+ const selfCheckFix = {
+ stylex:
+ 'replace any className=, style={{…}}, raw <div>/<span> layout, imported .css/@apply, or hardcoded #hex/px with the component or the xstyle prop + a token',
+ tailwind:
+ 'replace any style={{…}}, raw <div>/<span> layout, imported .css/@apply, or hardcoded/arbitrary value (e.g. bg-[#fff], p-[13px]) with the component or a token-backed utility',
+ css: 'replace any raw <div>/<span> layout, imported .css/@apply, or hardcoded value (#hex, 16px) with the component or a token (var(--color-*|--spacing-*|…))',
+ };
+ lines.push(
+ `- SELF-CHECK before you finish: re-read the file and ${selfCheckFix[stylingSystem] ?? selfCheckFix.css}. If unsure a component/prop exists, run \`astryx component <Name>\` / \`astryx search "<thing>"\`; don't hand-roll CSS.`,
+ );
lines.push('');
// Command reference — build/template/component are covered in WORKFLOW above.
diff --git a/packages/cli/src/commands/agent-docs.test.mjs b/packages/cli/src/commands/agent-docs.test.mjs
index bdd2c9124b..56e79317e8 100644
--- a/packages/cli/src/commands/agent-docs.test.mjs
+++ b/packages/cli/src/commands/agent-docs.test.mjs
@@ -43,6 +43,26 @@ describe('generateCompressedIndex', () => {
expect(result).toMatch(/never override --color-/);
});
+ it('includes the post-generation self-check rule', () => {
+ const result = generateCompressedIndex('1.0.0');
+ expect(result).toContain('SELF-CHECK before you finish');
+ expect(result).toMatch(/re-read the file/);
+ expect(result).toMatch(/don't hand-roll CSS/);
+ });
+
+ it('tailors the self-check to the styling system (xstyle for StyleX, not className for Tailwind)', () => {
+ // StyleX path: className/inline style are veers; the fix is the xstyle prop + a token
+ const stylex = generateCompressedIndex('1.0.0', {stylingSystem: 'stylex'});
+ const stylexSelfCheck = stylex.split('\n').find(l => l.includes('SELF-CHECK'));
+ expect(stylexSelfCheck).toMatch(/xstyle/);
+ expect(stylexSelfCheck).toMatch(/className=/);
+ // className IS the system in Tailwind — it must NOT be flagged
+ const tailwind = generateCompressedIndex('1.0.0', {stylingSystem: 'tailwind'});
+ const tailwindSelfCheck = tailwind.split('\n').find(l => l.includes('SELF-CHECK'));
+ expect(tailwindSelfCheck).toBeDefined();
+ expect(tailwindSelfCheck).not.toMatch(/className=/);
+ });
+
it('defaults to the CSS-variable styling path (no compiler)', () => {
const result = generateCompressedIndex('1.0.0');
expect(result).toMatch(/style\/className with tokens/);